Top 20 IPs Making Requests To Apache [on hold]
I'm trying to get the top 20 IPs that are making requests to my server. I've written a grep with awk and split to get the IP down to a class b range. I then try to sort it and count the number of requests but I think something is incorrect.
grep '2018-12-19' /var/log/httpd/example.com/access.log | awk '{split($3, a, "."); print a[1] "." a[2] }' | sort | uniq -c | sort --numeric | grep '::1' -v | tail -20
My log format is:
2018-12-19 15:49:27 121.57.166.207 etc....
I suspect something is wrong because I did it for the all of today (log rotation is on) and blocked the top IP ranges. I then limited it down to a minute:
grep '2018-12-19 12:05' /var/log/httpd/example.com/access.log | awk '{split($3, a, "."); print a[1] "." a[2] }' | sort | uniq -c | sort --numeric | grep '::1' -v | tail -20
and got back traffic from one of the blocked ranges..BUT.. I then re-ran my original grep for the full day and the count I get back for that range is the same as when I blocked it so it supposedly hasn't made any more requests.
First time full day:
15502 XXX.YYY
One minute interval (after blocking):
256 XXX.YYY
Second time full day:
15502 XXX.YYY
Some sample records:
2018-12-19 15:49:27 121.57.166.207 - 192.168.0.1 443 GET /page 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36" 56994254
2018-12-19 15:50:24 42.156.138.115 - 192.168.0.1 443 GET /js/file.woff2 200 "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1" 2413
2018-12-19 15:50:24 42.120.161.114 - 192.168.0.1 443 POST /api/error.php 200 "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1" 1400455
2018-12-19 15:50:33 111.206.198.4 - 192.168.0.1 443 POST /etc/file.php 200 "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)" 2368
2018-12-19 15:50:33 220.181.108.113 - 192.168.0.1 443 GET /test ?page=1 200 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 1262749
2018-12-19 15:50:36 111.206.198.18 - 192.168.0.1 443 GET /d.jpg 200 "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)" 287688
2018-12-19 15:50:44 106.120.173.105 - 192.168.0.1 443 GET /page/2678 ?&language=Russian 301 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 968756
2018-12-19 15:50:45 220.181.108.147 - 192.168.0.1 443 GET /page ?id=20 200 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 77875
2018-12-19 15:50:45 106.38.241.100 - 192.168.0.1 443 GET /page/562 ?language=Italian 301 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 901742
2018-12-19 15:50:47 119.136.88.91 - 192.168.0.1 443 GET /page/logo.jpg 200 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0" 2367
I would expect the following counts to come back from these samples (row 1 is count, 2 is the class b range):
1 121.57
1 42.156
1 42.120
2 111.206
2 220.181
1 106.38
1 119.136
awk grep sort split uniq
asked yesterday
user3783243
1012
put on hold as unclear what you're asking by Rui F Ribeiro, RalfFriedl, DarkHeart, Jeff Schaller, Shadur 14 hours ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
add a comment |
I'm trying to get the top 20 IPs that are making requests to my server. I've written a grep with awk and split to get the IP down to a class b range. I then try to sort it and count the number of requests but I think something is incorrect.
grep '2018-12-19' /var/log/httpd/example.com/access.log | awk '{split($3, a, "."); print a[1] "." a[2] }' | sort | uniq -c | sort --numeric | grep '::1' -v | tail -20
My log format is:
2018-12-19 15:49:27 121.57.166.207 etc....
I suspect something is wrong because I did it for the all of today (log rotation is on) and blocked the top IP ranges. I then limited it down to a minute:
grep '2018-12-19 12:05' /var/log/httpd/example.com/access.log | awk '{split($3, a, "."); print a[1] "." a[2] }' | sort | uniq -c | sort --numeric | grep '::1' -v | tail -20
and got back traffic from one of the blocked ranges..BUT.. I then re-ran my original grep for the full day and the count I get back for that range is the same as when I blocked it so it supposedly hasn't made any more requests.
First time full day:
15502 XXX.YYY
One minute interval (after blocking):
256 XXX.YYY
Second time full day:
15502 XXX.YYY
Some sample records:
2018-12-19 15:49:27 121.57.166.207 - 192.168.0.1 443 GET /page 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36" 56994254
2018-12-19 15:50:24 42.156.138.115 - 192.168.0.1 443 GET /js/file.woff2 200 "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1" 2413
2018-12-19 15:50:24 42.120.161.114 - 192.168.0.1 443 POST /api/error.php 200 "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1" 1400455
2018-12-19 15:50:33 111.206.198.4 - 192.168.0.1 443 POST /etc/file.php 200 "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)" 2368
2018-12-19 15:50:33 220.181.108.113 - 192.168.0.1 443 GET /test ?page=1 200 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 1262749
2018-12-19 15:50:36 111.206.198.18 - 192.168.0.1 443 GET /d.jpg 200 "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)" 287688
2018-12-19 15:50:44 106.120.173.105 - 192.168.0.1 443 GET /page/2678 ?&language=Russian 301 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 968756
2018-12-19 15:50:45 220.181.108.147 - 192.168.0.1 443 GET /page ?id=20 200 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 77875
2018-12-19 15:50:45 106.38.241.100 - 192.168.0.1 443 GET /page/562 ?language=Italian 301 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 901742
2018-12-19 15:50:47 119.136.88.91 - 192.168.0.1 443 GET /page/logo.jpg 200 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0" 2367
I would expect the following counts to come back from these samples (row 1 is count, 2 is the class b range):
1 121.57
1 42.156
1 42.120
2 111.206
2 220.181
1 106.38
1 119.136
awk grep sort split uniq
asked yesterday
user3783243
1012
put on hold as unclear what you're asking by Rui F Ribeiro, RalfFriedl, DarkHeart, Jeff Schaller, Shadur 14 hours ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
2
It looks like you're trying to summarize one day's worth of logs? Across some sort of network range? Could you clarify the requirements and provide sample log data to test against?
– Jeff Schaller
yesterday
Yes, that is correct. If the first (last depending on how you read it) two octets of an IP are the same I want it to be considered the same.192.168.0.1and192.168.0.2should count as2requests for192.168.192.167.0.1should count as 1 request for192.167. Adding sample records now.
– user3783243
yesterday
add a comment |
I'm trying to get the top 20 IPs that are making requests to my server. I've written a grep with awk and split to get the IP down to a class b range. I then try to sort it and count the number of requests but I think something is incorrect.
grep '2018-12-19' /var/log/httpd/example.com/access.log | awk '{split($3, a, "."); print a[1] "." a[2] }' | sort | uniq -c | sort --numeric | grep '::1' -v | tail -20
My log format is:
2018-12-19 15:49:27 121.57.166.207 etc....
I suspect something is wrong because I did it for the all of today (log rotation is on) and blocked the top IP ranges. I then limited it down to a minute:
grep '2018-12-19 12:05' /var/log/httpd/example.com/access.log | awk '{split($3, a, "."); print a[1] "." a[2] }' | sort | uniq -c | sort --numeric | grep '::1' -v | tail -20
and got back traffic from one of the blocked ranges..BUT.. I then re-ran my original grep for the full day and the count I get back for that range is the same as when I blocked it so it supposedly hasn't made any more requests.
First time full day:
15502 XXX.YYY
One minute interval (after blocking):
256 XXX.YYY
Second time full day:
15502 XXX.YYY
Some sample records:
2018-12-19 15:49:27 121.57.166.207 - 192.168.0.1 443 GET /page 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36" 56994254
2018-12-19 15:50:24 42.156.138.115 - 192.168.0.1 443 GET /js/file.woff2 200 "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1" 2413
2018-12-19 15:50:24 42.120.161.114 - 192.168.0.1 443 POST /api/error.php 200 "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1" 1400455
2018-12-19 15:50:33 111.206.198.4 - 192.168.0.1 443 POST /etc/file.php 200 "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)" 2368
2018-12-19 15:50:33 220.181.108.113 - 192.168.0.1 443 GET /test ?page=1 200 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 1262749
2018-12-19 15:50:36 111.206.198.18 - 192.168.0.1 443 GET /d.jpg 200 "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)" 287688
2018-12-19 15:50:44 106.120.173.105 - 192.168.0.1 443 GET /page/2678 ?&language=Russian 301 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 968756
2018-12-19 15:50:45 220.181.108.147 - 192.168.0.1 443 GET /page ?id=20 200 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 77875
2018-12-19 15:50:45 106.38.241.100 - 192.168.0.1 443 GET /page/562 ?language=Italian 301 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 901742
2018-12-19 15:50:47 119.136.88.91 - 192.168.0.1 443 GET /page/logo.jpg 200 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0" 2367
I would expect the following counts to come back from these samples (row 1 is count, 2 is the class b range):
1 121.57
1 42.156
1 42.120
2 111.206
2 220.181
1 106.38
1 119.136
awk grep sort split uniq
asked yesterday
user3783243
1012
I'm trying to get the top 20 IPs that are making requests to my server. I've written a grep with awk and split to get the IP down to a class b range. I then try to sort it and count the number of requests but I think something is incorrect.
grep '2018-12-19' /var/log/httpd/example.com/access.log | awk '{split($3, a, "."); print a[1] "." a[2] }' | sort | uniq -c | sort --numeric | grep '::1' -v | tail -20
My log format is:
2018-12-19 15:49:27 121.57.166.207 etc....
I suspect something is wrong because I did it for the all of today (log rotation is on) and blocked the top IP ranges. I then limited it down to a minute:
grep '2018-12-19 12:05' /var/log/httpd/example.com/access.log | awk '{split($3, a, "."); print a[1] "." a[2] }' | sort | uniq -c | sort --numeric | grep '::1' -v | tail -20
and got back traffic from one of the blocked ranges..BUT.. I then re-ran my original grep for the full day and the count I get back for that range is the same as when I blocked it so it supposedly hasn't made any more requests.
First time full day:
15502 XXX.YYY
One minute interval (after blocking):
256 XXX.YYY
Second time full day:
15502 XXX.YYY
Some sample records:
2018-12-19 15:49:27 121.57.166.207 - 192.168.0.1 443 GET /page 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36" 56994254
2018-12-19 15:50:24 42.156.138.115 - 192.168.0.1 443 GET /js/file.woff2 200 "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1" 2413
2018-12-19 15:50:24 42.120.161.114 - 192.168.0.1 443 POST /api/error.php 200 "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1" 1400455
2018-12-19 15:50:33 111.206.198.4 - 192.168.0.1 443 POST /etc/file.php 200 "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)" 2368
2018-12-19 15:50:33 220.181.108.113 - 192.168.0.1 443 GET /test ?page=1 200 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 1262749
2018-12-19 15:50:36 111.206.198.18 - 192.168.0.1 443 GET /d.jpg 200 "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)" 287688
2018-12-19 15:50:44 106.120.173.105 - 192.168.0.1 443 GET /page/2678 ?&language=Russian 301 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 968756
2018-12-19 15:50:45 220.181.108.147 - 192.168.0.1 443 GET /page ?id=20 200 "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 77875
2018-12-19 15:50:45 106.38.241.100 - 192.168.0.1 443 GET /page/562 ?language=Italian 301 "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 901742
2018-12-19 15:50:47 119.136.88.91 - 192.168.0.1 443 GET /page/logo.jpg 200 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0" 2367
I would expect the following counts to come back from these samples (row 1 is count, 2 is the class b range):
1 121.57
1 42.156
1 42.120
2 111.206
2 220.181
1 106.38
1 119.136
awk grep sort split uniq
awk grep sort split uniq
asked yesterday
user3783243
1012
asked yesterday
user3783243
1012
edited yesterday
asked yesterday
user3783243
1012
asked yesterday
user3783243
1012
asked yesterday
user3783243
1012
1012
put on hold as unclear what you're asking by Rui F Ribeiro, RalfFriedl, DarkHeart, Jeff Schaller, Shadur 14 hours ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
put on hold as unclear what you're asking by Rui F Ribeiro, RalfFriedl, DarkHeart, Jeff Schaller, Shadur 14 hours ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
2
It looks like you're trying to summarize one day's worth of logs? Across some sort of network range? Could you clarify the requirements and provide sample log data to test against?
– Jeff Schaller
yesterday
Yes, that is correct. If the first (last depending on how you read it) two octets of an IP are the same I want it to be considered the same.192.168.0.1and192.168.0.2should count as2requests for192.168.192.167.0.1should count as 1 request for192.167. Adding sample records now.
– user3783243
yesterday
add a comment |
2
It looks like you're trying to summarize one day's worth of logs? Across some sort of network range? Could you clarify the requirements and provide sample log data to test against?
– Jeff Schaller
yesterday
Yes, that is correct. If the first (last depending on how you read it) two octets of an IP are the same I want it to be considered the same.192.168.0.1and192.168.0.2should count as2requests for192.168.192.167.0.1should count as 1 request for192.167. Adding sample records now.
– user3783243
yesterday
2
2
It looks like you're trying to summarize one day's worth of logs? Across some sort of network range? Could you clarify the requirements and provide sample log data to test against?
– Jeff Schaller
yesterday
It looks like you're trying to summarize one day's worth of logs? Across some sort of network range? Could you clarify the requirements and provide sample log data to test against?
– Jeff Schaller
yesterday
Yes, that is correct. If the first (last depending on how you read it) two octets of an IP are the same I want it to be considered the same.
192.168.0.1 and 192.168.0.2 should count as 2 requests for 192.168. 192.167.0.1 should count as 1 request for 192.167. Adding sample records now.– user3783243
yesterday
Yes, that is correct. If the first (last depending on how you read it) two octets of an IP are the same I want it to be considered the same.
192.168.0.1 and 192.168.0.2 should count as 2 requests for 192.168. 192.167.0.1 should count as 1 request for 192.167. Adding sample records now.– user3783243
yesterday
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
2
It looks like you're trying to summarize one day's worth of logs? Across some sort of network range? Could you clarify the requirements and provide sample log data to test against?
– Jeff Schaller
yesterday
Yes, that is correct. If the first (last depending on how you read it) two octets of an IP are the same I want it to be considered the same.
192.168.0.1and192.168.0.2should count as2requests for192.168.192.167.0.1should count as 1 request for192.167. Adding sample records now.– user3783243
yesterday