Saturday, January 03, 2009

December 2008 Bots

Seems like we had quite a few bots hitting our web sites last month. Taking a look at some prior months it turns out the number of bot-like user agents seems to go up and down a lot as shown below. This may be due to updating the user agent file periodically to tell certain bots to go away but not sure without further tracking and analysis. I would guess that some bot makers simply rename their bots and send out new ones when too many of their bots end up in user agent files.

Bot Count - month - year
1138 12 2008
932 11 2008
1068 10 2008
683 9 2008
1032 8 2008
859 7 2008
783 6 2008
1553 5 2008
894 4 2008
1057 3 2008
439 2 2008

Here's a run down of last month's bot-ish traffic:

430 Moozilla
289 Mozilla/4.0 (compatible;)
65 Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
41 SurveyBot/2.3 (Whois Source)
36 MSR-ISRCCrawler
34 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
32 Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)
24 Mozilla/5.0 (compatible; WebDataCentreBot/1.0; +http://WebDataCentre.com/)
21 libwww-perl/5.814
15 Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
14 Gigabot/3.0 (http://www.gigablast.com/spider.html)
10 AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.)
9 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Alexa Toolbar)
9 REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project)
9 CazoodleBot/0.0.2 (http://www.cazoodle.com/contact.php; cbot@cazoodle.com)
8 Mozilla/5.0 (compatible; OpenX Spider; http://www.openx.org)
8 Mozilla/5.0 (compatible; LocalBot/2.1; +http://www.seattlekit.com)
6 libwww-perl/5.805
6 libwww-perl/5.803
5 Java/1.6.0_07
5 Axonize-bot
4 kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
3 Mozilla/5.0 (compatible; OnTownsBot/1.2; +http://www.ontowns.com/)
3 Snoopy v1.2
3 Java/1.5.0_11
3 ecxi/Nutch-1.0-dev (esCERT-UPC-ecxi; http://escert.upc.edu/; admin escert edu)
3 Site-Perf.com performance testing bot
3 libwww-perl/5.806
3 Mozilla/5.0 (compatible; LocalBot/2.1; +http://www.None)
2 BobCrawl/Nutch-0.9 (Test/Development crawler; http://notavalable.com; notavailable@notavailable.com)
2 SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)
2 Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
2 Mozilla/5.0 (compatible; del.icio.us-thumbnails/1.0; FreeBSD) KHTML/4.3.2 (like Gecko)
2 Mozilla/5.0 (compatible; Snappybot/0.1)
2 Mozilla/5.0 (compatible; SuchbaerBot/0.4; +http://bot.suchbaer.de/info.html)
2 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.17) Gecko/0829 SeaMonkey/1.1.12
2 Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows)
2 libwww-perl/5.820
2 Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
2 Horny Sex Search/Nutch-0.9 (HornySexSearch.com Crawler; http://www.hornysexsearch.com; Contact HornySexSearch.com)
2 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; TuneUp HTML Client Embedded Web Browser from: http://bsalsa.com/; SLCC1; .NET CLR 2.0.50727; Media
1 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/052906 Firefox/3.0/Nutch-0.9
1 USCity Link Checker - libwww-perl/5.65
1 Wget/1.5.3.1
1 libwww-perl/5.79
1 Crawler for Sika Solutions (http://www.sika-sol.co.uk/)
1 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; InfoPath.2)
1 MSRBOT (http://research.microsoft.com/research/sv/msrbot/
1 Netintelligence LiveAssessment - www.netintelligence.com
1 Jakarta Commons-HttpClient/3.1
1 Mozilla/5.0 (compatible; http://www.whoisde.de/2.1; +http://www.whoisde.de)
1 betaBot
1 SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)
1 Isidorus/2.0 (Isidorus; http://www.isidorus.com; crawler@isidorus.com)