Thursday, June 18, 2009

Mail.Ru/1.0 not obeying Robots.txt

This bot Mail.Ru/1.0 read robots.txt, ignored it and tried to access a page anyway from this IP address 94.100.181.242