Thursday, June 11, 2009

MSR-ISRCCrawler not obeying robots.txt

Strangely, MSR-ISRCCrawler checked robots.txt. It clearly says in our robots.txt file that this bot is disallowed. Then it proceeded to crawl our site anyway. Hmm...