Friday, May 29, 2009


The Twiceler robot is not obeying robots.txt file.

This bot repeatedly hits our web sites when we have listed in each one in the robots.txt file that they should go stay off our sites.

Additionally when I went to the link below - it says page not found. Bad bot all around!

It's coming from that same old repeat offender network that starts with the number 38 -- which you might want to block if you are experiencing the same problems:

OrgName: PSINet, Inc.
Address: 1015 31st St NW
City: Washington
StateProv: DC
PostalCode: 20007
Country: US

ReferralServer: rwhois://

NetRange: -