crawler

You are currently browsing articles tagged crawler.

If you read this…

2008/04/07 in Journal | No comments

Then you have passed the robots.txt file :) I have disabled the site from all crawlers. Using robots.txt.