Monday, August 13, 2007

Google Robots.txt Blog collection

Google is very respectful of the robots.txt file.  As such they have produced many Blog entries about best practices, tutorials and extensions to the protocol.  So in my own hope of organizing the worlds information.  Or at least Google's information on robots.txt.  I have links.

If you unsure of what a robots.txt file does then check out The Web Robots Pages at robotstext.org.  They give some down to earth explanations of the standards.

As to Google.  Here a few places they mention robots.txt and how they use them.

Webmasters overview All about robots
Okay not really robots.txt, but Robots meta tags, yes Google respects them to.

I am sure there are more and I will try and post them here.

Thanks for reading.

No comments: