Friday, September 26, 2008

Robots exclusion file

Did you know you can put a text file on your web page that will prevent robots/spiders from indexing your pages?

Check it:

http://en.wikipedia.org/wiki/Robots.txt


No comments: