Blog

SEO - robots.txt

Posted by Viv on 25 July 2013 | 0 Comments

Tags:

The robots.txt file is a simple little text file that helps to prevent the search engine bots from indexing pages that you don't want to have indexed (maybe things like the login page to the CMS).

One useful tool to generate and check the robots.txt file is at SEO Book.

Note: Save the plain text file as UTF-8 format.