Invite Google to index your pages

To invite Google to come and index your page webmasters will generally add a ‘robots.txt’ file. This file tells the Googlebot that you want it to read and index your page.

Now most people will not know how to write a ‘robot.txt’ file or maybe have never even heard of it. Well Google have come up with an ingenious translator that turns your requests into ‘robot.txt’ file, it is called generate robots text. Now you don’t even need to learn the language used all you have to do is know what you want indexed.

There are however some problems relating to privacy that some robots from other sources will just ignore the file and search everything. To overcome this you should protect private information by passwords and not rely on the ‘robots.txt’ file.

For more information have a look at Google Webmaster Blog.