How to prevent robots.txt from getting indexed in google or yahoo?
Use X-Robots-Tag to prevent google or yahoo from indexing documents with file extensions like .txt, .doc, .pdf etc? Google and yahoo can index all documents uploaded on a website, whether it be .txt files or .doc files or .pdf files.
However you can use X-Robots-Tag to prevent SERPs like google and yahoo from indexing your website.Here is an example of using X-Robots-Tag in your .htaccess file, to tell Google or yahoo not to index the robots.txt file.
Header set X-Robots-Tag “noindex, follow”
</FilesMatch>
Here is another example of using X-Robots-Tag in your .htaccess file, to tell Google or yahoo not to index files with .doc extensions.
Header set X-Robots-Tag “index, noarchive, nosnippet”
</Files>
We can also try using the “noindex” directive in the robots.txt.However only google obeys this directive though they have not yet announced full support for it in future. Thus X-Robots-Tag seem to be the best option to prevent google or yahoo from indexing robots.txt or any other document.
Filed under: Wordpress SEO Wordpress SEO & Plugins


Useful tip for beginners.
Nirmal’s last blog post..Customize Windows Live Messenger with Messenger Plus!
I don’t think it is just a tip for the beginners…very very few know the usage X-Robots-Tag…