How to prevent robots.txt from getting indexed in google or yahoo?

Recommended Videos

How to Optimize Your Robots txt File
How to Optimize Your Robots txt File
Set Robots.txt So Your Website Gets Crawled
Set Robots.txt So Your Website Gets Crawled
Robots.txt   How to Use Robots.txt files
Robots.txt How to Use Robots.txt files
How to Avoid Duplicate Content and Let Google Better Index Your Site
How to Avoid Duplicate Content and Let Google Better Index Your Site
Why aren't geographic target and query parameters specified in robots.txt?
Why aren't geographic target and query parameters specified in robots.txt?

Use X-Robots-Tag to prevent google or yahoo from indexing documents with file extensions like .txt, .doc, .pdf etc? Google and yahoo can index all documents uploaded on a website, whether it be .txt files or .doc files or .pdf files.

However you can use X-Robots-Tag to prevent SERPs like google and yahoo from indexing your website.Here is an example of using X-Robots-Tag in your .htaccess file, to tell Google or yahoo not to index the robots.txt file.

<FilesMatch “robots\.txt”>
Header set X-Robots-Tag “noindex, follow”
</FilesMatch>

Here is another example of using X-Robots-Tag in your .htaccess file, to tell Google or yahoo not to index files with .doc extensions.

<FilesMatch “\.doc$”>
Header set X-Robots-Tag “index, noarchive, nosnippet”
</Files>

We can also try using the “noindexdirective in the robots.txt.However only google obeys this directive though they have not yet announced full support for it in future. Thus X-Robots-Tag seem to be the best option to prevent google or yahoo from indexing robots.txt or any other document.

Related Videos

How to Optimize Your Robots txt File
How to Optimize Your Robots txt File
Set Robots.txt So Your Website Gets Crawled
Set Robots.txt So Your Website Gets Crawled
Robots.txt   How to Use Robots.txt files
Robots.txt How to Use Robots.txt files
How to Avoid Duplicate Content and Let Google Better Index Your Site
How to Avoid Duplicate Content and Let Google Better Index Your Site
Why aren't geographic target and query parameters specified in robots.txt?
Why aren't geographic target and query parameters specified in robots.txt?
2 comments on “How to prevent robots.txt from getting indexed in google or yahoo?
  1. Useful tip for beginners.

    Nirmal’s last blog post..Customize Windows Live Messenger with Messenger Plus!

  2. I don’t think it is just a tip for the beginners…very very few know the usage X-Robots-Tag…

Leave a Reply

Your email address will not be published. Required fields are marked *

*