How to prevent robots.txt from getting indexed in google or yahoo?

Use X-Robots-Tag to prevent google or yahoo from indexing documents with file extensions like .txt, .doc, .pdf etc? Google and yahoo can index all documents uploaded on a website, whether it be .txt files or .doc files or .pdf files.

However you can use X-Robots-Tag to prevent SERPs like google and yahoo from indexing your website.Here is an example of using X-Robots-Tag in your .htaccess file, to tell Google or yahoo not to index the robots.txt file.

<FilesMatch “robots\.txt”>
Header set X-Robots-Tag “noindex, follow”
</FilesMatch>

Here is another example of using X-Robots-Tag in your .htaccess file, to tell Google or yahoo not to index files with .doc extensions.

<FilesMatch “\.doc$”>
Header set X-Robots-Tag “index, noarchive, nosnippet”
</Files>

We can also try using the “noindexdirective in the robots.txt.However only google obeys this directive though they have not yet announced full support for it in future. Thus X-Robots-Tag seem to be the best option to prevent google or yahoo from indexing robots.txt or any other document.

Related Videos

Robots.txt Formatting | Affilorama
Robots.txt Formatting | Affilorama
How to Optimize Your Robots txt File
How to Optimize Your Robots txt File
How To Stop Search Engine From Crawling  (Part 7) | Digital Marketing | In Hindi | Go Digital |
How To Stop Search Engine From Crawling (Part 7) | Digital Marketing | In Hindi | Go Digital |
Robots.txt   How to Use Robots.txt files
Robots.txt How to Use Robots.txt files
HOW TO Create and Add Custom Robots.txt File on Blogger Site – Indexing Your Blog | SEO Blogger - 2
HOW TO Create and Add Custom Robots.txt File on Blogger Site – Indexing Your Blog | SEO Blogger - 2
2 comments on “How to prevent robots.txt from getting indexed in google or yahoo?
  1. Useful tip for beginners.

    Nirmal’s last blog post..Customize Windows Live Messenger with Messenger Plus!

  2. I don’t think it is just a tip for the beginners…very very few know the usage X-Robots-Tag…

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Recommended Videos

Robots.txt Formatting | Affilorama
Robots.txt Formatting | Affilorama
How to Optimize Your Robots txt File
How to Optimize Your Robots txt File
How To Stop Search Engine From Crawling  (Part 7) | Digital Marketing | In Hindi | Go Digital |
How To Stop Search Engine From Crawling (Part 7) | Digital Marketing | In Hindi | Go Digital |
Robots.txt   How to Use Robots.txt files
Robots.txt How to Use Robots.txt files
HOW TO Create and Add Custom Robots.txt File on Blogger Site – Indexing Your Blog | SEO Blogger - 2
HOW TO Create and Add Custom Robots.txt File on Blogger Site – Indexing Your Blog | SEO Blogger - 2