Home » Wordpress SEO, Wordpress SEO & Plugins

How to prevent robots.txt from getting indexed in google or yahoo?

 May 23, 2008 2 Comments

Use X-Robots-Tag to prevent google or yahoo from indexing documents with file extensions like .txt, .doc, .pdf etc? Google and yahoo can index all documents uploaded on a website, whether it be .txt files or .doc files or .pdf files. 
 

However you can use X-Robots-Tag to prevent SERPs like google and yahoo from indexing your website.Here is an example of using X-Robots-Tag in your .htaccess file, to tell Google or yahoo not to index the robots.txt file. 
 

<FilesMatch “robots\.txt”>
Header set X-Robots-Tag “noindex, follow”
</FilesMatch>

Here is another example of using X-Robots-Tag in your .htaccess file, to tell Google or yahoo not to index files with .doc extensions. 
 

<FilesMatch “\.doc$”>
Header set X-Robots-Tag “index, noarchive, nosnippet”
</Files>

 
 

We can also try using the “noindex” directive in the robots.txt.However only google obeys this directive though they have not yet announced full support for it in future. Thus X-Robots-Tag seem to be the best option to prevent google or yahoo from indexing robots.txt or any other document.

Related posts:
  1. Meta ROBOTS Tag (Index,Follow,Noindex,NoFollow) in Platinum SEO Plugin
  2. Prevent directory listing in wordpress
  3. Prevent Yahoo! pipes from mashing up your full feeds
  4. How to noindex your feedburner feed content
  5. Find posts/pages recently indexed by google
  6. stop calculating your google supplemental ratio
  7. Download Google Documents from Google Docs
  8. Secure Gmail (Google) account with HTTPS connection
  9. Virtues of a sitemap for your blog/website
  10. Google Keyword Search Monitor

Filed under: Wordpress SEO Wordpress SEO & Plugins

2 Responses to “How to prevent robots.txt from getting indexed in google or yahoo?”

  1. Nirmal said:

    Useful tip for beginners.

    Nirmal’s last blog post..Customize Windows Live Messenger with Messenger Plus!

    # 23 May 2008 at 2:15 pm
  2. techblissonline.com said:

    I don’t think it is just a tip for the beginners…very very few know the usage X-Robots-Tag…

    # 23 May 2008 at 4:16 pm

Leave a Reply