Search engine optimization for small and very small businesses.

Yahoo & Bing May Now Require Robots.txt File

Robots.txt File May No Longer Be Optional

Bing logoIt’s come to my attention very recently that Yahoo and Bing are purging websites from their listings if they don’t have a robots.txt file. One of our clients just got a notice from Bing that their website couldn’t be crawled because there was no robots.txt file.

Yahoo logoThis used to be optional, but apparently that’s no longer the case for Yahoo and Bing. We’ve seen a couple of websites where there are plenty of keywords ranking well in Google but nothing (or almost nothing) showing up in Yahoo and Bing. Upon checking, we’ve found they’re missing the robots.txt file.

Please check to see if you have a robots.txt file, and if not then ask your webmaster to create one for you. Your webmaster will understand what that is; they’re not difficult to create.

7 Comments
  1. Most of the SEO experts are focusing on Google while they forget that Yahoo and Bing still alive. Thanks for the great information about Yahoo and Bing.

  2. Thanks! Yes, Yahoo and Bing are certainly very much alive, and they account for about half as many searches as Google does. That’s a lot of searches.

  3. How is this going to impact CMS sites where the hosting system has a robots.txt file which is not accessible by their individual account holders?

    Only 1 robots.txt file is recommended per site so would you recommend that one be added to the individual account anyway?

    • Hi, Diane. There should be one robots.txt file per domain, not at some aggregate hosting account with multiple domains. So you should be able to add one to the root of your domain on the server. Hope that helps, but please let me know if I misunderstood your question.

  4. Thanks! That’s just what I needed to know. Looks like I will be adding robots.txt files to several of my websites.

  5. Very good post. I certainly love this site.
    Keep writing!

Leave a Reply