Robots.txt File May No Longer Be Optional
It’s come to my attention very recently that Yahoo and Bing are purging websites from their listings if they don’t have a robots.txt file. One of our clients just got a notice from Bing that their website couldn’t be crawled because there was no robots.txt file.
This used to be optional, but apparently that’s no longer the case for Yahoo and Bing. We’ve seen a couple of websites where there are plenty of keywords ranking well in Google but nothing (or almost nothing) showing up in Yahoo and Bing. Upon checking, we’ve found they’re missing the robots.txt file.
Please check to see if you have a robots.txt file, and if not then ask your webmaster to create one for you. Your webmaster will understand what that is; they’re not difficult to create.
March 6th, 2013 on 6:08 pm
Thanks! That’s just what I needed to know. Looks like I will be adding robots.txt files to several of my websites.
March 6th, 2013 on 4:20 pm
Hi, Diane. There should be one robots.txt file per domain, not at some aggregate hosting account with multiple domains. So you should be able to add one to the root of your domain on the server. Hope that helps, but please let me know if I misunderstood your question.
March 6th, 2013 on 12:38 pm
How is this going to impact CMS sites where the hosting system has a robots.txt file which is not accessible by their individual account holders?
Only 1 robots.txt file is recommended per site so would you recommend that one be added to the individual account anyway?
January 10th, 2013 on 10:44 am
Thanks! Yes, Yahoo and Bing are certainly very much alive, and they account for about half as many searches as Google does. That’s a lot of searches.
January 10th, 2013 on 6:53 am
Most of the SEO experts are focusing on Google while they forget that Yahoo and Bing still alive. Thanks for the great information about Yahoo and Bing.