Robots.txt File May No Longer Be Optional
It’s come to my attention very recently that Yahoo and Bing are purging websites from their listings if they don’t have a robots.txt file. One of our clients just got a notice from Bing that their website couldn’t be crawled because there was no robots.txt file.
This used to be optional, but apparently that’s no longer the case for Yahoo and Bing. We’ve seen a couple of websites where there are plenty of keywords ranking well in Google but nothing (or almost nothing) showing up in Yahoo and Bing. Upon checking, we’ve found they’re missing the robots.txt file.
Please check to see if you have a robots.txt file, and if not then ask your webmaster to create one for you. Your webmaster will understand what that is; they’re not difficult to create.