Google 'unable to access robots.txt'

[Chabrenas]

I do not have robots.txt files on any of my sites. However, over the past week, Webmaster's Tools reports that gbots didn't crawl the site because they were unable to access robots.txt. Most times, they failed on all of several attempts, but once they succeeded in 1 of 7 tries. Anyone seen this kind of thing? The e-commerce site discussed on an earlier thread is one of the ones most affected - could there be a connection between the two anomalies?

[akirk]

stick a robots.txt on there?!
Google - despite all the negative comments about them is quite obedient about using sitemaps / robots.txt etc. - you can also use them to show priority / and other information...
think of it as a guide to Google telling them how you want your site indexed

Alasdair

[tke71709]

I would make things as easy as possible for Gbot if I were you.

Just add a generic robots.txt file that allows access to everything that Google can get to if you don't have folders and the such that you're concerned about indexing (which you shouldn't seeing as you hadn't taken the time to exclude them via the robots.txt file before).

So either

User-agent: *
Disallow:

So basically all robots, nothing is disallowed.

[Chabrenas]

Thanks. I usually give Google sitemaps, but didn't think I needed to provide robots.txt.

However, I've just uploaded robots.txt and checked that mysite.com/robots.txt exists and contains

User-agent: *
Disallow:

Then I asked Webmaster Tools several times to 'Fetch as Googlebot'. Same result: 'unreachable robots.txt'. I checked both www and non-www URLs in my browser: the file is accessible.

Since the crawl errors file showed that on one occasion the bot succeeded once in 7 tries, I wonder if it is actually a case of it failing to access the site at all. A few days ago, I couldn't access it for 18 hours, although people in other countries could.

I see that every time I ask Tools to do a fetchm it counts down the 500 'fetches remaining'. Is this a function of a free version of WMT?

[Clinton]

In what program did you create the robots.txt? Word etc., are useless. You need to use something like Notepad++ or even Notepad. Further, try running it through a validator (example) for other errors. Invisible characters on the page, format in which it was uploaded, misspelling as robot.txt etc., are common errors.

[Chabrenas]

Good points, but I use Notepad++ all the time. G didn't say the robots.txt file had invalid content, it said it had failed to access it. Take a look at the file for yourself: http://www.ropeysoles.com/robots.txt

I've just checked GWT again, in case it was working from cache (which it shouldn't do, of course). Still the same result.

Tomorrow I'll check Awstats to see if the gbot visit is recorded or whether there were any 500 errors. I've had 2,000 of those in the past 18 days - is that about normal for around 600 pages/4,000 hits a day?

[monty]

This sounds like just a WMT anomaly which often sort themselves out given a few days, but one other thing to check is whether you have any bot/IP blocking rules in .htaccess or firewalls and so on.

[Chabrenas]

Thanks, Monty. No I don't block anything in .htaccess, etc. Nothing has changed since all was running smoothly before.

[crabfoot]

Them gargle bots "time out" too quick, no frickin patience, worse'n like a MSN download.

Them ther things want all the world signed up on a plate waitin an ready for them. If they get served one in seven times, it show nuthin is broke except frikin Gargle. Rest of the time the dang things got no patience. Typical gargle behaviour - sod yr customers, we was here second and we don't wait.

If we ever do a group project on E-P, we have to name it "sod off Gargle" - whatever the real subject. They is a pack of winkers, as grynge would siy ..