Robots.txt

[AL123al]

Hello,

My client has a robots.txt file which says this:

User-agent: *

Crawl-delay: 2

I put it through a robots checker which said that it must have a **disallow command**. So should it say this:

User-agent: *

Disallow:

crawl-delay: 2

What effect (if any) would not having a disallow command make?

Thanks

[MichaelC-15022]

Oops, good catch Paul, you're correct!

[ThompsonPaul]

Michael - you are _incorrect, _I'm afraid! You need to read up on the specifics of the robots exclusion protocol.

A blank Disallow directive absolutely does NOT match all URLs on the site. In order to match all URLs on the site, the configuration would have to be:

User-agent: *
Disallow: /

Note the slash denoting the root of the site. If the field after disallow: is blank, that specifically means no URLs should be blocked. To quote www.robotstxt.org:

Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record.

The second part of that statement is equally important. For a record to be valid, it must include at least one user agent declaration and at least one disallow statement. If you want the file to not block any URLs, you must include the disallow: statement, but leave its value empty.

For more proof of this, here's the exact example, also from robotstxt.org:

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

The main reason for including a robots.txt which doesn't block anything is to help clean up a server's error logs. With no robots.txt in place, an error will be inserted into the logs every time a crawler visits and can't find the file, bloating the logs and obscuring the real errors that might be present. A blank file may lead someone to believe that the robots.txt just hasn't been configured, leading to unnecessary confusion. So a file configured as above is preferable even if no blocking is desired.

Hope that clears things up?

Paul

[edited to replace line breaks in the code examples that were stripped out by Moz text editor]

[MichaelC-15022]

Caroline,

REMOVE THE DISALLOW LINE.

I am concerned that that line will match all URLs on the site, and disallow the ENTIRE site.

Michael.

[AL123al]

Thanks to both of you. I will recommend that the Robots.txt is changed to:

User-agent: *
Disallow:

in order to configure it right and miss out the crawl delay.

Caroline

[ThompsonPaul]

Your second version is correct, ALBA123 - the robots protocol does require you to include a disallow statement in order to be correctly configured, even if it's blank to indicate crawling the full site.

I really question the wisdom of having a crawl delay in place though. What's the reason for doing so? I never want anything to get in the way of the search crawlers "doing their thing" as effectively as possible.

It's also rather strange to go to a crawl delay, but not be blocking the crawling of any of the non-essential sections of the site. Usually a crawl delay is in place to reduce the resource use by crawlers (vastly better to improve the efficiency of the site or get stronger hosting) but delaying crawl for the whole site instead of saving resources by blocking the non-essential areas first is pretty heavy-handed.

Doers that make sense?

Paul

[MichaelC-15022]

I'd be really, REALLY careful about a disallow statement like that: you run the risk of disallowing your entire website.

FYI I'm not sure putting a crawl delay in your robots.txt file is the right answer.  I saw an example a week or so ago where Google (I think, but maybe it was Bing) explicitly said somewhere that it had ignored the crawl delay in the robots.txt. I would specify the crawl delay in Webmaster Tools instead.  It's hard to find, but it's there

• in Webmaster Tools, select the site you want to set the crawl rate for
• click the Gear icon in the upper right
• you'll see the option there to set the crawl rate