Robots.txt - Allow and Disallow. Can they be the same?

[Presenter]

Hi All,

I need some help on the following:

Are the following commands the same?

User-agent: *

Disallow:

or

User-agent: *

Allow: /

I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots.

Is that correct?

Many thanks,

Aidan

[eventurerob]

Hi Aidan

I'm getting a similar problem on a site I'm working on. The on page rank checker "can't reach the page". I've checked everything obvious (at least I think I have!)

May I ask how you eventually resolved it?

Thanks Aidan

[malecce]

Hi

you can use this tool for be sure that the crawler see your files

http://pro.seomoz.org/tools/crawl-test

but you must wait for receive the report to a email.

when you say:

"get the following msg when I try to run On Page Analysis:"

the tools is this?

http://pro.seomoz.org/tools/on-page-keyword-optimization/new

for check the website you can use this:

http://www.opensiteexplorer.org

Ciao

Maurizio

[Presenter]

Hi,

Thanks for the clarification. So the Robots.txt isn't blocking anything.

Do you know why then i cannot use SEOMoz On Page Analysis and Xenu and Screaming Frog only return 3 URLs?

I get the following msg when I try to run On Page Analysis:

"Oops! We were unable to reach the papge you requested for your report. Please try again later."

Would there be something else blocking me? GWMT Parameters maybe?

[malecce]

E' un piacere.

but I don't understand the problem.

if the site have this robots.txt

**User-agent: ***

Allow: /

every crawler can index and see all files of the this website and Seo moz also.

Maybe the problem is different?

Ciao

[Presenter]

Thanks Maurizio,

I need to do some analysis on this site. Is there a way to use my SEO tools (screaming frog, SEOMoz) to ignore the robots.txt to enable me to do a good site audit?

Thanks again for the answers. Much appreciated

Aidan

[malecce]

Hi Aidan

User-agent: *
Disallow:

and

User-agent: *
Allow: /

are the same

Ciao
Maurizio

[Presenter]

Hi Maurizio,

The reason I asked is because I am working on a site and it's robots.txt is :

User-agent: *

Allow: /

Why would they have this?

I can't use On-Page Analysis or Screaming Frog as it only results in 3 URLs.

Thanks again,

Aidan

[malecce]

Hi

1° example:

User-agent: *
Disallow:

all User-agent can index your files

2° example

User-agent: *
Disallow: /

never User-agent"can index you files

other example here:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

Ciao

Maurizio