Welcome to the Q&A Forum

Hi,

Thanks for the clarification. So the Robots.txt isn't blocking anything.

Do you know why then i cannot use SEOMoz On Page Analysis and Xenu and Screaming Frog only return 3 URLs?

I get the following msg when I try to run On Page Analysis:

"Oops! We were unable to reach the papge you requested for your report. Please try again later."

Would there be something else blocking me? GWMT Parameters maybe?

Thanks Maurizio,

I need to do some analysis on this site. Is there a way to use my SEO tools (screaming frog, SEOMoz) to ignore the robots.txt to enable me to do a good site audit?

Thanks again for the answers. Much appreciated

Aidan

Hi Maurizio,

The reason I asked is because I am working on a site and it's robots.txt is :

User-agent: *

Allow: /

Why would they have this?

I can't use On-Page Analysis or Screaming Frog as it only results in 3 URLs.

Thanks again,

Aidan

Hi All,

I need some help on the following:

Are the following commands the same?

User-agent: *

Disallow:

or

User-agent: *

Allow: /

I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots.

Is that correct?

Many thanks,

Aidan