Robots.txt - Allow and Disallow. Can they be the same?
-
Hi All,
I need some help on the following:
Are the following commands the same?
User-agent: *
Disallow:
or
User-agent: *
Allow: /
I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots.
Is that correct?
Many thanks,
Aidan
-
Hi Aidan
I'm getting a similar problem on a site I'm working on. The on page rank checker "can't reach the page". I've checked everything obvious (at least I think I have!)
May I ask how you eventually resolved it?
Thanks Aidan
-
Hi
you can use this tool for be sure that the crawler see your files
http://pro.seomoz.org/tools/crawl-test
but you must wait for receive the report to a email.
when you say:
"get the following msg when I try to run On Page Analysis:"
the tools is this?
http://pro.seomoz.org/tools/on-page-keyword-optimization/new
for check the website you can use this:
http://www.opensiteexplorer.org
Ciao
Maurizio
-
Hi,
Thanks for the clarification. So the Robots.txt isn't blocking anything.
Do you know why then i cannot use SEOMoz On Page Analysis and Xenu and Screaming Frog only return 3 URLs?
I get the following msg when I try to run On Page Analysis:
"Oops! We were unable to reach the papge you requested for your report. Please try again later."
Would there be something else blocking me? GWMT Parameters maybe?
-
E' un piacere.
but I don't understand the problem.
if the site have this robots.txt
**User-agent: ***
Allow: /
every crawler can index and see all files of the this website and Seo moz also.
Maybe the problem is different?
Ciao
-
Thanks Maurizio,
I need to do some analysis on this site. Is there a way to use my SEO tools (screaming frog, SEOMoz) to ignore the robots.txt to enable me to do a good site audit?
Thanks again for the answers. Much appreciated
Aidan
-
Hi Aidan
User-agent: *
Disallow:and
User-agent: *
Allow: /are the same
Ciao
Maurizio -
Hi Maurizio,
The reason I asked is because I am working on a site and it's robots.txt is :
User-agent: *
Allow: /
Why would they have this?
I can't use On-Page Analysis or Screaming Frog as it only results in 3 URLs.
Thanks again,
Aidan
-
Hi
1° example:
User-agent: *
Disallow:all User-agent can index your files
2° example
User-agent: *
Disallow: /never User-agent"can index you files
other example here:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Ciao
Maurizio
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I increase my website speed?
Hi, I hope you are doing good. My website speed is too much slow. Mobile speed is 12 and Desktop speed is 39. Please check my website speed.
Web Design | | Bigbrand
&
What can I do for my website to increase speed?
What is best way to increase website speed. Here is my website: https://www.myqurantutor.com/0 -
Can we link back from help documents to product or features pages on website?
Hi, We have all our help documents on subdirectory linked for all the features or products we provide. Like we linked website.com/help/seo-guide from website.com/services/seo-product as that is relevant guide. Do we need to link back from all help guide pages to product pages? Thanks
Web Design | | vtmoz0 -
Can forwarding users from one domain to a different domain damage rank and authority of first domain?
Preliminary Explanation: We launched a new website a couple months back but haven't had much luck in Google taking notice. One of the main attractions to our site is an old flash app that was made nearly a decade ago. As the original developer has long ago moved on and we are unable to figure out how to integrate it with our new site, we've been stuck hosting the flash app on a different domain. As such, users who come to our site and want to use the app must immediately navigate away from our site to this other domain. This has caused our primary domain's bounce rate and average site time to plummet while raising it for the other domain. My question: is this damaging our search rank and page authority with Google for this primary domain/site and counter-acting any other positive SEO changes we can make? How much weight does Google give towards bounce rate/average site time spent by users in its overall calculations for search rank and page authority? Our average site time for this primary domain is resting currently at 50-60 seconds, while for the secondary domain that hosts the old flash app it is 4-5 minutes.
Web Design | | Closetstogo0 -
How we can check whether website design is good for SEO or not?
Is there any tool available to check website design whether it is good for SEO or not?
Web Design | | ross254sidney0 -
Can only get a few pages indexed on by google
Hi I've touched upon this before on previous questions so apologies for repeating myself. In a nutshell out of the 60 webpages submitted to Google 11 have been indexed and out of the 140 images submitted none have indexed any ideas would be great! Here is a screen shot of what Google Webmaster is showing http://www.tidy-books.com/sitemapshow.png and here is the sitemap - > http://www.tidy-books.com/sitemap/us/sitemap.xml Thanks
Web Design | | tidybooks0 -
We believe we accomplished an SEO Parallax site with a nice balance. Can the MOZ community critique this site from an SEO perspective?
Our goal was to accomplish a site that has parallax scrolling and great onsite optimization. We noticed that most Awwward winning sites www.awwwards.com have great parallax scrolling but no SEO. Can the MOZ community critique this site from an SEO perspective? (Note this site was optimized for Chrome or Firefox. If you are using IE, you will be redirected to the old site.) www.posicionamientowebenbuscadores.com Note the site is in BETA still. It has the following technologies CSS3 HTML5 REsponsive Wordpress Parallax Scrolling Onsite Optimization (SEO) No mobile (ran out of funds...)
Web Design | | Carla_Dawson0 -
URLs appear in Google Webmaster Tools that I can't find on my own site?!?
Hi, I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site. For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like: http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs. I think I need to: 1. Find how Googlebot is finding these urls? 2. Find out how to keep out of index (e.g. robots.txt, canonical etc.... Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know? Thanks, Darrell
Web Design | | clickyleap0 -
Search directory - How to apply robots
Hi. On the site I'm working on, we use a search directory to display our search results. It displays as follows - Mydomain.com/search-results/# With the dynamic search results appearing after the hash tag. Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav. How should I handle this? Thanks.
Web Design | | Blenny0