Robots.txt - Allow and Disallow. Can they be the same?
-
Hi All,
I need some help on the following:
Are the following commands the same?
User-agent: *
Disallow:
or
User-agent: *
Allow: /
I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots.
Is that correct?
Many thanks,
Aidan
-
Hi Aidan
I'm getting a similar problem on a site I'm working on. The on page rank checker "can't reach the page". I've checked everything obvious (at least I think I have!)
May I ask how you eventually resolved it?
Thanks Aidan
-
Hi
you can use this tool for be sure that the crawler see your files
http://pro.seomoz.org/tools/crawl-test
but you must wait for receive the report to a email.
when you say:
"get the following msg when I try to run On Page Analysis:"
the tools is this?
http://pro.seomoz.org/tools/on-page-keyword-optimization/new
for check the website you can use this:
http://www.opensiteexplorer.org
Ciao
Maurizio
-
Hi,
Thanks for the clarification. So the Robots.txt isn't blocking anything.
Do you know why then i cannot use SEOMoz On Page Analysis and Xenu and Screaming Frog only return 3 URLs?
I get the following msg when I try to run On Page Analysis:
"Oops! We were unable to reach the papge you requested for your report. Please try again later."
Would there be something else blocking me? GWMT Parameters maybe?
-
E' un piacere.
but I don't understand the problem.
if the site have this robots.txt
**User-agent: ***
Allow: /
every crawler can index and see all files of the this website and Seo moz also.
Maybe the problem is different?
Ciao
-
Thanks Maurizio,
I need to do some analysis on this site. Is there a way to use my SEO tools (screaming frog, SEOMoz) to ignore the robots.txt to enable me to do a good site audit?
Thanks again for the answers. Much appreciated
Aidan
-
Hi Aidan
User-agent: *
Disallow:and
User-agent: *
Allow: /are the same
Ciao
Maurizio -
Hi Maurizio,
The reason I asked is because I am working on a site and it's robots.txt is :
User-agent: *
Allow: /
Why would they have this?
I can't use On-Page Analysis or Screaming Frog as it only results in 3 URLs.
Thanks again,
Aidan
-
Hi
1° example:
User-agent: *
Disallow:all User-agent can index your files
2° example
User-agent: *
Disallow: /never User-agent"can index you files
other example here:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Ciao
Maurizio
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can wordpress actually be bad for sites if it static?
Hello, So last year I did rank for my website. Yet I switched from Adobe Muse to wordpress. I thought it would be great for updating and blobbing if I ever do it. So I got a theme, and went for it I have Yoast and that's it for plugins. But if I take say another couple of years to blog, am I hurting myself with wordpress? Like Google knows I am using wordpress so it expects me to be creating content? I know its an odd question, just had to ask
Web Design | | Berner0 -
I can’t understand, please help?
Our old <acronym title="Search Engine Optimization">SEO</acronym> built a lot of bad links (web 2.0, article ect) and in April of 2012 we received the dreaded Google Webmaster Tools notice of detected unnatural links to XXXXXXXX After 4 requests and disavowing all of the links he created we have finally received Reconsideration request for xxxx Manual spam action revoked The only problem is that this feels more of a penalty than when the received the detected links email? Everything across the board as dropped, I know we added more and more links to the disavow but everything has been going up with the quality we have been putting out until today when we received the manual action revoked. Has anyone experienced this, I feel like I wish we were still under the penalty as the rankings have dived.
Web Design | | BobAnderson0 -
Can white text over images hurt your SEO?
Hi everyone, I run a travel website that has about 30 pre-search city landing pages. In a redesign last year we added large "hero" images to the top of the page, and put our h1 headlines on top of them in white. The result is attractive, but I'm wondering if Google could be reading this page as "white text on white page", which is an obvious no-no, especially if it could seem that we're trying to hide text. Here's an example: http://www.eurocheapo.com/paris/ H1: Expert reviews of cheap hotels in Paris I should add that our SERPs for these city pages has dropped (for "Cheap hotels in X"), but it could obviously be related to other issues. Any advice would be appreciated. Many thanks! Tom
Web Design | | TomNYC0 -
How can we improve our e-commerce site architecture to help best preserve Page Authority?
Today I installed the SEOMoz toolbar for Firefox (very cool, highly recommended). I was comparing our site http://www.ccisolutions.com to this competitor: http://www.uniquesquared.com For the most part, the deeper I go in our site the more the page authority drops. We have a few exceptions where the page authority of a subcategory page is actually better than the cat. page one level up. In comparison, when I was looking at http://www.uniquesquared.com I noticed that their page authority stays at "21" on every single category page I visit. Are you seeing what I'm seeing? Is this potentially a problem with the tool bar or, is there something significantly different about their site architecture that allows them to maintain that PA across all category and sub category pages? Is there something fundamentally wrong with our (http://www.ccisolutions.com) site architecture? I understand that we have longer URLs, but this is an old store with a lot of SKUs, so we have decided not to remove the /category/ and /product/ from the URLs because the 301 redirects that would result wouldn't pass all of the authority they've built up over the years. Interested to know viewpoints on the site architecture and how it might be improved. Thanks!
Web Design | | danatanseo0 -
Can you recommend a responsive web design company of template to accommodate mobile searches
I need to change my old Dreamweaver website www.endeavourcottage.co.uk into a responsive website that can be viewed and used easily on mobile phones. Four years ago when the site was originally built it was one cottage on one basic website but it's grown into a small letting agency with several properties, so I think it's time to invest in a fully functional website. Looking at Google analytics around the fifth of the searches and now done on a mobile phone each month on this particular site, and obviously that's going to become a larger percentage in the future. Does anybody know of a particular company that can produce good quality responsive websites? Or alternatively used a particular responsive website template which was good. One of the guys that work for me is quite a good web designer if we got the right template. The old website is grown into quite a large beast with a blog so I think the idea would be to change the main pages with the properties on, just linking back to the old websites quite extensive blog / historical interest pages which I've built over the last few years. I need to keep the old blog webpages indexed because they have hundreds of back links coming with longtail keywords producing traffic. Have you gone through the process of changing your main business website for a resonsive redesign recently and how did you approach it?
Web Design | | whitbycottages0 -
Can i do this? Will Google penalize me?
I have a page for a Criminal Defense Attorney and i set up a list of the type of criminal charges he is certified to deal with. I wanted to use title tags and put the Keyword "Miami Criminal Defense Attorney" & "Miami Traffic Defense Lawyer"... My question is will Google penalize me for plugging the same Key words over and over on the title tag for each ?? CHECK THE IMAGE to see what I'm talking about... thanks guys. x97dl
Web Design | | marig0 -
Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS. Thanks
Web Design | | DCondon0