Robots.txt - Allow and Disallow. Can they be the same?
-
Hi All,
I need some help on the following:
Are the following commands the same?
User-agent: *
Disallow:
or
User-agent: *
Allow: /
I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots.
Is that correct?
Many thanks,
Aidan
-
Hi Aidan
I'm getting a similar problem on a site I'm working on. The on page rank checker "can't reach the page". I've checked everything obvious (at least I think I have!)
May I ask how you eventually resolved it?
Thanks Aidan
-
Hi
you can use this tool for be sure that the crawler see your files
http://pro.seomoz.org/tools/crawl-test
but you must wait for receive the report to a email.
when you say:
"get the following msg when I try to run On Page Analysis:"
the tools is this?
http://pro.seomoz.org/tools/on-page-keyword-optimization/new
for check the website you can use this:
http://www.opensiteexplorer.org
Ciao
Maurizio
-
Hi,
Thanks for the clarification. So the Robots.txt isn't blocking anything.
Do you know why then i cannot use SEOMoz On Page Analysis and Xenu and Screaming Frog only return 3 URLs?
I get the following msg when I try to run On Page Analysis:
"Oops! We were unable to reach the papge you requested for your report. Please try again later."
Would there be something else blocking me? GWMT Parameters maybe?
-
E' un piacere.
but I don't understand the problem.
if the site have this robots.txt
**User-agent: ***
Allow: /
every crawler can index and see all files of the this website and Seo moz also.
Maybe the problem is different?
Ciao
-
Thanks Maurizio,
I need to do some analysis on this site. Is there a way to use my SEO tools (screaming frog, SEOMoz) to ignore the robots.txt to enable me to do a good site audit?
Thanks again for the answers. Much appreciated
Aidan
-
Hi Aidan
User-agent: *
Disallow:and
User-agent: *
Allow: /are the same
Ciao
Maurizio -
Hi Maurizio,
The reason I asked is because I am working on a site and it's robots.txt is :
User-agent: *
Allow: /
Why would they have this?
I can't use On-Page Analysis or Screaming Frog as it only results in 3 URLs.
Thanks again,
Aidan
-
Hi
1° example:
User-agent: *
Disallow:all User-agent can index your files
2° example
User-agent: *
Disallow: /never User-agent"can index you files
other example here:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Ciao
Maurizio
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Block parent folder in robot.txt, but not children
Example: I want to block this URL (which shows up in Webmaster Tools as an error): http://www.siteurl.com/news/events-calendar/usa But not this: http://www.siteurl.com/news/events-calendar/usa/event-name
Web Design | | Zuken0 -
Can anyone help me detect some SEO improvements onpage please...
Can anyone help me detect some SEO improvements onpage please... I have shortened the website URl so its not easily found when searched via search engines.. http://goo.gl/GlfMRl Please have a look and give me some tips. Thanks
Web Design | | Nettv0 -
How Can I Make My Site iPhone Friendly?
I have been looking into making my website for iphone friendly as my analytics are not great for the iphone and I know when I try to navigate around it on an iphone it can be tough. I was told that if I make changes to the layout that it would affect my layout across everything, which I did not want to do. So I have two questions: Is this correct regarding the layout? If so, if you did something like m.waikoloavacationrentals.com which would be the mobile version how would that possibly effect your rankings with regards to the traffic distribution? Any feedback would be appreciated. Also if anyone has any experience in doing this I would be interested in discussing further.
Web Design | | RobDalton0 -
Can white text over images hurt your SEO?
Hi everyone, I run a travel website that has about 30 pre-search city landing pages. In a redesign last year we added large "hero" images to the top of the page, and put our h1 headlines on top of them in white. The result is attractive, but I'm wondering if Google could be reading this page as "white text on white page", which is an obvious no-no, especially if it could seem that we're trying to hide text. Here's an example: http://www.eurocheapo.com/paris/ H1: Expert reviews of cheap hotels in Paris I should add that our SERPs for these city pages has dropped (for "Cheap hotels in X"), but it could obviously be related to other issues. Any advice would be appreciated. Many thanks! Tom
Web Design | | TomNYC0 -
Does anyone know how much a wordpress site can store (in terms of data) I want to put all my movies on it and use it as a personal global external hard drive! Thanks!!
So basically, I have about 500 GB of movies on my computer and I don't want to buy an external hardrive. I don't want to spend the money A website I could access anytime, and anywhere, without having to carry my external with me everywhere I go. Thanks in advance for any help/ references.
Web Design | | TylerAbernethy1 -
Can you recommend a responsive web design company of template to accommodate mobile searches
I need to change my old Dreamweaver website www.endeavourcottage.co.uk into a responsive website that can be viewed and used easily on mobile phones. Four years ago when the site was originally built it was one cottage on one basic website but it's grown into a small letting agency with several properties, so I think it's time to invest in a fully functional website. Looking at Google analytics around the fifth of the searches and now done on a mobile phone each month on this particular site, and obviously that's going to become a larger percentage in the future. Does anybody know of a particular company that can produce good quality responsive websites? Or alternatively used a particular responsive website template which was good. One of the guys that work for me is quite a good web designer if we got the right template. The old website is grown into quite a large beast with a blog so I think the idea would be to change the main pages with the properties on, just linking back to the old websites quite extensive blog / historical interest pages which I've built over the last few years. I need to keep the old blog webpages indexed because they have hundreds of back links coming with longtail keywords producing traffic. Have you gone through the process of changing your main business website for a resonsive redesign recently and how did you approach it?
Web Design | | whitbycottages0 -
Search directory - How to apply robots
Hi. On the site I'm working on, we use a search directory to display our search results. It displays as follows - Mydomain.com/search-results/# With the dynamic search results appearing after the hash tag. Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav. How should I handle this? Thanks.
Web Design | | Blenny0