Robots.txt - Allow and Disallow. Can they be the same?
-
Hi All,
I need some help on the following:
Are the following commands the same?
User-agent: *
Disallow:
or
User-agent: *
Allow: /
I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots.
Is that correct?
Many thanks,
Aidan
-
Hi Aidan
I'm getting a similar problem on a site I'm working on. The on page rank checker "can't reach the page". I've checked everything obvious (at least I think I have!)
May I ask how you eventually resolved it?
Thanks Aidan
-
Hi
you can use this tool for be sure that the crawler see your files
http://pro.seomoz.org/tools/crawl-test
but you must wait for receive the report to a email.
when you say:
"get the following msg when I try to run On Page Analysis:"
the tools is this?
http://pro.seomoz.org/tools/on-page-keyword-optimization/new
for check the website you can use this:
http://www.opensiteexplorer.org
Ciao
Maurizio
-
Hi,
Thanks for the clarification. So the Robots.txt isn't blocking anything.
Do you know why then i cannot use SEOMoz On Page Analysis and Xenu and Screaming Frog only return 3 URLs?
I get the following msg when I try to run On Page Analysis:
"Oops! We were unable to reach the papge you requested for your report. Please try again later."
Would there be something else blocking me? GWMT Parameters maybe?
-
E' un piacere.
but I don't understand the problem.
if the site have this robots.txt
**User-agent: ***
Allow: /
every crawler can index and see all files of the this website and Seo moz also.
Maybe the problem is different?
Ciao
-
Thanks Maurizio,
I need to do some analysis on this site. Is there a way to use my SEO tools (screaming frog, SEOMoz) to ignore the robots.txt to enable me to do a good site audit?
Thanks again for the answers. Much appreciated
Aidan
-
Hi Aidan
User-agent: *
Disallow:and
User-agent: *
Allow: /are the same
Ciao
Maurizio -
Hi Maurizio,
The reason I asked is because I am working on a site and it's robots.txt is :
User-agent: *
Allow: /
Why would they have this?
I can't use On-Page Analysis or Screaming Frog as it only results in 3 URLs.
Thanks again,
Aidan
-
Hi
1° example:
User-agent: *
Disallow:all User-agent can index your files
2° example
User-agent: *
Disallow: /never User-agent"can index you files
other example here:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Ciao
Maurizio
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can the design still be considered adaptive if the URL is different?
I was under the impression our site had a mobile dedicated design, but my developers are telling me we have an adaptive design. The mobile site is set up different and has different content and the url is as follows: www.site.com/MobileView/MobileHome.aspx Can it still be considered adaptive if the URL is not the exact same? Hopefully this make sense and I appreciate anyone's input!
Web Design | | AliMac260 -
How to find out that none of the images on my site violates copyrights? Is there any tool that can do this without having to check manually image by image?
We plan to add several thousand images to our site and we outsourced the image search to some freelancers who had instructions to just use royalty free pictures. Is there any easy and quick way to check that in fact none of these images violates copyrights without having to check image by image? In case there are violations we are unaware of, do you think we need to be concerned about a risk of receiving Takedown Notices (DMCA) before owner giving us notification for giving us opportunity to remove the photo?
Web Design | | lcourse1 -
Can a cloud based firewall affect my search ranking?
Hi, I recently implemented a firewall on my website to prevent hacking attacks. We were getting a crazy amount of people per day trying to brute force our website. I used the sucuri cloud proxy firewall service which they claim because of the super fast caching actually helps SEO. I was just wondering is this true? Because we're slowly falling further and further down the SERPS and i really don't know why. If not, is there any major google update recently I don't know about? Thanks, Robert
Web Design | | BearPaw880 -
How Can I Make My Site iPhone Friendly?
I have been looking into making my website for iphone friendly as my analytics are not great for the iphone and I know when I try to navigate around it on an iphone it can be tough. I was told that if I make changes to the layout that it would affect my layout across everything, which I did not want to do. So I have two questions: Is this correct regarding the layout? If so, if you did something like m.waikoloavacationrentals.com which would be the mobile version how would that possibly effect your rankings with regards to the traffic distribution? Any feedback would be appreciated. Also if anyone has any experience in doing this I would be interested in discussing further.
Web Design | | RobDalton0 -
Can anyone recommend a tool that will identify unused and duplicate CSS across an entire site?
Hi all, So far I have found this one: http://unused-css.com/ It looks like it identifies unused, but perhaps not duplicates? It also has a 5,000 page limit and our site is 8,000+ pages....so we really need something that can handle a site larger than their limit. I do have Screaming Frog. Is there a way to use Screaming Frog to locate unused and duplicate CSS? Any recommendations and/or tips would be great. I am also aware of the Firefix extensions, but to my knowledge they will only do one page at a time? Thanks!
Web Design | | danatanseo0 -
How can i embed my video into a table using SEO embed setting?
We use Wistia.com to embed our videos. They have different options for embed settings and we prefer to use the SEO embed setting, however, when we use that setting we aren't able to insert the video in a table side by side with another image or text. When we try, the video jumps out of the table and the table gets (for lack of a better work) out of wack. When we embed the video with the iframe embed setting, the video can be placed in a table with no issues, but then we don't get the SEO credit. We have our site in wordpress. I'm not sure if that has something to do with the tables getting messed up. Check out this link to see an example of how we want the video to show up. http://www.3000doorhangers.com/ Any suggestions as to how we can use the SEO embed setting within a table as shown in the above link?
Web Design | | JimDirectMailCoach0 -
Can't figure out what's going on with these strange 403's
The last crawl found a good number of 404's and I can't figure out what's going on. I don't even recognize these links. What's really strange is a few of them high a fairly decent page authority. For instance this one has a PA of 55: http://noahsdad.com/?path=http%3A%2F%2Feurosystems.it%2Fconf_commerciale%2Fimages%2Fdd.gif%3F%3F There are several more like this one also, it seems most of these new ones have "Feurosystems" in the link...I have no idea what that is. Just curious what you guys think is going on, why these are 404'ing, and how to fix it. Thanks. Edit: I took out the "%" from the links and I get this: http://noahsdad.com/?path=http3A2F2Feurosystems.it2Fconf_commerciale2Fimages2Fdd.gif3F3F which takes me to a page on my site. I have no idea what's going on, or what that link is. Hoping someone can chime in because this is strange. Another edit: I just checked out the Google Webmaster's and it looks like these errors are 403's and all started around March 21st. I have no idea what happend on March 21st to start causing all of these errors, but I'd sure like to get it fixed. 🙂
Web Design | | NoahsDad0 -
How can the Web site designer and the SEO strategist work together peacefully?
The organization I work for has decided to re-design or re-develop the existing company Web site. My part in this project is to come up with new features to add to the site, as well as making the site SEO-friendly (copywriting, link-building, keyword research, etc.). I don’t know a thing about Web site design, coding, format, etc., and I guess I will have to work with a designer on this project. How would I go about finding a Web site designer? Should they have some SEO knowledge? How much designer, coding and site structure knowledge should I have? And how do we not infringe on one another as we work together? (Sorry so many questions.)
Web Design | | Obie0