Can't work out robots.txt issue.
-
Hi
I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?
-
I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.
-
Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler
- Remember to put your file is in the top-level directory of your web server.
- Also, check if your hosting provider is not blocking third-party crawlers (server level)
Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz help@moz.com
Also, you should check
Best of luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My site's domain authority is 1\. why is that
Hi Guys My website's domain authority is 1 no matter i try www or non www.. why is that? can you guys please help? Thanks a lot in advance. http://www.opensiteexplorer.org/links?site=autoproject.com.au
Moz Pro | | JazzJack
http://www.opensiteexplorer.org/links?site=www.autoproject.com.au Jazz0 -
Can you highlight a list of keywords in reports?
Hello all, My client recently asked me if there was a way to highlight a specific list of keywords to put on the first page of every report. These would be the highest priority keywords that we would always want to know the status of for his site. In the reports section, I am only seeing options for organizing the keywords by rankings improved/declined and comparing to competitors. Would anyone know how to label/categorize this list of keywords and then produce them as part of the monthly reports? Thank you,
Moz Pro | | Level2Designs
Daniel0 -
Can SEOMOZ Pro suggest me the best keyword to work for?
Hi, how SEOMOZ PRO can suggest me the best keyword to rank easily in the first position of Google's SERP? Is there a tool to find good keyword and got some ideas? Regards
Moz Pro | | jadlib0 -
How can you set SEOmoz to work with your dev site behind an htpasswd?
All sites need to be developed from the small to the grand - and this takes time. Development usually takes place on a subdomain different from our live domain. It is locked down behind an htpasswd during development so its not picked up by searching engines - that may create duplicate content issues if when the site goes live it has already scanned our site on the development server. Its also a security implementation to keep the site away from prying eyes before its ready for launch There could be security holes that have not been tweaked. Whats the best strategy to get SEOmoz involved in this scenario. Its tools are invaluable to the SEO part of the build - but the seomoz crawler bot has a different IP address (being cloud based) - so we cannot just let a single IP address through our htpasswd. Also is there a way to link the dev and live site in seomoz - so when it goes live to maintain all teh same logs without having to create two seperate site campaigns? Thanks!
Moz Pro | | dseo2410 -
Can Open Site Explorer Do This?
Is there any way to set up Open Site Explorer to show these things for competitor external backlinks: Google Page Rank of the page the backlink is on Google Page Rank of the domain the backlink is on Whether the backlink is a follow or no follow Is this possible in OSE? If not, are there any other SEOMOZ Tools that will do this? Thanks.
Moz Pro | | N5c0 -
How can I get ranking keys of competitors
Sometimes it might be clever to analyse the stategy of a competitor to see with which keys they rank successfully. For example I have a webshop, so I type in amazon.com and seomoz spits out: books, dvds, ... I can now see with what keys i wont have a chance and can think of missing keys where my chance might be. Is there a possibility to find out with seomoz?
Moz Pro | | SEOBavaria820 -
Is the Competitive Link Finder having issues?
Hi there, Been trying to use the Competitive Link Finder off-and-on for about an hour now. It keeps telling me it's busy. Is there something known going on right now? And if so, do we know when it comes back? Best Regards, Daniel @ Path Marketing.
Moz Pro | | PathMarketing0
