Can't work out robots.txt issue.
-
Hi
I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?
-
I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.
-
Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler
- Remember to put your file is in the top-level directory of your web server.
- Also, check if your hosting provider is not blocking third-party crawlers (server level)
Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz help@moz.com
Also, you should check
Best of luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my site page don't appeare in the ranking ?
why my site page don't appear in the ranking, which type of domain tracking should i choose to track all my site pages ( my site is a .blogspot.com ) ? when i check it i found that they appear in the first 30 results although seomoz saying they are not in the first 50 result !
Moz Pro | | AmrKamal0 -
Why can't I get the search volume data from Google in the Keyword difficulty tool?
Hi, For some reason I can't get the traffic data from Google. when I'm doing a keyword research via the Difficult tool then i need to check the keywords volume in Google Adwords and copy the words into MOZ's keyword tool to check feasibility. mZGRSbC.png?1
Moz Pro | | Indiatravelz0 -
Merged websites issue
Hi Before my time, company big.com took over company small.com. They decided to replicate big.com web pages onto small.com - so both websites have identical pages and copy, just different domains. Within the small.com sitmap.xml they list only big.com urls. They are also using big.com google analytics tracking code on small.com. I have no idea what happened to the original content on small.com or if they put 301 redirects on. I am thinking: do a 301 redirect on small domain to big domain. A) Agree? Small domain is likely to have valuable historic inbound links which are now going to 404 pages. After I do the 301 should these then appear in big.com SEOMoz campaign and on big.com webmaster tool for me to fix? B) Views? Or should I get up webmaster tool on small.com and fix that first? C) Views? Many thanks in advance guys, sorry its a long statement! Richard
Moz Pro | | Richard5550 -
Can't get Page Analysis for homepage - any ideas?
Using seomoz toolbar and can't get Page Analysis for homepage - it comes up blank for "page elements", "page attributes" and "link data". Any help would be great. Cheers, Don
Moz Pro | | nositeleftbehind0 -
How Does On Page Analysis work
Hi guys, I just need to run something past you. when I look at my on page analysis I have 5 key terms I am focusing on. For instance one of them is "computer backup". According to the report the current grade is 'F' when looking at site page "/" which I assume is the home page.
Moz Pro | | cubetech
When I do a lookup on other pages of the site it gets a ranking of A. Which is good. But since the hompage ranking went from A to F my rankings have definitely been affect. So i guess my questions are: does "/" mean the hompage, or all pages overall. What should I really be looking at here. I am assuming that you select certain pages to target certain key words. Should i be focusing like this, or more to the "/". Thanks Guys hoping to clear this one up.0 -
How can I track multiple pages using SEOMoz
Hi there, I setup my domain along with 20 keywords to target. One of the first things I learned is not to focus on too many keywords per page. For this reason I decided to create 5 sub pages and link these pages on my index page. My questions to you: 1. Does SEOMoz track the keyword rankings for the URL entered only?
Moz Pro | | debeenus
2. If the answer for 1 is YES, Would I have to setup those 5 extra URL's as new sites along with their 3 to 5 keywords? Thank you,0 -
Fetch googlebot for sites you don't own?
I've used the "fetch as googlebot" tool in Google webmaster tools to submit links from my site, but I was wondering if there was any type of tool or submission process like this for submitting links from other sites that you do not own? The reason I ask is, I worked for several months to get a website to accept my link as part of their dealer locator tool. The link to my site was published a few months ago, however I don't think google has found it and the reason could be because you have to type in your zip code to get the link to appear. This is the website that I am referencing: http://www.ranchhand.com/dealers.php?zip=78070&radius=20 (my website is www.rangeroffroad.com) Is there any way for Google to index the link? Any ideas?
Moz Pro | | texmeix0 -
Can SEOMOZ do Local SEO ?
Most of our clients are local but we can not get local or regional (for example VA State or Washington DC) SERP results within SEOMOZ. it would be nice to create custom search profiles based on region (State or County) and track SERP within SEOMOZ. Ranking reports and even keyword difficulty analysis etc. within SEOMOZ are misleading for local web sites. SEOMOZ default site setup assumes every business doing SEO serves all over the US. I think it will be very beneficial if we have a choice to select State or County of local web sites during initial campaing set up at SEOMOZ. what do you think? do you know any work around of this problem? Thanks,
Moz Pro | | CertifiedSEO
Lewis2