Can't work out robots.txt issue.
-
Hi
I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?
-
I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.
-
Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler
- Remember to put your file is in the top-level directory of your web server.
- Also, check if your hosting provider is not blocking third-party crawlers (server level)
Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz help@moz.com
Also, you should check
Best of luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can we discuss Moz's Forecast? It's been over 100 degrees for a month.
Hey everyone! I just wanted to start a discussion on Moz's Forecast. If you haven't noticed, it has been over 100 degrees for an entire month now, which I have personally never seen before... Why do you think this might be happening? Any ideas or theories? I'd love to hear them!
Moz Pro | | TaylorRHawkins1 -
Why does one page rank while a similar page doesn't?
We have a blog post (actually several of them that have the same SEO characteristics) that brings a fair amount of traffic to our site (relatively speaking), and according to Google Webmaster Tools it averages in the top 10 in SERPs for various terms. This page has no external links to it, and very few internal links pointing to it. When I run Moz's On-Page Grader for the various keywords it ranks for, the page get's an F on all of them. It was not optimized for any keyword, and it isn't our best content; it was a blog post written a few years ago and forgotten about and was never promoted in any way. The topic does happen to be about something that people search for frequently. According to the keyword difficulty tool, all of the keywords it ranks for have 40-45% difficulty. We have lots of other pages on our site that we have tried to optimize and that get A's and B's in On-Page Grader, that have both internal (from the home page and main menu) and external links pointing at them, etc, but they don't rank well at all. Keyword difficulty for these keywords is in the same range, from 37 - 53%. Why does this one page rank so well when the other pages don't? Additionally, we have been looking at a competitor who has a page that ranks #1 in universal results for numerous keywords according to SEMRush, yet the page gets an F On-page Grader for those keywords. The page has 3 links to it, all from the same domain and it has a very low domain and page authority. The Domain Authority of this page is 47 and the page authority is 33 according to Open Site Explorer (compared with our DA of 30, and PA of 1), and the social metrics are a bit higher than ours, but neither has a lot (they may have 15 likes to our 10). Why does this page rank so well for them? How can we get our Page Authority higher? Thanks for any and all help.
Moz Pro | | mukunig1 -
Blocked by Meta Robots.
Hi, I get this warning on my reporting. Blocked by Meta Robots - This page is being kept out of the search engine indexes by meta-robots. what does that means ? and how to solve that, if i using wordpress as my website engine. and about rel=canonical , in which page I should put this tag, in original page, or in copy page ? thanks for all of your answer, it will be means a lot
Moz Pro | | theconversion0 -
Why my site page don't appeare in the ranking ?
why my site page don't appear in the ranking, which type of domain tracking should i choose to track all my site pages ( my site is a .blogspot.com ) ? when i check it i found that they appear in the first 30 results although seomoz saying they are not in the first 50 result !
Moz Pro | | AmrKamal0 -
Do the SEOmoz Campaign Reports follow Robots.txt?
Hello, Do the SEOmoz Campaign Reports (that track errors and warnings for a website) follow rules I write in the robots.txt file? I've done all that I can to fix the legitimate errors with my website, as reported by the fabulous SEOmoz tools. I want to clean up my pages indexed with the search engines so I've written a few rules to exclude content from Wordpress tag URLs for instance. Will my campaign report errors and warnings also drop as a result of this?
Moz Pro | | Flexcin0 -
How can I export from Followerwonk into a Twitter List?
I use Hootsuite but can't see anywhere in there or find any other tools to import a text file of Twitter usernames into a Twitter list. Does anyone know how?
Moz Pro | | benners0 -
How well is your Internet marketing working for your business?
Not asking about specifics of money, but this is just a general question to see what kind of return people get for their Internet marketing time investment.. I would say I'm getting over 40,000 hits per year for one business which is holiday cottages but I only own one cottage and act as an agent for three other clients. Because the campaign is working well for me at the moment I get probably 95% occupancy throughout the year, which you would have no chance of accomplishing without SEO intensive input. If I had to pay £0.50 a click approximately obviously it wouldn't be worthwhile, especially when you realise that a percentage of the traffic and not related to your product. I also have a second business which is installing wireless security alarms over the Internet and once again approximate traffic at the moment is almost approaching 40,000 again per year. Now I'm sure there are some big hitters out but my general question is how productive, is you're organic results and Internet marketing for your personal business in comparison to what it would cost you if you had to pay per click? I'm kind of wondering how well the Internet is working people on this forum for their particular businesses? Good, Bad ? whats your story ? Cheers alan
Moz Pro | | whitbycottages0 -
How can a competitor show no internal links?
I'm comparing metrics between my site and competitors. Statistics on Site Explorer show that one competitor has no internal links. How is that possible? http://screencast.com/t/P9WX7R9hpdD Thanks for your insight. P9WX7R9hpdD
Moz Pro | | addventures0