How do can the crawler not access my robots.txt file but have 0 crawler issues?
-
So I'm getting this errorOur crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.https://www.evernote.com/l/ADOmJ5AG3A1OPZZ2wr_ETiU2dDrejywnZ8kHowever, Moz is saying I have 0 Crawler issues. Have I hit an edge case? What can I do to rectify this situation? I'm looking at my robots.txt file here: http://www.dateideas.net/robots.txt however, I don't see anything that woudl specifically get in the way.I'm trying to build a helpful resource from this domain, and getting zero organic traffic, and I have a sinking suspicion this might be the main culprit.I appreciate your help!Thanks!
-
Hey there! Tawny from Moz's Help Team here.
It looks like your site's robots.txt file is returning some errors to our tools. When I try to visit the robots.txt file from the root domain, which is where our crawler starts, I get a warning that the DNS address can't be found: https://www.screencast.com/t/ezfsiyVso4B9 That same file is returning a 503 error to our crawler: https://www.screencast.com/t/ROlNo8AQz
That robots.txt file doesn't redirect anywhere, so you may want to consider putting in a redirect there to your robots.txt file at http://www.dateideas.net/robots.txt.
The reason you're seeing 0 issues reported is that we weren't able to reach your robots.txt file, so we stopped crawling and didn't have any issues to report.
I would speak to your web developer or whoever manages your site for you about making sure that your robots.txt file is fully accessible to our crawlers and can be reached in a browser.
I hope this helps! If you've still got questions, feel free to shoot us a note at help@moz.com and we'll do our best to sort everything out with you!
-
This just an advice that worked for me in the past for the same issue and of course is not in the user guide. Simple a delete the website or project from Moz Pro and create a new one for the same domain, I dont know why but all the errors and Issues disappeared.
-
Well technically if it can't crawl your site, it won't be able to find any issues...
I think this may be an error with the Moz crawler rather than any crawlers - checking with site:dateideas.net pulls up some results, although they seem to be tag and archive pages at the moment, which I'd be looking to noindex to avoid duplicate results in the future... You can always check Google's crawling by making sure you've registered your site with Google Search Console, and I'd suggest also making sure you've submitted your sitemap.
I have had issues in the past with the Moz bot struggling to crawl on specific hosts, so it might be that... The new crawl seems to work fine for me at the moment, but is it possible your site was down at the moment Moz was crawling it? You can always request a new crawl and monitor your site uptime to see if there's an issue...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I retrieve city based keywords from Moz Keyword Explorer?
I was trying to take keywords based on cities, but could not find an option to set the city based target. Please help!
Moz Bar | | S-A-Marketing1 -
On Page Grader - URL not accessible
We have tried to use the On Page Grader today and it is coming back with URL not accessible for all pages on our website. We previously used the On Page Grader on Friday 10th Nov for a couple of product pages with no issues. Since then, the only changes we have made on the websites is updating some downloadable documents. We have done this several times before and it has never affected Moz. We have not changed the page URLs, and therefore do not know why it is now not working. The pages are working fine on the website with no issues. A link to one of the pages is below. http://www.processinstruments.co.uk/products/dissolved-oxygen-monitor/ Any help would be greatly appreciated.
Moz Bar | | PiMike0 -
Moz crawler only crawls one page?!
Hello there, I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled. We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working. Hopefully someone can point us at the right direction. Thanks in advance, Jeremy
Moz Bar | | mediaxplain.nl0 -
Duplicate Page and Title Issues
On the last crawl, we received errors for duplicate page titles and some duplicate content pages. Here is the issue: We went through our page titles that were marked as duplicate and changed them to make sure their titles were different. However, we just received a new crawl this week and it is saying there are even more duplicate page title errors detected than before. We're wondering if this is a problem with just us or if it has been happening to other Moz users. As for the duplicate content pages, what is the best way to approach this and see what content is being looked at as a "duplicate" set?
Moz Bar | | Essential-Pest0 -
Can you give some examples of MOZ Local Partners who have an API and update quickly?
In answering questions about the speed of MOZ Local, I've seen this a few times: "The time it takes our partners to update the listing really depends on whether or not they have an API. If they do have an API, the listing will be updated pretty much immediately." Can someone provide a list of which Partners have an API, and which do not?
Moz Bar | | irapasternack0 -
How can the Moz Page Grader support a 'keyword portfolio' approach?
I used to use the Page Grader tools to support the old philosophy of one page - one keyword. With more focus now being given to a portfolio of keywords around a topic area - what would be a good approach to using the page grader tool? Obviously getting A's and B's is impossible for multiple keywords. The only way i've seen suggested in moz tools to help with keyword portfolios is to use labels in the ranking measurement and then find averages of the results. Are there other strategies that I can try?
Moz Bar | | AISFM0 -
Issues with on-page report
My on page reports are really bad (F´s all over the place). When I go to a specific page to check why, it seems that Moz is not properly tracking some of the key metrics. For example for the following page (https://www.iguanafix.com.ar/servicios/pisos/palermo?loc=AR.CF.CF.PA) and keyword "pisos palermo" it is tracking that the keyword is not in the title (which it is), the keyword is not in the URL (which clearly is), not in H1, again it is, etc. I guess this is what is affecting my ranking and thus preventing me from defining a strategy to improve my SEO. Any ideas much appreciated. Thanks
Moz Bar | | IguanaFix0 -
Can we export branded reports from Moz Analytics Beta yet?
Hey guys, Just looking around the new Moz-alytics (droooool) and it is superb, but are we able to export and build reports from it yet? Or is this something that will be in the final release? Apologies if this has already been asked. Regards! Joe Sturgess
Moz Bar | | Sturge
Twitter: @Sturge871