Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi there! Tawny from Moz's Help Team here!
I think I can help you figure out what's going on with your robots.txt file. First things first: we're not starting at the robots.txt URL you list. Our crawler always starts from your Campaign URL and goes from there, and it can't start at an HTTPS URL, so it starts at the HTTP version and crawls from there. So, the robots.txt file we're having trouble accessing is khadination.shop/robots.txt.
I ran a couple of tests, and it looks like this robots.txt file might be inaccessible from AWS (Amazon Web Services). When I tried to curl your robots.txt file from AWS I got a 302 temporary redirect error (https://www.screencast.com/t/jy4MkDZQNbQ), and when I ran it through hurl.it, which also runs on AWS, it returned an internal server error (https://www.screencast.com/t/mawknIyaMn).
One more thing — it looks like you have a wildcard character ( * ) for the user-agent as the first line in this robots.txt file. Best practices indicate that you should put all your specific user-agent disallow commands before a wildcard user-agent; otherwise those specific crawlers will stop reading your robots.txt file after the wildcard user-agent line, since they'll assume that those rules apply to them.
I think if you fix up those things, we should be able to access your robots.txt and crawl your site!
If you still have questions or run into more trouble, shoot us a note at help@moz.com and we'll do everything we can to help you sort everything out.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Localized pages with hreflang markup being reported by Moz as duplicate content
We have 5 websites, each targeted at a different geography. They are all in English, but targeted per country (e.g. US, Canada, UK, Australia). And we've properly implemented hreflang tags on each page, with the US site being the x-default, and with each country-specific page having a self-referencing hreflang tag as well as individual hreflang tags pointing to the other country-specific versions. All seems to be working properly with the search engines. But, each week in our Moz Pro campaigns, Moz reports the pages as "duplicate content". It seems that Moz does not regard the hreflang tags when deciding if content is duplicate or not. I'm not 100% sure that's what's causing Moz to report them as duplicate, but it's my best guess. To date, I've been marking these as "ignore". But, that creates two problems. First, we have new pages all the time, and so this gets to be laborious. Second, it makes it somewhat likely we might miss a real duplicate content issue. Can someone confirm whether Moz should be looking at hreflang tags before considering pages as dupliacte? And possibly offer any suggestions to us if Moz doesn't do that?
Feature Requests | | seoelevated0 -
Can someone give details about Moz Pro Tool in Depth ?
Hi, I have been using some other tool for my monthly SEO activities, due to my advice of friend, thinking to move to Moz Pro. Can Someone give the answers for the questions SERP Ranking Re-freshness ( For how often it would update ) Mobile SERP Ranking ( Whether the tool shows the SERP Rankings on Mobiles ) Backlinks ( Does Backlinks Data has the Historical Index ) Alerts ( The important thing i need was to get instant alerts on Rankings, Backlinks Toxic Score, so how often i would receive those alerts
Feature Requests | | NishanthRaj0 -
Can Moz add an alert to email us when a competitor's site gets a new backlink?
This would be a very useful feature, and other sites are doing this, including Ahrefs.
Feature Requests | | rabbit5190 -
Will MOZ be updating there tools with regards to the new Meta Description length?
Now that Google has increased the number of characters allowed for meta descriptions/snippets when will MOZ be updating the tools to cater for the new lengths? I'm sure a lot of my Meta descriptions that are being flagged as too long with disappear once updated. Cheers Lee B
Feature Requests | | lbagley1 -
Moz crawler is not able to crawl my website
Hello All, I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. " We changed the robots.txt file and checked it . but still the issue is not resolved. URL : https://www.khadination.shop/robots.txt Do let me know what went wrong and wjhat needs to be done. Any suggestion is appreciated. Thank you.
Feature Requests | | Harini.M0 -
Discrepancy in "np" within in my Moz Report?
| Keyword |
Feature Requests | | jessential
| np - /google-structured-data-update/ | In my "New Rankings & Insights | Hill Web Creations" report, while "Google structured data update" moved up significantly, at the same time, "np - /google-structured-data-update/" moved down significantly. Any insights that would offer a deeper understanding of the discrepancy and how to pull both up? If this corresponds with Google Analytics, the "np" Acquisitions > Campaigns > Organic Keywords > "Organic Search Traffic" Report, how is Moz rendering is New Rankings & Insights?0 -
Suggestion to Improve Moz's Link Data
Hello All. The recent Index update (29th January) seems to have produced quite a few shocks to Moz Pro users. Index updates always cause panic and joy every time they are released. However, I feel this is a big improvement that could be made. In this last index, the DA for my site has remained the same. I was expecting a rise, simply because I have been doing some link building since October and was expecting the results to start showing around this time. However, the link metrics for my site looked worse this month than ever before. The index clearly didn't crawl sites and pages that it has crawled the last couple of indexes, and I noticed a few links disappeared from the metrics altogether, links that I know still exist. So I have a suggestion... If Moz has collected link data in previous indexes, can it not preserve that data rather than starting afresh with every new index. Regular checks could be conducted on existing link data to identify expired or broken links, but to disregard them and start anew with every index seems a little odd. That way, Moz would have much more accurate and comprehensive link data which would help to avoid the DA inaccuracy, and provide users with a better measure of how their site is performing against the rest of the web. This may be naive because I do not know the technical side to index updates. Any thoughts, advice, suggestions etc would be appreciated.
Feature Requests | | AHC_SEO0 -
Do MOZ have an affiliate platform?
Do MOZ have an affiliate platform? If so, it's hiding from me, would somebody care to point me the right way? Shall I assume this is the page / they MOZ don't have a direct affiliate program? https://moz.com/partners Many Thanks
Feature Requests | | Whittie0