Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi there! Tawny from Moz's Help Team here!
I think I can help you figure out what's going on with your robots.txt file. First things first: we're not starting at the robots.txt URL you list. Our crawler always starts from your Campaign URL and goes from there, and it can't start at an HTTPS URL, so it starts at the HTTP version and crawls from there. So, the robots.txt file we're having trouble accessing is khadination.shop/robots.txt.
I ran a couple of tests, and it looks like this robots.txt file might be inaccessible from AWS (Amazon Web Services). When I tried to curl your robots.txt file from AWS I got a 302 temporary redirect error (https://www.screencast.com/t/jy4MkDZQNbQ), and when I ran it through hurl.it, which also runs on AWS, it returned an internal server error (https://www.screencast.com/t/mawknIyaMn).
One more thing — it looks like you have a wildcard character ( * ) for the user-agent as the first line in this robots.txt file. Best practices indicate that you should put all your specific user-agent disallow commands before a wildcard user-agent; otherwise those specific crawlers will stop reading your robots.txt file after the wildcard user-agent line, since they'll assume that those rules apply to them.
I think if you fix up those things, we should be able to access your robots.txt and crawl your site!
If you still have questions or run into more trouble, shoot us a note at help@moz.com and we'll do everything we can to help you sort everything out.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Localized pages with hreflang markup being reported by Moz as duplicate content
We have 5 websites, each targeted at a different geography. They are all in English, but targeted per country (e.g. US, Canada, UK, Australia). And we've properly implemented hreflang tags on each page, with the US site being the x-default, and with each country-specific page having a self-referencing hreflang tag as well as individual hreflang tags pointing to the other country-specific versions. All seems to be working properly with the search engines. But, each week in our Moz Pro campaigns, Moz reports the pages as "duplicate content". It seems that Moz does not regard the hreflang tags when deciding if content is duplicate or not. I'm not 100% sure that's what's causing Moz to report them as duplicate, but it's my best guess. To date, I've been marking these as "ignore". But, that creates two problems. First, we have new pages all the time, and so this gets to be laborious. Second, it makes it somewhat likely we might miss a real duplicate content issue. Can someone confirm whether Moz should be looking at hreflang tags before considering pages as dupliacte? And possibly offer any suggestions to us if Moz doesn't do that?
Feature Requests | | seoelevated0 -
Is there a moz tool to optimize original advertorial content?
We are launching a new module that will involve paid advertorial along with unpaid articles. Does Moz have a tool that will help us with optimizing these articles for the English speaking Hong Kong market? We are not interested in optimizing for the US or other markets as people there would not be searching for the content we are producing so the optimization would not be very relevant. Example: https://hongkong.asiaxpat.com/other/263101/guide-to-buying-and-riding-a-motorcycle-in-hong-kong/
Feature Requests | | HKPaul0 -
High Fives for new Moz Pro interface
No question - just happy to see the latest changes to Moz Pro - Nice job all around.
Feature Requests | | 7thPowerInc3 -
Are there no tools for evaluating/recommending internal links in Moz?
Everything I've seen using Moz has been great, but there's one thing I can't figure out how to do. All I want to see is some kind of overview or display of my site's internal link architecture, but there doesn't appear to be anything like that in any of the Moz tools? Am I missing it, or do they just not have anything for that?
Feature Requests | | A.S.0 -
Can Moz add an alert to email us when a competitor's site gets a new backlink?
This would be a very useful feature, and other sites are doing this, including Ahrefs.
Feature Requests | | rabbit5190 -
Why Moz Update has 48 days this time?
Hi There! We have been Monitoring our DA in Moz.com, from past 6 Months and we have observed that every Moz update happens between 25 to 30 days. Why is the next update has been scheduled for 48 days of time? Is there any specific reason for this? Awaiting your response Thanks Malik Zakaria
Feature Requests | | mzakaria0 -
Moz crawler is not able to crawl my website
Hello All, I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. " We changed the robots.txt file and checked it . but still the issue is not resolved. URL : https://www.khadination.shop/robots.txt Do let me know what went wrong and wjhat needs to be done. Any suggestion is appreciated. Thank you.
Feature Requests | | Harini.M0 -
MOZ Site Crawl - Ignore functionality Request
I now understand that the ignore option found in the MOZ Site Crawl tool will permanently remove the item from ever showing up in our Issues again. We desire to use the issues list as kind of like a To-Do checklist with the ultimate goal to have no issues found and would like to "Temporarily Remove" an issue to see if it is shows back up in future crawls. If we properly fix the issue it shouldn't show back up. However, based on the current Ignore function, if we ignore the issue it will never show back up, even if the issue is still a problem. At the same time, the issue could be a known issue that the end user doesn't want to ever show back up and they desire to never have it show again. In this case it might be nice to maintain the current "Permanently Ignore" option. Use the following imgur to see a mockup of my idea for your review. pzdfW
Feature Requests | | StickyLife0