Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi there! Tawny from Moz's Help Team here!
I think I can help you figure out what's going on with your robots.txt file. First things first: we're not starting at the robots.txt URL you list. Our crawler always starts from your Campaign URL and goes from there, and it can't start at an HTTPS URL, so it starts at the HTTP version and crawls from there. So, the robots.txt file we're having trouble accessing is khadination.shop/robots.txt.
I ran a couple of tests, and it looks like this robots.txt file might be inaccessible from AWS (Amazon Web Services). When I tried to curl your robots.txt file from AWS I got a 302 temporary redirect error (https://www.screencast.com/t/jy4MkDZQNbQ), and when I ran it through hurl.it, which also runs on AWS, it returned an internal server error (https://www.screencast.com/t/mawknIyaMn).
One more thing — it looks like you have a wildcard character ( * ) for the user-agent as the first line in this robots.txt file. Best practices indicate that you should put all your specific user-agent disallow commands before a wildcard user-agent; otherwise those specific crawlers will stop reading your robots.txt file after the wildcard user-agent line, since they'll assume that those rules apply to them.
I think if you fix up those things, we should be able to access your robots.txt and crawl your site!
If you still have questions or run into more trouble, shoot us a note at help@moz.com and we'll do everything we can to help you sort everything out.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No access to the Analytics part of the website. Help?
I have accepted a seat on my managers subscription to help look into our website and the analytics part specifically but every time I try to access wither the keyword explorer or any other of the analytics part of the website it appears that there is a broken link and will just show a white screen with only the headers and footers intact.
Feature Requests | | libbyh2019
Could someone please let me know how to resolve the issue?
Kind regards,
Olivia Houghton.0 -
High Fives for new Moz Pro interface
No question - just happy to see the latest changes to Moz Pro - Nice job all around.
Feature Requests | | 7thPowerInc3 -
Moz Keyword Explorer - Explore by Folder
When using Moz Keyword Explorer to search for my ranking keywords is there any way to see the ranking keywords for a directory or folder on a site? For example: if a blog is located at a subdomain, I can see all the ranking keywords for the blog by selecting subdomain and searching for https://blog.example.com. However, I would like to be able to search for all of a blog's ranking keywords if the blog URL is something like https://example.com/blog/ instead. If this is not feature, can I request its addition? Thank you!
Feature Requests | | LEAP_US_Marketing1 -
Can Moz add an alert to email us when a competitor's site gets a new backlink?
This would be a very useful feature, and other sites are doing this, including Ahrefs.
Feature Requests | | rabbit5190 -
Crawl error : 804 https (SSL) error
Hi, I have a crawl error in my report : 804 : HTTPS (SSL) error encountered when requesting page. I check all pages and database to fix wrong url http->https but this one persist. Do you have any ideas, how can i fix it ? Thanks website : https://ilovemypopotin.fr/
Feature Requests | | Sitiodev0 -
MOZ Campaigns Down?
MOZ Campaigns has been displaying a blank page in the last 2 hours. Is anyone else experiencing this outage? I have attempted to access MOZ via an iPad and an iPhone using Safari and Chrome. My geographic location is France.
Feature Requests | | Kingalan10 -
Moz Local - Does it do the work for you?
Good Morning Everyone, I have a quick question regarding Moz Local ... I understand what it does and the purpose and how great it is... My question is this: If you sign up for Moz local, does it actually send all of the proper and necessary info to each of the aggregators or sites that it lists or does it simply tell me what information i need to send them and i have to provide this info to each one myself? For example, does it say something like "Inconsistent info found on Yelp, please do this to correct" .... OR, does it actually make the corrections and send to yelp for me? Thanks
Feature Requests | | Prime850 -
Is there a way to add custom dashboards into Moz's "Custom Reporting"?
I have dashboards set up in Google Analytics that track conversions and year over year traffic data and just wanted to see if their was a way to add those extra metrics Moz's Custom Report? Any help or suggestions would be much appreciated!
Feature Requests | | Gruenagency0