Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi Harini,
Jo from the Moz help team here.
I've had a look at your site and it looks like there is something server side that is blocking our bot.
When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1
https://screencast.com/t/J3hhDTCM
I'm also seeing this message in this third party tool.
"The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"
All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.
I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2
You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2
You'll also want to make sure you are not blocking AWS (Amazon Web Services).
Best of luck!
Jo
-
Thank you Andy . But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?color=
Disallow: /?manufacturer=
Disallow: /?filter_material-fabric=
Disallow: /?filter_color=
Disallow: /?query_type_color=
Disallow: /?filter_size=
Disallow: /?taxonomy=
Disallow: /?view_mode=
Disallow: /?query_type_material-fabric=
Disallow: /?orderby=
Disallow: /?source_id=
Disallow: /?source_tax=
Disallow: /?shop-2__trashed?
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.khadination.shop/sitemap.xml
this was the previous version of robots.txt that were been used ....
-
Hi - As Andy has said, you're not allowing Moz to crawl the site.
Read up on Rogerbot here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot
-
Hi there,
You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.
Allow: /
add this on line 2 of your robots.txt file.
Good luck
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Access all crawl tests
How can I see all crawl tests ran in the history of the account? Also, can I get them sent to an email that isn't the primary one on the account? Please advise as I need this historical data ASAP.
Feature Requests | | Brafton-Marketing0 -
Moz is flagging my callto: links as 404s. I "fixed" with tel: & nofollow and lost 2 DA. Help?
I changed the callto: links to tel: links in the footer and nofollowed them and my domain authority dropped 2. I'm assuming this is because we lost 2.8K "external links" Advice on how to remove 404s for Phone#'s without affecting DA? Will just using the tel: format stop Moz from crawling them as 404's? It was adding links to the 404 list that look like this: www.ocp.org/missals/callto:########### I assumed it was because to the crawlers the links looked like relative links this is what our links look like now: <a <span="" class="html-tag">href</a><a <span="" class="html-tag">="</a>tel:###########" rel="nofollow" title="Call us">########### Old footer links: <a <span="" class="html-tag">href</a><a <span="" class="html-tag">="</a>callto:###########" title="Call us">###########
Feature Requests | | OCPmusic0 -
MOZ Site Crawl - Ignore functionality Request
I now understand that the ignore option found in the MOZ Site Crawl tool will permanently remove the item from ever showing up in our Issues again. We desire to use the issues list as kind of like a To-Do checklist with the ultimate goal to have no issues found and would like to "Temporarily Remove" an issue to see if it is shows back up in future crawls. If we properly fix the issue it shouldn't show back up. However, based on the current Ignore function, if we ignore the issue it will never show back up, even if the issue is still a problem. At the same time, the issue could be a known issue that the end user doesn't want to ever show back up and they desire to never have it show again. In this case it might be nice to maintain the current "Permanently Ignore" option. Use the following imgur to see a mockup of my idea for your review. pzdfW
Feature Requests | | StickyLife0 -
Moz Crawler failing with https redirect?
Is there a way to get the Moz Crawl Test to work with HTTPS? I just got back this error: 902 : Network errors prevented crawler from contacting server for page. Site is set up with a standard 301 to redirect http to https - or at least I certainly hope it is! Rex Swain's HTTP Header Checker took shows a standard 301. Anyone else experiencing this error? btw - this is both a specific question and an opportunity for open discussion... Thanks!
Feature Requests | | seo_plus0 -
Can Moz change this on the Subscriptions dashboard?
Hey Moz, Bit of a random bug bear of mine, each month I log in to download the latest invoice receipt to hand over to accounts for processing etc. After I click the billing and subscriptions link from the profile drop down menu, I am presented with the subscriptions page. The payment history panel placed located on the right all ways shows my first 3 invoices. As this is a snippet in time, would it not be more beneficial from a UX perspective to show the most recent 3 invoices. That way I can get the info I require that tiny bit quicker rather than be presented with invoices that are 3 years old. I know it is a tiny thing but wondered i it could be accommodated. 🙂 Many thanks Tim ypjDGDk
Feature Requests | | TimHolmes1 -
Way to track public's use of keywords over time in MOZ?
I apologize for the basic question but is there a way to track within MOZ the public's use of my keywords over time? My traffic understandably goes down certain times of the year -- holidays, vacation season, back to school, etc. What I want to see is whether my share of the traffic available is staying steady, slipping or even increasing. Sometimes I'll see a big spike in traffic that I suspect is related more to a general increase in traffic in my keywords than anything else. Thanks for any help.
Feature Requests | | NCCompLawyer0 -
Can I embed a Moz report in my own admin site?
I'd like to be able to pull one of the reports that I have on my campaign dashboard on another site (preferably via iframe). Is this possible?
Feature Requests | | m4change0 -
My moz.com profile page authority has vanished.
Hi there Anybody has seen his page authority regarding his own profile page from moz.com vanishing? Mine was aproximately 46, now PA=1 I´ve checked it and nobody is making any comments about this so maybe.... it´s just me :S? Any clues? I loved my link from moz.com!!! Cheers Claudio Heilborn
Feature Requests | | ClaudioHeilborn0