Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi Harini,
Jo from the Moz help team here.
I've had a look at your site and it looks like there is something server side that is blocking our bot.
When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1
https://screencast.com/t/J3hhDTCM
I'm also seeing this message in this third party tool.
"The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"
All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.
I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2
You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2
You'll also want to make sure you are not blocking AWS (Amazon Web Services).
Best of luck!
Jo
-
Thank you Andy
. But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?color=
Disallow: /?manufacturer=
Disallow: /?filter_material-fabric=
Disallow: /?filter_color=
Disallow: /?query_type_color=
Disallow: /?filter_size=
Disallow: /?taxonomy=
Disallow: /?view_mode=
Disallow: /?query_type_material-fabric=
Disallow: /?orderby=
Disallow: /?source_id=
Disallow: /?source_tax=
Disallow: /?shop-2__trashed?
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.khadination.shop/sitemap.xml
this was the previous version of robots.txt that were been used ....
-
Hi - As Andy has said, you're not allowing Moz to crawl the site.
Read up on Rogerbot here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot
-
Hi there,
You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.
Allow: /
add this on line 2 of your robots.txt file.
Good luck
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Pricing Needs Update
Hi I am coming from Ahrefs and Moz plans need serious update about the limits especially rank tracking. Ahrefs is providing more keywords $0.08/keyword (5000 keywords for the advanced plan) and update interval of 3 days. Moz large plan is $0.13/keyword with intervals of 1 week. Can this be looked into? Hopefully Moz can improve the limits for keywords ranking/tracking. In 2020 where many rank tracking tools provide daily update, 1 week is a bit ridiculously long. Hopefully Moz can make some improvement in this area,
Feature Requests | | EdmondHong0 -
No access to the Analytics part of the website. Help?
I have accepted a seat on my managers subscription to help look into our website and the analytics part specifically but every time I try to access wither the keyword explorer or any other of the analytics part of the website it appears that there is a broken link and will just show a white screen with only the headers and footers intact.
Feature Requests | | libbyh2019
Could someone please let me know how to resolve the issue?
Kind regards,
Olivia Houghton.0 -
Does/could the MpzBar highlight your website in the SERPs?
Is anyone aware of a setting in the MozBar which would highlight your website within SERPs? Or, even more exciting, display a number on the bar when looking at a SERP, showing your best current position for that query (in real time)? If not, it'd be interesting to know your thoughts on the idea? Great for quick checks and references or would it discourage people from staying vigilant in the SERPs and keeping an eye on the competition? (Note: we do have MOZ Pro and are aware that it does something similar. I'm thinking more for a quick check of those difficult targets where you're not hitting the first couple of pages).
Feature Requests | | MSGroup0 -
Get the keywords Google is ranking my website
Hi, I'm using the Moz keyword explorer to know which keywords Google is ranking my website. It says my website is ranked just for 7 words which I find very estrange... Any idea? If I use the google AdWords keyword explorer I get a lot more of keywords for my site. Which one should I trust to improve my seo? Thanks, Regards
Feature Requests | | jcobo0 -
Will MOZ be updating there tools with regards to the new Meta Description length?
Now that Google has increased the number of characters allowed for meta descriptions/snippets when will MOZ be updating the tools to cater for the new lengths? I'm sure a lot of my Meta descriptions that are being flagged as too long with disappear once updated. Cheers Lee B
Feature Requests | | lbagley1 -
How to reschedule a missed free moz pro walk through with the expert ?
I scheduled a moz pro walk through few days back, but by mistake I chose wrong timings for the same. How Can I rescheduled my free moz pro walk through now ?
Feature Requests | | Anupmehta0 -
Posted by Link on Moz - Broken
I wasn't sure which category to place this in as Support doesnt feature Q&A or the Moz site in general so I dropped it under other research tools to which the Q&A kind of is 🙂 Now I am not sure if it is just me and that you have rectified the issue, but when ever I click on the "posted by" links to the user. I am getting a page not found error. The links in question can be seen in my two grabs and effect all posted by links on the Q&A section of Moz. A simply trailing slash after .com "/" will do the trick 🙂 https://moz.comusers/view/636129 - BROKEN
Feature Requests | | TimHolmes
https://moz.com/users/view/636129 - FIXED Aj5S2Pj,YfSQEmZ#0 Aj5S2Pj,YfSQEmZ#10 -
Way to track public's use of keywords over time in MOZ?
I apologize for the basic question but is there a way to track within MOZ the public's use of my keywords over time? My traffic understandably goes down certain times of the year -- holidays, vacation season, back to school, etc. What I want to see is whether my share of the traffic available is staying steady, slipping or even increasing. Sometimes I'll see a big spike in traffic that I suspect is related more to a general increase in traffic in my keywords than anything else. Thanks for any help.
Feature Requests | | NCCompLawyer0