Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawler is not able to crawl my website
-
Hello All,
I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "
We changed the robots.txt file and checked it . but still the issue is not resolved.
URL : https://www.khadination.shop/robots.txt
Do let me know what went wrong and wjhat needs to be done.
Any suggestion is appreciated.
Thank you.
-
Hi Harini,
Jo from the Moz help team here.
I've had a look at your site and it looks like there is something server side that is blocking our bot.
When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1
https://screencast.com/t/J3hhDTCM
I'm also seeing this message in this third party tool.
"The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"
All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.
I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2
You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2
You'll also want to make sure you are not blocking AWS (Amazon Web Services).
Best of luck!
Jo
-
Thank you Andy . But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?color=
Disallow: /?manufacturer=
Disallow: /?filter_material-fabric=
Disallow: /?filter_color=
Disallow: /?query_type_color=
Disallow: /?filter_size=
Disallow: /?taxonomy=
Disallow: /?view_mode=
Disallow: /?query_type_material-fabric=
Disallow: /?orderby=
Disallow: /?source_id=
Disallow: /?source_tax=
Disallow: /?shop-2__trashed?
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.khadination.shop/sitemap.xml
this was the previous version of robots.txt that were been used ....
-
Hi - As Andy has said, you're not allowing Moz to crawl the site.
Read up on Rogerbot here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot
-
Hi there,
You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.
Allow: /
add this on line 2 of your robots.txt file.
Good luck
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Matomo Data Integration
As GA4 is not easily usable for all clients in the EU, it would be great if you could offer a Matomo integration. Matomo has recently been more and more popular and is used by a lot of companies, that are required to adhere to EU standards in data privacy.
Feature Requests | | datenwerk0 -
Server blocking crawl bot due to DOS protection and MOZ Help team not responding
First of all has anyone else not received a response from the help team, ive sent 4 emails the oldest one is a month old, and one of our most used features on moz on demand crawl to find broken links doesnt work and its really frustrating to not get a response, when we're paying so much a month for a feature that doesnt work. Ok rant over now onto the actual issue, on our crawls we're just getting 429 errors because our server has a DOS protection and is blocking MOZ's robot, im sure it will be as easy as whitelisting the robots IP, but i cant get a response from MOZ with the IP. Cheers, Fergus
Feature Requests | | JamesDavison0 -
Keyword keyword rank checker
Where is the Moz tool/page where you can check rankings of keywords for any site or page? Thanks Rachel
Feature Requests | | RaquelSaiz0 -
Is there an API to download keyword data?
Howdy fellow Mozzers, We're tracking a client's keyword rankings in one of our campaigns and want to export that data on a regular basis to display in Tableau. Tableau let's us quickly match up different data sources and create visuals with the data. It's useful for marrying different data sources that don't normally talk to each other. We know we can pull keyword tracking data manually, but we're hoping there's an API or scheduler that we can use to download the data automatically. Automating the process would be super helpful. Thank you!
Feature Requests | | brettmandoes0 -
Does seo moz toolbar work in safari? Can I avoid using Chrome or firefox?
I am using a Mac and use Safari for web. I am not very familiar with Google Chrome or Firefox. I started my trial with SEOMoz Pro and want to use it (loved it) but do not want to adapt myself to Google Chrome or Firefox - thus my hesitation. What limitations do I have with SEOMoz Pro in safari - e.g. Moz add-on in the toolbar? What would I be missing if I continue using Safari? I would much appreciate an honest and complete answer so that I can decide whether to use Google Chrome of Firefox. Thank you
Feature Requests | | Cenktukel0 -
Keyword Volume Data for Austria
Hi, we can't find any Volume Data for Keywords in Austria.
Feature Requests | | DIGIHOUSE
If we upload a Keyword List in the Keyword Explorer there is always US shown. Any suggestions? thxChris0 -
External Link Detection Very Slow... why?
Hello. I'm new to MOZ and I'm now comparing it with ahrefs. I can't afford to have an account with all SEO tools that are out there, but I have to start somewhere. I really like MOZ a lot. However, there is one major issue - it updates backlinks every 3 weeks, or what? I registered to ahrefs trial today just to make sure I don't regret in the future that I bought the first service that I tried (MOZ), and I was amazed that ahrefs showed all my website's backlinks straight away. MOZ is saying that next scheduled update for Externals Links is going to be on Sept 14.... emm sorry? My trial will run out by then. It now shows exactly 0. Pricing is nearly the same and in Today's fast paced world ahrefs seems to tick the ever so important backlinks quality box. So here is my question. Dear MOZ, are you going to speed up the backlink data delivery? Are you working on it, or you think it's just ahrefs's gimmick? If you can promise faster speeds, I'll be happy to stay subscribed with MOZ and wait because I like the UX a lot. However, at the end of the day speed is important and if my client requests the latest report, I don't want to tell him that he needs to wait a month for it. Many thanks for reading and I hope to hear back from someone.
Feature Requests | | NewWindow1 -
Do MOZ have an affiliate platform?
Do MOZ have an affiliate platform? If so, it's hiding from me, would somebody care to point me the right way? Shall I assume this is the page / they MOZ don't have a direct affiliate program? https://moz.com/partners Many Thanks
Feature Requests | | Whittie0