Unsolved Site Crawler not working but on-demand crawler working
-
Hi,
In Moz pro, when using Site crawler (or recrawl), we are seeing message site is banned. But when using on-demand crawler, it could generate report successfully.
I just like to know if in both these cases, it is roberbot that is used!
And kindly note, site crawler was perfectly working before. So the required setup is already in place from long time. Site crawler ban issue started appearing from nov/dec 2023. .
Could you please us understand how could we possibly make site-crawler work?
I am happy to provide more details if you need any.Thanks
-
Hi,
This question requires help from MozPro.
Site Crawler is not working because it is missing request header 'user-agent' when we investigated the logs in our system and it got banned because of this reason.
On-demand crawler is still working because it has request header 'user-agent' and our system approved it hence able to generate report.Could you please look into this issue of no-user-agent request header?
Your response is much appreciated.Thanks
-
Hi,
I will double-check with firewall settings in our servers. Could you please share moz-pro site-crawler roger bot IP addresses/range? We will verify against our firewall rules.
Thanks
Shashi -
I am looking for roger bot site crawler IP addresses Please provide.
Thanks
-
@Aditi_08
Could you please help me on how to get IP addresses of Site Crawler? Just please note, Site Crawler is working before November so IP addresses were not blocked.Like it is mentioned before,
- no change in robots.txt
- no issue with rate limiting
- no changes in site-crawler configuration
-
@gilesd If you're experiencing issues with Moz Pro's Site Crawler showing that the site is banned while the On-Demand Crawler works fine, it might be due to changes or updates since November/December 2023. Both tools likely use the same crawler, "rogerbot," but differ in their operational schedules. The problem could be due to rate limiting or blocking by your server, IP blocking, changes in your robots.txt file, or updates in the Site Crawler configuration. To resolve this, check your robots.txt file to ensure it allows Moz's crawler, review server logs and firewall settings to ensure the crawler’s IP addresses aren’t blocked, and adjust rate limiting settings if necessary. Also, double-check the settings in Moz Pro to make sure there are no configurations causing the issue. If the problem persists, contact Moz support with detailed information about the error messages and any recent changes to your site’s configuration. Regular monitoring of your site’s interactions with automated tools and coordinating with your hosting provider can help prevent such issues in the future.
-
I am not sure why my reply not appearing here. Just for confirmation, replying again,
I like to confirm you -
There is no modification in Robots.txt
No issues with rate limit
Moz Pro settings are not changedWe are looking for your help to identify the issue.
Thanks
-
Thanks for your trouble shooting tips.
I assure you there has been nothing changed in robots.txt file or any settings in MozPro.
And there is frequency limit, Site Crawler triggers only once in 2 weeks.Thanks
-
Hi, gilesd
In Moz Pro, when using the Site Crawler or Recrawl, we also received a message indicating the site was banned. However, the on-demand crawler could generate the report successfully.
To address your question:
Robots.txt Configuration: Both the Site Crawler and on-demand crawler should be using the same robots.txt file unless there's been a recent change. Ensure your robots.txt hasn't been updated to block specific user agents.
IP Blocking or Rate Limiting: Some web servers or security settings might block or limit access based on IP or request frequency. The Site Crawler might be hitting these limits, whereas the on-demand crawler, being less frequent, avoids these blocks.
Moz Pro Settings: Double-check the Moz Pro settings to see if there have been any changes or updates to how the Site Crawler operates compared to the on-demand crawler. Any recent updates might have altered how the Site Crawler interacts with your site.
Thanks,
Hamza Zubair -
Hi, gilesd
In Moz Pro, when using the Site Crawler or Recrawl, we also received a message indicating the site was banned. However, the on-demand crawler could generate the report successfully.
To address your question:
Robots.txt Configuration: Both the Site Crawler and on-demand crawler should be using the same robots.txt file unless there's been a recent change. Ensure your robots.txt hasn't been updated to block specific user agents.
IP Blocking or Rate Limiting: Some web servers or security settings might block or limit access based on IP or request frequency. The Site Crawler might be hitting these limits, whereas the on-demand crawler, being less frequent, avoids these blocks.
Moz Pro Settings: Double-check the Moz Pro settings to see if there have been any changes or updates to how the Site Crawler operates compared to the on-demand crawler. Any recent updates might have altered how the Site Crawler interacts with your site.
Thanks,
Hamza Zubair
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Duplicate Content
We have multiple collections being flagged as duplicate content - but I can't find where these duplications are coming from? The duplicate content has no introductory text, and no meta description. Please see examples:- This is the correct collection page:-
Technical SEO | | Caroline_Ardmoor
https://www.ardmoor.co.uk/collections/deerhunter This is the incorrect collection page:-
https://www.ardmoor.co.uk/collections/vendors How do I stop this incorrect page from showing?0 -
Unsolved Moz Crawl seems to be stuck?
Hi all, It seems like moz has been stuck on crawling our site for a while now - I had a message of 'you will get a notification for when your site crawl is complete' for about 2 weeks now, and it doesn't seem to finish it? Any ideas why this happens and how to fix it? Thank you in advance.
Moz Tools | | StevenWalley0 -
Unsolved Site Crawl Stalled and Can't Restart
In my GreenSeed campaign, the site crawl continues to say "in progress." I can't figure out how to stop it or how to restart the site crawl. Can you please help?
Moz Pro | | Winger1 -
Links to Your Site: No Data Available in Google Search Console
The site I am working on did not have their site submitted to Google Search Console (formerly Google Webmaster Tools). I submitted the site and a sitemap that auto updates. Google is crawling the site daily (about 30 pages a day). Under Search Traffic > Links to Your Site it shows no data is availible. I thought it was because it was a newly submitted site, but it has been two months now. Moz seems to have the same issue. Moz does show inbound links, but their are some that we think should really help us that are not shown. For instance, the Dallas Morning News wrote this article. They have a high DA and PA. Also, iliveindallas.com has an article about us that is still on the front page. That was a few weeks ago but also does not show up on Moz or Google SC. We are trying to be selective about the links we are getting. That they are follow links from reputable sites. Worried that both Google and Moz are not showing them.
Moz Pro | | TapGoods1 -
We have a Wix site but I wonder if we would be better suited with square space
Our SEO has taken a hit in moving our website from homestead website builder to Wix this last year. 1,500 unique visitors a month to 400 on our website www.bestlifeint.com using much of the same SEO and content. I think 2 of our main issues now (site speed and SEO friendliness) would be addressed in the switch to squarespace. But I guess is it worth it? I do not think google likes the whole hash bang html5 ajax thing wix uses and our website relies heavily on good seo. Any advice? I am starting link building but it might be a waste if we switched the site and in turn changed all the page url's as wix uses ?! for it's pages. Just need some clarification ideas on getting traffic to our website, also SEO MOZ software is hard to use report card with wix. We also switched our ecommerce from homestead to shopify and this has been a good move. Shopify (our ecommerce actually gets more traffic than our website now. But the problem I feel is wix hurts our seo just by being on their platform. Seriously thinking of squarespace but feel we would have to start from scratch and just match some of the elements on current site. Any insight, ideas? Am I on the right path in my thinking? Should I just stick it out with Wix? Thanks. -Brian
Moz Pro | | SammisBest0 -
When is rank tracker going to work
has anyone heard when rank tracker under the research tools is going to be working, been a very long time. I use this tool to check on the progress of sites, I know that it is in the campaigns per week but i want the tool under the research tools and feel a bit let down by semoz over this
Moz Pro | | ClaireH-1848860 -
Google and Open Site Explorer not showing as many links
I've noticed this past week that when you search for the links pointing to a given site, by using the "link:" operator, that Google not showing as many links as they use to. I noticed this also with Open Site Explorer, it is not showing the detail link information as much as it did before. Is Google trying to mask what we can view now on competitors backlinks? If so, how can we see the backlink building that our competitors are doing?
Moz Pro | | tdawson090 -
How often does Open Site Explorer Update?
How often does Open Site Explorer Update? Just trying to get a rough idea. Great tool btw.
Moz Pro | | seo3210