Unsolved Crawling error emails
-
Recently we start having random error messages about crawling issue:
2024-08-30 edweek:Ok
2024-08-29 marketbrief:Err. advertise: Err, edweek:Err, topschooljobs:Ok
2024-08-23 edweek:Ok
2024-08-22 marketbrief:Err. advertise: Err, edweek:Err
2024-08-21 topschooljobs:Ok, edweek:Ok
2024-08-15 marketbrief:Ok. advertise:OK
2024-08-13 edweek:Ok
2024-08-12 marketbrief:Ok
2024-08-08 marketbrief:Ok, advertise:Ok
2024-08-03 edweek:Ok, topschooljobs:Ok
All for 2024-07 - are OkYesterday I set 2 more crawls for the same sites (edweek and marketbrief) and I get a morning email about original edweek site is ok (still have some problem but crawl occurs and all is fine) but for test crawl for the same site "EW Test" I just got error email.
Also I suppressed ALL email communications and frankly surprised by this email.Can you please check what is wrong with a crawler or stat collection or I don't know who produced the issues.
-
@DTashjian said in Crawling error emails:
Recently we start having random error messages about crawling issue:
2024-08-30 edweek:Ok
2024-08-29 marketbrief:Err. advertise: Err, edweek:Err, topschooljobs:Ok
2024-08-23 edweek:Ok
2024-08-22 marketbrief:Err. advertise: Err, edweek:Err
2024-08-21 topschooljobs:Ok, edweek:Ok
2024-08-15 marketbrief:Ok. advertise:OK
2024-08-13 edweek:Ok
2024-08-12 marketbrief:Ok
2024-08-08 marketbrief:Ok, advertise:Ok
2024-08-03 edweek:Ok, topschooljobs:Ok
All for 2024-07 - are OkYesterday I set 2 more crawls for the same sites (edweek and marketbrief) and I get a morning email about original edweek site is ok (still have some problem but crawl occurs and all is fine) but for test crawl for the same site "EW Test" I just got error email.
Also I suppressed ALL email communications and frankly surprised by this email.Can you please check what is wrong with a crawler or stat collection or I don't know who produced the issues.
@DTashjian said in Crawling error emails:
Recently we start having random error messages about crawling issue:
2024-08-30 edweek:Ok
2024-08-29 marketbrief:Err. advertise: Err, edweek:Err, topschooljobs:Ok
2024-08-23 edweek:Ok
2024-08-22 marketbrief:Err. advertise: Err, edweek:Err
2024-08-21 topschooljobs:Ok, edweek:Ok
2024-08-15 marketbrief:Ok. advertise:OK
2024-08-13 edweek:Ok
2024-08-12 marketbrief:Ok
2024-08-08 marketbrief:Ok, advertise:Ok
2024-08-03 edweek:Ok, topschooljobs:Ok
All for 2024-07 - are OkYesterday I set 2 more crawls for the same sites (edweek and marketbrief) and I get a morning email about original edweek site is ok (still have some problem but crawl occurs and all is fine) but for test crawl for the same site "EW Test" I just got error email.
Also I suppressed ALL email communications and frankly surprised by this email.Can you please check what is wrong with a crawler or stat collection or I don't know who produced the issues.
Hi,
I had a similar issue with my site and resolved it by first reviewing the specific error messages to understand the problem. I then verified that the new crawl configurations were correct and matched the settings of the working ones. Ensuring that the test environment was properly set up and permissions were correctly configured was also crucial. Additionally, I compared logs to identify any discrepancies and made sure the crawler software and libraries were up to date. I suggest trying these steps to address your issue. Let me know if you need further assistance!
-
@DTashjian
I had a similar issue with my site and resolved it by first reviewing the specific error messages to understand the problem. I then verified that the new crawl configurations were correct and matched the settings of the working ones. Ensuring that the test environment was properly set up and permissions were correctly configured was also crucial. Additionally, I compared logs to identify any discrepancies and made sure the crawler software and libraries were up to date. I suggest trying these steps to address your issue. Let me know if you need further assistance! -
Hi DTashjian,
I had a similar issue with my [site](link url) and resolved it by first reviewing the specific error messages to understand the problem. I then verified that the new crawl configurations were correct and matched the settings of the working ones. Ensuring that the test environment was properly set up and permissions were correctly configured was also crucial. Additionally, I compared logs to identify any discrepancies and made sure the crawler software and libraries were up to date. I suggest trying these steps to address your issue. Let me know if you need further assistance!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Rogerbot blocked by cloudflare and not display full user agent string.
Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
Moz Pro | | BB_NPG
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
Ben Cloudflare activity log, showing differences in user agent strings
2022-07-01_13-05-59.png0 -
Unsolved /%25s
Hi Community, has anyone else had a 404 error reported by Moz, where the end of the domain is /%25s? The error comes from my blog home page https://kaydee.net/blog/ But when I look at the source code, I can't see anything that has a space at the end of the URL. I wonder if it is to do with the WordPress search? Thanks in advance for any insight.
Moz Pro | | kaydeeweb0 -
DA error in my website
Hi
Product Support | | Bdgbye
there are some errors in this website. When i check this with my friends laptop it shows different DA PA
but if i check this on my PC it shows different. Now which one is perfect?
Here's the Name and link of website Juicks
Please check this and guide me where i am doing wrong. Thank you.0 -
How to block Rogerbot From Crawling UTM URLs
I am trying to block roger from crawling some UTM urls we have created, but having no luck. My robots.txt file looks like: User-agent: rogerbot Disallow: /?utm_source* This does not seem to be working. Any ideas?
Product Support | | Firestarter-SEO0 -
I have removed a subdomain from my main domain. We have stopped the subdomain completely. However the crawl still shows the error for that sub-domain. How to remove the same from crawl reports.
Earlier I had a forum as sub-domain and was mentioned in my main domain. However i have now discontinued the forum and have removed all the links and mention of the forum from my main domain. But the crawler still shows error for the sub-domain. How to make the crawler issues clean or delete the irrelevant crawl issues. I dont have the forum now and no links at the main site, bu still shows crawl errors for the forum which doesnt exist.
Product Support | | potterharry0 -
Why is Report Email not White labeled
I'm at a loss here as I've just discovered (after running my new account for about a month) that the white label of reporting is incomplete and you didn't even warn me. I just saw an email delivered from the custom report section. I went through and set it up including custom logo so everything would be white labeled and even checked everywhere else in the preview to ensure that nothing about Moz showed up. This I was greeted this morning with a bunch of reports for multiple campaigns going to multiple clients and in all cases the email was branded as Moz. It's from a Moz email, has a Moz header and footer, and even has links for managing moz email preferences as well as Moz's privacy policy. I'd love to know who thought this was white label? Seriously, this is absurdly ridiculous. Is there a fix? Micha
Product Support | | Multiverse-Media-Group2 -
Analytics Linking Error
Our team keeps getting this Analytics error, "It appears there's a problem with our connection to your Google Analytics account. Please go to your Settings page to update your connection," The issue is that we have linked appropriately several times now. Anyone else having this issue? Glitch?
Product Support | | jgrammer0 -
Traffic Data Report Error: Anyone else?
I included a screenshot of the report. I had a 111 organic visits for last Mondays past week report, this Monday it is telling me I had 0. It's also telling me I have -10 non-branded keywords. I know the organic search visits isn't 0. Also, (this part I'm not sure about) but I don't see how I could have a negative amount of non-branded keywords. I would think it would just stop at 0. Has anyone had something like this happen? Or am I just missing something? Thanks for any help or comments. zK3FbG4.png
Product Support | | KempRugeLawGroup0