Bogus Crawl Errors in Webmaster Tools?
-
I am suddenly seeing a ton of crawl errors in webmaster tools.
Almost all of them are URL links coming from scraper sites.that I do not own.
Do you see these in your Webmaster Tools account?
Do you mark them as "fixed" if they are on a scraper site? There are waaaay too many of these to make redirects.
Thanks!
-
Thanks, Marcus,
My numbers are rising rapidly right now... but hopefully the trend will reverse.
I'll let you know if I learn anything.
-
Hey, I know, it's kind of bonkers but I certainly think that assuming Google does not know what they are doing is a good place to start.
For us they just cleared up in time, obviously, this is webmaster tools so it was a good old bit of time (months rather than weeks) but it did sort itself out.
Take care!
Marcus -
Hello Marcus,
Thank you for sharing your experience and finding those posts. I appreciate it.
I think I am going to ignore these and assume that Google doesn't know what they are doing.
It surprises me that the URL errors on spammer sites are being presented to me as something that should be fixed.
Thanks again!
-
Hey EGOL
I have seen this in the past on my own site and on a few client sites in the past (which is not to say I have an answer here).
We were seeing completely random looking URLs that at first made me think the site had been somehow hacked or compromised but further investigation revealed that was not the case. We were just getting the strangest of links to pages that did not exist like xhyx.php?id=jamesbrown (that kind of thing).
We did nothing here and over time it seems to have resolved itself and these pages are not listed any longer. I tend to think of the webmaster tools data as diagnostics and it is telling me these pages don't exist so I can check for problems. Well, there is no problem, they don't exist and I am happy about that. Still, whether to mark them as fixed or not, I am unsure and would err towards not doing anything with them as they are not 'errors' as far as I am concerned. Likewise, I don't want to redirect them in most cases as I don't like the linking sites and have better things to do with my working day (I am not getting that time back - it's the digital equivalent or ironing clothes or some such laborious grind).
I had a look around again and whilst I can't find any specific answers regarding whether to mark them as fixed the following posts are of interest:
- http://productforums.google.com/forum/#!topic/webmasters/3GTOLCE-8pk
- https://productforums.google.com/forum/?hl=en#!category-topic/webmasters/webmaster-tools/rKI-38ohfbc
Particularly this quote from John Mueller at Google (webmaster tools guy I believe):
"In general, if a URL is really a 404, that's fine for us, and not something that would cause your site any problems in the long run. At any rate, you don't need to "fix" this problem (eg with a 301 redirect), if you're sure that the URL should really not exist. Having 404s listed in Webmaster Tools will generally not affect your site's crawling, indexing, or ranking; it's normal for websites to return 404 for URLs that don't exist."
So, my take is not to bother but would be interesting to ask the question in webmaster tools section of the Google product forums: https://productforums.google.com/forum/?hl=en#!categories/webmasters/webmaster-tools
Not an answer as such but hope that helps.
Marcus
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Webmaster tools: Sitemap.xml not processed everyday
Hi, We have multiple sites under our google webmaster tools account with each having a sitemap.xml submitted Each site's sitemap.xml status ( attached below ) shows it is processed everyday except for one _Sitemap: /sitemap.xml__This Sitemap was submitted Jan 10, 2012, and processed Oct 14, 2013._But except for one site ( coed.com ) for which the sitemap.xml was processed only on the day it is submitted and we have to manually resubmit every day to get it processed.Any idea on why it might?thank you
Technical SEO | | COEDMediaGroup0 -
Google webmaster tools says access denied error 403
Hi, this keeps on happening, just check early today and it tells me i have access denied and 403 errors I have this from time to time in my google webmaster tools and i have checked the pages and they work properly, so i am puzzled why this has happened. I have contacted my hosting company who have said there is not a problem but there must be a problem somewhere which could affect my site rankings. can anyone let me know what this could be please. i work in joomla | parenting-magazine | 403 | 8/10/13 |
Technical SEO | | ClaireH-184886
| | 2 | personal-finance-money-advice | 403 | 8/10/13 |
| | 3 | 201308081607/emmerdale/emmerdale-chas-confronts-cameron-over-affair-with-debbie | 403 | 8/10/13 |
| | 4 | 201308081606/emmerdale/emmerdale-declan-gets-a-visit-from-the-police | 403 | 8/10/13 |
| | 5 | 201308081608/emmerdale/emmerdale-cameron-debbie-affair-is-out-in-the-open | 403 | 8/10/13 |
| | 6 | 201308081614/uk-holiday-news/visitscotland-launch-campaign-to-boost-tourism | 403 | 8/10/13 |
| | 7 | dog-advice/training-your-puppy-a-beginners-guide | 403 | 8/10/13 |
| | 8 | gadgets/hp-envy-13-laptop-review | 403 | 8/10/13 |
| | 9 | gadget-talk/everyday-smartphone-gadgets-which-could-revolutionise-your-life | 403 | 8/10/13 |
| | 10 | news-gadgets/the-htc-one-mobile-phone-review | 403 | 8/10/13 |
| | 11 | gadget-talk/five-iphone-apps-for-home-improvement | 403 | 8/10/13 |
| | 12 | gadget-talk/are-android-apps-useful-for-business-success | 403 | 8/10/13 |
| | 13 | gadget-talk/television-gadgets-the-future-of-television-is-coming | 403 | 8/10/13 | | | |0 -
Soft 404 errors
Hello Everyone, I recently removed some pages and made a custom 404 page by putting "ErrorDocument 404 http://www.site.com/404.htm" in the htaccess file but WMT now reports soft 404 errors, how do I do this properly? Thanks
Technical SEO | | jwdl0 -
Crawl rate
Hello, In google WMT my site has the following message. <form class="form" action="/webmasters/tools/settings-ac?hl=en&siteUrl=http://www.prom-hairstyles.org/&siteUrl=http://www.prom-hairstyles.org/&hl=en" method="POST">Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate.Why would this be?A bit of backgound - this site was hammered by Penguin or maybe panda but seems to be dragging itself back up (maybe) but has dropped from several thousand visitors/day to 100 or so.Cheers,Ian</form>
Technical SEO | | jwdl0 -
500 Server Error on RSS Feed
Hi there, I am getting multiple 500 errors on my RSS feed. Here is the error: <dt>Title</dt> <dd>500 : Error</dd> <dt>Meta Description</dt> <dd>Traceback (most recent call last): File "build/bdist.linux-x86_64/egg/downpour/init.py", line 391, in _error failure.raiseException() File "/usr/local/lib/python2.7/site-packages/twisted/python/failure.py", line 370, in raiseException raise self.type, self.value, self.tb Error: 500 Internal Server Error</dd> <dt>Meta Robots</dt> <dd>Not present/empty</dd> <dt>Meta Refresh</dt> <dd>Not present/empty</dd> Any ideas as to why this is happening, they are valid feeds?
Technical SEO | | mistat20000 -
seo moz crawl diagnosis
Hi seomozzers, We are creating a brand new website for a client and I would like to run an seo moz crawl to fix what has been done wrong. So my question is it ok to run an SEO moz crawl with a dev URL? Are final URLs and dev URLs will give me the same results or not? Basically, Should I wait for getting the final URL or is it ok to run a crawl under a dev URL such as www.dev2.example.com or http://183.2564.2864? Thank you 🙂
Technical SEO | | Ideas-Money-Art0 -
Webmaster tools lists a large number (hundreds)of different domains linking to my website, but only a few are reported on SEOMoz. Please explain what's going on?
Google's webmaster tools lists hundreds of links to my site, but SEOMoz only reports a few of them. I don't understand why that would be. Can anybody explain it to me? Is there someplace to I can go to alert SEOMoz to this issue?
Technical SEO | | dnfealkoff0 -
How is my competition causing bad crawl errors and links on my site
We have a compeditor who we are in a legal dispute at the moment, and they are using under hand tactics to cause us to have bad links and crawl errors and i do not know how they are doing it or how to stop it. The crawl errors we are getting is the site having two urls together, for example www.testsite.com/www.testsite.com and other errors are pages that we do not even have or pages that are spelt wrong or have a dot after the page name. We have been told off a number of people in our field that this has also happened to them and i would like to know how they are doing it so we can have this stopped Since they have been doing this our traffic has gone down by half
Technical SEO | | ClaireH-1848860