404s effecting crawl rate?
-
We made a change to our site where we all of a sudden we are creating a large number of 404 pages. Is this effecting the crawl/indexing rate?
Currently we've submitted 3.4 million pages, have over 834K indexed but have over and 330K pages not found. Since the large increase in 404s we've noticed a decrease in pages crawled per day. I found this Q & A in Webmasters (http://googlewebmastercentral.blogspot.com/2011/05/do-404s-hurt-my-site.html) but it seems like the 404s should not have an effect. Is this article out of date?
What do you think fellow Moz-ers? Is this a problem?
-
It's not a problem, just fix those as soon as you can. And yes, it does affect crawl rate from what I've seen.
-
That article you mention is very up to date.
but if you got "hit" by Google bot several times a day for those pages that now you return a 404 response code you will see a decrease in pages crawled per day since once Google sees a 404 response code it will not visit / hit that page that often aftre that... -
Yes i've seen this numerous of times. Is it just 404's are are there also things like DNS playing along?
But if the amount of 404's jump up really high then for sure google turns down the speed. i guess this gives you some air to fix it in time.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Value of having a good crawl budget?
Hi, I've seen several questions where people give advice on how to increase the crawl budget. What I haven't seen anyone comment is what the value of this really is if you have many pages that doesn't get updated very often. Take for example the typical agency page - 50 pages, most of them rarely gets updates. In a monthly basis normally 10% of the website gets updated. Is there really any value then of having 100% of the website crawled on a daily basis?
Technical SEO | | Inevo0 -
Redirect effecting ranking?
I manage the SEO for several different regions which are also based on the same site e.g. example.com/au, example.com/us The /us site has pretty good rankings and changes I'm making to the site are having an impact. The /au site has really bad rankings, even though much of the content is the same. (The /uk site is also awful but we had an issue with 4,500 duplicate pages which were only resolved last week). Crawl diagnostics are only showing 1 major error for a 404 response, I'm receiving a domain authority of 43 and A grade page ranking for some of our targeted keywords. I could believe that this isn't necessarily going to get us a top 10 rating but I would have thought we would be in the top 50, especially for branded keywords. Could the lack of ranking be to do with how our domain redirects? If you go to example.com.au you are taken to the home page rather than being redirected to example.com/au. Once you head to an internal page the URL changes to example.com/au/page
Technical SEO | | ahyde0 -
Google WMT continues reporting fixed 404s - why?
I work with a news site that had a heavy restructuring last spring. This involved removing many pages that were duplicates, tags, etc. Since then, we have taken very careful steps to remove all links coming into these deleted pages, but for some reason, WMT continues to report them. By last August, we had cleared over 10k 404s to our site, but this lasted only for about 2 months and they started coming back. The "linked from" gives no data, and other crawlers like seomoz aren't detecting any of these errors. The pages aren't in the sitemap and I've confirmed that they're not really being linked from from anywhere. Why do these pages keep coming back? Should I even bother removing them over and over again? Thanks -Juanita
Technical SEO | | VoxxiVoxxi0 -
Can too many pages hurt crawling and ranking?
Hi, I work for local yellow pages in Belgium, over the last months we introduced a succesfull technique to boost SEO traffic: we have created over 150k of new pages, all targeting specific keywords and all containing unique content, a site architecture to enable google to find these pages through crawling, xml sitemaps, .... All signs (traffic, indexation of xml sitemaps, rankings, ...) are positive. So far so good. We are able to quickly build more unique pages, and I wonder how google will react to this type of "large scale operation": can it hurt crawling and ranking if google notices big volumes of content (unique content)? Please advice
Technical SEO | | TruvoDirectories0 -
Have a client that migrated their site; went live with noindex/nofollow and for last two SEOMoz crawls only getting one page crawled. In contrast, G.A. is crawling all pages. Just wait?
Client site is 15 + pages. New site had noindex/nofollow removed prior to last two crawls.
Technical SEO | | alankoen1230 -
What is the most effective way to migrate an ecommerce site?
I am about migrate a 1500 product ecommerce site from Netsuite to Volusion. The url structure is not going to be the same so I need to know the most effective way of redirecting the old urls to the new site. Is there an easier method than collecting the most popular pages and creating a 301 xml page and upload it once the site goes live?
Technical SEO | | BenRWoodard0 -
Tracking a Crawl error
Hi All, If you find a crawl error on your page. How do you find it? The error only says the URL that is wrong but this is not the location. Can i drill down and find out more information? Thank you!
Technical SEO | | wedmonds0 -
How can I get Google to crawl my site daily?
I was wndering if there was a trick to getting google to crawl my website daily?
Technical SEO | | labradoodlelocator0