High Number of Crawl Errors for Blog
-
Hello All,
We have been having an issue with very high crawl errors on websites that contain blogs. Here is a screenshot of one of the sites we are dealing with: http://cl.ly/image/0i2Q2O100p2v .
Looking through the links that are turning up in the crawl errors, the majority of them (roughly 90%) are auto-generated by the blog's system. This includes category/tag links, archived links, etc. A few examples being:
http://www.mysite.com/2004/10/
http://www.mysite.com/2004/10/17/
As far as I know (please correct me if I'm wrong!), search engines will not penalize you for things like this that appear on auto-generated pages. Also, even if search engines did penalize you, I do not believe we can make a unique meta tag for auto-generate pages. Regardless, our client is very concerned seeing these high number of errors in the reports, even though we have explained the situation to him.
Would anyone have any suggestions on how to either 1) tell Moz to ignore these types of errors or 2) adjust the website so that these errors now longer appear in the reports?
Thanks so much!
- Rebecca
-
Hi Rebecca
What are the crawl errors exactly? From that report screenshot it looks like you have a variety of them, so the fixes will all be different.
Let me know, and in the meantime you might want to check out my article on Moz about setting up WordPress
-Dan
-
It is true that you will most likely not be penalized for these pages, Google is pretty good at figuring out common canonicalization problems in my opinion and would most likely not penalize you for having duplicate content. I would encourage you to dig a little deeper and see what additional problems these pages could create though.
Consider that Google will waste valuable crawl bandwidth crawling these meaningless pages, rather than focusing on the important content you want them too. If Google is crawling them, you can most likely bet that PageRank is flowing through these pages as well, diluting the link equity of your site.
Are you using Wordpress? There are a lot of great plug ins that can help you manage these pages. You could control how Google crawls these pages with your robots.txt, by placing meta robots tags on the pages using a plug in, or by placing rel=canonical tags on the pages pointing back to the page that is the original source.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Crawl - Crawls only homepage?
Hi Moz Comunity! Joined Moz just 2 weeks ago and slowly trying to get used to tools available in here! Great tools and info available on this site! My concern is that Site Crawl of Moz in my Campaign seems to have crawled only my homepage and no other sub-domains, is there any reason for this? FOr some reason it seems that Moz interacts only with my homepage? Even when I tried the Keyword Exlporer set on Keyword to see if any of my pages rank for any keywords, it seems only my homepage was ranking for a few keywords. It's possible my other sub-domains don't rank for any keywords yet but still, seems suspicious... I have added a link to Site Crawl that says it has crawled only 3 pages on my site, and all are my homepage... Thanks for any help! Jacob s!AlxV7sobbcgmhJB_fXcF4EPzbPSovA
Getting Started | | Shotlife_Studio0 -
SSL - green padlock but Moz say there's an 804 error?
Hi, my site has a green padlock and no SSL errors but Moz are reporting an 804 error. I use CloudFlare with fairly complex settings. I've read this thread but it's quite old and I don't understand which parts of it are still valid. I'd love to know whether this can be sorted before I spend hours setting up Moz's features as if they can't crawl my site then I would obviously need to cancel my subscription. Thanks
Getting Started | | Barn2Plugins0 -
Error code 804
URL Page Authority Linking Root Domains Status Code | Error Code 804: HTTPS (SSL) Error Encountered https://www.alleninv.com |
Getting Started | | alleninv0 -
Crawl test
Can anyone give me an idea how to use the MOZ crawl test results...I'm a little confused on how to read it? I have a lot of "no's"...I think this is good?
Getting Started | | sdwellers0 -
Why wont rogerbot crawl my page?
How can I find out why rogerbot won't crawl an individual page I give it to crawl for page-grader? Google, bing, yahoo all crawl pages just fine, but I put in one of the internal pages fo page-grader to check for keywords and it gave me an F -- it isn't crawling the page because the keyword IS in the title and it says it isn't. How do I diagnose the problem?
Getting Started | | friendoffood0 -
Error help for newbie please
Hi, I signed up after seeing the videos on Udemy and YouTube (white board Friday) So I've started the free trial and am looking forward to getting my site ranking higher. I've crawled my site www.sussexchef.com and its come back with the following errors (please see below. 608 I'm sure this information is very important but I have no idea how to fix the 608, I found a robots.txt in my directory and deleted it as I think that maybe the problem? I crawled the site twice by accident so will have to wait till tomorrow to find out? 404 I found it quite hard to find the broken links at first but once I realized all the information I needed was in the table I think I got them all. did I miss a tutorial or am I just a little out of my depth here? 503 I have no idea how to fix these, I can click the links and it takes my to that page or file. so how can it be the server down? Or is this because they are links to PDF's? should i convert them to jpegs and give them meta data? I'd be grateful for any help anyone has to offer as I'm keen to learn how to promote my site better. Crawl Error Moz encountered an error on one or more pages on your site608 Page not Decodable as Specified Content EncodingInvestigate the cause of this issue on the Help Hub.Discovered: Sep 2 - 8Crawl Diagnostics Crawl Issue Found: 404 Errors 10% of site pages served 404 errors during the last crawlA high percentage of 404 pages can indicate a problem with the internal link structure.Crawl Diagnostics 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/wedding-caterers.aspx4041215http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/dinner-party-catering.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/christmas-party-catering.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/wedding-cakes.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/outdoor-catering-specialists.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/hen-party-cupcake-classes.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/funeral-caterers.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/drinks-service.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/private-party-catering.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/corporate-catering.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/caterers.aspx40401http://sussexchef.com/wedding-catering/N/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/funeral-caterers-brighton.aspx Crawl Issue Found: 500 Errors More than 5% of site pages served 500 errors during the last crawlExcessive 500 errors impact search engine indexation. Double check that your website is serving pages properly to both users and crawlers. 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Wedding-Packages-2014.pdf141N/A50302N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Vegetarian-BBQ-Menu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/AllInclusiveMenuPrices1.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Finger%20Buffet.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Dessert.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/SummerMenu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/HotorColdBuffet.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Canape%20Menu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Susex-Chef-Xmas-Dinner-artwork-file.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/BBQ.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Fun-Finger-Buffets.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/HogRoast.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Private-Chef-Dinner-Packages.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Salad%20Menu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Childrens-Menu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/HotForkBuffetDelivererd.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Afternoon%20Tea.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Christmas-Promo.pdf141N/A50301N/A 500 : Received 500 (Internal Server Error) error response for page. http://sussexchef.com/?attachment_id=51110N/A50000N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Biography.pdf10N/A50300N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/?attachment_id=62810N/A50300N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2014/02/2014-02-01-11.12.00.jpg
Getting Started | | SussexChef830 -
Number of on-page keywords outnumbers total number of keywords
On my weekly report, my number of keywords on the first page outnumber the total number of keywords. Anyone know why?
Getting Started | | NicoleCriona0 -
Campaign.crawl-seed.bad-response ???
Hello Guys I have just tried to set-up a new campaign for this site http://www.emsababies.co.uk/ HoweverI keep getting this error ?? campaign.crawl-seed.bad-response Anyone know what I am doing wrong ?? Cheers James
Getting Started | | BlueNinja1