Increase in Crawl Errors
-
I had a problem with a lot of crawl errors (on Google Search Console) a while back, due to the removal of a shopping cart. I thought I'd dealt with this & Google seemed to agree (see attached pic), but now they're all back with a vengeance!
The crawl errors are all the old shop pages that I thought I'd made clear weren't there anymore. The sitemaps (using Yoast on Wordpress to generate these) all updated 16 Aug but the increase didn't happen till 18-20.
How do I make it clear to Google that these pages are gone forever?
-
Many thanks Britney
-
This is a great robots.txt cheat sheet to help figure out what robot exclusions might be right for your pages.
-
Hi Martijn
Many thanks for your response. We have been redirecting old product URLs to the shop URL (not the front page, but the front page of the shop) but I think ideally we just want rid of all these pages as the client is considering getting rid of the shop completely.
What would we do with robots.txt in this instance? Is there a tutorial you could point me at?
-
HI,
Did you redirect the old pages to another page, like worst case the home page? The alternative would be to look into excluding these pages if they have a similar page path from the search engines by adding them to your robots.txt.
Martijn.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Webmaster message - increase in 404 pages
I've had a message in webmaster tools: Increase in “404” pages on http://www.ethicaredental.co.uk/There are 1000 pages in the crawl error list. But all of them direct to t a 404 page, i.e http://www.ethicaredental.co.uk/search?searchword=toothwhich, which as far as I can tell has all the necessary features of a good 404 page (clear message saying the page doesnt exist anymore, navigation, and business details.The webiste was built in Joomla previous to a re-design in Wordpress. Is this a Joomla issue? How can I satisfy webmaster and Googles crawl to understand these are decent 404 pages? Or dop all 1000 pages need to be 301 redirected.??Any thoughts appreciated
Technical SEO | | dentaldesign1 -
I've had a sudden a increase in crawl issues as of yesterday (like 300 from a steady 10, does anyone else have this issue?
the main issue is that it's now indexing both www and http:// - anyone else got this issue or had any changes suddenly on their crawl results?
Technical SEO | | beckyhy0 -
Weird, long URLS returning crawl error
Hi everyone, I'm getting a crawl error "URL too long" for some really strange urls that I'm not sure where they are being generated from or how to resolve it. It's all with one page, our request info. Here are some examples: http://studyabroad.bridge.edu/request-info/?program=request info > ?program=request info > ?program=request info > ?program=request info > ?program=programs > ?country=country?type=internships&term=short%25 http://studyabroad.bridge.edu/request-info/?program=request info > ?program=blog > notes from the field tefl student elaina h in chile > ?utm_source=newsletter&utm_medium=article&utm_campaign=notes%2Bfrom%2Bthe%2Bf Has anyone seen anything like this before or have an idea of what may be causing it? Thanks so much!
Technical SEO | | Bridge_Education_Group0 -
Yoast and Standard theme: Fatal error
Hi all- A client has tried installing Yoast on her site and received at fatal error (below). She's been able to restore her site and get it functioning again, but I'm wondering if there's a work around so we can use the plugin. It's a Wordpress site using the Standard Theme. I've searched the forums (and here!) and haven't found anything helpful yet. Do you have any suggestions? Thanks! "Fatal error: Cannot redeclare yoast_breadcrumb() (previously declared in /vservers/nwconstructi/htdocs/NWCL/wp-content/plugins/wordpress-seo/inc/wpseo-functions.php:108) in /vservers/nwconstructi/htdocs/NWCL/wp-content/themes/StandardTheme_272/lib/standard_yoast_breadcrumbs.php on line 280"
Technical SEO | | DonnaDuncan0 -
Strange 404 Error(Answered)
Hi everyone! I recently took over a new account and I was running an initial crawl on the site and a weird 404 error popped up. http://www.directcolors.com/products/liquid-colored-antique/top
Technical SEO | | rblake
http://www.directcolors.com/applications/concrete-antiquing/top
http://www.directcolors.com/applications/concrete-countertops/top I understand that the **top **could be referring to an actual link that brings users to the top of a page, but on these pages there is no such link. Am I missing something?1 -
Sitemap error
Hi, When i search for my blog post in google i get sitemap results, and when i click on it i get an error, here is the screen shot http://screencast.com/t/lXOIiTnVZR1 http://screencast.com/t/MPWkuc4Ocixy How can i fix that, it loos like if i just add www. it work just fine. Thanks
Technical SEO | | tonyklu0 -
Suggested crawl rate in google webmaster tools?
hey moz peeps, got a general question: what is the suggested custom crawl rate in google webmaster tools? or is it better to "Let Google determine my crawl rate (recommended)" If you guys have any good suggestions on this and site why that would be very helpful, thanks guys!
Technical SEO | | david3050 -
Most Common Errors & Warnings
Hello there, i would like to ask some basic tips.. regarding found common errors & Warnings. list : Tittle Element Too Long
Technical SEO | | Bretly
Duplicate Page Content
and Duplicate Page Tittle. how could i fixed this one? any help would be greatly appreciated regards,0