Increase in Crawl Errors
-
I had a problem with a lot of crawl errors (on Google Search Console) a while back, due to the removal of a shopping cart. I thought I'd dealt with this & Google seemed to agree (see attached pic), but now they're all back with a vengeance!
The crawl errors are all the old shop pages that I thought I'd made clear weren't there anymore. The sitemaps (using Yoast on Wordpress to generate these) all updated 16 Aug but the increase didn't happen till 18-20.
How do I make it clear to Google that these pages are gone forever?
-
Many thanks Britney
-
This is a great robots.txt cheat sheet to help figure out what robot exclusions might be right for your pages.
-
Hi Martijn
Many thanks for your response. We have been redirecting old product URLs to the shop URL (not the front page, but the front page of the shop) but I think ideally we just want rid of all these pages as the client is considering getting rid of the shop completely.
What would we do with robots.txt in this instance? Is there a tutorial you could point me at?
-
HI,
Did you redirect the old pages to another page, like worst case the home page? The alternative would be to look into excluding these pages if they have a similar page path from the search engines by adding them to your robots.txt.
Martijn.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
4xx errors
I"m trying to fix 4xx errors but I"m not finding the pages in my admin. Where can I find this page? https://cracklefireplaces.com/collections/ethanol-wall-mounted-fireplaces/products/ignis-maximum-wall-mounted-ethanol-fireplace
Technical SEO | | carlbrekjern0 -
Help Crawl friendliness for large site
After watching Rand's video I am trying to think of the best way to make my large site more crawl friendly. Background I have a large site with over 100k product skus and so when you get to a particular page of products there are tons of different refinements and options that help you sort the products. Most of these are noindex followed, but I was wondering if I should be nofollowing the internal links as well in order to keep bots out of those pages and going to the pages that I want them to go too. Is this a good way to handle it? Also, does anyone have good recommendations of links to posts that deal with helping the crawl friendliness of a large site? Thanks!
Technical SEO | | Gordian0 -
Noticed a lot of duplicate content errors...
how do I fix duplicate content errors on categories and tags? I am trying to get rid of all the duplicate content and I'm really not sure how to. Any suggestions, advice and/or help on this would be greatly appreciated. I did add the canonical url through the SEO Yoast plugin, but I am still seeing errors. I did this on over 200 pages. Thanks for any assistance in advance. Jaime
Technical SEO | | slapshotstudio0 -
Crawl errors: 301 (permanent redirect)
Hi, here are some questions about SEO Crawl Diagnostics. We've recently found out this 301 (permanent redirect) errors in our website and we concluded that the two factors below are the causes. 1. Some of our URLs that has no / at the end is automatically redirected to the same URL but with / at the end. 2. For SEO reasons we have designed our website in a way that when we type in a URL it will automatically redirect to a more SEO friendly URL. For example, if one of the URLs is www.example.com/b1002/, it will automatically redirect to www.example.com/banana juice/. The question is, are these so significant for our SEO and needs to be modified? One of the errors in our blog was having too many on-page links. Is this also a significant error and if so, how many on-page links are recommended from the SEO perspective? Thanks in advance.
Technical SEO | | Glassworks0 -
Changes to website haven't been crawled in over a month
We redesigned our website at http://www.aptinting.com a few months ago. We were fully expecting the crawl frequency to be very low because we had redesigned the website from a format that had been very static, and that probably has something to do with the problem we're currently having. We made some important changes to our homepage about a month ago, and the cached version of that page is still from April 2nd. Yet, whenever we create new pages, they get indexed within days. We've made a point to create lots of new blog articles and case studies to send a message to Google that the website should be crawled at a greater rate. We've also created new links to the homepage through press releases, guest blog articles, and by posting to social media, hoping that all of these things would send a message to Google saying that the homepage should be "reevaluated". However, we seem to be stuck with the April 2nd version of the homepage, which is severely lacking. Any suggestions would be greatly appreciated. Thanks!
Technical SEO | | Lemmons0 -
How does Google Crawl Multi-Regional Sites?
I've been reading up on this on Webmaster Tools but just wanted to see if anyone could explain it a bit better. I have a website which is going live soon which is going to be set up to redirect to a localised URL based on the IP address i.e. NZ IP ranges will go to .co.nz, Aus IP addresses would go to .com.au and then USA or other non-specified IP addresses will go to the .com address. There is a single CMS installation for the website. Does this impact the way in which Google is able to search the site? Will all domains be crawled or just one? Any help would be great - thanks!
Technical SEO | | lemonz0 -
404 Errors - How to get rid of them?
Hi, I am starting an SEO job on an academic site that has been completely redone. The SEOMoz crawl detected three 404 Errors to pages that cannot be found anywhere on either Joomla or the server. What can I do to solve this? Thanks!!
Technical SEO | | michalseo0 -
Having some weird crawl issues in Google Webmaster Tools
I am having a large amount of errors in the not found section that are linked to old urls that haven't been used for 4 years. Some of the ulrs being linked to are not even in the structure that we used to use for urls. Never the less Google is saying they are now 404ing and there are hundreds of them. I know the best way to attack this is to 301 them, but I was wondering why all of these errors would be popping up. I cant find anything in the google index searching for the link in "" and in webmaster tools it shows unavailable as where these are being linked to from. Any help would be awesome!
Technical SEO | | Gordian1