Welcome to the Q&A Forum

Hi mike, we do have webmaster tools set up and I will take a look at Screaming Frog, as it looks like an efficient piece of software.

I should have probably been a bit clearer in my query though. Will Screaming Frog tell me which internal links are pointing to 404 pages?

The only way I can think of checking this at the moment is to go through every page on the site, checking each of the links to see if they go to the correct pages.

We possibly have internal links on our site that point to 404 pages as well as links that point to old pages.

I need to tidy this up as efficiently as possible and would like some advice on the best way to go about this.

Brilliant thanks Dan.

This is what I'm going to do.

Looking at this video - https://www.youtube.com/watch?v=KBdEwpRQRD0&feature=youtu.be Matt Cutts advises to use the noindex tag on every individual page. However, this is very time consuming if you're dealing wit a large volume of pages.

The other option he recommends is to use the robots.txt file as well as the URL removal tool in GWMT, Although this is the second choice option, it does seem easier for us to implement than the noindex tag.

Hi Nico

Great response thanks.

This is certainly something I'm taking into consideration and will question my developer about this.

Thanks Thomas.

I'm now finding out from my developer is we are able to noindex these pages with the meta robots.

If this is something that isn't possible, it's likely that we'll add to the robots.txt as you did.

Either way I think will be progress to different degrees.

Thanks Martijn. This is what I was assuming would happen. However, I got a confusing message from my developer which said the following,

"won't remove the URL's from the index but it will mean that they will only show up for very specific searches that customers are extremely unlikely to use. It will also increase Asgard's crawl budget as Google and Bing won't try to crawl these URLs. Would you be happy with this solution?"

I would tend to still agree with your statement though.

Our developer tells us that we have a lot of 404 pages that are being included in our sitemap and the reason for this is because we have put 301 redirects on the old pages to new pages.

We're using Magento and our current process is to simply disable, which then makes it a a 404. We then redirect this page using a 301 redirect to a new relevant page. The reason for redirecting these pages is because the old pages are still being indexed in Google.

I understand 404 pages will eventually drop out of Google's index, but was wondering if we were somehow preventing them dropping out of the index by redirecting the URL's, causing the 404 pages to be added to the sitemap.

My questions are:

1. Could we simply delete the entire unwanted page, so that it returns a 404 and drops out of Google's index altogether?

2. Because the 404 pages are in the sitemap, does this mean they will continue to be indexed by Google?

I found a lot of duplicate title tags showing in Google Webmaster Tools. When I visited the URL's that these duplicates belonged to, I found that they were just images from a gallery that we didn't particularly want Google to index. There is no benefit to the end user in these image pages being indexed in Google.

Our developer has told us that these urls are created by a module and are not "real" pages in the CMS.

They would like to add the following to our robots.txt file

Disallow: /catalog/product/gallery/

QUESTION: If the these pages are already indexed by Google, will this adjustment to the robots.txt file help to remove the pages from the index?

We don't want these pages to be found.