Recovering from index problem (Take two)

[Blink-SEO]

Hi all. This is my second pass at the problem. Thank you for your responses before, I think I'm narrowing it down!

Below is my original message. Afterwards, I've added some update info.

For a while, we've been working on http://thewilddeckcompany.co.uk/. Everything was going swimmingly, and we had a top 5 ranking for the term 'bird hides' for this page - http://thewilddeckcompany.co.uk/products/bird-hides.

Then disaster struck! The client added a link with a faulty parameter in the Joomla back end that caused a bunch of duplicate content issues. Before this happened, all the site's 19 pages were indexed. Now it's just a handful, including the faulty URL (thewilddeckcompany.co.uk/index.php?id=13‎)

This shows the issue pretty clearly.

https://www.google.co.uk/search?q=site%3Athewilddeckcompany.co.uk&oq=site%3Athewilddeckcompany.co.uk&aqs=chrome..69i57j69i58.2178j0&sourceid=chrome&ie=UTF-8

I've removed the link, redirected the bad URL, updated the site map and got some new links pointing at the site to resolve the problem. Yet almost two month later, the bad URL is still showing in the SERPs and the indexing problem is still there.

UPDATE

OK, since then I've blocked the faulty parameter in the robots.txt file. Now that page has disappeared, but the right one - http://thewilddeckcompany.co.uk/products/bird-hides - has not been indexed. It's been like this for several week.

Any ideas would be much appreciated!

[Blink-SEO]

Thank you all, this is brilliant.

[CleverPhD]

Your problem is with the robots.txt file.  You are blocking the URL

thewilddeckcompany.co.uk/index.php?id=13

That URL 301 redirects to the correct URL of

http://thewilddeckcompany.co.uk/products/bird-hides

Google cannot "see" the 301 redirect from the old "bad" URLs to the new "good" URL.

You have to let Google crawl the old URLs and see the 301 redirects so that it knows how things need to forward.

I would do this for all the duplicate pages, make sure they 301 to the correct pages and do not put the "bad" pages in robots.txt - otherwise the indexing will not be updated.

Something separate to check.  We have seen Google taking a while to acknowledge some of our 301s.  Go into your GWT and look at your duplicate title reports.  You may see the old and new URLs showing as duplicates, even with the 301s in place.   We had to setup a self canonicalizing link on the "good" pages to help get that cleaned up.

[RobertFisher]

Blink-SEO

Jonathan is correct to try a Fetch as Google in WMT for the urls you need re indexed. (Note, that is not really the purpose of a Fetch as Google, but sometimes it works.)
I would also resubmit the sitemap now that you have blocked the offending url with robots.txt. It is likely the resubmission will help you the quickest IMO.

Best,

Robert

[Jonathan1979]

It sounds like you just need to wait for Google to recrawl your robots.txt file.  I saw this error in the serps:

www.thewilddeckcompany.co.uk/products/timber-water...

A description for this result is not available because of this site's robots.txt – learn more.So it is clear that the robots.txt file has not updated with the changes, after the mistake was made.  Try fetching as Googlebot within webmaster tools, but it may take a little time to update.  But at least it would seem that the robots.txt error is still a cause of the problem, just need to wait a little longer.