Recovering from disaster
-
Short Question: What's the best way to get Google to re-index duplicate URLs?
Long Story:
We have a long ago (1997) established website with a proprietary CMS. Never paid much attention to SEO (other than creating a sitemap) until four months ago. After learning some we started modifying the engine to provide better site to google (proper HTTP codes, consistent URLs to eliminate duplicates - we had something like 15,000 duplicates - etc...)
Things went great for three and half months and we reached the first page on google for our main keyword (very, very competitive keyword). Before the SEO we were getting around 25,000 impressions and 3000 clicks on google. After our SEO efforts, we reached 70,000 daily impressions and more than 7000 daily clicks.
On Aug 30th, 2014, one of our programmers committed a change to the live server by mistake. This small change effectively changed every article's URL by adding either a dash at its end or a dash and a keyword '-test-keyword' (literally).
Nobody noticed anything until two days later as the site worked perfectly for humans. The result of this small code change is that within five days our site practically disappeared from Google's results pages except when one searched for our site's name. Our rank dropped from 8 and 10 to 80 and 100 for our main keywords.
We reverted the change as soon as we noticed the problem, but during those two days, Google's bots went on a binge crawling five times the usual number of page crawled per day.
We've been trying to recover and nothing seems to be working so far. Google's bots aren't crawling the repaired URLs to get the 301 headers back to the original URL and now we still have over 2300 duplicates as reported by the webmaster tools.
Our Google impressions and clicks dropped to way below what we had before we did any SEO, down to 5000 impressions and 1200 clicks (inclusive of our direct domain name search).
During the last 15 days (after we fixed the problem), our duplicate count went from a maximum of 3200, down to 1200, then back up to 2300 without any changes on our end.
we've redone our sitemap and resubmitted it on day 3.
So, what do we do? Do we go through the URLs with 'fetch as Google' function? (that's a bit tedious for 2300 URLs) or we wait for the bots to come around whenever they feel like it? if we do this, should we submit the bad URL, have google fetch it, get the redirect, follow it and then submit the followed URL to the index?
Or is there a better solution that I'm unaware of?
Second question: Is this something to be expected when something like this happens knowing that our inbound link rarely link to the actual articles?
-
Well, after submitting multiple temporary sitemaps and having Google index them, our duplicate counts dropped back to pre-event levels.
However, our rankings haven't improved at all. Actually, if anything, they dropped even further.
At this point it's really starting to look like this is a hit from Panda 4.1 and that we had our URLs change was merely a coincidence. From the looks of it, Google is now marking our site as a low quality site. Now that we know about such a thing, we definitely experienced a 'sinister surge' prior to disaster striking.
Since we've never engaged in any bad behavior on the site and we've always followed google's best practice advice, we're currently at a loss of what could be the reason that we're hit that way. Our content is fresh and high quality (arguably the highest quality in our domain), we have a very decent link profile according to MajesticSEO, so for now, no clue about what's going on really.
Attached is the site's impressions and clicks graph from Webmaster Tools.
-
["knowing that our inbound link rarely link to the actual articles" --> not sure I follow.]
I asked whether it's normal for all ranking to drop even for unaffected pages when pages with no inbound link have issues. For example, our top ranked page for our main keyword didn't change in anyway, not its URL, its description nor its title, yet it's rank tanked after this event.
I like the temporary sitemap idea. Thanks.
-
Once you have all the 301 redirects set up, create a sitemap with all of the old urls and submit that. Google will crawl them and see that they are now 301 redirects and process the data faster. then delete the sitemap.
You should also have a canonical tag on the article pages with the new/current link that should be indexed.
"knowing that our inbound link rarely link to the actual articles" --> not sure I follow.
In general, your rankings should bounce back once google picks up on all of the fixes.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Never recovered from loss of visibility in Google, what did we miss?
Hello, I hope someone can help me, or can recommend someone who could look into this in more detail a a consultancy service or something. Our client https://bit.ly/2IjLD7X experienced a bad drop in visibility early 2017 which we didn't manage to recover from. We had a problem with one main category page that wasn't indexed anymore (it was #1 for a major keyword in the Netherlands). We were able to fix the problem eventually (had to do with the .be being indexed instead due to no hreflang) but never really recovered with that page, only the entire site around that time saw a major drop in visibility. The client went to another agency (a renowned agency in the Netherlands) but after over half a year they haven't been able to recover. Organic visits are still down 45% compared to the same period last year. Another major Dutch keyword they ranked #1 for is now #9 for instance. So basically two SEO agencies (we and the other one) have tried everything we could think of for the past year, but none of us seem to know what the problem is and the owner is quite desperate (it's a small store with only 3 employees and his monthly revenue is only half of what it once was. I'm afraid a quick look into the site won't help anyone find the issue, so if someone has a good idea of how we can help this previous client I would love to hear so!
Technical SEO | | WebdesignTilburg0 -
Do people recover from an unnatural outbound links penalty?
Dear Friends I received an unnatural outbound links penalty on 4th of June 2016. I immediately acted on it, and analysed a lot of outbound links and removed most of them. I had to file reconsideration 2 times, but on 26th of June my penalty was revoked.However, traffic drop was even worse 27th June onwards and it is still dropping. Now, i dont know why the traffic is dropping. Any idea how much time it takes to come back from an unnatural outbound link penalty? PHd8BzX.png
Technical SEO | | marketing910 -
Slowly recovering from algorithm penalty
Hi , over the years a website we took over was hit by an algorithm penalty (a combination of penguin and panda). We managed to bring rankings back (after 6 months) from page 5/6 to page 2 after we used the google disavow tool. now for the past 9 months we have been stuck on page 2.. is there anything you think can be done to bring it back to page 1? we are building quality links now and moved away from low quality links other link builders were making. We are managing the process much closer and ensuring we maintain good standards of links. also making the pages flatter and merging short page content to larger content pages now we are looking at site structure and creating structured internal link flow is there anything we should be aware of and any recommendations to get back on page 1.. this is a tailor-made travel related website with a small selection of destinations
Technical SEO | | Direct_Ram0 -
Need 301 Advice with a Recovered URL from a Domain Typosquatter
I am new to a SMB and someone bought the plural version of our domain back in 2005 and has yet to let it expire. The domain was just renewed for another year so we finally decided to contact a lawyer and go through the domain name dispute process. This seems like a pretty cut an dry case and the lawyer is very confident that we'll have the domain within 30-40 days. Currently the plural version domain 303s to spammy web pages, shows shady ads and is just a malicious looking page in general. I am not savvy enough to know the exact complexities of what's happening on the backend but it's spammy. Knowing the history of the plural version domain, how would you treat it after we acquire it? Obviously, I wouldn't want to put our site in jeopardy by 301ing the plural version of our URL to our current healthy site but at the same time many customers might go to that domain by accident so eventually I'd like to 301 it. If it's any help, the plural version has a robots.txt that prevent G from crawling it..thank you in advance for your guidance!
Technical SEO | | ssimarketing0 -
How long to recover from Panda Update
Hi there, I think I was affected by the recent Panda update as I had a lot of duplicate content for my product descriptions (about 300). I'm going through and rewriting these to be both helpful and unique. I was ranking quite nicely for a big spread of keywords, but have been seeing my rankings drop day after day since the update. Is it possible to see my rankings improve again after Google re-crawls my site, or would a penalty have been applied to my site preventing me to re-gain my positions for sometime. It's probably worth noting that I have a lot of unique and helpful content, it was just my product pages that had duplicate content, but I've seen my rankings across the board drop. Any discussion and insight would be much appreciated.
Technical SEO | | BlueTree_Sean0 -
Internal linking disaster
Can someone help me understand what my devs have done? The site has thousands of pages but if there's an internal homepage link on all of the pages (click on the logo) shouldn't that count for internal links? Could it be because they are nonfollow? http://goo.gl/0pK5kn I've attached my competitors opensiteexplorer rankings (I'm the 2nd column) .. so despite the face the site is new you can see where I'm getting my ass kicked. Thanks! psRsQtH.png
Technical SEO | | bradmoz0 -
Is this a Penguin hit; how to recover?
Our site is: www.kibin.com Our organic search traffic has dropped significantly in the last few days (about 50%). Although we don't have a ton of traffic coming in via organic search, this is a huge blow. We we previously the top spot for keywords like 'edit my essay', 'fix my paper', and 'paper revision'. These kw phrases linked to smaller services pages on our site such as: kibin.com/s/edit-my-essay kibin.com/s/fix-my-paper kibin.com/s/paper-revision We no longer rank for these keywords... at all. My first question/concern is... is this a Penguin penalty? I'm not sure. Honestly, I know we have some directory links, but we've never been aggressive on the backlink front. In fact, in the last few months we've been investing quite a bit into content: kibin.com/blog If this is a Penguin penalty, how should I best go about cleaning this up? I'm not even sure what links Google would be considering spammy, and again... we really don't have that extensive of a backlink profile. Please help, this is a real blow to our business and it's got me a big freaked out. Thanks!
Technical SEO | | Kibin0 -
Redesign an SEO-Disaster | Help with Redirects of Gray Hat Pages
Hi gang. I'm a new SEO and I'm currently working on the redesign of a website. I have just discovered a ton of hidden pages that are filled with duplicate content, basically reiterating the main keyword in a variety of different variations. Each page is titled with the variation on the keyword phrase and then has one paragraph of text very similar to the previous page, etc. Here is an example of one of the offensive pages (nice lookin' site, eh?): http://www.vasectomy-reversals.com/vasectomy_reversal_surgery.html The new site will not have any of these pages. I'm writing the 301 redirects now and want to redirect these offensive pages to the most relevant page on the new site. But, I'm afraid to redirect the offensive pages. Should I leave them alone, or can I have the former developer remove them? Help. Don't know how to handle these pages and their redirects. Thanks for your help! ~ Mills
Technical SEO | | Mills0