Recovering from disaster
-
Short Question: What's the best way to get Google to re-index duplicate URLs?
Long Story:
We have a long ago (1997) established website with a proprietary CMS. Never paid much attention to SEO (other than creating a sitemap) until four months ago. After learning some we started modifying the engine to provide better site to google (proper HTTP codes, consistent URLs to eliminate duplicates - we had something like 15,000 duplicates - etc...)
Things went great for three and half months and we reached the first page on google for our main keyword (very, very competitive keyword). Before the SEO we were getting around 25,000 impressions and 3000 clicks on google. After our SEO efforts, we reached 70,000 daily impressions and more than 7000 daily clicks.
On Aug 30th, 2014, one of our programmers committed a change to the live server by mistake. This small change effectively changed every article's URL by adding either a dash at its end or a dash and a keyword '-test-keyword' (literally).
Nobody noticed anything until two days later as the site worked perfectly for humans. The result of this small code change is that within five days our site practically disappeared from Google's results pages except when one searched for our site's name. Our rank dropped from 8 and 10 to 80 and 100 for our main keywords.
We reverted the change as soon as we noticed the problem, but during those two days, Google's bots went on a binge crawling five times the usual number of page crawled per day.
We've been trying to recover and nothing seems to be working so far. Google's bots aren't crawling the repaired URLs to get the 301 headers back to the original URL and now we still have over 2300 duplicates as reported by the webmaster tools.
Our Google impressions and clicks dropped to way below what we had before we did any SEO, down to 5000 impressions and 1200 clicks (inclusive of our direct domain name search).
During the last 15 days (after we fixed the problem), our duplicate count went from a maximum of 3200, down to 1200, then back up to 2300 without any changes on our end.
we've redone our sitemap and resubmitted it on day 3.
So, what do we do? Do we go through the URLs with 'fetch as Google' function? (that's a bit tedious for 2300 URLs) or we wait for the bots to come around whenever they feel like it? if we do this, should we submit the bad URL, have google fetch it, get the redirect, follow it and then submit the followed URL to the index?
Or is there a better solution that I'm unaware of?
Second question: Is this something to be expected when something like this happens knowing that our inbound link rarely link to the actual articles?
-
Well, after submitting multiple temporary sitemaps and having Google index them, our duplicate counts dropped back to pre-event levels.
However, our rankings haven't improved at all. Actually, if anything, they dropped even further.
At this point it's really starting to look like this is a hit from Panda 4.1 and that we had our URLs change was merely a coincidence. From the looks of it, Google is now marking our site as a low quality site. Now that we know about such a thing, we definitely experienced a 'sinister surge' prior to disaster striking.
Since we've never engaged in any bad behavior on the site and we've always followed google's best practice advice, we're currently at a loss of what could be the reason that we're hit that way. Our content is fresh and high quality (arguably the highest quality in our domain), we have a very decent link profile according to MajesticSEO, so for now, no clue about what's going on really.
Attached is the site's impressions and clicks graph from Webmaster Tools.
-
["knowing that our inbound link rarely link to the actual articles" --> not sure I follow.]
I asked whether it's normal for all ranking to drop even for unaffected pages when pages with no inbound link have issues. For example, our top ranked page for our main keyword didn't change in anyway, not its URL, its description nor its title, yet it's rank tanked after this event.
I like the temporary sitemap idea. Thanks.
-
Once you have all the 301 redirects set up, create a sitemap with all of the old urls and submit that. Google will crawl them and see that they are now 301 redirects and process the data faster. then delete the sitemap.
You should also have a canonical tag on the article pages with the new/current link that should be indexed.
"knowing that our inbound link rarely link to the actual articles" --> not sure I follow.
In general, your rankings should bounce back once google picks up on all of the fixes.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Home Page Deindexed Only at Google after Recovering from Hack Attack
Hello, Facing a Strange issue, wordpress blog hghscience[dot]com was hacked by someone, when checked, I found index.php file was changed & it was showing some page with a hacked message, & also index.html file was added to the cpanel account.All pages were showing same message, when I found it, I replaced index.php to default wordpress index.php file & deleted index.htmlI could not find any other file which was looking suspicious. Site started working fine & it was also indexed but cached version was that hacked page. I used webmaster tool to fetch & render it as google bot & submitted for indexing. After that I noticed home page get deindexed by google. Rest all pages are indexing like before. Site was hacked around 30th July & I fixed it on 1st Aug. Since then home page is not getting indexed, I tried to fetch & index multiple time via google webmasters tool but no luck as of now. 1 More thing I Noticed, When I used info:mysite.com on google, its showing some other hacked site ( www.whatsmyreferer.com/ ) When Searching from India But when same info:mysite.com is searched from US a different hacked site is showing ( sigaretamogilev.by )However when I search "mysite.com" my site home page is appearing on google search but when I check cached URL its showing hacked sites mentioned above.As per my knowledge I checked all SEO Plugins, Codes of homepage, can't find anything which is not letting the homepage indexed.PS: webmaster tool has received no warning etc for penalty or malware. I also noticed I disallowed index.php file via robots.txt earlier but now I even removed that. 7Dj1Q0w.png 3krfp9K.png
Technical SEO | | killthebillion0 -
Slowly recovering from algorithm penalty
Hi , over the years a website we took over was hit by an algorithm penalty (a combination of penguin and panda). We managed to bring rankings back (after 6 months) from page 5/6 to page 2 after we used the google disavow tool. now for the past 9 months we have been stuck on page 2.. is there anything you think can be done to bring it back to page 1? we are building quality links now and moved away from low quality links other link builders were making. We are managing the process much closer and ensuring we maintain good standards of links. also making the pages flatter and merging short page content to larger content pages now we are looking at site structure and creating structured internal link flow is there anything we should be aware of and any recommendations to get back on page 1.. this is a tailor-made travel related website with a small selection of destinations
Technical SEO | | Direct_Ram0 -
How to recover search volume after domain name change?
On the 3rd of November we changed our company name and domain. The new site was not changed at all so the 301 process was quite straightforward. The change over was successful, no downtime, all pages redirected correctly (with a few minor exceptions). However, after a few days we started to see more and more links into the new site from the old site. They now stand at over 3 million. And links from the new site to the old site of over 200K. Links from the new site back to the old, were due to us having left a lot of links tucked away on various pages which were possibly causing loops with the 301 redirects on the old site. We fixed these and now there are no remaining links back to the old site, though we are still showing just over 200K links back to the old site. We are also seeing a LOT more back-links on the new site from old junk sites, which are not showing for the old site. A couple of years ago we went through about a year of trying to track down and remove thousands of spam backlinks. We did what we could, got a lot removed, showed Google the evidence, then Google lifted the penalty and said they had made some changes that meant the links were no longer causing the penalty. I added the old disavow file to the new site, but it doesn't cover a fraction of the sites which are being displayed as providing backlinks... many of which are clearly spammy. Is it possible that Google made some manual actions to lift the penalties but failed to associate these changes with the new domain? Changes that were not included in the disavow file? All help appreciated.
Technical SEO | | Exotissimo0 -
How long to recover from Panda Update
Hi there, I think I was affected by the recent Panda update as I had a lot of duplicate content for my product descriptions (about 300). I'm going through and rewriting these to be both helpful and unique. I was ranking quite nicely for a big spread of keywords, but have been seeing my rankings drop day after day since the update. Is it possible to see my rankings improve again after Google re-crawls my site, or would a penalty have been applied to my site preventing me to re-gain my positions for sometime. It's probably worth noting that I have a lot of unique and helpful content, it was just my product pages that had duplicate content, but I've seen my rankings across the board drop. Any discussion and insight would be much appreciated.
Technical SEO | | BlueTree_Sean0 -
How Long To Recover Rankings After Multi-Day Site Outage?
Hi, A site we look after for a client was down for almost 3 days at the start of this month (11th - 14th of May, to be exact). This was caused by my client's failure to verify their domain name in accordance with the new ICANN procedures. The details are unimportant, but it took a long while for them to get their domain name registration contact details validated, hence the outage. Very soon after this down time we noticed that the site has slipped back in the Google rankings for most of the target keywords, sometimes quite considerably. I guess this is Google penalizing this client for their failure to keep their site live. (And they really can't have too many complaints about this, in my opinion). The good news is that the rankings show signs of improving again slightly. However, they have not recovered all the way to where they were before the outage, two weeks ago. My question is this ... do you expect that the site will naturally re-gain the previous excellent rankings without us doing anything? If so, how long do you estimate this could take? On the other hand, if Google typically penalizes this kind of error by 'permanently', is there is anything we can do to help signal to Google that the site deserves to get back up to where is used to be? I am keen to get your thoughts, and especially to hear from anyone who has faced a similar problem in the past. Thanks
Technical SEO | | smaavie0 -
Redesiging website and how to prevent a SEO disaster?
Hey guys, I'm in the process of re designing my company website. This website runs in Kentico CMS and now I'm moving to drupal CMS. The main reason for the move is that I need a more flexible design with responsiveness and integrate better social aspects. Currently I'm having to deal with a developer whom can't be bothered to update the site. I also find kentico extremely difficult to deal with. The site has good Google rankings and I was thinking about setting up some 301 redirects when the move is completed. Now the question is this site is been live for over 2 years and I have tons of blog posts sitting in here. How can I get these moved? When this is done will they lose their 'age' ? What if I re create the blog posts as it is and then setup 301 re-directs on this? I have G+ authorship and will they be affected by this? Last question is having a responsive site will affect my SEO rankings once moved? I heard the 301 is not considered as a 'safe' thing with Google so what are my options? If anyone can share a link(s) to site upgrades/moving to new design tutorials/best practices articles based on SEO ill be grateful. Thank you very much.
Technical SEO | | Suganthan0 -
Can I disallow my subdomain for penguin recover?
Hi, I have a site like BannerBuzz.com, before last penguin my site's all keywords were in good position in google, but after penguin hit on my website, my all keywords are going down and down day by day, i have done some changes in my website for improvement, but in 1 change i have some confusion. i have one sub domain (http://reviews.bannerbuzz.com/), which display my websites all keywords user reviews, in which every category's 15 reviews are display in my website http://www.bannerbuzz.com so are those user reviews consider as duplicate content between sub domain and main website. can i disallow sub domain from all search engine? currently sub domain is open for all search engine, is that helpful to block it? Thanks
Technical SEO | | CommercePundit0 -
Traffic stats disaster
Hi, We have recently moved a few old websites to an existing one and redesigned the existing website too. The visits in Analytics have fallen though the roof in the last few days. I would expect the traffic to have gone up because visitors from the old websites should now be on the redesigned one. Not sure what the problem is entirely. GA code is on the site but not before the closing head where I asked for it to be put. It's just before closing body tag. Does it make a difference? (It's a CMS site). GA code must be working because visits are being recorded (tens rather than the expected hundreds). Redirects have not been done properly - not sure what IT have done there. Would the lack of proper redirects be the sole reason for the drop? I am still hassling for 301's to be sorted. Thanks all
Technical SEO | | Houses0