Recovering from disaster
-
Short Question: What's the best way to get Google to re-index duplicate URLs?
Long Story:
We have a long ago (1997) established website with a proprietary CMS. Never paid much attention to SEO (other than creating a sitemap) until four months ago. After learning some we started modifying the engine to provide better site to google (proper HTTP codes, consistent URLs to eliminate duplicates - we had something like 15,000 duplicates - etc...)
Things went great for three and half months and we reached the first page on google for our main keyword (very, very competitive keyword). Before the SEO we were getting around 25,000 impressions and 3000 clicks on google. After our SEO efforts, we reached 70,000 daily impressions and more than 7000 daily clicks.
On Aug 30th, 2014, one of our programmers committed a change to the live server by mistake. This small change effectively changed every article's URL by adding either a dash at its end or a dash and a keyword '-test-keyword' (literally).
Nobody noticed anything until two days later as the site worked perfectly for humans. The result of this small code change is that within five days our site practically disappeared from Google's results pages except when one searched for our site's name. Our rank dropped from 8 and 10 to 80 and 100 for our main keywords.
We reverted the change as soon as we noticed the problem, but during those two days, Google's bots went on a binge crawling five times the usual number of page crawled per day.
We've been trying to recover and nothing seems to be working so far. Google's bots aren't crawling the repaired URLs to get the 301 headers back to the original URL and now we still have over 2300 duplicates as reported by the webmaster tools.
Our Google impressions and clicks dropped to way below what we had before we did any SEO, down to 5000 impressions and 1200 clicks (inclusive of our direct domain name search).
During the last 15 days (after we fixed the problem), our duplicate count went from a maximum of 3200, down to 1200, then back up to 2300 without any changes on our end.
we've redone our sitemap and resubmitted it on day 3.
So, what do we do? Do we go through the URLs with 'fetch as Google' function? (that's a bit tedious for 2300 URLs) or we wait for the bots to come around whenever they feel like it? if we do this, should we submit the bad URL, have google fetch it, get the redirect, follow it and then submit the followed URL to the index?
Or is there a better solution that I'm unaware of?
Second question: Is this something to be expected when something like this happens knowing that our inbound link rarely link to the actual articles?
-
Well, after submitting multiple temporary sitemaps and having Google index them, our duplicate counts dropped back to pre-event levels.
However, our rankings haven't improved at all. Actually, if anything, they dropped even further.
At this point it's really starting to look like this is a hit from Panda 4.1 and that we had our URLs change was merely a coincidence. From the looks of it, Google is now marking our site as a low quality site. Now that we know about such a thing, we definitely experienced a 'sinister surge' prior to disaster striking.
Since we've never engaged in any bad behavior on the site and we've always followed google's best practice advice, we're currently at a loss of what could be the reason that we're hit that way. Our content is fresh and high quality (arguably the highest quality in our domain), we have a very decent link profile according to MajesticSEO, so for now, no clue about what's going on really.
Attached is the site's impressions and clicks graph from Webmaster Tools.
-
["knowing that our inbound link rarely link to the actual articles" --> not sure I follow.]
I asked whether it's normal for all ranking to drop even for unaffected pages when pages with no inbound link have issues. For example, our top ranked page for our main keyword didn't change in anyway, not its URL, its description nor its title, yet it's rank tanked after this event.
I like the temporary sitemap idea. Thanks.
-
Once you have all the 301 redirects set up, create a sitemap with all of the old urls and submit that. Google will crawl them and see that they are now 301 redirects and process the data faster. then delete the sitemap.
You should also have a canonical tag on the article pages with the new/current link that should be indexed.
"knowing that our inbound link rarely link to the actual articles" --> not sure I follow.
In general, your rankings should bounce back once google picks up on all of the fixes.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Recover google INdexing issue after fixing malware attack.
Dear My Niche site attacked by malware on 1 st march 2018. Hacker inject a php file on my blogpage. Injected link like: mydomain.com/blog/dmy4xa.php? Then I scan My site by wordfence. Identifying all malware code.Then manually clean whole site with database. My site is completely free from malware. and remove all malware link from webmaster tools. Even Block my blog page by robots.txt . But new malware link index every week. So i need to remove those link every week. So this issue I decided to rebuild my site. Finally I rebuild my site another server. Then I flash my current server and migrate my site from those server on 10th january 2019 . I wait 1 month to deindex malware link. But new link are indexing every week. I discourage site for over 1 week and even delete site from google webmaster tools with all properties as well as verification file from server. Over 1 week , Link are showing. I feel boar to delete malware link every week. I need permanent solution. Please give me a perfect solution for this malware link index. Google index about 100 url .After that I clean my site with some tools. My site was free from malware. But Ne
Technical SEO | | Gfound1230 -
I have lose my ranking Via 301 Redirection - How To Recover?
Hey, Folks! I Have Used 301 Redirection Method to Increase My Rankings but When i applied this Method My Website Ranked Down To 55 Numbers. Can anyone Suggest me How to Recover it?
Technical SEO | | SumitJiGupta0 -
Recovering an Almost Dead Blog?
Hello, I wanted to ask this from long time but finally i gathered my energy to ask this long question at moz. Well, like almost all newbies with little knowledge of SEO & google I started my first blog in 2009, as things were very different that time & with posting more and more, I was getting good results & started to build decent traffic but with poor content ( I really din't care about it ) as I was getting organic traffic. But things changed with Google Panda Completely after 11th April 2011, Since the time Traffic keep on falling, I never made backlinks so Penguin Updates never hit us but because of Poor & thin Content Site went down lower & lower. I took some steps like increasing word count of posts, removing some posts but nothing worked so far but nothing worked. Blog has almost 1200 articles & most important it was my first blog so I was bit attached with it. Now my Question is, Should I just dispose the blog & move on or There is something which I can try to recover it. The blog is 6 years old as of of now & received 2 million organic traffic as of now. ( attached organic Traffic screenshot ) My question is, Can something be done Seriously for this blog or I should just let it go. I will appreciate some genuine advice on that. Thanks ZhS1xIS.png
Technical SEO | | killthebillion0 -
Home Page Deindexed Only at Google after Recovering from Hack Attack
Hello, Facing a Strange issue, wordpress blog hghscience[dot]com was hacked by someone, when checked, I found index.php file was changed & it was showing some page with a hacked message, & also index.html file was added to the cpanel account.All pages were showing same message, when I found it, I replaced index.php to default wordpress index.php file & deleted index.htmlI could not find any other file which was looking suspicious. Site started working fine & it was also indexed but cached version was that hacked page. I used webmaster tool to fetch & render it as google bot & submitted for indexing. After that I noticed home page get deindexed by google. Rest all pages are indexing like before. Site was hacked around 30th July & I fixed it on 1st Aug. Since then home page is not getting indexed, I tried to fetch & index multiple time via google webmasters tool but no luck as of now. 1 More thing I Noticed, When I used info:mysite.com on google, its showing some other hacked site ( www.whatsmyreferer.com/ ) When Searching from India But when same info:mysite.com is searched from US a different hacked site is showing ( sigaretamogilev.by )However when I search "mysite.com" my site home page is appearing on google search but when I check cached URL its showing hacked sites mentioned above.As per my knowledge I checked all SEO Plugins, Codes of homepage, can't find anything which is not letting the homepage indexed.PS: webmaster tool has received no warning etc for penalty or malware. I also noticed I disallowed index.php file via robots.txt earlier but now I even removed that. 7Dj1Q0w.png 3krfp9K.png
Technical SEO | | killthebillion0 -
Slowly recovering from algorithm penalty
Hi , over the years a website we took over was hit by an algorithm penalty (a combination of penguin and panda). We managed to bring rankings back (after 6 months) from page 5/6 to page 2 after we used the google disavow tool. now for the past 9 months we have been stuck on page 2.. is there anything you think can be done to bring it back to page 1? we are building quality links now and moved away from low quality links other link builders were making. We are managing the process much closer and ensuring we maintain good standards of links. also making the pages flatter and merging short page content to larger content pages now we are looking at site structure and creating structured internal link flow is there anything we should be aware of and any recommendations to get back on page 1.. this is a tailor-made travel related website with a small selection of destinations
Technical SEO | | Direct_Ram0 -
How to recover from duplicate subdomain penalty?
Two and half a weeks ago, my site was slapped with a penalty -- 60% of organic traffic disappeared over 2-3 days. After investigating we discovered that our site was serving the same content for all subdomains, and Google somehow had two additional subdomains it was crawling and indexing. We solved the issue with 301 redirects to our main site (www) a couple of days after the drop -- about two weeks ago. Our rankings have not recovered, and the subdomains are still indexed per Webmaster Tools. Yesterday we submitted a Reconsideration Request. Will that help? Is there any other way to speed up the process of lifting the penalty? This is the site: http://goo.gl/3DCbl Thank you!
Technical SEO | | tact0 -
A huge drop in rankings since last 10 days, and not recovered yet.
Hi Mozzers, I have a serious topic to discuss and want help from the experts here. Our website has 6 PR and we have been consistency staying at the top for very competitive terms in the niche. Since last Friday (24th February, 2012) we have been facing massive fluctuation in the rankings for most of the keywords we are focusing on. After this fall, we checked the following details but didn’t find any serious/critical issue that might be contributing towards these fluctuations:- We analyzed Google webmaster tools, there’s no update/warning from Google regarding any negative activity and other things seem to be normal. We checked our website through site search (site: www.domain.com) and found that we haven’t lost any indexed pages and things appear normally as they used to. So, we are sure that we haven’t been banned or penalized. We also cross verified our link building and other promotional activities and we didn’t find anything suspicious that could lead to such a big fluctuation. The drop is really big, some keywords went to 5th or 6th page from top 3 position; some keywords are not in top 200 or 300 spots which were usually staying put between 5th to 10th position. We have analyzed a lot but haven’t come to know the reason why we are facing this fluctuation. Our website is 4 years old and this kind of fluctuation has happened for the first time. Has anyone faced this kind of issue before? I’m looking forward to your support in identifying this trouble. Thanks
Technical SEO | | ValSmith0 -
Redesign an SEO-Disaster | Help with Redirects of Gray Hat Pages
Hi gang. I'm a new SEO and I'm currently working on the redesign of a website. I have just discovered a ton of hidden pages that are filled with duplicate content, basically reiterating the main keyword in a variety of different variations. Each page is titled with the variation on the keyword phrase and then has one paragraph of text very similar to the previous page, etc. Here is an example of one of the offensive pages (nice lookin' site, eh?): http://www.vasectomy-reversals.com/vasectomy_reversal_surgery.html The new site will not have any of these pages. I'm writing the 301 redirects now and want to redirect these offensive pages to the most relevant page on the new site. But, I'm afraid to redirect the offensive pages. Should I leave them alone, or can I have the former developer remove them? Help. Don't know how to handle these pages and their redirects. Thanks for your help! ~ Mills
Technical SEO | | Mills0