Help recover lost traffic (70%) from robots.txt error.
-
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change.
Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated.
On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL?
Thanks,
-
Firstly, I would definitely take the opportunity to switch to SSL. A migration to SSL shouldn't be something to worry about if you set up your redirects properly, but given that most of your pages aren't indexed at all, it is even less risky.
You will eventually get the traffic back, as far as how long, it's very difficult to say.
I would concentrate on crawlability, and make sure your structure makes sense, and that you aren't linking any 404's or worse. Given the size of your site, that wouldn't be a bad thing anyway.
From your description of your pages, I'm not sure there is any "importance hierarchy", so my suggestion may not help, but you could make use of Google's API to submit pages for crawling. Unfortunately, you can only submit in batches of 100 and you are limited to 200 a day. You could, of course, prioritise or cherry pick some important pages and "hub" pages, if such things exist within your site, and then start working through those.
Following the recent Google blunder where they deindexes huge swathes of the web and, in the short term, the only way to get them back in the index was to resubmit them, someone has provided a tool to interact with the API, which you can find here: https://github.com/steve-journey-further/google-indexing-api-bulk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does sending ecom traffic away from a site impact rankings?
I have a client that represents a few big brands but doesn't offer ecommerce capability onsite, they essentially send traffic to retailers and distributors that do sell their products online - they even point to an amazon store that they own. My question is, if they did offer ecom and the conversions took place on their site, rather than direct traffic away from them, would this impact on their rankings at all?
On-Page Optimization | | Algorhythm_jT0 -
Need Professional Help with Site Structure, Page Authority, and Internal Linking
We are a 10 year + small (1000 pages) niche ecommerce site (Magento) that has recently lost rankings to a competitor. We out perform them in every metric so I do not understand them leap frogging us to the top spot. This has forced me to look at my site structure, page authority (rank), and internal linking. After reviewing a moz crawl issues report, here are some of my observations: Root domain has a PA of 40 Top 3 $ Category pages have a PA of 22, 18, 18 Multiple meaningless blog posts and other category/product pages have PA’s of 30+ Here is a screenshot of the crawl report with internal links, links, etc showing. I need some help - thoughts, suggestions, next steps in analysis?
On-Page Optimization | | SammyT0 -
Static or Dynamic Home Page? Please Help!
Hi again 🙂 I ask for ur input again my friends I finally decided to go for the shorter .COM domain, and I put my domain online. Since it will be an online games site, every post will be in fact a new game. And of course, every game, will be in fact a post. I put my home page as "latest posts" this way, all my posts are shown in home page, and posts being for me games, all my games appear on my home page. So my home page always has the latest games and is always fresh. The theme I use has the option to ad some "static content" at the end of the posts, on home page, for SEO. I my home page I have also about 300 words of "article" content, that never changes, and is for seo, since my post content, is in fact java script, and other html codes ( flash games). So my home pages is that article plus excerpts of latest post. But the problem is, that I cannot control the seo, on page this way, and I cannot control my keywords very well, or the density of them, since the home page isnt a page. Now the question is, to let my Home page as it is, or change it and set it up as a page. I can put games on pages to, that is not a big problem. What is best, for the future of the site, google ranking, and seo, in ur opinion. I ad that I recived "A" Grade on moz for Page Seo Grade! Thank You!
On-Page Optimization | | Catinas970 -
Traffic Down - May Need Outside Help
Hi Moz Community - Our website (www.motivators.com) experienced a small traffic drop in mid-March. This was followed by a steady decline in traffic through May, June and July. Please note that a site redesign went live on April 4th. Starting in mid-July, we began implementing aggressive site improvements (mostly based upon site speed), but our traffic is still down. Can anyone recommend a service or company that can look at our site and determine the root cause / more strategies for improvement? Thanks for your suggestions!
On-Page Optimization | | Motivators0 -
Authorship and 50% Drop in Search Traffic
Hi. I had some problems with my content being copied by other sites, and was suggested to claim my pages. Immediately after Google started listing the posts with my photo next to them my traffic picked up slightly but since, over the past few weeks, it has dropped by about 50%! It seems that the average position has declined (as has the CTR) At the same time as claiming authorship I added the post date to my articles which I have now removed. Does anyone have any ideas for me? Thanks!
On-Page Optimization | | ben10000 -
Canonical tag help
Hi, We have a product which is marketed by affiliates . Affiliates send referrals to our sale page by adding their affiliate IDs to our product page like http://www.mysite.com/products.php?ref= 12345. We want to avoid the content duplication impression to Google by using canonical tags but we are not clear about its use. Should we use it on http://www.mysite.com/products.php ( actual page) or we should create temporary pages for each referral id i.e http://www.mysite.com/products.php?ref= 12345 and then add canonical tags to all those pages linking to proper page i.e http://www.mysite.com/products.php ? Thanks, shaz
On-Page Optimization | | shaz_lhr0 -
What's the best practice for implementing a "content disclaimer" that doesn't block search robots?
Our client needs a content disclaimer on their site. This is a simple "If you agree to these rules then click YES if not click NO" and you're pushed back to the home page. I have this gut feeling that this may cause an upset with the search robots. Any advice? R/ John
On-Page Optimization | | TheNorthernOffice790 -
Crawl Diagnostics - Duplicate Content and Duplicate Page Title Errors
I am getting a lot of duplicate content and duplicate page title errors from my crawl analysis. I using volusion and it looks like the photo gallery is causing the duplicate content errors. both are sitting at 231, this shows I have done something wrong... Example URL: Duplicate Page Content http://www.racquetsource.com/PhotoGallery.asp?ProductCode=001.KA601 Duplicate Page Title http://www.racquetsource.com/PhotoGallery.asp?ProductCode=001.KA601 Would anyone know how to properly disallow this? Would this be as simple as a robots.txt entry or something a little more involved within volusion? Any help is appreicated. Cheers Geoff B. (a.k.a) newbie.
On-Page Optimization | | GeoffBatterham0