Help recover lost traffic (70%) from robots.txt error.
-
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change.
Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated.
On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL?
Thanks,
-
Firstly, I would definitely take the opportunity to switch to SSL. A migration to SSL shouldn't be something to worry about if you set up your redirects properly, but given that most of your pages aren't indexed at all, it is even less risky.
You will eventually get the traffic back, as far as how long, it's very difficult to say.
I would concentrate on crawlability, and make sure your structure makes sense, and that you aren't linking any 404's or worse. Given the size of your site, that wouldn't be a bad thing anyway.
From your description of your pages, I'm not sure there is any "importance hierarchy", so my suggestion may not help, but you could make use of Google's API to submit pages for crawling. Unfortunately, you can only submit in batches of 100 and you are limited to 200 a day. You could, of course, prioritise or cherry pick some important pages and "hub" pages, if such things exist within your site, and then start working through those.
Following the recent Google blunder where they deindexes huge swathes of the web and, in the short term, the only way to get them back in the index was to resubmit them, someone has provided a tool to interact with the API, which you can find here: https://github.com/steve-journey-further/google-indexing-api-bulk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is robots.txt file issue?
I hope you are well. Mostly moz send me a notification that your website can,t be crawled and it says me o check robots.txt file. Now the Question is how can solve this problem and what should I write in robots.txt file? Here is my website. https://www.myqurantutor.com/ need your help brohers.... and Thanks in advance
On-Page Optimization | | matee.usman0 -
Increasing in 404 errors that doesnt exist
Hi First of all I should say that I have error in the old webmaster not the new one. I have two WordPress blog in root and subfolder. Today I checked my webmaster and recognize that I had 100 errors (404) that found in few days ago. My root WordPress is OK but subfolder WordPress has error. Let me show you by example. http://example.com/subfolder/article15245 I had error for this page: http://example.com/article15245 looks like this subfolder deleted I checked my links, but all of them were OK and linked to the right URL. unfortunately this errors dont have "linked from" section
On-Page Optimization | | meysamzare7112365410 -
Drop in Traffic
We recently went through a server upgrade, including SSL, Encryption, Browser Caching and Image Compression. Most articles I've read, state there will be a slight decrease in traffic shortly after launching before an increase from the normal volume. However, we launched over 3 weeks ago and our traffic has dropped consistently since. Question1: What would be an average length of time after implementing this type of update to rebound from the drop? Question 2: I follow Moz to the letter and have scores of 97 and 99 ,from the optimize a page section, for most of our product pages targeted keywords. What recommendations do you, my friends in the Moz Community, suggest we do next? TY in advance, you input is very valuable to me. KJr 7NSVm
On-Page Optimization | | KevnJr1 -
We´re in trouble with our on site internal link optimization - please help
Dear Moz community, We have made a great mistake. Looking at keyword search volumes somehow Moz showed volumes for two keywords which only differentiate by an '**s (plural) as same. **Now we optimized our internal links (all links have the keyword) for the singular word. Now looking at other search volume estimations from competitors we see, that the plural has 5 times bigger volume. Our issue: If we change some of the category links now to another keyword, we will loose our ranking with the singular word. Correct? If we do not change any of our internal links, we will never rank in top 6 with the keyword. (currently singular is 6 and plural is 15) What would you reccomend?
On-Page Optimization | | advertisingcloud0 -
Homepage SEO: Does Text Content Help Traffic?
Hi Mozzers! My employers homepage (www.swarovski.com) is - amongst other problems we're about to fix - very thin (not to say empty!) in text content. If we were to put relevant text on the next version of the page, would that be beneficial in terms of traffic to that page? Thanks and cheers, Chris
On-Page Optimization | | Diderino0 -
Hi can anyone help me with navigating through my business admin page? having a little trouble. Many Thanks
It looks like this... The person that created it has put the term "Hand painted ceramics from spain - inspired ceramica" on every page title and i cant find where it is to delete it. If anyone can help that will be a real help. Many Thanks, Robbie oOU4YDY
On-Page Optimization | | allenr120 -
404 Error to homepage
Is there any risk of forwarding 404 links directly to homepage. I already have 404 page but now google showing you have lots of 404 links and some of them i can't control to fix this issue. Is there any problem if i do so for SEO ?
On-Page Optimization | | chandubaba0 -
Help Please! - Anchor Text in the Menu
Hi everyone, I am a SEOMOZ newbie and I have been learning about SEO for a while now whilst working on my site - lockcity.co.uk - I already understood the importance of anchor text but was amazed to learn how google only count the anchor text used in the first link (http://www.seomoz.org/learn-seo/anchor-text). My questions are - does this rule still apply? and if so do the menus really count as the first link? If I went on this approach, this would make my menus too long for e.g. on my 'Auto Locksmith' page, my targetted keywords are 'Auto Locksmith' but also the town keywords need to be included. I really thought I had this covered on the home page by making sure the anchor text and alt text were keyword relevant to the link, but looks like Ive been missing out on an opportunity. Our business is slightly complicated in that the 25 mile radius we cover includes 4 different regions - therefore I feel like I always have to get these keywords in as well to make sure we get traffic from our area. Thanks for any advice you can give!
On-Page Optimization | | LockCity0