Help recover lost traffic (70%) from robots.txt error.
-
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change.
Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated.
On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL?
Thanks,
-
Firstly, I would definitely take the opportunity to switch to SSL. A migration to SSL shouldn't be something to worry about if you set up your redirects properly, but given that most of your pages aren't indexed at all, it is even less risky.
You will eventually get the traffic back, as far as how long, it's very difficult to say.
I would concentrate on crawlability, and make sure your structure makes sense, and that you aren't linking any 404's or worse. Given the size of your site, that wouldn't be a bad thing anyway.
From your description of your pages, I'm not sure there is any "importance hierarchy", so my suggestion may not help, but you could make use of Google's API to submit pages for crawling. Unfortunately, you can only submit in batches of 100 and you are limited to 200 a day. You could, of course, prioritise or cherry pick some important pages and "hub" pages, if such things exist within your site, and then start working through those.
Following the recent Google blunder where they deindexes huge swathes of the web and, in the short term, the only way to get them back in the index was to resubmit them, someone has provided a tool to interact with the API, which you can find here: https://github.com/steve-journey-further/google-indexing-api-bulk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help optimizing website for speed
Hello, My website is www.likechimp.com and is a University project. I need to optimise the website for speed as the bounce rate is fairly quick - I feel this could be due to how long it takes web site to load? Any tips in increasing internet speed. I am willing to higher someone if they feel they can help! Thanks, L
On-Page Optimization | | xlucax0 -
Help with Temporary Redirects on Pages
Hi Guys, My latest crawl shows this: "44% of site pages served 302 redirects during the last crawl" When I click this to investigate the issue I see: URL:
On-Page Optimization | | jeeyer
https://www.....
https://www.....
https://www.....
https://www.....
https://www..... And under Redirect URL:
http://www....
http://www....
http://www.... I've recently read an article from Yoast that this is a https:// to http:// redirect (ofcourse 😉 ).
but why is this an issue that needs to be solved and how do I solved it? Thanks again for your help and thoughts. Joost0 -
*** Please HELP *** A/B tests and optimisation implications
Hi Mozzers, We've been A-B testing landing pages, and have had some success. The changes we've been making have been quite radical in some instances - for example we tested this page: https://www.turnkeymortgages.co.uk/todays-mortgage-deals/ against this one: https://www.turnkeymortgages.co.uk/mortgage-quote/ (Today's best deals won, but we've decided to keep the quote page as it does work for some channels). The decision was made to try and optimise Today's best deals for 'best mortgage deals today' rather than 'mortgage quote' because it offers so much more than simply a quote. The quote page is optimised for 'mortgage quote', though it doesn't rank particularly well (I'm not overly concerned by this as even though you'd think that when people are looking for a quote that they would fill in the form, they don't - people are strange!) As a result of the change above we changed all links that originally went to the quote page to go to Today's best deals instead. As we go through the process of optimising for best conversion will it be damaging if we don't change the url as well. As I can see lots of iterations and lots of work whenever we make changes to the pages (going through the entire site to change the links). I am worried though that we'll end up with hundreds of landing pages and changing links all over the site - do you think we should keep the URLs the same from now on, unless the content changes as radically as it did in the instance I've highlighted above? Thanks, Amelia
On-Page Optimization | | CommT0 -
Google Crawl Errors from vbseo change
We have vbseo setup on our site and for some reason a setting was changed unexpectedly and was un-noticed where it changed the URL of all the pages and so none of our pages were getting indexed by google any longer due to 401 errors. Most of our SE traffic fell off. We discovered the issue a couple weeks ago and we changed the setting back so that the URLs are the same as they were originally before but in Google webmasters it's still showing crawl errors and our search engine traffic hasn't recovered at all. We have sitemaps being sent daily.
On-Page Optimization | | RudySF0 -
I need some help...
I am completely perplexed here guys. I have accomplished all of the the things that the On- Page Analysis tool says that we need to perform as far as( Keyword laden page titles and webpages) yet the report comes back and gives the webpage a C and says that we still need to correct these issues. Can anyone explain this? The keywords are: " real estate augusta ga" " property management augusta ga" the address is: www.aubenrealty.com Thanks in advance, C
On-Page Optimization | | AubbiefromAubenRealty0 -
Should I index news and blog posts which receive little traffic
Hi all, I have a very large site at the moment with a handful of high authority pages with great content which describe the charity's work. But the main content on the site is blogs and news article which, while being good quality content from a reader's point of view, receive little traffic from search engines as they are so niche and long-tail. They do, however, get ok internal traffic from other pages. Should these pages be indexed still or should I remove them? Is there a rule of thumb regarding minimum clicks/bounce rates that an indexed page should have? What do big news agencies do with really niche articles that may get next to no traffic but have valuable content on for those that do click through? Thanks in advance! Den
On-Page Optimization | | Deniz0 -
Long tail traffic - what is the best way to go back and add focus to repetitive long tail keywords?
Hey everybody, So, our niche doesn't have a million and a half searches per month, which makes a handle full of visitors look mighty enticing to a CMO Our price point is very high too, so to the question, is it worth taking the time to put a whole new content strategy in line for a few new visitors, the answer is yes. Now's the hard part. How on earth do I make 1,000 pages for similar topics? Is making new pages the best way to go about this? (probably so right? It's the only thing that I can see that would certainly increase likelihood of being more relevant, plus if I don't I will be missing out on the benefits of beefing up our site, AND the opportunity to more specifically answer a users query.) With phrases like "keyword" and "aftermarket keyword," the searcher is asking for two totally separate collections of results. I'm always reading about the importance of being there throughout the buyers complete purchasing /research process, which makes me think that considering doing anything other than creating unique pages is simply missing out.. Suggestions? Massive Content Strategy Help? Anybody? Thanks, TA
On-Page Optimization | | TylerAbernethy0 -
Detecting SE traffic on a landing page
Hi all, I am trying to optimize our landing pages in order to getting new visitors to make an action instead of leaving. But in order to so, I need to be able to identify when a user comes from a Google search and when a user does not. Can anyone help me? The site is written in PHP. Best regards, Rasmus
On-Page Optimization | | rasmusbang0
