Help recover lost traffic (70%) from robots.txt error.
-
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change.
Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated.
On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL?
Thanks,
-
Firstly, I would definitely take the opportunity to switch to SSL. A migration to SSL shouldn't be something to worry about if you set up your redirects properly, but given that most of your pages aren't indexed at all, it is even less risky.
You will eventually get the traffic back, as far as how long, it's very difficult to say.
I would concentrate on crawlability, and make sure your structure makes sense, and that you aren't linking any 404's or worse. Given the size of your site, that wouldn't be a bad thing anyway.
From your description of your pages, I'm not sure there is any "importance hierarchy", so my suggestion may not help, but you could make use of Google's API to submit pages for crawling. Unfortunately, you can only submit in batches of 100 and you are limited to 200 a day. You could, of course, prioritise or cherry pick some important pages and "hub" pages, if such things exist within your site, and then start working through those.
Following the recent Google blunder where they deindexes huge swathes of the web and, in the short term, the only way to get them back in the index was to resubmit them, someone has provided a tool to interact with the API, which you can find here: https://github.com/steve-journey-further/google-indexing-api-bulk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will German meta data help an English USA-based site get found in Germany?
I think I already know the answer to this, but I'd love a second opinion. If you add German meta data to some web pages will it help those pages turn up in search results in Germany?I'm assuming not. It's an English, USA based site, doing well here for English searches - and I don't want to mess that up. A German site is being planned, but this client is hoping the English site could be found in the meantime. ~Caro
On-Page Optimization | | Caro-O0 -
Strange error on my website
Hi Wondering if anyone could help out here please. I tried to crawl my website using screaming frog and got a 500 error warning (odd considering it worked yesterday). I tried google speed test and got the following message - PageSpeed Insights received a 500 response from the server. To analyze pages behind firewalls or that require authentication. However, it also crawled the site and gave me a score of 70/100. It showed the site thumbnail, as normal. This was updated this morning and it showed the latest one so must have crawled. I can access the site, as can my 'people', the site is getting hits from real users in Google Analytics so must be working. Am very puzzled how the site can be working and not be working at the same time! If possible i would rather not share the url. The site is a test of concept and is very messy looking. it is more a test to see what happens if i do x... Many thanks Carl
On-Page Optimization | | daedriccarl0 -
When I changes Template, why traffic goes down?
I've noticed that when I change my blog's template the traffic goes down dramatically, about of 40% decrease. I know that new themes can have some problems but I have tried this with 2 different themes. First try was with genesis framework(Paid one) and just in one day traffic went down and when I reverted the old theme, the traffic became normal. Should I wait for 1 week to see what happens? What could be the potential reason of this?
On-Page Optimization | | hammadrafique0 -
Help an SEO-DUMMY : ) Established hyphenated domain...redirect?!...new domain?!
Hello, everybody. I am definitely not an SEO specialist. My family owns a transportation business (since 2010) and i am the one responsible for the website (until we find a good SEO company). My question: Several years ago i did not know much about SEO and have chosen a domain name www.airporttransportation-limo.com (it is not the actual domain...just an example...i'm not sure if i can post the real website here) and another domain that is just the name of our company (it also has hyphen in it). Both websites are still doing good and we receive quite a bit of traffic, but i read more an more about how hyphenated domains and domains with more then two worlds can be bad for your SEO/business/traffic. I feel like the websites are stuck and not moving up any more..could that be because of the hyphens? I registered another domain that is the name of our company (which is well known by now) without any hyphens. Now i have no idea what to do. Should i redirect both old domains (old websites are different and do not have duplicate content) to the new one, or should i just redirect the old domain (just the name of our company with hyphen) to a new one (without hyphen) and leave the www.airportransportation-limo.com as is... Or maybe i should register another domain without any hyphens (two words only) and redirect the www.airporttransportation-limo.com to it... I am very nervous to make any changes and loose all the traffic. My family will kill me. Please help! I'm lost!
On-Page Optimization | | KL20140 -
Page Analysis - Helping Product Pages Outrank Search Results Pages
Hi! We have a lot of our search results pages that have been indexed and outrank our product pages and in some instance the actual product pages barely show up at all. Here is an example query that includes our brand name: http://goo.gl/cgB6W So, we have loads of actual product pages, video pages, etc that should be showing up here, but are not and this is just one example. Unfortunately, there are a LOT of these Search Results pages out there and utlimately we would love to de-index them altogether, but it's going to have to be carefully done. So, was wondering if anyone would want to check out one of our product pages and give any feedback as to what we could change to possibly improve rank or to make them more search friendly or hopefully to help them rise above these indexed search results pages? Here is an example product page: http://goo.gl/2R4IT Thanks!! Craig
On-Page Optimization | | TheCraig0 -
How to Recover From Unstable Site Structure History
I have a site that has suffered several phases of restructuring. Apparently its owners were unsure as to which direction to take when it came to structuring their content and URL schema and subjected the site to several rounds of poorly thought through implementations (i.e. example.com/content/page-title, example.com/page-title, example.com/"silo"/page-title, etc.), all within a 8 month period. I posted the originating question here on this Q&A Forum. I want to thank EGOL and Cody for taking a stab at it. What would be a good strategy to help a site like the one I describe above begin ranking again?
On-Page Optimization | | UplinkSpyder0 -
404 errors on page urls that don't even exist
The Seomoz crawler found 404error of pages dont even exist. Ho can that be possible?? Pages like: URL: http://www.yoxo.it/catalog/seo_sitemap/category/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/falli/
On-Page Optimization | | yoxo0 -
Unable to see internal link numbers on Opensiteexplorer - Need help
I'm Anuj, a regular user of SEOMOZ. I need some SEO guidance from SEO experts. I'm trying to optimize a webstore for few keywords. I am facing some issues on SEO I was using https all over the webstore and was advised by the community members to not have https through out the site (Due to various reasons). The internal links were not showing up in opensiteexplorer & Google Webmaster Tools too when the site was with https (They were just showing 1 or 2). After changing the pages from https to http, I'm now able to see all the internal links of my website on GWT. Unfortunately, the internal link count on opensiteexplorer shows a very small fraction when compared to the # of internal links shown on GWT. The link update from Opensiteexplorer was on 27th FEB 2013. I had done the https to http (for all pages) somewhere between 17-24th of JANUARY 2013. I wanted to know if I have missed something as I am unable to see those numbers on Opensiteexplorer or will it take time for opensiteexplorer to show the internal link numbers ?
On-Page Optimization | | Pepperjet0