Accidental No Index
-
Hi everyone,
We control several client sites at my company. The developers accidentally had a no index robot implemented in the site code when we did the HTTPS upgrade without knowing it (yes it's true). Ten days later we noticed traffic was falling. After a couple days we found the no index tags and removed them and resumbitted the sitemaps. The sites started ranking for their own keywords again within a day or two. The organic traffic is still down considerably and other keywords they are not ranking for in the same spot as they were before or at all.
If I look in Google Search console, it says we submitted for example 4,000 URLs and only 160 have been indexed. I feel like maybe Google is taking a long time to re-index to remainder of the sites?? Has anyone has this issue?? We're starting to get very concerned so any input would be appreciate. I read an article on here from 2011 about a company that did the same and they were ranking for their keywords within a week. It's been 8 days since our fix.
-
- Make sure the redirects from http --> https are 301 redirects
- Make sure the canonical URLs have been updated to https
- Make sure your sitemap URLs have been updated to https
-
Thank you for your reply and sorry for the delay in mine. We have the https versions of our site in GA and GSC. The https versions of the sites were not fully indexed. We noticed when traffic was falling significantly on all the sites.
Unfortunately according to GSC, Google has still only crawled a small fraction of all client sites and rankings/organic traffic are slightly improving but still not back to normal. We submitted the site maps and have re-submitted some. We can try to again?
Any other input would be appreciated.
-
Great suggestions by Oleg.
Yoast wrote an article last fall that talks about some of the reasons a site might be slow getting indexed and how to get it indexed faster. They have some worthwhile suggestions.
Did you submit the new (https) version of your website to both google search console and google analytics? Is that what you're looking at? Was the https version of the site fully indexed before you noticed the noindex tags?
Have you confirmed every page on the site was converted to https and there aren't any remaining pages (images or pdfs, for example) that are still http and therefore not showing up in your reports?
-
- Crawl the entire site and make sure no noindex tags remain
- Keep resubmitting the sitemap(s)
- If you have an HTML sitemap, you can do "Fetch as Google" then "Index this page and internal links" which will help recrawl the pages
Don't think there is much else you can do but send Google signals to recrawl the site and see that the noindex tag was removed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Historic issue with incomplete indexing
Hi there We run quite a big site in the UK in the commercial real-estate space. Historically we have always had a challenge getting our "primary" landing pages indexed, which are location based property result pages. e.g. https://realla.co/to-rent/commercial-property/oxford For example, for the "towns" category we have 8,549 submitted in our xml sitemap, with only 3,171 indexed. This is a general issue across all our sitemaps. 120k submitted, 80k indexed. Our pages are linked through breadcrumbs, and nearby links. In the new search console these pages are reported as "crawled - currently not indexed" These all sit under the folder: site:https://realla.co/to-rent/commercial-property/* site:https://realla.co/to-rent/office/* We have done extensive work to optimise performance, including AMP pages. Each location page has many details pages for individual properties e.g. https://realla.co/to-rent/details/0ffbbd0a1a1147edb8847c5ce6179509 One action we have remaining is to nest the details under the locations pages, which may help. These details pages are indexed fully. Any feedback much appreciated
Technical SEO | | ianparryuk0 -
Index problems, Part 2
Hi Guy's A few weeks ago i posted a question:
Technical SEO | | Happy-SEO
https://moz.com/community/q/index-problems After some good advice, we changed a few things: www.domain.com <<< NL version www.domain.com/fr/ <<<< French version (domain.com/nl/ 301 redirect to domain.com). So the SERPS for keyword ‘shutters’ went from #32 to #8...... for 2 day's.... and gone.... and not comming back anymore.... Did we missed something? Help is much appreciated, thanks 🙂3 -
Homepage indexed and cached as the wrong domain
I'm a bit baffled by this one and would love if someone in the community could help provide some clarity! In general, my website (PSG1.com) is indexed and cached correctly. The exception is that the homepage is actually cached as plasticsurgerygroupnewjersey.com, another domain we own. Header checkers all confirm that plasticsurgerygroupnewjersey.com redirects to PSG1.com, not the other way around No canonical is set for that domain. At one time, I used the Moz toolbar to view attributes and it registered PSG1.com as having a response code of both 200 and 301 to plasticsurgerygroupnewjersey.com. However, I cannot replicate this. Any idea why the homepage of PSG1.com is not indexed/cached correctly? I appreciate your wisdom!
Technical SEO | | BTeubner0 -
How to Stop Google from Indexing Old Pages
We moved from a .php site to a java site on April 10th. It's almost 2 months later and Google continues to crawl old pages that no longer exist (225,430 Not Found Errors to be exact). These pages no longer exist on the site and there are no internal or external links pointing to these pages. Google has crawled the site since the go live, but continues to try and crawl these pages. What are my next steps?
Technical SEO | | rhoadesjohn0 -
301 Redirect with index.asp
I am very new to all of this so forgive the newbie questions I will get better. Ok so after starting a campaign I see that I have many issues including where some pages are being deemed as duplicate content. 1. The report says the http://lucid8.com has duplicate content on 2 other pages 2. When I look at them it shows that http://lucid8.com/index.asp and http://www.lucid8.com are duplicates. 3. Really these are the exactly the same page because the default page that is opened for www.lucid8.com http://www.lucid8.com etc always opens the index.asp page. 4. Now I read that I should do permanent redirects and how to do this via IIS and I tried to do a redirect from index.asp to www.lucid8.com but that does not work because www.lucid8.com is pointing to index.asp and so we end up in a circle. So the question is how do I get rid of these duplicate page references without causing problems. Thanks
Technical SEO | | TroyW0 -
Google indexing tags help
Hey everyone, So yesterday someone pointed out to me that Google is indexing tags and that will likely hurt search engine results. I just did a "site:thetechblock.com" and I notice that tags are still being pulled. http://d.pr/i/WmE6 Today, I went into my Yoast settings and checked "noindex,follow" tags in the Taxomomies settings. I just want to make sure what I'm doing is right. http://d.pr/i/zmbd Thanks guys
Technical SEO | | ttb0 -
Getting querystring indexed?
Hi everybody! I work with tags a lot on my photo blog but I haven't gotten Google to index one tag so far. Any tips on how to do this? Thanks / Niklas
Technical SEO | | KAN-Malmo0 -
Development site accidentally got indexed and now appears in SERPs. How to fix?
I work at a design firm, and we just redesigned a website for a client. When it came time for the coding, we initially built a development site to work out all the kinks before going live. Then we relaunched the actual site about a week ago. Here's the problem: Somehow, the developer who coded the site for us (a freelancer) allowed the development site to be indexed by Google. Now, when you enter the client's name into Google, the development site appears higher in the results pages than the real site! In fact, the real site isn't even in the top 50 search results. The client is understandably angry about this for multiple reasons. We quickly added a robots.txt file to the development site and a 301 redirect to the real site. However, that did seemed to have no effect on the problem. Any ideas on how to fix this mess? Thank you in advance!
Technical SEO | | matt-145670