Website dropped out from Google index
-
Howdy, fellow mozzers.
I got approached by my friend - their website is https://www.hauteheadquarters.com
She is saying that they dropped from google index over night - and, as you can see if you google their name, website url or even site: , most of the pages are not indexed. Home page is nowhere to be found - that's for sure.
I know that they were indexed before. Google webmaster tools don't have any manual actions (at least yet). No sudden changes in content or backlink profile. robots.txt has some weird rule - disallow everything for EtaoSpider. I don't know if google would listen to that - robots checker in GWT says it's all good.
Any ideas why that happen? Any ideas what I should check?
P.S. Just noticed in GWT there was a huge drop in indexed pages within first week of August. Still no idea why though.
P.P.S. Just noticed that there is noindex x-robots-tag in headers... Anyone knows where this can be set?
-
"P.P.S. Just noticed that there is noindex x-robots-tag in headers"
That will do it. You are telling Google to take all of your pages out of Google. You set that at the web server level and so you will need to get into your apache or nginx setup
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
Get on this ASAP!
-
Hi Dmitri,
I also see the homepage in Google, but very few pages indexed beyond that, so there does appear to be a serious problem. I don't see anything immediately regarding problems with robots.txt or no index tags. Screaming Frog was able to crawl this site without any problems.
One thing I did see in the few pages that are indexed is the presence of a lot of internal search results pages being indexed.
For example:
https://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descThese two pages are exactly the same products, just in different order. This page also exists: https://www.hauteheadquarters.com/shop/rings/2 - the same products again. For all practical purposes all three of these pages are exactly the same content. Unfortunately, they price sort pages are not blocked from being crawled and indexed AND they are using self-referencing canonical tags.Based on pages like these and other duplicate/thin content issues across the site, I wouldn't rule out a Panda Penalty. It is highly likely that this site may have been penalized. Just because there is no manual action doesn't mean a penalty isn't in play.Recommendations:1. Audit sitewide content and determine which pages should be in Google2. Implement directives in the robots.txt file to prevent the URLs containing query parameters that don't provide unique content from being crawled.3. Implement canonical tags referencing the original URL without query parameters. Examplehttps://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descShould both be canonicalized to https://www.hauteheadquarters.com/shop/rings/24. Rebuild the XML sitemap and include only important URLs5. Resubmit the XML sitemap in GSC Wait a anywhere from a couple of days to a couple of weeks after resubmitting the sitemap, then evaluate if this has remedied the problem.Don't file a reconsideration request. This won't do any good because if it is a penalty, it was done via the algorithm and not manually.Hope that helps a little and good luck!Sincerely,Dana
-
Me too!
-
Absolutely, I'm glad you got things squared!
-
Thanks for response!
Well, basically, as I mentioned, the problem was due to http-header robots tag. So, after removing it, and requesting "fetch as google", it's all up and running now. The crawl time proves that as well.
Thanks for giving me idea for looking into cache times in the future though!
-
I see the homepage in my results - https://www.google.com/#q=site%3Ahttps%3A%2F%2Fwww.hauteheadquarters.com
Homepage was also cached today: http://webcache.googleusercontent.com/search?q=cache:https://www.hauteheadquarters.com&bav=on.2,or.r_cp.&biw=1920&bih=955&dpr=1&ion=1&ech=1&psi=EYbEV6CLN8aweJDvktgD.1472497162096.3&ei=EYbEV6CLN8aweJDvktgD&emsg=NCSR&noj=1
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Indexing Request - Typical Time to Complete?
In Google Search Console, when you request the (re) indexing of a fetched page, what's the average amount of time it takes to re-index and does it vary that much from site to site or are manual re-index request put in a queue and served on a first come - first serve basis despite the site characteristics like domain/page authority?
Intermediate & Advanced SEO | | SEO18050 -
Can you index a Google doc?
We have updated and added completely new content to our state pages. Our old state content is sitting in a our Google drive. Can I make these public to get them indexed and provide a link back to our state pages? In theory it sounds like a great link building strategy... TIA!
Intermediate & Advanced SEO | | LindsayE1 -
Should I worry about rendering problems of my pages in google search console fetch as google?
Some elements are not properly shown when I preview our pages in search console (fetch as google), e.g.
Intermediate & Advanced SEO | | lcourse
google maps, css tables etc. and some parts are not showing up since we load them asynchroneously for best page speed. Is this something should pay attention to and try to fix?0 -
Http and https protocols being indexed for e-commerce website
Hi team, Our new e-commerce website has launched and I've noticed both http and https protocols are being indexed. www.mountainjade.co.nz Our old website was http with only the necessary pages running https (cart, checkout etc). No https pages were indexed and you couldn't access a https page if you manually typed it into the browser. We outrank our competition by a mile, so I'm treading carefully here and don't want to undo the progress we made on the old site, so I have a few questions: 1. How exactly do we remove one protocol from the index? We are running on Drupal. We tried a hard redirect from https to http and excluded the relevant pages (cart, login etc from the redirect), but found that you could still access https pages if you we're in the cart (https) and then pressed back on the browser button for example. At that point you could browse the entire site on https. 2. Is the safer option to emulate what we had in place on the old website e.g http with only the necessary pages being https, rather than making the switch to sitewide https? I've been struggling with this one, so any help would be much appreciated. Jake S
Intermediate & Advanced SEO | | Jacobsheehan0 -
Website Ranks and gets de indexed ??
Hi My website is almost 3-4 months old . Whats strange is that as soon as it get Crawled it ranks for few terms for 1-2 days and all of a sudden gets de Indexed for these same terms or Rank drops like drops from page 5 to page 10 . Nothing shows up in Webmater tools under Manual Action . Assuming its a Algorithmic penalty, How to deal with this kind of stuff. Should I stop working on this site all together ? Or assuming its a New website, google does not want it to rank for medium or high volume keywords ? What keywords I am after have 300 -2k searches per month .
Intermediate & Advanced SEO | | aus00070 -
Google Places Listing Active In Two Seperate Google Places Accounts?
Hi is there any issues with having a google places listing in two seperate google places accounts. For example we have a client who cannot access their old google places account (ex-employee had their login details which they can't get) and want us to take control over the listing. If we click the "is this your listing" manage this page button - and claim the listing, will this transfer the listing to our control? Or will it create a duplicate? Are there any problems having the listing in different separate accounts. Is it a situation in which the last person who manages the listing takes control? And the listing automatically deactivates from the old account? Do all the images remain aswell? Thanks,
Intermediate & Advanced SEO | | MBASydney
Tom0 -
We're indexed in Google News, any tips or suggestions for getting traffic from news?
We have a news sitemap, and follow all best practices as outlined by Google for news. We are covering breaking stories at the same time as other publications, but have only made it to the front page of Google News once in the last few weeks. Does anyone have any tips, recommended reading, etc for how to get to the front page of Google News? Thanks!
Intermediate & Advanced SEO | | nicole.healthline0 -
How to get content to index faster in Google.....pubsubhubbub?
I'm curious to know what tools others are using to get their content to index faster (other than html sitmap and pingomatic, twitter, etc) Would installing the wordpress pubsubhubbub plugin help even though it uses pingomatic? http://wordpress.org/extend/plugins/pubsubhubbub/
Intermediate & Advanced SEO | | webestate0