Website dropped out from Google index
-
Howdy, fellow mozzers.
I got approached by my friend - their website is https://www.hauteheadquarters.com
She is saying that they dropped from google index over night - and, as you can see if you google their name, website url or even site: , most of the pages are not indexed. Home page is nowhere to be found - that's for sure.
I know that they were indexed before. Google webmaster tools don't have any manual actions (at least yet). No sudden changes in content or backlink profile. robots.txt has some weird rule - disallow everything for EtaoSpider. I don't know if google would listen to that - robots checker in GWT says it's all good.
Any ideas why that happen? Any ideas what I should check?
P.S. Just noticed in GWT there was a huge drop in indexed pages within first week of August. Still no idea why though.
P.P.S. Just noticed that there is noindex x-robots-tag in headers... Anyone knows where this can be set?
-
"P.P.S. Just noticed that there is noindex x-robots-tag in headers"
That will do it. You are telling Google to take all of your pages out of Google. You set that at the web server level and so you will need to get into your apache or nginx setup
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
Get on this ASAP!
-
Hi Dmitri,
I also see the homepage in Google, but very few pages indexed beyond that, so there does appear to be a serious problem. I don't see anything immediately regarding problems with robots.txt or no index tags. Screaming Frog was able to crawl this site without any problems.
One thing I did see in the few pages that are indexed is the presence of a lot of internal search results pages being indexed.
For example:
https://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descThese two pages are exactly the same products, just in different order. This page also exists: https://www.hauteheadquarters.com/shop/rings/2 - the same products again. For all practical purposes all three of these pages are exactly the same content. Unfortunately, they price sort pages are not blocked from being crawled and indexed AND they are using self-referencing canonical tags.Based on pages like these and other duplicate/thin content issues across the site, I wouldn't rule out a Panda Penalty. It is highly likely that this site may have been penalized. Just because there is no manual action doesn't mean a penalty isn't in play.Recommendations:1. Audit sitewide content and determine which pages should be in Google2. Implement directives in the robots.txt file to prevent the URLs containing query parameters that don't provide unique content from being crawled.3. Implement canonical tags referencing the original URL without query parameters. Examplehttps://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descShould both be canonicalized to https://www.hauteheadquarters.com/shop/rings/24. Rebuild the XML sitemap and include only important URLs5. Resubmit the XML sitemap in GSC Wait a anywhere from a couple of days to a couple of weeks after resubmitting the sitemap, then evaluate if this has remedied the problem.Don't file a reconsideration request. This won't do any good because if it is a penalty, it was done via the algorithm and not manually.Hope that helps a little and good luck!Sincerely,Dana
-
Me too!
-
Absolutely, I'm glad you got things squared!
-
Thanks for response!
Well, basically, as I mentioned, the problem was due to http-header robots tag. So, after removing it, and requesting "fetch as google", it's all up and running now. The crawl time proves that as well.
Thanks for giving me idea for looking into cache times in the future though!
-
I see the homepage in my results - https://www.google.com/#q=site%3Ahttps%3A%2F%2Fwww.hauteheadquarters.com
Homepage was also cached today: http://webcache.googleusercontent.com/search?q=cache:https://www.hauteheadquarters.com&bav=on.2,or.r_cp.&biw=1920&bih=955&dpr=1&ion=1&ech=1&psi=EYbEV6CLN8aweJDvktgD.1472497162096.3&ei=EYbEV6CLN8aweJDvktgD&emsg=NCSR&noj=1
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When searching for related:katom.com on google, why isn't our website coming up?
A lot of our competitors come up but we aren't coming up. What do we need to do so that google considers us related? Our website is culinarydepotinc.com And I believe not being related to those big competitors affects our SEO, is that correct?
Intermediate & Advanced SEO | | Sammyh2 -
International website. Di I need a new website
i am looking to expand from the UK and open a location in the US. i curretly have a .co.uk domain. what would you recommend I do with th website, create a new one wth a .com domain?
Intermediate & Advanced SEO | | Caffeine_Marketing0 -
Client rebranded with a new website but can't migrate now defunct franchise website to new website.
Hi everyone, My client is a chain of franchised restaurants with a local domain website named after the franchise. The franchise exited the market while the client stayed and built its own brand with a separate website. The franchise website (which is extremely popular) will be shut down soon but the client will not be able to redirect the franchise website to the new website for legal reasons. What can I do to ensure that we start ranking immediately for the franchise keyphrase as soon as the franchise website is shutdown. We currently have the new website and access to the old website (which we can't redirect) Thanks, T
Intermediate & Advanced SEO | | Tarek_Lel0 -
What to do when you buy a Website without it's content which has a few thousand pages indexed?
I am currently considering buying a Website because I would like to use the domain name to build my project on. Currently that domain is in use and that site has a few thousand pages indexed and around 30 Root domains linking to it (mostly to the home page). The topic of the site is not related to what I am planing to use it for. If there is no other way, I can live with losing the link juice that the site is getting at the moment, however, I want to prevent Google from thinking that I am trying to use the power for another, non related topic and therefore run the risk of getting penalized. Are there any Google guidelines or best practices for such a case?
Intermediate & Advanced SEO | | MikeAir0 -
Website Survived Last Year Penguin & Panda Updates but dropped Jan 2013?
Hi, I have a website which was gaining good rankings through 2012 and even looked in pretty good shape after all of the Penguin and Panda updates in 2012 but in Jan 2013 the sites rankings have been very poor as it dropped for all terms. It is still in the index and webmaster tools shows the sites keywords but is showing places of 100 or so rather than 10 - 16 in the seeps. The website is www.mywebsitehosting.org Any ideas why this has dropped in January and what can I do to fix it? Thanks. P.s I have made a change to the homepage content but inner pages for the website also dropped to which have the same content as they did a few months ago.
Intermediate & Advanced SEO | | firebubble0 -
What is better for google: keep old not visited content deeply in the website, or to remove it?
We have quite a lot of old content which is not visited anymore. Should we remove it and have a lot of 410 errors which will be reported in GWT? Or should we keep it and forget about it?
Intermediate & Advanced SEO | | bele0 -
Getting Google in index but display "parent" pages..
Greetings esteemed SEO experts - I'm hunting for advice: We operate an accommodation listings website. We monetize by listing position in search results, i.e. you pay more to get higher placing in the page. Because of this, while we want individual detailed listing pages to be indexed to get the value of the content, we don't really want them appearing in Google search results. We ideally want the "content value" to be attributed to the parent page - and google to display this as the link in the search results instead of the individual listing. Any ideas on how to achieve this?
Intermediate & Advanced SEO | | AABAB0 -
Does Google index url with hashtags?
We are setting up some Jquery tabs in a page that will produce the same url with hashtags. For example: index.php#aboutus, index.php#ourguarantee, etc. We don't want that content to be crawled as we'd like to prevent duplicate content. Does Google normally crawl such urls or does it just ignore them? Thanks in advance.
Intermediate & Advanced SEO | | seoppc20120