Website dropped out from Google index
-
Howdy, fellow mozzers.
I got approached by my friend - their website is https://www.hauteheadquarters.com
She is saying that they dropped from google index over night - and, as you can see if you google their name, website url or even site: , most of the pages are not indexed. Home page is nowhere to be found - that's for sure.
I know that they were indexed before. Google webmaster tools don't have any manual actions (at least yet). No sudden changes in content or backlink profile. robots.txt has some weird rule - disallow everything for EtaoSpider. I don't know if google would listen to that - robots checker in GWT says it's all good.
Any ideas why that happen? Any ideas what I should check?
P.S. Just noticed in GWT there was a huge drop in indexed pages within first week of August. Still no idea why though.
P.P.S. Just noticed that there is noindex x-robots-tag in headers... Anyone knows where this can be set?
-
"P.P.S. Just noticed that there is noindex x-robots-tag in headers"
That will do it. You are telling Google to take all of your pages out of Google. You set that at the web server level and so you will need to get into your apache or nginx setup
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
Get on this ASAP!
-
Hi Dmitri,
I also see the homepage in Google, but very few pages indexed beyond that, so there does appear to be a serious problem. I don't see anything immediately regarding problems with robots.txt or no index tags. Screaming Frog was able to crawl this site without any problems.
One thing I did see in the few pages that are indexed is the presence of a lot of internal search results pages being indexed.
For example:
https://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descThese two pages are exactly the same products, just in different order. This page also exists: https://www.hauteheadquarters.com/shop/rings/2 - the same products again. For all practical purposes all three of these pages are exactly the same content. Unfortunately, they price sort pages are not blocked from being crawled and indexed AND they are using self-referencing canonical tags.Based on pages like these and other duplicate/thin content issues across the site, I wouldn't rule out a Panda Penalty. It is highly likely that this site may have been penalized. Just because there is no manual action doesn't mean a penalty isn't in play.Recommendations:1. Audit sitewide content and determine which pages should be in Google2. Implement directives in the robots.txt file to prevent the URLs containing query parameters that don't provide unique content from being crawled.3. Implement canonical tags referencing the original URL without query parameters. Examplehttps://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descShould both be canonicalized to https://www.hauteheadquarters.com/shop/rings/24. Rebuild the XML sitemap and include only important URLs5. Resubmit the XML sitemap in GSC Wait a anywhere from a couple of days to a couple of weeks after resubmitting the sitemap, then evaluate if this has remedied the problem.Don't file a reconsideration request. This won't do any good because if it is a penalty, it was done via the algorithm and not manually.Hope that helps a little and good luck!Sincerely,Dana
-
Me too!
-
Absolutely, I'm glad you got things squared!
-
Thanks for response!
Well, basically, as I mentioned, the problem was due to http-header robots tag. So, after removing it, and requesting "fetch as google", it's all up and running now. The crawl time proves that as well.
Thanks for giving me idea for looking into cache times in the future though!
-
I see the homepage in my results - https://www.google.com/#q=site%3Ahttps%3A%2F%2Fwww.hauteheadquarters.com
Homepage was also cached today: http://webcache.googleusercontent.com/search?q=cache:https://www.hauteheadquarters.com&bav=on.2,or.r_cp.&biw=1920&bih=955&dpr=1&ion=1&ech=1&psi=EYbEV6CLN8aweJDvktgD.1472497162096.3&ei=EYbEV6CLN8aweJDvktgD&emsg=NCSR&noj=1
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Gradual traffic drop of personal finance website in the last three months
Dear All, I have personal finance website https://mymoneysouq.com and the traffic dropped by less than half of what is was before last three months. I am figuring out all the possible issues and doing everything that comes to our mind to improve the quality of our website. I tried the following before posting here:1. Tried contacting website owner which we think spam and add all such domains to our disavow list2. We found little duplicate content on sites like Quora, we made those answers down by reporting to Quora3. Reported to DMCA on 3 articles articles(partial) from our website.4. We are trying improving user experience5. Removed one of our page that shared by many people but our page was not indexed by Google.6. Checked and modified content if any our articles are having more keywords than what SEO experts recommend. 7. We are working on researching more and figuring our what else can might have gone wrong with our traffic.8. Working on improving EAT I attached our traffic drop graph. I believe this drop is not natural it happened because of some issue at our end and we are not able to figure out the exact reasons.Surprisingly another site with not so high quality content started ranking now in the top.I am here to get community members/experts help on this. I could provide you if you need any further details. Thanks a lot for your time. We really appreciate any tips that you can share with us.Q2S1tlK Q2S1tlK
Intermediate & Advanced SEO | | swamyallamraju0 -
Mobile First Timeline - When Will Google Move The Remainder of Websites?
A common question from anxious webmasters who have separate/smaller mobile websites is: _"Will Google move my site over to mobile-first, even though I'm not "ready" yet?" _It's not an easy question to answer. Here's what we know so far: Google is currently already migrating websites over that have a strong correlation between mobile and desktop content Google is taking the migration process very slowly, and for now, is not migrating sites over that are not "ready". Google has not announced any firm timeline for when they plan to migrate the remainder of all websites. In answering this question, I typically mention all of the above to help allay any fears. I then state that it could maybe be anywhere from one year to several years before the process is over - but with a huge disclaimer that this is pure speculation, and that only Google knows for sure. Lastly, I reiterate that Google in the meantime is strongly encouraging webmasters w mobile sites to ensure that they match the desktop version (URLs, schema, video, metadata, etc). So the choice is to either to upgrade to responsive/adaptive, or upgrade the mobile site. This is where the future is going. STILL - any additional feedback / thoughts / ideas / tips on this are welcome, because I continue to struggle with answering this question for clients. Thanks!
Intermediate & Advanced SEO | | mirabile1 -
How to improve or fix my website ? for an unknown reason bad ranks on Google
Hello, I know that I need to have good quality backlinks, and I am getting backlinks sometime more sometime less. but that is not a problem at this moment i belive. So I am asking someone who can help me and check my website MajorDroid and tells me how can I improve or fix my problem if there is a problem. Because I am very sad when I see this for example: if you search for keyword: oukitel k3 review my website is at page 3, but at same time on page 1 there are only three relevant results, and all other results on page 1 are only partial like OUKITEL K3 Specs Confirmed.... and they are not relevant for users... because they are looking for review. I have the same problem with other keywords also where my page is on second page or third page of Google... and again some other websites with only partial keyword are on first page.... and google claims to offer relevant info for users... now i do not know how google looks my website, or where is the problem ? why quite often my website ranks on second page or third page although competition is not great, and google shows only several relevant results on first page... for example this is review for oukitel k3 I hope you can help me Thank you
Intermediate & Advanced SEO | | zgb0 -
Redirecting Old Websites to New Websites
Hi Everyone, We are about to take down a number of websites in favour of a new singular B2B hub and would be looking to redirect all of these sites to the new home. For SEO purposes, what would be the best way to do this? Due to the difference in setups and scale of the site, it would be difficult to correctly match up each page to page between the sites for individual 301 redirects. Could someone advise on the best plan of action? Thanks.
Intermediate & Advanced SEO | | chbiz0 -
Why isn't Google indexing this site?
Hello, Moz Community My client's site hasn't been indexed by Google, although it was launched a couple of months ago. I've ran down the check points in this article https://moz.com/ugc/8-reasons-why-your-site-might-not-get-indexed without finding a reason why. Any sharp SEO-eyes out there who can spot this quickly? The url is: http://www.oldermann.no/ Thank you
Intermediate & Advanced SEO | | Inevo
INEVO, digital agency0 -
Meta NOINDEX... how long before Google drops dupe pages?
Hi, I have a lot of near dupe content caused by URL params - so I have applied: How long will it take for this to take effect? It's been over a week now, I have done some removal with GWT removal tool, but still no major indexed pages dropped. Any ideas? Thanks, Ben
Intermediate & Advanced SEO | | bjs20100 -
Huge Google index on E-commerce site
Hi Guys, I got a question which i can't understand. I'm working on a e-commerce site which recently got a CMS update including URL updates.
Intermediate & Advanced SEO | | ssiebn7
We did a lot of 301's on the old url's (around 3000 /4000 i guess) and submitted a new sitemap (around 12.000 urls, of which 10.500 are indexed). The strange thing is.. When i check the indexing status in webmaster tools Google tells me there are over 98.000 url's indexed.
Doing the site:domainx.com Google tells me there are 111.000 url's indexed. Another strange thing which another forum member describes here : Cache date has been reverted And next to that old url's (which have a 301 for about a month now) keep showing up in the index. Does anyone know what i could do to solve the problem?0 -
Getting a site to rank in both google.com and google.co.uk
I have a client who runs a yacht delivery company. He gets business from the US and the UK but due to the nature of his business, he isn't really based anywhere except in the middle of the ocean somewhere! His site is hosted in the US, and it's a .com. I haven't set any geographical targeting in webmaster tools either. We're starting to get some rankings in google US, but very little in google UK. It's a small site anyway, and he'd prefer not to have too much content on the site saying he's UK based as he's not really based anywhere. Any ideas on how best to approach this?
Intermediate & Advanced SEO | | PerchDigital0