Organic listings disappeared I don't know why!
-
Brief history:
I am MD of a medium sized health organisation in the UK. We have one of the leading websites in the world for our industry. We were hit by a Google algorithm update last year (Penguin or Panda, I can't remember, but that's not relevant here I don't think) and our daily visits went down from around 10,000 to around 5,000 in two separate hits over a couple of months. Then there was a steady decrease to about 3,000-4,000 visits a day until we totally updated the design of the site and did some good work on the content. We have always been white-hat and the site has around 3,000 pages with unique content added daily.
So things have really been on the up for the past couple of months. We have been receiving around 6,000 visits a day in recent weeks (a slow incline over the past few months), until Sunday. Sunday morning around 10am all of our organic listings pretty much disappear, including for our brand name. Monday morning a few come back, including our brand name and our main, most competitive keyword, which we were showing up on the third page for and we returned to this page. Then Tuesday morning another few of our most competitive keywords show up, back where they were before. This includes images which had disappeared from Google images.
Our PPC and business listings were not really affected at all.
My developer submitted a site map through webmaster tools on Monday morning and I'm not sure if this is the reason pages started to show up again. In our Webmaster tools the indexed pages are about a quarter of all of the ones on the site - all pages were indexed before. I just don't know what has happened! It doesn't make any sense as 1. Google don't seem to have rolled out any algorithm updates on that day 2. we do not have any messages in Webmaster Tools 3. a number of our main keywords have re-appeared - why would that happen if we had been hit by a Google update?!
Our organic hits, which previously made up about 80% of all our hits, have gone down by 80% and this is drastically affecting business. If this continues it is likely we will have to downsize the business and I'm not sure what to do.
When I saw that the 'indexed pages' in Webmaster tools started to increase (they were around 600 on Monday, around 900 yesterday and then this morning, around 1,300), I thought that we were on our way up and maybe this problem would just resolve itself and our listings would re-appear, but now our indexed pages have reduced slightly since this morning, back down to around 1,100 so the increase has stalled.
Can anybody help?! Do you have any idea what could be causing this? Apparently there have been no changes made to robots.txt and my developer says that no changes were made that could have affected our listings.
ANY ADVICE WOULD BE GREATLY APPRECIATED.
-
Interesting situation...and very frustrating for you, I'm sure.
You mentioned this below:
"I checked 'cached snapshot of page' in Google Toolbar for the pages that weren't being indexed, and it showed up as a 404 error. "
This sounds like you had some sort of technical error. But some things still don't add up for me. It kind of sounds like your pages were not resolving for Google. But, the odd thing is that if Google sees a 404 error, they keep trying for days, weeks or even months before they conclude that the pages should be removed from the index.
I don't have an answer for you but the first place I'd look is to make sure that your robots.txt file is not blocking Googlebot in some way. I'd also check server logs and perhaps check with your host to see if there was some significant down time for the site.
If there was a technical glitch, and the problem is now fixed, then your pages should come back into the index without you doing anything.
I'm pretty certain this isn't a penalty issue though.
-
Thank you. I will look into this, although I don't think the pages are set to no follow because there has been a further development. I checked 'cached snapshot of page' in Google Toolbar for the pages that weren't being indexed, and it showed up as a 404 error. These are pages that have always been cached before this problem occurred. I then went to 'submit URL to Google' and submitted a couple of URLs. They instantly showed up in Google's listings in the same spot as they were before and the cached snapshot of page then showed up correctly. I could do that for every page but 1. that would be a HUGE job and 2. would that look spammy or suspicious to Google? 3. is there a way of doing that for multiple pages at a time?! I feel like this problem is very close to being solved but I just don't quite know how to solve it.
-
The only time I've seen this type of thing happen - all of the pages in a site are no longer indexed, yet PPC still works, is when something on the site has been set to no-index / no-follow.
If you had a manual penalty from Google, that would show up in Google Webmaster Tools. Plus, the site would still be indexed, just ranked really, really low. If everything was missing from Google's cache, then the most likely explanation is that it was set accidentally to no-index / no-follow.
This is a very easy thing to mess up, and it's possible that someone might have hit the wrong button by accident, or updated the robots.txt file.
In the past, I had a project manager who messed this up for a client while doing a content update on the site, and it was about a week before anyone noticed. She's no longer here (not due just to that issue). But this is so critical for me and my company that we've put an automated and human testing check in place each day:
For our company, we have an automated script that runs through all of our sites (and client's sites) each day to make sure that the site is set to index / follow, both on the pages and in the robot.txt file. We also check the title tag and make sure that the name servers haven't changed.
I also pay someone on my team to run through a 12 step checklist each and every day to make sure that things like the site search are working, contact forms go through properly, and that pages are set to index / follow.
I hope this helps...
Thanks,
-- Jeff
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I use the on classified listing pages that have expired?
We have went back and forth on this and wanted to get some outside input. I work for an online listing website that has classified ads on it. These ads are generated by companies on our site advertising weekend events around the country. We have about 10,000 companies that use our service to generate their online ads. This means that we have thousands of pages being created each week. The ads have lots of content: pictures, sale descriptions, and company information. After the ads have expired, and the sale is no longer happening, we are currently placing the in the heads of each page. The content is not relative anymore since the ad has ended. The only value the content offers a searcher is the images (there are millions on expired ads) and the descriptions of the items for sale. We currently are the leader in our industry and control most of the top spots on Google for our keywords. We have been worried about cluttering up the search results with pages of ads that are expired. In our Moz account right now we currently have over 28k crawler warnings alerting us to the being in the page heads of the expired ads. Seeing those warnings have made us nervous and second guessing what we are doing. Does anybody have any thoughts on this? Should we continue with placing the in the heads of the expired ads, or should we be allowing search engines to index the old pages. I have seen websites with discontinued products keeping the products around so that individuals can look up past information. This is the closest thing have seen to our situation. Any help or insight would be greatly appreciated! -Matt
Intermediate & Advanced SEO | | mellison0 -
All urls seem to exist (no 404 errors) but they don't.
Hello I am doing a SEO auditing for a website which only has a few pages. I have no cPanel credentials, no FTP no Wordpress admin account, just watching it from the outside. The site works, the Moz crawler didn't report any problem, I can reach every page from the menu. The problem is that - except for the few actual pages - no matter what you type after the domain name, you always reach the home page and don't get any 404 error. I.E. Http://domain.com/oiuxyxyzbpoyob/ (there is no such a page, but i don't get 404 error, the home is displayed and the url in the browser remains Http://domain.com/oiubpoyob/, so it's not a 301 redirect). Http://domain.com/WhatEverYouType/ (same) Could this be an important SEO issue (i.e. resulting in infinite amount of duplicate content pages )? Do you think I should require the owner to prevent this from happening? Should I look into the .htaccess file to fix it ? Thank you Mozers!
Intermediate & Advanced SEO | | DoMiSoL0 -
Are iframes really an organic search problem?
I'm helping someone with a new site that will have pages for organic search that contain embedded video. Some will be youtube embeds and some will be wistia embeds. These pages will have several hundred words of transcript text and the embeds (iframes) iframes themselves will be rather small, but expandable and possibly more than one per page. The transcript text area is more like 80% of the page. Do you think this is an organic search problem? I use one site audit tool that calls this out as a serious warning. Currently, the embedded player(s) are a column down the left side, about 1/4th of the width of the page, and the transcripts are everything else, wrapping around it. The transcripts are fully readable and not hidden in some kind of expandable accordion or anything. Does layout matter in this issue? Thanks... Darcy
Intermediate & Advanced SEO | | 945010 -
Why isn't www.devonshiredentalcare.co.uk ranking?
Hi, devonshiredentalcare.co.uk was hit badly by one of Google's algorithm updates due to some prior poor seo (by another company). We took this client on and followed all of Google's guidelines and after a lot of work, managed to lift the penalty. It's been almost a year since the penalty was lifted, but it seems to be impossible to get this website ranking for 'Dentist Glasgow', they are currently page 4 in the SERPs! They have 33 Google reviews, we've built good quality links and citations, they have a 'Grade A' for on page optimisation in moz, we are also about to make the website responsive due to the recent mobilegeddon update. Do you have any further suggestions to help get this website ranking? Thanks in advance, Faye
Intermediate & Advanced SEO | | dentaldesign0 -
Is ok to add 'no follow' to every outbound link?
How do you handle outbound links from your site?.. do you no follow them all to be on the safe side?
Intermediate & Advanced SEO | | nick-name1230 -
Why my site it's not being indexed?
Hello.... I got to tell that I feel like a newbie (I am, but know I feel like it)... We were working with a client until january this year, they kept going on their own until september that they contacted us again... Someone on the team that handled things while we were gone, updated it´s robots.txt file to Disallow everything... for maybe 3 weeks before we were back in.... Additionally they were working on a different subdomain, the new version of the site and of course the didn't block the robots on that one. So now the whole site it's been duplicated, even it´s content, the exact same pages exist on the suddomain that was public the same time the other one was blocked. We came in changes the robots.txt file on both server, resend all the sitemaps, sent our URL on google+... everything the book says... but the site it´s not getting indexed. It's been 5 weeks now and no response what so ever. We were highly positioned on several important keywords and now it's gone. I now you guys can help, any advice will be highly appreciated. thanks Dan
Intermediate & Advanced SEO | | daniel.alvarez0 -
Best solutions when homepage won't rank in Google?
My homepage (www.LeatherHideStore.com) will not rank for my keywords in Google - with Google mostly pulling product pages and some categories for serp results. In contrast, my homepage consistently shows for Yahoo and Bing with exceptions where a category is a better match for the keyword. In other words, it is working exactly as it should in Yahoo and Bing. After a year of this frustration I just upgraded to a new site on Magento Community and surprise, the same problem! The SEO moz analyzer has flagged significant duplicate content issues which I think is at the heart of my problem. I have asked my developer to address these but let's just say that customer service is not his forte. I am even starting to doubt he knows what to do although the site appears is well done. Given that it is a brand new site and duplicate content in Magento is to be expected (from what I have now read), I am deeply discouraged that my developer did not or could not plan for this so here I am again! Can anyone give me guidance on what to do? I have read a lot about canonicalization and it seems complicated especially if you have 1000 duplicate page titles. I have seen that there are some extensions (i.e. Ultimate SEO Suite by aheadWorks) for Magento that claim to be able to solve duplicate content problems but I am really just grasping at straws and do not have the confidence or skills to implement this on my own. Can anyone please help? Thanks! Hunter
Intermediate & Advanced SEO | | leatherhidestore0 -
Has anyone else seen a Google Plus Local listing displace a regular search listing?
I have a particular site that I have been working on for about eight months and had the site on Page 1 of Google search results for eight keywords (they are fairly small local-based keywords, so I'm really not trying to boast). Perhaps six weeks ago for two of the keywords we popped into the #2 position for Google Plus Local results. When this happened the site completely disappeared from the regular search results. A couple weeks later, the Google Plus Local listing was gone, and the site was back on Page 1 in the regular listings. This has gone back and forth several times, with either a very high Local result or a very high regular search result, but only one at a time. I suppose it would make sense for the same site to only be able to have one position on the front page at any given time, but my searches for info on this have been entirely fruitless. Has anyone else seen anything like this or have any thoughts? Cheers.
Intermediate & Advanced SEO | | IanKietzman271