Blog posts not getting indexed and being outranked by scrapper sites.
-
Our Google traffic has dropped significantly over the last year and now we're struggling to even get our blog posts indexed. It's been extremely discouraging and we're trying to do what ever we can to fix it.
I've included a screenshot of our Google traffic as well as Pages Indexed according to WebmasterTools.
The Problem
- Our blog posts are frequently not getting indexed.
- Many times they are outranked by low authority scraper sites, our Twitter/FB account, etc.
- Sometimes our homepage will rank instead of the blog post.
- Sometimes we'll break a news story, get tons of quality backlinks, and still be nowhere in Google.
- Pretty much the only Google traffic we see is from existing posts.
- Still 3,200 pages indexed when we have only 1,600 posts. I guess this isn't really a problem... just waiting for the meta noindex to take effect.
More details
- We've seen no duplicate content or other warnings from WebmasterTools.
- We've been constantly acquiring quality backlinks from credible sites.
- We deleted the useless content and fixed the canonical issues that were a result of switching servers.
History
Our site is a news/entertainment blog. The traffic usually has spikes depending on what's going on in the news.
- Nov 1, 2011 - Site kept maxing out at 30k+ visits so we switched servers.
- Jan 30, 2012 - Hired a writer so we could focus on other aspects of the site.
- Apr 19, 2012 - Noticed our posts weren't getting indexed like they used to. Suspected our writer was spinning articles but couldn't find any evidence. 90% of our blog posts were nowhere to be found in Google. Scrapper sites would outrank us for our own stories... even our Twitter account was ranking ahead of us. IF our story would show up in Google it would usually be the home page instead of the blog post.
- Sep 2012 - Finally got more serious about addressing the problem. Noticed a couple potentially big problems and started making changes.
Canonical Issues
- non-www site didn't redirect to www. It showed 2 different link profiles according to OpenSiteExplorer and 0 backlinks according to Webmaster Tools.
- Wordpress shortlinks weren't redirecting to the actual permalink. For instance http://www.domain.com/?p=123 and http://www.domain.com/post-example were both getting indexed.
For every post there were 4 different versions that Google had to choose from.
http://domain.com/?p=123, http://www.domain.com/?p=123, http://domain.com/post-example, and http://www.domain.com/post-example
I figured the canonical issues must have happened when we switched servers which was the reason for the drop in WebmasterTools indexed pages and increase in Not Selected pages.
FIXED (Sep 15): One we fixed the canonical issue the Indexed Pages went back up however the Not Selected is still the same.
Duplicate Content
When we first created our site we wanted to have tons of images for each musician/athlete/actor/etc. so we uploaded about 5-10 for each person. We created a blog post for each image with no writing and the exact same post titles. As a result there were TONS of low-quality, similar posts, with virtually identical permalinks. e.g. http://www.domain.com/james-smith1, http://www.domain.com/james-smith2, http://www.domain.com/james-smith3, etc.
A crawl on Sep 26 showed over 550 duplicate content warnings.
FIXED (Oct 1): We deleted/301 redirected the useless pages (they weren't getting traffic anyways) and by the next crawl the number was almost to 0... which it's at now.
We also had TONS of tags (since there're constantly new names in the media) that were getting indexed so we had meta robots noindex them.
Questions:
- Why aren't a majority of our posts getting indexed?
- Were we penalized or just stuck because of a filter?
- How long should it take for meta robots to noindex the tags pages? (I did it on Sep 25 but they are still there)
- If a site is scraping our content (same title, image, excert) but linking to us, should we contact them and tell them to remove it?
- Is there anything else we need to do start getting our blog posts indexed like they used to?
- Should we try contacting Google to re-evaluate our site?
Sorry, that was a LOT of writing. If anyone wants the URL please let me know so I can PM it to you. Any help would be greatly appreciated!
-
Thanks for the quick response Dana!
They are sourcing us but at the moment they're constantly showing up instead of our site. There's no reason they should outrank us, but we figured they couldn't really be hurting us since they are linking to us.
It's weird that the site is able to scrape content and still get good indexes... maybe Google just hasn't picked up on it yet.
We are hoping that we don't have to worry about scrapers once our blog posts start getting indexed like they used to.
I think we'll send them a friendly email like you suggested.
-
I sympathize with your frustration. I know what it's like to come into a situation after years and years of technical missteps made by folks who never took SEO into consideration and then have to start cleaning up the mess.
You have a lot of hard technical problems and I'm not a developer so I will let those more technically gifted than myself address some of those.
There is one question I felt I could answer and that is #4: "If a site is scraping our content (same title, image, excert) but linking to us, should we contact them and tell them to remove it?"
You could go that way. But in the event that it is a decent site (despite the fact they scraped your content), first make sure you have a canonical tag properly implemented on your page, then, contact them and say "I see you found my content interesting enough to share on your site. Instead of me asking you to remove it, would you mind adding in an attribution line, giving credit to my site as being the source of the content and including a link back to me please? I think we can both agree this would be better than having me file a DMCA request with Google."
Just a thought
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How & What is the best advice on Keyword Cannibalization & get onpage optimized perfectly?
Hi all mozzers, I am having confusion to understand the fact and importance to target a single or related grouped keywords which is quite broader in terms of relevancy being found within our business. Let's explain more in detail:
Search Behavior | | KammySEO
Suppose we have a website: abc.com deal businesses in "Party Supplies, Party Decorations" Where the term "Party Supplies" being used exact or randomly many places, please see below finalized Titles respective to each landing page: abc.com/birthday/
title - Birthday Party Supplies - Kids Birthday Party Decorations Ideas abc.com/wedding/
title - Wedding Favors - Wedding Party Decorations & Centerpieces abc.com/baking/
title - Buy Baking Supplies - Cupcake & Cake Decorating Supplies abc.com/occasions/
title - Special Occasions Parties Supplies & Events - Party Time My main concern is, do our keyword party supplies gets stuck with "Keyword Cannibalization" ? If yes then what is the best advice you folks like to input here in order to safeguard and optimize best our landing pages for the such broader related search terms within the businesses. I am looking for best answer here0 -
Moz Showing 1500+ errors on my site but webmaster showing 130ish. Whats going on?
This Morning when I check my moz account it said I had 1500+ high priority issues, The break down of issues were 1432 4xx Client errors, 49 duplicate issues, 31 title page missing. but when I check my webmaster account is it only showing I have around 135 issues. I have been monitoring is the ranking for my homepage and it's been fluctuating between appearing around the 10th page for me to not at all. which begs the question that I'm being penalized or what is going on? dor4 7xtd
Search Behavior | | Nicktaylor10 -
Google site: search showing twice amount of indexed pages. why?
I have around 50k pages indexed on my site but when I do a google site: of my site it shows around 100k pages indexed. Why is it showing so much more? It is also only showing around 700 pages indexed in my web masters account for the site. Background: We have a custom site map being generated automatically. Let me know if you would like more info, Thanks.
Search Behavior | | Nicktaylor10 -
Google De-Indexed Our SIte for Branded Terms?
Hello all, As of 10am Pacific on September 12th, 2013, my team has noticed that our site, www.wirelessemporium.com, does not show up on the first 5 or 6 pages of SERPs for branded terms like "wireless emporium." We have not received any messages from Google via Webmaster Tools regarding this. Major activity that we've been doing to our site is updating content, meta tags, and h1 tags, along with removing/301 redirecting certain pages that did not meet Google quality guidelines. We've also been purging our backlink portfolio of toxic links and URLs, both manually and through the disavow tool. No blackhat has been done to this site for a very long time (more than 8 months now). One thing to note is that we did have a manual spam penalty placed on us back in July of 2012, it expired in early August of 2013 after a reconsideration request was submitted, and a 2nd manual spam penalty was placed on us again later that month. We are submitting a 2nd reconsideration request this Monday. Could this or the recent Panda update have anything to with this? We are very much in need of opinions as to why this is happening to our site. 5adbd14a31de3a78b998df94f0b6d2be
Search Behavior | | eugeneku0 -
My satellite website is ranking better than my actual site. Why and what to do?
This is our website: www.sauspiel.de and this is the satellite website. As the satellite site has a better ranking for the keyword "schafkopf" since two or three weeks we wonder how this could happen and what would be best to change it or improve the situation. Shall we take down the satellite site, redirect to the "real" site or try to downgrade? And what would you consider to be the (strongest) reason? Any suggestions? Thanks a lot for your help and assessment!
Search Behavior | | sauspiel0 -
Using Google Analytics to See What Time of Day Visitors View My Site
Hi folks, My company has Google Analytics setup for all of our websites, but I am a bit stumped on something. Now, this may not be possible, but am I able to see what time of day visitors most frequently view my blog? I would like to optimize blog post publishing for when I know we have in influx of visitors, yet I cannot find this information on GA. Any input would be much appreciated. Regards,
Search Behavior | | Instabill
Meghan0 -
How to test a site for usability with hundreds users and get genuine feedback?
My question is twofold; firstly is there a recommended company that I could use to get feedback on my site usability. Ideally I would like this company to arrange a focus group of a couple hundred people that fit into the right demographic and organise a report on the opinions of the group and the issues faced. Secondly an in-house understanding of usability Now I would like to think that when I review a site for usability I am not biased in any way. But I believe that if I look at the page that I have created or even collaborated on that I will always miss usability issues as I frankly think my/our creation is amazing. I'm interested to know how people rank and check their site usability and what factors rank highest in their assessments i.e. the tools you use and how you use them.
Search Behavior | | Stefan-Thorpe0 -
Why would my Bounce Rate drop from 40% to 2% over 3 days without making any major changes to my site www.hiltonhead.com?
Our Wordpress site has had an historical bounce rate of 40% for the last 3 years but last Thursday it dropped to 17%, Friday to 7% and now is running at 2%. The pageviews per visit have also increased from 3 to almost 6. Thanks
Search Behavior | | CorkyGifford0