Blog posts not getting indexed and being outranked by scrapper sites.
-
Our Google traffic has dropped significantly over the last year and now we're struggling to even get our blog posts indexed. It's been extremely discouraging and we're trying to do what ever we can to fix it.
I've included a screenshot of our Google traffic as well as Pages Indexed according to WebmasterTools.
The Problem
- Our blog posts are frequently not getting indexed.
- Many times they are outranked by low authority scraper sites, our Twitter/FB account, etc.
- Sometimes our homepage will rank instead of the blog post.
- Sometimes we'll break a news story, get tons of quality backlinks, and still be nowhere in Google.
- Pretty much the only Google traffic we see is from existing posts.
- Still 3,200 pages indexed when we have only 1,600 posts. I guess this isn't really a problem... just waiting for the meta noindex to take effect.
More details
- We've seen no duplicate content or other warnings from WebmasterTools.
- We've been constantly acquiring quality backlinks from credible sites.
- We deleted the useless content and fixed the canonical issues that were a result of switching servers.
History
Our site is a news/entertainment blog. The traffic usually has spikes depending on what's going on in the news.
- Nov 1, 2011 - Site kept maxing out at 30k+ visits so we switched servers.
- Jan 30, 2012 - Hired a writer so we could focus on other aspects of the site.
- Apr 19, 2012 - Noticed our posts weren't getting indexed like they used to. Suspected our writer was spinning articles but couldn't find any evidence. 90% of our blog posts were nowhere to be found in Google. Scrapper sites would outrank us for our own stories... even our Twitter account was ranking ahead of us. IF our story would show up in Google it would usually be the home page instead of the blog post.
- Sep 2012 - Finally got more serious about addressing the problem. Noticed a couple potentially big problems and started making changes.
Canonical Issues
- non-www site didn't redirect to www. It showed 2 different link profiles according to OpenSiteExplorer and 0 backlinks according to Webmaster Tools.
- Wordpress shortlinks weren't redirecting to the actual permalink. For instance http://www.domain.com/?p=123 and http://www.domain.com/post-example were both getting indexed.
For every post there were 4 different versions that Google had to choose from.
http://domain.com/?p=123, http://www.domain.com/?p=123, http://domain.com/post-example, and http://www.domain.com/post-example
I figured the canonical issues must have happened when we switched servers which was the reason for the drop in WebmasterTools indexed pages and increase in Not Selected pages.
FIXED (Sep 15): One we fixed the canonical issue the Indexed Pages went back up however the Not Selected is still the same.
Duplicate Content
When we first created our site we wanted to have tons of images for each musician/athlete/actor/etc. so we uploaded about 5-10 for each person. We created a blog post for each image with no writing and the exact same post titles. As a result there were TONS of low-quality, similar posts, with virtually identical permalinks. e.g. http://www.domain.com/james-smith1, http://www.domain.com/james-smith2, http://www.domain.com/james-smith3, etc.
A crawl on Sep 26 showed over 550 duplicate content warnings.
FIXED (Oct 1): We deleted/301 redirected the useless pages (they weren't getting traffic anyways) and by the next crawl the number was almost to 0... which it's at now.
We also had TONS of tags (since there're constantly new names in the media) that were getting indexed so we had meta robots noindex them.
Questions:
- Why aren't a majority of our posts getting indexed?
- Were we penalized or just stuck because of a filter?
- How long should it take for meta robots to noindex the tags pages? (I did it on Sep 25 but they are still there)
- If a site is scraping our content (same title, image, excert) but linking to us, should we contact them and tell them to remove it?
- Is there anything else we need to do start getting our blog posts indexed like they used to?
- Should we try contacting Google to re-evaluate our site?
Sorry, that was a LOT of writing. If anyone wants the URL please let me know so I can PM it to you. Any help would be greatly appreciated!
-
Thanks for the quick response Dana!
They are sourcing us but at the moment they're constantly showing up instead of our site. There's no reason they should outrank us, but we figured they couldn't really be hurting us since they are linking to us.
It's weird that the site is able to scrape content and still get good indexes... maybe Google just hasn't picked up on it yet.
We are hoping that we don't have to worry about scrapers once our blog posts start getting indexed like they used to.
I think we'll send them a friendly email like you suggested.
-
I sympathize with your frustration. I know what it's like to come into a situation after years and years of technical missteps made by folks who never took SEO into consideration and then have to start cleaning up the mess.
You have a lot of hard technical problems and I'm not a developer so I will let those more technically gifted than myself address some of those.
There is one question I felt I could answer and that is #4: "If a site is scraping our content (same title, image, excert) but linking to us, should we contact them and tell them to remove it?"
You could go that way. But in the event that it is a decent site (despite the fact they scraped your content), first make sure you have a canonical tag properly implemented on your page, then, contact them and say "I see you found my content interesting enough to share on your site. Instead of me asking you to remove it, would you mind adding in an attribution line, giving credit to my site as being the source of the content and including a link back to me please? I think we can both agree this would be better than having me file a DMCA request with Google."
Just a thought
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blog Post SEO - Bounce Rate & Layout
Mozzers, I wrote the following blog post about Microsoft’s current offer to receive $100 free in bing ads. http://www.cildermansolutions.com/blog/free-bing-ads-coupon-for-2017-get-100-in-free-advertising-credits I noticed that the bounce rate on the post is higher than i was expecting; I am wondering if this is because I put the eligibility rules for collecting the ads coupon in the 2nd paragraph...and people are just seeing that right away and bouncing. I obviously don’t want to lie or trick readers, but i am wondering if the community has any idea/suggestions. Thanks, Matt
Search Behavior | | matt.nails0 -
Google smacked my site and dropped all rankings, can't find out why
I have checked out everything, I mean everything. We have no dupe content, our content is a little thin, but it is ours and accurate enough to help our customers.We follow all SEO guidelines and make sure we de-index any pages with no / little content (like privacy, or faq) All in all, we haven't done any major updates to the sites, and everything was great (page one on almost all kw) but beginning of the month, all kw wiped to the second page, than third, than back to one and now pretty much gone (rank 100-200) I really don't know what to do. We didn't receive a manual action, and the last algo update was nothing big to cause such a drastic change. Meanwhile our competition (multiple sites) are gaining in ranks and nothing happened to them (most of them have even less content and not even SSL) Negative SEO is out of question, I check all links via ahrefs every other day. Any help is appreciated Thanks
Search Behavior | | s-s0 -
How come some local 7 pack listings link to site and some link to the G+ page?
Does anyone know how to fix this issue? Even though a site profile has had the website added to it Google continues to link the main "title tag" link to the G+ page and not the actual website domain. Thanks for any info in advance! https://www.google.com/#sclient=psy-ab&q=dog+sitting+in+rockaway&oq=dog+sitting+in+rockaway&gs_l=hp.3..0i22i30l4.14871.16189.1.16397.8.8.0.0.0.0.296.2042.2-7.7.0...0.0...1c.1.15.psy-ab.Y1db0jo77V0&pbx=1&bav=on.2,or.r_cp.r_qf.&bvm=bv.47244034,d.bmk&fp=722b460c2153b7be&biw=1920&bih=910
Search Behavior | | irvingw0 -
Google indexing PDF's
Hello, We work heavily on E-commerce SEO and recently Google has started to index PDF pages (Datasheets) added to the product pages instead of the actual product pages. Has anyone else noticed this at all? Seems to have got worse over the last month or so. Thanks
Search Behavior | | voipme0 -
E-commerce SEO when products are shared between sites
We have a couple of e-commerce sites that sell the same product. One focuses more on a niche category of the products, while the others contains a wide variety of products. Will we be hit with a penalty from google if the product descriptions are the same? If so, how much of a difference in content does there need to be? There are only so many ways to describe the product and it is usually done in a couple sentences. The rest of the product description contains bullet points of the products and is the same for each product. We are also looking for a SEO company to work with and wondered if there are companies out there that specialize in e-commerce seo that can work with a small ($500 month) budget. I've looked through the recommended companies directory here, but it seems most have minimum retainers around $1000 a month. Also, what should I look for in an SEO company?
Search Behavior | | toomeycustoms0 -
Internal Site Search Analysis
Hi Folks, I have about 6,000 internal site search phrases that I want to analyze. There are many variations and duplicates that have similar intent within the data, e.g. Employment, Employment Opportunities, Employment Application. Does anyone have any thoughts on how I can aggregate the data to get an idea of user intent. There's a lot of long-tail in there. The data does not come from Google's site search tool. I just have a spreadsheet of the terms and the number of times they were searched. Cheers!
Search Behavior | | BedeFahey0 -
How do I get my report to show ranking for a location such as pittsburgh or columbus?
I am not interested in US ranking only Pittsburgh and Columbus as that's where my client's offices are located. Are there settings to help me view our rank on our areas? Thank you!!
Search Behavior | | IEInteractiveServices0