Panda Updates - robots.txt or noindex?
-
Hi,
I have a site that I believe has been impacted by the recent Panda updates. Assuming that Google has crawled and indexed several thousand pages that are essentially the same and the site has now passed the threshold to be picked out by the Panda update, what is the best way to proceed?
Is it enough to block the pages from being crawled in the future using robots.txt, or would I need to remove the pages from the index using the meta noindex tag? Of course if I block the URLs with robots.txt then Googlebot won't be able to access the page in order to see the noindex tag.
Anyone have and previous experiences of doing something similar?
Thanks very much.
-
This is a good read. http://www.seomoz.org/blog/duplicate-content-in-a-post-panda-world I think you should be careful with robot.txt because blocking access to the bot will not cause them to remove the content from their index. They will simply include a message saying not quite sure what's on this page.. I would use noindex to clear out the index first before attempting robot.txt exclusion.
-
Yes, both because if a page is linked to on another site google with spider that other site and follow your link without hitting the robots.txt and the page could get indexed if there is not a noindex on it.
-
Indeed try both.
Irving +1
-
both. block the lowest quality lowest traffic pages with nodindex and block the folder in robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Whats up with the last google update.
I have numerous clients who were at the top of page in the top 3 spots. They all dropped to page 3 or 4 or 2 and now they are number 1 in maps or in the top 3. Content is great on all these sites. backlinks are high quality and we do not build high quantity, we always focus on quality. the sites have authorship information. trust . we have excellent content written by professionals in the industry for each of the websites. The sites load super fast. they are very mobile friendly. we have CDN installed. content is organized per topic. all of our citations are setup properly and no duplicates, or missing citations. code is good on the websites. we do not have anchor text links pointing to the site from gust posts or whatever. we have plenty of content. our DA/PA is great. Audits of the website are great. I've been doing this a long time and ive never been so dumb founded as to what google has done this time. Or better yet what exactly is wrong with our clients websites today that was working perfectly for the last 5 years. I really am getting frustrated. im comparing my sites to competitors and everything's better. Please someone guide me here and tell me what im missing or tell me what you have done to recover from this nonsense.
Intermediate & Advanced SEO | | waqid0 -
Ecommerce catalog update: 301 redirects?
Hello mozers, We run an ecommerce store and are planning a massive catalog update this month. Essentially, 100% of our product listings will be deleted, and an all new catalog will be uploaded. The new catalog contains mostly new products, however there are some products that already existing in the old catalog as well. The new catalog has a bunch of improvements to the product pages, included optimized meta titles and descriptions, multiple language, optimized URLs and more. My question is the following: When we delete the existing catalog, all indexed URLs will return 404 errors. Setting up 301 redirects from old to new products (for products which existing previously) is not feasible given the number of products. Also, many products are simply being remove entirely. So should we go ahead and delete all products, upload the new catalog, update the sitemap, resubmit it for crawling, and live with a bunch of 404 errors until these URLs get dropped from Google? The alternative I see is setting 301 redirects to the home page, but I am not sure this would be correct use of 301 redirects. Thanks for your input.
Intermediate & Advanced SEO | | yacpro130 -
Noindex search pages?
Is it best to noindex search results pages, exclude them using robots.txt, or both?
Intermediate & Advanced SEO | | YairSpolter0 -
How long does it take for google to update title and metadata?
I updated my site's title and description about a week ago, however for some reason it's still not reflected in google search results. Here's my site and try searching for 'shopious directory'. Any idea why this is? I tried looking at webmaster tool and it seems that google didn't have any errors. Why is it still showing the old data?
Intermediate & Advanced SEO | | herlamba0 -
I want to Disavow some more links - but I'm only allowed one .txt file?
Hey guys, Wondering if you good people could help me out on this one? A few months back (June 19) I disavowed some links for a client having uploaded a .txt file with the offending domains attached. However, recently I've noticed some more dodgy-looking domains being indexed to my client's site so went about creating a new "Disavow List". When I went to upload this new list I was informed that I would be replacing the existing file. So, my question is, what do I do here? Make a new list with both old and new domains that I plan on disavowing and replace the existing one? Or; Just replace the existing .txt file with the new file because Google has recognised I've already disavowed those older links?
Intermediate & Advanced SEO | | Webrevolve0 -
Robots.txt: Syntax URL to disallow
Did someone ever experience some "collateral damages" when it's about "disallowing" some URLs? Some old URLs are still present on our website and while we are "cleaning" them off the site (which takes time), I would like to to avoid their indexation through the robots.txt file. The old URLs syntax is "/brand//13" while the new ones are "/brand/samsung/13." (note that there is 2 slash on the URL after the word "brand") Do I risk to erase from the SERPs the new good URLs if I add to the robots.txt file the line "Disallow: /brand//" ? I don't think so, but thank you to everyone who will be able to help me to clear this out 🙂
Intermediate & Advanced SEO | | Kuantokusta0 -
Hit by Google updates; Some good advice needed
Hi, Here`s my domain http://www.kent-website-designer.co.uk/. Registered in 2007. We have took a big hit from the updates in the last 6 months and its really affecting revenue. I know when you look at the site you may well think WOW this is 2007 SEO and youre right it hasnt been updated in some time as of last year we ranked very highly and it gave us enough business to concentrate on. However up until last year many of my competitors were using same onpage and offpage strategies....and probably a few of you were too! So now the inquiries and income is drying up. However, I provided myself with an income from my efforts, rather than be unemployed, so I want to get it back on track. I visited the Google webmaster forums to query a couple of webmaster account queries and basically got beat up by the rude and arrogant google forum admins. Basically they said I was a spam site who shouldnt be in business. How very nice! 1. I have EMD - but domain age should mean something? 2. I lost a few links from https://www.getsafeonline.org/partners-and-supporters/ in the last year which hasnt helped when they reorganised their content. Same with other trusted sites we lost links. We are left with low quality links. 3. Some CMS sites have replicated our footer links on a large scale, which wasnt intentional but may look as link spam, plus they arent no followed as G prefers. 4. Google seems to have become intelligent? Apparently it can detect content which is negative in outdated seo advice. How, can it understand context and meaning so older seo advice isdetected as spam content? 5. No pages are de indexed just a rank drop to 30 - 60 positions. 6. Over optimised H1`s? 7. Is Pipe command in titles now negative? So its sink or swim time I guess. The siteand domain is honest but neglected and probably should re align the business with what we can offer. We got away with that SEO but clearly things have changed. However with no grey or black hat at least we arent overly worried by removing links. Also looking for an SEO company who we can outsource with a white label solution in order to offer SEO. I dont need beating up, short and to the point critiques please. Pros and Cons Many thanks.
Intermediate & Advanced SEO | | xtopher661 -
How can scraper sites be successful post Panda?
I read this article on SEJ: http://www.searchenginejournal.com/scrapers-and-the-panda-update/34192/ And, I'm a bit confused as to how a scraper site can be successful post Panda? Didn't panda specifically target sites that have duplicate content & shouldn't scraper sites actually be suffering?
Intermediate & Advanced SEO | | nicole.healthline0