NoIndex user generated pages?
-
Hi,
I have a site, downorisitjustme (dot) com
It has over 30,000 pages in google which have been generated by people searching to check if a specific site is working or not and then possibly adding a link to a msg board to the deeplink of the results page or something which is why the pages have been picked up.
Am I best to noindex the res.php page where all the auto generated content is showing up and just have the main static pages as the only ones available to be indexed?
-
Yes, add it to the robots.txt (use a Disallow and a NoIndex statement). I did find that Bing for example has not reliably in the past honoured robots.txt (especially in the case where you have an explicit "index" tag on the page and a noindex for a URL path).
-
Thanks for the replies Gerg & Irving.
The robots.txt block/exclude I take it I can just do that to the res.php page and not have to individually for the 30k generated dynamic pages off it (probably a silly question I know but wanted to double check).
-
I'd noindex the page, block in robots.txt, make sure your sitemap.xml is not generating these URLs if automated, and if there is a main folder where all the user generated pages are then request removal of that content in Google WMT.
-
I would also exclude them via robots.txt and then push through a sitemap with your static content to "nudge" Google to recrawl your content (and hopefully drop the other pages off quickly over time).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Only fraction of the AMP pages are indexed
Back in June, we had seen a sharp drop in traffic on our website. We initially assumed that it was due to the Core Update that was rolled out in early June. We had switched from http to https in May, but thought that should have helped rather than cause a problem. Until early June the traffic was trending upwards. While investigating the issue, I noticed that only a fraction (25%) of the AMP pages have been indexed. The pages don't seem to be getting indexed even though they are valid. Accordingly to Google Analytics too, the percentage of AMP traffic has dropped from 67-70% to 40-45%. I wonder if it is due to the indexing issue. In terms of implementation it seems fine. We are pointing canonical to the AMP page from the desktop version and to the desktop version from the AMP page. Any tips on how to fix the AMP indexing issue. Should I be concerned that only a fraction of the AMP pages are indexed. I really hope you can help in resolving this issue.
Technical SEO | | Gautam1 -
Why google does not remove my page?
Hi everyone, last week i add "Noindex" tag into my page, but that site still appear in the organic search. what other things i can do for remove from google?
Technical SEO | | Jorge_HDI0 -
Thousands of 404-pages, duplicate content pages, temporary redirect
Hi, i take over the SEO of a quite large e-commerce-site. After checking crawl issues, there seems to be +3000 4xx client errors, +3000 duplicate content issues and +35000 temporary redirects. I'm quite desperate regarding these results. What would be the most effective way to handle that. It's a magento shop. I'm grateful for any kind of help! Thx,
Technical SEO | | posthumus
boris0 -
Noindex PPC landing pages or optimise for SEO?
Organic seems to be down YoY on one of the categories of a large ecommerce website that I work on. This particular category has multiple landing pages set up for PPC consisting of filtered products. So these landing pages are prone to duplicate content due to the products listed. e.g. Blue Thingamajigs White Thingamajigs Black Thingamajigs High Gloss Thingamajigs Oak Thingamajigs Glass Thingamajigs etc These landing pages do well for PPC, but are nowhere to be seen in organic (51+). The main category page however ranks quite well for quite a variety of root and longtail keywords, though not as well as it used to. For example, it does rank for "thingamajigs", "white thingamajigs", "white gloss thingamajigs" and "white gloss thingamajigs with cherries on top". Would it benefit the main category page if the PPC landing pages were noindexed? Or, despite Google's preference for the main category, work on further optimising the landing pages for SEO? Or is there another solution that I'm completely overlooking? (It is a Friday afternoon after all...)
Technical SEO | | Ria_0 -
Noindex meta tag
Hi When following Webmaster Tools/Optimization/HTML Improvements it says that we have duplicate title tags and duplicate meta descriptions for hundreds of pages, As corrective action we have added to those pages and also changed title tags to make sure that they are different but still Webmaster keeps reporting that the duplication exist. Is it possible that google bot doesn't see our noindex code while crawling? By the way our seomoz report says that there is no duplicate title tag or meta description on our site google has crawled our site today and we received our report from seomoz today thanks
Technical SEO | | iskq0 -
132 pages reported as having Duplicate Page Content but I'm not sure where to go to fix the problems?
I am seeing “Duplicate Page Content” coming up in our
Technical SEO | | danatanseo
reports on SEOMOZ.org Here’s an example: http://www.ccisolutions.com/StoreFront/product/williams-sound-ppa-r35-e http://www.ccisolutions.com/StoreFront/product/aphex-230-master-voice-channel-processor http://www.ccisolutions.com/StoreFront/product/AT-AE4100.prod These three pages are for completely unrelated products.
They are returning “200” status codes, but are being identified as having
duplicate page content. It appears these are all going to the home page, but it’s
an odd version of the home page because there’s no title. I would understand if these pages 301-redirected to the home page if they were obsolete products, but it's not a 301-redirect. The referring page is
listed as: http://www.ccisolutions.com/StoreFront/category/cd-duplicators None of the 3 links in question appear anywhere on that page. It's puzzling. We have 132 of these. Can anyone help me figure out
why this is happening and how best to fix it? Thanks!0 -
Too many on page links
Hello I have about 800 warnings with this. Example of one url with this problem is: http://www.theprinterdepo.com/clearance?dir=asc&order=price I was checking and I think all links are important. But I suppose that if I put a nofollow on the links on the left which are only for navigation purposes I can get rid of these warnings. Any other idea?
Technical SEO | | levalencia10 -
Page load speed?
Hi everyone, hope you've all had a good new years celebration. Just a quick one on page load speed? how important do you think it is to organic rankings, especially on competitive terms? Any tips on improving it? do i need to improve just the landing page im trying to rank or every page as its looked at as a qualtiy score as a whole? Kind Regards,
Technical SEO | | pauledwards0