When you add 10.000 pages that have no real intention to rank in the SERP, should you: "follow,noindex" or disallow the whole directory through robots? What is your opinion?
-
I just want a second opinion
The customer don't want to loose any internal linkvalue by vaporizing link value though a big amount of internal links. What would  you do?
-
Hi Jeff,
Thanks for your answer. Please take a look to the reply above on Fredrico
-
Hi Federico,
In this case it's an affiliate website and the 10.000 pages are all prodcutpages. It's all coming from datafeeds so it's duplicate content.
We don't want to index this that's for sure.
So noindex,follow or disallow the whole directory or both...
We have our own opinion about this but I want to hear what others are thinking about this
Thanks in advanced!
-
Yep, I agree with belt and suspenders.
-
Wesley - I do agree with Federico.
That said, if they really don't want those pages indexed, use the belt-and-suspender method (if you wear both a belt and suspenders, chances are greater that your pants won't fall down).
I'd put a robot.txt file to disallow the indexing of the directory, and also no-index / no-follow each of the pages, too.
That way when they have someone working on the pages in the site and they change things to followed, you're still covered. Â Likewise, if someone blows away the robot.txt file.
Just my $0.02, but hope it helps…
-- Jeff -
What do they have? 10,000 pages of uninteresting content? a robots tag noindex,follow will do to leave them our of engines. But to decide you really need to know what those pages have. 10,000 isn't a few, and if there's value content worth sharing, a page could get a link, that if you disallow it through the robots, won't even flow pagerank.
It all comes down to what are those pages for...?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blog page and homepage ranking next to each other for same keyword
Hello, I have my homepage that has been existing for 10 years that is ranked in 18 th position on google for the keyword luxury bike tours. This homepage doesn't have any external link or internal links saying luxury bike tours and nowhere in the title or on the page do I have the word luxury. I only have the words bike and tours. I created a blog page 24 hours ago that has the word luxury, bike and tours in the title and it is ranked in 19 th position just behind my homepage. I am wondering how it can be there and my homepage just be one spot above with all the history and linking it has ? Is it due to the fact that I have the word luxury in the title ? Is it just because my internal linking structure is correct and this blog page is brand new and will my homepage rank higher in the near future but see that I just redid the structure I need to wait a few months ?
Intermediate & Advanced SEO | | seoanalytics0 -
Author Byline in Page Title in SERP
I was exploring my company's visibility in Google News results, and I noticed the author byline in a recently published article was being pulled into the page title in the SERP. See the attached image for a screenshot. It makes it sound awkward: "How to Find the Best Cannabis Experience and High for You Patrick..." - as if we're explaining it to some guy named Patrick? We have the byline the same way in all other posts, but this is the first I've seen this happen. Has anyone seen/had this happen, and if so, have any ways to prevent it? Thanks in advance for any insights! Here's the post URL:Â https://www.leafly.com/news/cannabis-101/how-to-find-best-cannabis-experience-high csvmF
Intermediate & Advanced SEO | | davidkaralisjr0 -
How to Improve Search Ranking from 10 to Top 5
Hi, I was just optimizing a page and have improved it from Page 2 -> Page 1 ; Position 15->10 for now by basic onpage seo edits. I want to understand how I would take it to next level by getting it to Top 5 or Top 3 results. My keyword and page are as follows (all checked in Google India (.in) ) Page - https://nirogam.com/ayurvedic-treatment-home-remedies-chikungunya/
Intermediate & Advanced SEO | | pks333
Keyword - ayurvedic treatment for chikungunya What steps can I take to go from 10 - Top 3 or Top 5 results here? I was checking the Rank tracker & moz Grader for this - got all ticks except adding keyword in URL. Would this be recommended of changing the URL after it's ranking so well to just add this keyword in the same?0 -
Dilemma about "images" folder in robots.txt
Hi, Hope you're doing well. I am sure, you guys must be aware that Google has updated their webmaster technical guidelines saying that users should allow access to their css files and java-scripts file if it's possible. Used to be that Google would render the web pages only text based. Now it claims that it can read the css and java-scripts. According to their own terms, not allowing access to the css files can result in sub-optimal rankings. "Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings."http://googlewebmastercentral.blogspot.com/2014/10/updating-our-technical-webmaster.htmlWe have allowed access to our CSS files. and Google bot, is seeing our webapges more like a normal user would do. (tested it in GWT)Anyhow, this is my dilemma. I am sure lot of other users might be facing the same situation. Like any other e commerce companies/websites.. we have lot of images. Used to be that our css files were inside our images folder, so I have allowed access to that. Here's the robots.txt --> http://www.modbargains.com/robots.txtRight now we are blocking images folder, as it is very huge, very heavy, and some of the images are very high res. The reason we are blocking that is because we feel that Google bot might spend almost all of its time trying to crawl that "images" folder only, that it might not have enough time to crawl other important pages. Not to mention, a very heavy server load on Google's and ours. we do have good high quality original pictures. We feel that we are losing potential rankings since we are blocking images. I was thinking to allow ONLY google-image bot, access to it. But I still feel that google might spend lot of time doing that. **I was wondering if Google makes a decision saying, hey let me spend 10 minutes for google image bot, and let me spend 20 minutes for google-mobile bot etc.. or something like that.. , or does it have separate "time spending" allocations for all of it's bot types. I want to unblock the images folder, for now only the google image bot, but at the same time, I fear that it might drastically hamper indexing of our important pages, as I mentioned before, because of having tons & tons of images, and Google spending enough time already just to crawl that folder.**Any advice? recommendations? suggestions? technical guidance? Plan of action? Pretty sure I answered my own question, but I need a confirmation from an Expert, if I am right, saying that allow only Google image access to my images folder. Sincerely,Shaleen Shah
Intermediate & Advanced SEO | | Modbargains1 -
Yellow pages, how to improve rankings?
We have a huge database of companies in the baltic region (www.business-baltics.com) the page is completely yellow pages with no unique texts or anything. How would you improve the Search Engine Rankings for a website like this? And how do you do a link building for a page like this?
Intermediate & Advanced SEO | | MarkasKR0 -
Can a home page penalty cause a drop in rankings for all pages?
All my main keywords have dropped out of the SERPS. Could it be that the home page (the strongest) page has been devalued and therefore 'link juice' that used to spread throughout the site is no longer doing so. Would this cause all other pages to drop? I just can't understand how all my pages have lost rankings. The site is still indexed so there's no problem there.
Intermediate & Advanced SEO | | SamCUK0 -
How Can I know that a link placed is not lableld "No Follow"?
If someone wants to trade links, how can I be sure the link is followed?
Intermediate & Advanced SEO | | SEObleu.com0 -
Noindex term for pages
I have a client who has a virtuemart site and all of the "ask a question on this product" had been indexed on google. I have managed to get a noindex meta tag into the ask a question page, will these be dropped from the index next time they are crawled and google sees the noindex?
Intermediate & Advanced SEO | | webseoservices0