Can I use a "no index, follow" command in a robot.txt file for a certain parameter on a domain?
-
I have a site that produces thousands of pages via file uploads. These pages are then linked to by users for others to download what they have uploaded.
Naturally, the client has blocked the parameter which precedes these pages in an attempt to keep them from being indexed. What they did not consider, was they these pages are attracting hundreds of thousands of links that are not passing any authority to the main domain because they're being blocked in robots.txt
Can I allow google to follow, but NOT index these pages via a robots.txt file --- or would this have to be done on a page by page basis?
-
Since you have those pages blocked via robots.txt, the bots would never even crawl these pages in theory...which means the Noindex,follow is not helping.
Also, if you do a report on the domain on opensiteexplorer and dig, you should be able to find tons of those links already showing up. So if my site is linking to a page on that site, that page may not be cached/indexed because of the robots.txt exclusion, but that as long as my site is follow, your domain is still getting the credit for the link.
Does that make sense ?
-
Answered my own question.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Localized Domain Issue - Can I use Search Console to solve this?
Struggling through trying to resolve a complicated search issue - would appreciate any community input or suggestions. The Background Info We have several brand sites and each one has both a .ca and .com domain. For some reason, our website platform was created in a way that hundreds of pages on the .com domain have an equivalent page on the .ca domain, which are all 301'ed to the appropriate .com pages. Example below for clarity: www.domain.ca/gadget/brand - 301 Redirected to: www.domain.com/gadget/brand www.domain.ca/gadget/en/brandcanada = Proper .ca Canadian URL (where en is the language - fr exists as well) The Problem Because these .com pages exist under the .ca domain as well, they have started to outrank the correct .ca pages on Google. This has led to Canadian customers finding incorrect information, pricing, and reviews for these products - causing all sorts of customer service issues and therefore affecting our sales. I am being told that to properly fix the issue, and remove the incorrect URLs under the .ca domain would be prohibitively expensive in terms of resources, so I'm left trying to fix this via means available to me (i.e. anything but a change to how the platform is currently setup). The Attempted Fix I've submitted proper sitemaps for the .ca brand sites, and we have also created a robots.txt file to be accessed only when the site is crawled through the .ca domain. In that robots.txt, we have Disallowed crawling of any /gadget/brand/ URLs for the .ca domain. This was done a week ago and I am still seeing the .com URL show up in search results. The Question Should I be submitting any www.brand.ca/gadget/brand/ URLs to be temporarily removed from Google? Because of the 301 redirect in place from www.brand.ca/gadget/brand to www.brand.com/gadget/brand, I am hesitant to do so, as I do not want the .com URL removed. Will Google simply remove the .ca URL and not follow the 301 redirect to remove that URL as well? Any additional insight or feedback would be awesome as well.
Intermediate & Advanced SEO | | Trevor-O0 -
Robots.txt - Googlebot - Allow... what's it for?
Hello - I just came across this in robots.txt for the first time, and was wondering why it is used? Why would you have to proactively tell Googlebot to crawl JS/CSS and why would you want it to? Any help would be much appreciated - thanks, Luke User-Agent: Googlebot Allow: /.js Allow: /.css
Intermediate & Advanced SEO | | McTaggart0 -
meta robots no follow on page for paid links
Hi I have a page containing paid links. i would like to add no follow attribute to these links
Intermediate & Advanced SEO | | Kung_fu_Panda
but from technical reasons, i can only place meta robots no follow on page level (
is that enough for telling Google that the links in this page are paid and and to prevent Google penlizling the sites that the page link to? Thanks!0 -
Can pages compete with each other? Inbound links & domain authority, How to determine problem areas?
Heyy, I'm having some pretty big SEO issues. 😞 We have had some drops in our ranking. We're 5th page or worse depending on location for a few of our keywords that we used to rank well for. There are all sorts of random non relevant sites outranking us for the term "stickley" and "stickley furniture" One thing I noticed is that we are ranking for a different page for each keyphrase. Our home page is ranking for "Stickley" and our stickley page is ranking for "Stickley Furniture" Is this normal? I guess Google is just picking what it see's as what's more relevant. Is it possible that these two pages are "competing?" Do similar phrases linking to different pages cause pages to "fight" or unevenly disperse link juice? I'm having trouble knowing which page I should send inbound links to since Google seems to be linking similar keywords to different pages. How much should I stress about which pages I receive links on? Is it true that any inbound link to a site site will help increase its overall domain authority and overall SEO? What should I be focusing on? I've added 301 redirects for non WWW as well as tried to make the pages well optimized for SEO. Should I just add more related content to the pages? I know backlinks are important but I'm having a really hard time figuring out how to get links that aren't just spammy forum post footers or junk directory submissions. The thing that bothers me is we were ranking well and then suddenly are way back. We have never done any black hat SEO of any sort. I feel a bit stuck and confused at the moment 😞 Thanks in advance for any help!
Intermediate & Advanced SEO | | SheffieldMarketing
-Amy0 -
Can Someone Provide an Example of a Site that Indexes Search Results Successfully?
So, I know indexing search results is a big no-no, but I recently started working with a site that sees 50% of its traffic from search result pages. The user engagement on these pages is very high, and these pages rank well too. Unfortunately, they've been hit by Panda. They already moved the section of the site with search results to a subdomain, and saw temporary success. There must be a way to preserve their traffic from these search result pages and get out from under Panda.
Intermediate & Advanced SEO | | nicole.healthline0 -
Rel="prev" and rel="next" implementation
Hi there since I've started using semoz I have a problem with duplicate content so I have implemented on all the pages with pagination rel="prev" and rel="next" in order to reduce the number of errors but i do something wrong and now I can't figure out what it is. the main page url is : alegesanatos.ro/ingrediente/ and for the other pages : alegesanatos.ro/ingrediente/p2/ - for page 2 alegesanatos.ro/ingrediente/p3/ - for page 3 and so on. We've implemented rel="prev" and rel="next" according to google webmaster guidelines without adding canonical tag or base link in the header section and we still get duplicate meta title error messages for this pages. Do you think there is a problem because we create another url for each page instead of adding parameters (?page=2 or ?page=3 ) to the main url alegesanatos.ro/ingrediente?page=2 thanks
Intermediate & Advanced SEO | | dan_panait0 -
New Site: Use Aged Domain Name or Buy New Domain Name?
Hi,
Intermediate & Advanced SEO | | peterwhitewebdesign
I have the opportunity to build a new website and use a domain name that is older than 5 years or buy a new domain name. The aged domain name is a .net and includes a keyword.
The new domain would include the same keyword as well as the U.S. state abbreviation. Which one would you use and why? Thanks for your help!0 -
XML Sitemap instruction in robots.txt = Worth doing?
Hi fellow SEO's, Just a quick one, I was reading a few guides on Bing Webmaster tools and found that you can use the robots.txt file to point crawlers/bots to your XML sitemap (they don't look for it by default). I was just wondering if it would be worth creating a robots.txt file purely for the purpose of pointing bots to the XML sitemap? I've submitted it manually to Google and Bing webmaster tools but I was thinking more for the other bots (I.e. Mozbot, the SEOmoz bot?). Any thoughts would be appreciated! 🙂 Regards, Ash
Intermediate & Advanced SEO | | AshSEO20110