Robots.txt and redirected backlinks
-
Hey there,
since a client's global website has a very complex structure which lead to big duplicate content problems, we decided to disallow crawler access and instead allow access to only a few relevant subdirectories. While indexing has improved since this I was wondering if we might have cut off link juice. Since several backlinks point to the disallowed root directory and are from there redirected (301) to the allowed directory I was wondering if this could cause any problems?
Example: If there is a backlink pointing to example.com (disallowed in robots.txt) and is redirected from there to example.com/uk/en (allowed in robots.txt). Would this cut off the link juice?
Thanks a lot for your thoughts on this.
Regards,
Jochen
-
A noindexed page can still accumulate and pass link equity, although results vary on whether or not some of that link juice "evaporates" along the way. I'm inclined to agree with Chris, though, that there's probably no need to noindex a page that redirects to a page that you do want indexed.
-
Hi Jochen,
It's an interesting situation and to be honest, I don't know for sure how search engines will deal with that "link juice". This will come down to a question of whether search engines see robots.txt or htaccess first. If it looks at robots first (which is my suspicion), it can't see that page to pass the strength.
I suppose to test this, you could submit the redirected page to index via Search Console and see if it shows you the redirect or says it's blocked.
Interesting question aside, there's no real need to block access to a 301'd page
Also, apologies if I'm just highlighting the obvious here but it would be far better to clean up the site structure and remove that duplication rather than just masking it with robots; the user experience is at least as important as the algorithms!
Along the same lines, cleaning up those pages is going to help your crawl budget immensely.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Content From API - Remove or to Redirect ?
Hi Guys,
Intermediate & Advanced SEO | | PaddyM556
I am working on a site at the moment,Â
Previous developer used a API to pull in HealthCare content (HSE) .
So the API basically generates landing pages within the site, and generates the content.
To date it has over 2k in pages being generated.
Some actually rank organically and some don't. New site being launch: So a new site is being launched & the "health advice" where this content used to live be not included in the new site. So this content will not have a place to be displayed. My Query: Would you allow the old content die off in the migration process & just become 404's
OrÂ
Would you 301 redirect the all or only ranking pages to the homepage ? Other considerations, site will be moved to https:// so site will be submitted to search console & re-indexed by Google. Would love to hear if anyone had similar situation or suggestions.
Best Regards
Pat0 -
Blog subdomain not redirecting
Over the last few weeks I have been focused on fixing high and medium priority issues, as reported by the Moz crawler, after a recent transition to WordPress. I've made great progress, getting the high priority issues down from several hundred (various reasons, but many duplicates for things like non-www and www versions) to just five last week. And then there's this weeks report. For reasons I can't fathom, I am suddenly getting hundreds of duplicate content pages of the form http://blog.<domain>.com</domain> (being duplicates with the http://www.<domain>.com</domain> versions). I'm really unclear on why these suddenly appeared. I host my own WordPress site ie WordPress.org stuff. In Options / General everything refers to http://www.<domain>.com</domain> and has done for a number of weeks. I have no idea why the blog versions of the pages have suddenly appeared. FWIW, the non-www version of my pages still redirect to the www version, as I would expect. I'm obviously pretty concerned by this so any pointers greatly appreciated. Thanks. Mark
Intermediate & Advanced SEO | | MarkWill0 -
Clarity needed on 301 redirects
Looking to get a bit of clarity on redirects: We're getting ready to launch a new website with a simplified url structure (we're consolidating pages & content) & I already know that I'll have to employ 301 redirects from the old url structure to the new. What I'm not clear about is how specifc I should be. Here's an example of my file structure: Old website: www.website.com
Intermediate & Advanced SEO | | JSimmons17
New website: www.website.com Old website: www.website.com/vacations
New website: www.website.com/vacations Old website: www.website.com/vacations/costa-rica
New website: www.website.com/vacations/central-america Old website: www.website.com/vacations/costa-rica/guanacaste
New website: www.website.com/vacations/central-america Old website: www.website.com/vacations/mexico
New website: www.website.com/vacations/central-america Old website: www.website.com/vacations/mexico/cancun
New website: www.website.com/vacations/central-america Old website: www.website.com/vacations/bolivia
New website: www.website.com/vacations/south-america Old website: www.website.com/vacations/bolivia/la-paz
New website: www.website.com/vacations/south-america Do I need to redirect each and every page or would just redirecting just the folder be enough to keep my SEO juice? Many thanks in advance for any help!0 -
Does a 302 redirect pass penalties?
I'm having problems finding a definitive answer to this question, there is a lot of rumour and gossip out there but nothing I can rely on. I'm working with a site that received an unnatural links notice followed by a massive drop in search traffic. Looking at the link profile it's pretty much jacked beyond repair and I have recommended that we move over to a fresh domain. However, it's an established brand with many more sources of traffic than organic search. There's no way we can burn all their repeat visits, loyal customers, brand recognition that they've built up over the years so I want to redirect from the old domain to the new. Â This is not to try and make any SEO gain from the previous site, frankly we don't give a crap about that. We just want to maintain the brand. A 302 is a temporary redirect, this will be a permanent move BUT a 301 will pass on the penalty. Â So can we safely use a 302 redirect in this situation or is there a better alternative (meta refresh?) Thanks for your help! MB.
Intermediate & Advanced SEO | | MattBarker0 -
Should I redirect secondary keyword page
I have a reasonably high authority home page and have decided to optimise the home page to target a competitive keyword that previously had a specific page that was optimised on an internal page of my site that I have spent time building links to. The internal page has over 200 links to it so should I 301 redirect this internal page to the home page. Will that increase the auhority of the home page further? Or should I keep the internal page as a 'secondary' page for that keyword?. If I do have two pages don't I risk confusing google?
Intermediate & Advanced SEO | | SamCUK0 -
Setting up of 301 redirects
Good morning all, As part of the analysis of our website, we have realised that we are diluting our keyword strength in a particular area by having multiple zones all targeting the same keyword. We have decided to combine these zones into one, and set up 301 redirects so that the remaining zone gets the benefit of the other zones' link juice. When setting up a 301 redirect from zone "X" to zone "Y" say, do I need to keep all of the content in zone X, or should I remove all content before the redirect is set up? Does zone Y still get the benefit of zone X's link juice if the content is removed? Many thanks Guy
Intermediate & Advanced SEO | | Horizon0 -
Robots.txt & url removal vs. noindex, follow?
When de-indexing pages from google, what are the pros & cons of each of the below two options: robots.txt & requesting url removal from google webmasters Use the noindex, follow meta tag on all doctor profile pages Keep the URLs in the Sitemap file so that Google will recrawl them and find the noindex meta tag make sure that they're not disallowed by the robots.txt file
Intermediate & Advanced SEO | | nicole.healthline0 -
Multiple 301 redirects considered a redirection chain?
I need to redirect a ton of duplicate content, so I want to try redirect 301 /store/index.php /store redirect 301 /store/product-old /store/product-new redirect 301 /store/product-old1 /store/product-new1 redirect 301 /store/product-old2 /store/product-new2 redirect 301 /store/product-old3 /store/product-new3 redirect 301 /store/product-old4/file.html /store/product-old4/new4/file.html and then a whole bunch of old dead links to homepage. So we've had /index.php redirected to / on other parts of the site for awhile, and for the most part /store is a friendly URL, but then we have tons of dup content and work arounds that preceded my job here. I'm wondering if those redirects above would be considered a redirection chain? Since the all the redirects below the /index.php -> /store count on that one redirect. Thanks for any insight you may be able to give!
Intermediate & Advanced SEO | | Hondaspeder1