Robots txt. in page with 301 redirect
-
We currently have a a series of help pages that we would like to disallow from our robots txt.
The thing is that these help pages are located in our old website, which now has a 301 redirect to current site.
Which is the proper way to go around?
1- Add the pages we want to disallow to the robots.txt of the new website?
2- Break the redirect momentarily and add the pages to the robots.txt of the old one?
Thanks
-
In that case, you'd need to add the robots meta tag at the page level before the tag.
or
-
Hey, for some time we will keep the files in the old domain. Should we break the redirect and insert the disallows to the robot.txt of the old site?
-
So, the problem is that the robots.txt file can't be accessed because of the 301 redirect to the new domain?
Do you plan to keep the help files on the old domain, or will they be removed completely?
-
Hi Laura,
Thanks for your reply. I don't want to disallow the URLs these pages are being redirected to. Actually these URLs are in the old version but still can be accessed. So to put it simply, this is my case:
1- This was our current website: www.kilgray.com (With a 301 redirect)
2- This is our new website: www.memoq.com
3- I would like to disallow the following links on the old website that are still visible (haven't been redirected):
http://kilgray.com/memoq/2015-100/help-en/index.html
http://kilgray.com/memoq/2014/help-en/
-
Do you want to disallow the URLs that these pages are being redirected to? If not, there's no need to add anything to the robots.txt file.
If you do want to disallow the URLs that these pages are being redirected to, use relative URLs in your robots.txt file. For example, let's say olddomain.com/old-help-page/ is being redirected to newdomain.com/new-help-page/. If that's the case, add the following to your robots.txt file.
Disallow: /new-help-page/
There's no need to disallow the specific URLs that are being redirected to something else. Are you trying to get them removed from Google's index or something? If so, Google will update their index eventually based on your 301 redirects.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have two robots.txt pages for www and non-www version. Will that be a problem?
There are two robots.txt pages. One for www version and another for non-www version though I have moved to the non-www version.
Technical SEO | | ramb0 -
301 redirected all my pages to my new domain, now I have a problem with Google Search Console
Hi guys! I bought a new domain name and redirected all my URLs from the old domain to the new one. Everything worked perfectly but now I have a little problem. I want to use the option 'Address Change' in google search console. Step 1 Works (Select new website in the list) Step 2 Works (Confirm that the 301 are working) Step 3 Asks me to Verify the old domain (huh!?) in order to complete the request. Obviously that doesn't work because my 301s WORKS! So if I try to verify the old website by putting a google file in the root of my domain Google tries to access it and it automatically redirects to the new domain. I must be missing something lol help!
Technical SEO | | benoit_20180 -
301 Domain Redirect And Old Domain to a New one including pages
Hi, I need to 301 an old domain to a new one (new website) I need to 301 the domain to a new page not the new domain direct for example www.olddomain.co.uk to www.newdomain.co.uk/pagenew Then I need to also 301 all the other pages on the old domain to the new one for example... www.oldmain.co.uk/oldpage to www.newdomain.co.uk/newpage Issue is I can do one or the other not both, I can get the other pages to redirect but then the main domain wont redirect to the correct new page. Or I can get the old domain to redirect but not the internal pages. Thanks
Technical SEO | | David-Sharpe0 -
301 Redirects Relating to Your XML Sitemap
Lets say you've got a website and it had quite a few pages that for lack of a better term were like an infomercial, 6-8 pages of slightly different topics all essentially saying the same thing. You could all but call it spam. www.site.com/page-1 www.site.com/page-2 www.site.com/page-3 www.site.com/page-4 www.site.com/page-5 www.site.com/page-6 Now you decided to consolidate all of that information into one well written page, and while the previous pages may have been a bit spammy they did indeed have SOME juice to pass through. Your new page is: www.site.com/not-spammy-page You then 301 redirect the previous 'spammy' pages to the new page. Now the question, do I immediately re-submit an updated xml sitemap to Google, which would NOT contain all of the old URL's, thus making me assume Google would miss the 301 redirect/seo juice. Or do I wait a week or two, allow Google to re-crawl the site and see the existing 301's and once they've taken notice of the changes submit an updated sitemap? Probably a stupid question I understand, but I want to ensure I'm following the best practices given the situation, thanks guys and girls!
Technical SEO | | Emory_Peterson0 -
301 Redirect domain with penalty
Wondering if I could get a few views on this please... I have added an affiliate store to a domain I own, however I forgot to noindex the product pages which were duplicate content of the merchants. Despite a good deal of backlink building the site will not do much in the engines at all, doesn't even come up on the first few pages for it's own name! This suggests to me that I have a duplicate content penalty. Try as I may I cannot get it removed so am thinking of cloning the domain to a new domain, however, I do not want to lose the links I collected so I am planning on 301ing them. While I will not get all the link power moved over, I should at least get credit for some of them which will kick start the new domain. Can anyone forsee any potential issues with doing this? Is there a danger of 301ing a site with a penalty that the penalty would be carried over? I know there is no penalty on the links, no WMT warnings etc, it is the content causing the issue. Thanks, Carl
Technical SEO | | Grumpy_Carl0 -
Googlebot does not obey robots.txt disallow
Hi Mozzers! We are trying to get Googlebot to steer away from our internal search results pages by adding a parameter "nocrawl=1" to facet/filter links and then robots.txt disallow all URLs containing that parameter. We implemented this late august and since that, the GWMT message "Googlebot found an extremely high number of URLs on your site", stopped coming. But today we received yet another. The weird thing is that Google gives many of our nowadays robots.txt disallowed URLs as examples of URLs that may cause us problems. What could be the reason? Best regards, Martin
Technical SEO | | TalkInThePark0 -
301 Redirect & re-use
I have an old site which is being moved to a new tld due to re-branding. I understand I would do a series of 301 redirects from the pages of the old site to capture the authority and move to the new site. However, at some point in the future (probably 1-2 years) we may want to re-use the old site again for a different brand (it has a good brand, just not for what we're going after). Question is - can a redirected site be re-used at some point in the future? And if so, which site would new authority (links, etc.) go to?
Technical SEO | | uwaim20120 -
301 redirect .htaccess problem
Can anyone explain to me why this doesn't work? Redirect 301 /category/diamond-pendants/nstart/1/start/(.*) http://www.povada.com/category/pendants/nstart/1/start/$1 Im trying to replace everything after /start/ and insert it into the new url. Thanks in advance.
Technical SEO | | 13375auc30