Robots txt. in page with 301 redirect
-
We currently have a a series of help pages that we would like to disallow from our robots txt.
The thing is that these help pages are located in our old website, which now has a 301 redirect to current site.
Which is the proper way to go around?
1- Add the pages we want to disallow to the robots.txt of the new website?
2- Break the redirect momentarily and add the pages to the robots.txt of the old one?
Thanks
-
In that case, you'd need to add the robots meta tag at the page level before the tag.
or
-
Hey, for some time we will keep the files in the old domain. Should we break the redirect and insert the disallows to the robot.txt of the old site?
-
So, the problem is that the robots.txt file can't be accessed because of the 301 redirect to the new domain?
Do you plan to keep the help files on the old domain, or will they be removed completely?
-
Hi Laura,
Thanks for your reply. I don't want to disallow the URLs these pages are being redirected to. Actually these URLs are in the old version but still can be accessed. So to put it simply, this is my case:
1- This was our current website: www.kilgray.com (With a 301 redirect)
2- This is our new website: www.memoq.com
3- I would like to disallow the following links on the old website that are still visible (haven't been redirected):
http://kilgray.com/memoq/2015-100/help-en/index.html
http://kilgray.com/memoq/2014/help-en/
-
Do you want to disallow the URLs that these pages are being redirected to? If not, there's no need to add anything to the robots.txt file.
If you do want to disallow the URLs that these pages are being redirected to, use relative URLs in your robots.txt file. For example, let's say olddomain.com/old-help-page/ is being redirected to newdomain.com/new-help-page/. If that's the case, add the following to your robots.txt file.
Disallow: /new-help-page/
There's no need to disallow the specific URLs that are being redirected to something else. Are you trying to get them removed from Google's index or something? If so, Google will update their index eventually based on your 301 redirects.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 Redirects - Clearing out a 300 Page WordPress Website (HELP)
I'm working on a WordPress website that has around 300 pages (not posts, pages), many of which have very little content and / or content that is super outdated with zero relevancy. The site has been up and running for 10 years and ranks well overall, 1st in many instances for competitive keywords. The company holds yearly events and works with various other companies who exhibit at their events. In their very early years, they created pages on their website for each of these companies, some of which don't even exist anymore. If I had to delete all of the pages that are deprecated there would only be 10 pages of the site left (with blog posts counted separately). Would it be safe to remove these pages and set a 301 redirect to a semi-related page without hurting the website rankings? Any and all advice appreciated!!
Technical SEO | | enimmo19970 -
Removing Multiple 301 Redirects
During my last redesign (and migration to Drupal) some of the updated SEO friendly url's on the new site were misspelled. Rather than updating the 301 redirects to point to the correct page the developer just added an additional 301 redirect. So it was redirected like this website.com/oldpage (301 to) website.com/new-paige (301 to) website.com/new-page Instead of website.com/oldpage (301 to) website.com/new-page I'll be finishing another redesign and updating to https soon, should I remove the redirect to the misspelled domain and just have one 301 from the original page? These multiple redirects have been up for over a year. Thanks for any specific advice!
Technical SEO | | talltrees0 -
HTacess 301 redirect with special characters
Hello moz community ! I would to make a special 301 redirection through my htaccess file. I am a total noob concerning regexp and 301 redirection. I would like to redirect(301) this url : http://www.legipermis.com/stages-points/">http://www.legipermis.com/stages-points/</a></p>; yes yes it's in the index of google, this strange url includes the last ; to http://www.legipermis.com/stages-points/ I have already include a canonical tag by security, i would like to remove url with a 301 redirection and by remove this url through GWT (but the removal tool can't "eat' this kind of URL) Please consider the fact that i am not an expert about 301 redirections and regexps. No 301 redirect generator works properly for such a strange URL (which triggers content duplication corrected anyway with canonical tag). Thanks for your help.
Technical SEO | | LegiPermis0 -
Importance of 301 Redirects
Hello, I have been brought in at the last minute to consult for an e-commerce client who is about to relaunch their website. The site currently receives 8000 visits a month, 3100 of which are from organic search. They have a few thousand product pages. The web development firm they are using is changing all of the old product page urls and using 'search engine friendly' urls for the new site, which is expected to launch in a few weeks. However, they did not/are not planning on including 301 redirects from the old URLs. Other than simply stating 'this will be bad for your SEO', what would be a correct way of explaining to the client how much of a problem it will be if their new site launches without 301s. For example, is this a big enough issue to delay the launch of the site / get in a contract dispute with the web developer?
Technical SEO | | stageagent0 -
260k 301 redirects
Hello, I just found that some of the urls on my site have both ugly characters and some other things I'd like to fix (such as ---- into a single - ) After some local tests i've seen that If i leave some imperfections there will be 48k different urls on the other hand if the renaming procedure is strict i'll have around 260k out of 2.3M urls to be renamed. If I'm going to do this I'll create new canonicals meta tag and redirect old urls with 301 headers to the new location. The content will not change. My big doubt is SEO wise, I know that I'll have better urls, but aren't those too much redirects on a single day? what would you do if you wish to have shipshape urls and know some of these are crap? thanks
Technical SEO | | mylittlepwny0 -
Confirming Robots.txt code deep Directories
Just want to make sure I understand exactly what I am doing If I place this in my Robots.txt Disallow: /root/this/that By doing this I want to make sure that I am ONLY blocking the directory /that/ and anything in front of that. I want to make sure that /root/this/ still stays in the index, its just the that directory I want gone. Am I correct in understanding this?
Technical SEO | | cbielich0 -
Should i redirect my lost links to my home page
Hi, as some of you maybe aware, i had a major problem last year that has caused me nothing but trouble. in short, my hosting company lost me over 10,000 pages from my site and i had to rebuild the site from stratch which is still on going. I lost thousands of links to my site and i have been over the past week pointing the pages not found to the sections that is best suited to them. But i am just wondering if it would harm my site if i also point some of those links to my home page. I was a page rank four before disaster happened to my site and now i am a page rank two and i want to build this up. so i am just wondering if i should point some of those good links to my home page i am redirecting the pages using 301 in my htaccess file any advice would be great
Technical SEO | | ClaireH-1848860 -
Redirecting internal pages to another domain
Hi, I recently redirected the domain www.organic7thheaven.com to http://www.7thheavennaturals.com/ However, the internal pages of www.organic7thheaven.com such as http://www.7thheavennaturals.com/products/sensitivesouls/ are now displaying a 404 Not Found Message How can I have it so that pages such as the above automatically also redirect to http://www.7thheavennaturals.com/ Thanks!
Technical SEO | | MJMarketing0