Robots.txt and 301
-
Hi Mozzers,
Can you answer something for me please.
I have a client and they have 301 re-directed the homepage '/' to '/home.aspx'.
Therefore all or most of the linkjuice is being passed which is great. They have also marked the '/' as nofollow / noindex in the Robots.txt file so its not being crawled.
My question is if the '/' is being denied access to the robots is it still passing on the authority for the links that go into this page? It is a 301 and not 302 so it would work under normal circumstances but as the page is not being crawled do I need to change the Robots.txt to crawl the '/'?
Thanks
Bush
-
Thanks Yannick - really helpful
cheers and have a good weekend
Bush
-
Yes.
Remove the noindex nofollow for / from the robots.txt. It doesn't add anything. It can only confuse SE's and lose you rankings. -
Hi Yannick.
Thanks for your reply. The / redirect to /home redirect has been done and the / is excluded.
I'm limited as to what changes I can make. I can amend the robots.txt to crawl / which is 301's to /home and that way the links will be attributed and the 301 remains.
Will that be beneficial do you think?
Bush
-
I would suggest not excluding the / in the robots.txt.
First you redirect the / to home.aspx but then you tell SE's to exclude the home page which you are redirecting anyway. Makes no sense to me, so I would remove it in the robots.txt. No use anyway, because the url is being redirected.
I wouldn't use the 301 redirect to home.aspx. Why not make some (technical?) modifications so that the home page is available on the root of the domain? It's much more beneficial to you site's SEO. Or the other way around, redirect home.aspx to /. Because way more ppl will be linking to the root and not the home.aspx page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 Redirects
Hello, All. Hopefully this will be an easy question for some of you. I have a (WordPress) site with the format of: http://www.site.com/folder-old/page-old/ I have since re-named both the parent and sub-folders to (example): http://www.site.com/folder-new/page-new/ Everything is working well EXCEPT I am also trying to redirect visitors from the old URL/structure to the new. I have a 301 redirect setup as the following: Redirect 301 /folder-old/page-old/Â http://www.site.com/folder-new/page-new/ -- But it doesn't seem to be working. Not sure if this is something finicky with WordPress or if the redirect is incorrect. Thank you.
Technical SEO | | BrandBuilder0 -
The use of robots.txt
Could someone please confirm that if I do not want to block any pages from my URL, then I do not need a robots.txt file on my site? Thanks
Technical SEO | | ICON_Malta0 -
260k 301 redirects
Hello, I just found that some of the urls on my  site have both ugly characters and some other things I'd like to fix (such as ---- into a single - ) After some local tests i've seen that If i leave some imperfections there will be 48k different urls  on the other hand if the renaming procedure is strict i'll have around 260k out of 2.3M urls to be renamed. If I'm going to do this I'll create new canonicals meta tag and redirect old urls with 301 headers  to the new location. The content will not change. My big doubt is SEO wise, I know that I'll have better urls, but aren't those too much redirects on a single day? what would you do if you wish to have shipshape urls and know some of these are crap? thanks
Technical SEO | | mylittlepwny0 -
Robots.txt and joomla
Hello, I use joomla for my website and automatically all those files are blocked is that good or bad, so I remove anything and if so why ? User-agent: *
Technical SEO | | seoanalytics
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/ I also added to my robots.txt files my email address ( is that useful, I am afraid google passes PR to the email address )
and a javascript: void (0) because I have tabs on my webpage ( is that useful )
as well as a .pdf ( is it also useful ) any comments ? does anything need to be changed or is it ok ? Thank you,0 -
Confirming Robots.txt code deep Directories
Just want to make sure I understand exactly what I am doing If I place this in my Robots.txt Disallow: /root/this/that By doing this I want to make sure that I am ONLY blocking the directory /that/ and anything in front of that. I want to make sure that /root/this/ still stays in the index, its just the that directory I want gone. Am I correct in understanding this?
Technical SEO | | cbielich0 -
301 redirects
Hi, I am a working on a new web site, and I want to redirect all the urls of another site (on a different host) to this one. According to both hosts it is "impossible" to do this for all urls. I don't believe that to be the case, but how do I do this? And, should both sites be hosted on the same server first?
Technical SEO | | vibelingo0 -
301 redirect to new website
We are migrating to a new website that will be using entirely new URLs under the same domain as the old website. The old website is a custom PHP script and the new website uses Drupal. I know that I should use individual 301 redirects to the corresponding new pages. My question is just how to set up the hundreds of 301 redirects from the old website to the new one? Here is the process I've come up with. Please let me know if there is an easier and better way for this. Before actually changing to the new website: download an advanced report with all pages on this domain from OSE. Find corresponding pages on the new website Make the hundreds of 301 redirect lines in an .htaccess file with the following code: redirect 301 /oldurl.html http://domain.com/the-full-url Thanks in advance for your help!
Technical SEO | | qbeeker0 -
Prestashop, language and 301
Hello, A question about Prestashop: I noticed that there are some Prestashop websites, with only one language, that work with a redirection from the main domain (for exemple, www.domain-name.com) with a 301 redirection to the main language url (for example, www.domain-name.com/fr ) Here is an example: http://prestaplayers.com/fr/ Assuming that French is the only language on that website, and knowing that links are generally made to www.domain-name.com  (and not to www.domain-name.com/fr) I assume that some netlinking juice is lost for no reason, due to the 301 redirect ? In this case, and if the website exists for some years, what should be done ? Change all the urls because of that problem ? Or maybe change the stronger links to make them point to the real address (www.domain-name.com/fr) ? Thank you in advance for yout help dans suggestions !
Technical SEO | | Spleen0
