Robots.txt and 301
-
Hi Mozzers,
Can you answer something for me please.
I have a client and they have 301 re-directed the homepage '/' to '/home.aspx'.
Therefore all or most of the linkjuice is being passed which is great. They have also marked the '/' as nofollow / noindex in the Robots.txt file so its not being crawled.
My question is if the '/' is being denied access to the robots is it still passing on the authority for the links that go into this page? It is a 301 and not 302 so it would work under normal circumstances but as the page is not being crawled do I need to change the Robots.txt to crawl the '/'?
Thanks
Bush
-
Thanks Yannick - really helpful
cheers and have a good weekend
Bush
-
Yes. Remove the noindex nofollow for / from the robots.txt. It doesn't add anything. It can only confuse SE's and lose you rankings.
-
Hi Yannick.
Thanks for your reply. The / redirect to /home redirect has been done and the / is excluded.
I'm limited as to what changes I can make. I can amend the robots.txt to crawl / which is 301's to /home and that way the links will be attributed and the 301 remains.
Will that be beneficial do you think?
Bush
-
I would suggest not excluding the / in the robots.txt.
First you redirect the / to home.aspx but then you tell SE's to exclude the home page which you are redirecting anyway. Makes no sense to me, so I would remove it in the robots.txt. No use anyway, because the url is being redirected.
I wouldn't use the 301 redirect to home.aspx. Why not make some (technical?) modifications so that the home page is available on the root of the domain? It's much more beneficial to you site's SEO. Or the other way around, redirect home.aspx to /. Because way more ppl will be linking to the root and not the home.aspx page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 Redirect Review Nodes
I have a client who rents out vacation beach rentals. They currently have thousands of homes under management. Each property has its own internal reviewing platform. Reviews are not really intended to be viewed on their own, as in a stand alone page with just the review on it. The problem is that Drupal makes just about every type of node viewable on its own dedicated URL. I was just thinking about taking request to view stand alone reviews and 301’ing them to their respective property page, the context in which they are intended to be viewed. The website has about 2500 review nodes currently crawlable via Drupal that sit on their own URLs. Would there be a material impact to 301 them to their respective property page when any attempt to view them on their own is made to the site?
Technical SEO | | conversionpipeline20 -
Robots.txt vs. meta noindex, follow
Hi guys, I wander what your opinion is concerning exclution via the robots.txt file.
Technical SEO | | AdenaSEO
Do you advise to keep using this? For example: User-agent: *
Disallow: /sale/*
Disallow: /cart/*
Disallow: /search/
Disallow: /account/
Disallow: /wishlist/* Or do you prefer using the meta tag 'noindex, follow' instead?
I keep hearing different suggestions.
I'm just curious what your opinion / suggestion is. Regards,
Tom Vledder0 -
301 redirect homepage question
Hi If i have a homepage which is available at both www.homepage.com and www.homepage.com// should i 301 the // version to the first version. Im curious as to whether slashes are taking into consideration Thanks in advance
Technical SEO | | TheZenAgency0 -
Site Penalized - 301 Redirect Question
Hello, We have a website that was penalized roughly two years by Google for "Unnatural Links"... We are experiencing a lot of problems with this site, completely unrelated to the penalty or SERPS, and we're debating doing a 301 Re-direct to another site we own that is totally clean and has no "Unnatural Links". If we do a 301 from the penalized site to our alternative website, will there be any cross-contamination? Will the penalty carry over to our other site? Please let me know what you guys think. Thanks
Technical SEO | | Prime850 -
Is having no robots.txt file the same as having one and allowing all agents?
The site I am working on currently has no robots.txt file. However, I have just uploaded a sitemap and would like to point the robots.txt file to it. Once I upload the robots.txt file, if I allow access to all agents, is this the same as when the site had no robots.txt file at all; do I need to specify crawler access on can the robots.txt file just contain the link to the sitemap?
Technical SEO | | pugh0 -
Robots.txt
Hello Everyone, The problem I'm having is not knowing where to have the robots.txt file on our server. We have our main domain (company.com) with a robots.txt file in the root of the site, but we also have our blog (company.com/blog) where were trying to disallow certain directories from being crawled for SEO purposes... Would having the blog in the sub-directory still need its own robots.txt? or can I reference the directories i don't want crawled within the blog using the root robots.txt file? Thanks for your insight on this matter.
Technical SEO | | BailHotline0 -
Joomla 301 redirects
hi i am using joomla sef404 and i have 7000 not found pages in my webmaster google tool page. the trouble is i got rid of a lot of pages and also components so now i am left with loads of not found pages. what i want to try and do is to create 301 redirects so that i do not lose page rank. can anyone please let me know how to do this step by step please
Technical SEO | | ClaireH-1848860