Robots.txt and 301
-
Hi Mozzers,
Can you answer something for me please.
I have a client and they have 301 re-directed the homepage '/' to '/home.aspx'.
Therefore all or most of the linkjuice is being passed which is great. They have also marked the '/' as nofollow / noindex in the Robots.txt file so its not being crawled.
My question is if the '/' is being denied access to the robots is it still passing on the authority for the links that go into this page? It is a 301 and not 302 so it would work under normal circumstances but as the page is not being crawled do I need to change the Robots.txt to crawl the '/'?
Thanks
Bush
-
Thanks Yannick - really helpful
cheers and have a good weekend
Bush
-
Yes. Remove the noindex nofollow for / from the robots.txt. It doesn't add anything. It can only confuse SE's and lose you rankings.
-
Hi Yannick.
Thanks for your reply. The / redirect to /home redirect has been done and the / is excluded.
I'm limited as to what changes I can make. I can amend the robots.txt to crawl / which is 301's to /home and that way the links will be attributed and the 301 remains.
Will that be beneficial do you think?
Bush
-
I would suggest not excluding the / in the robots.txt.
First you redirect the / to home.aspx but then you tell SE's to exclude the home page which you are redirecting anyway. Makes no sense to me, so I would remove it in the robots.txt. No use anyway, because the url is being redirected.
I wouldn't use the 301 redirect to home.aspx. Why not make some (technical?) modifications so that the home page is available on the root of the domain? It's much more beneficial to you site's SEO. Or the other way around, redirect home.aspx to /. Because way more ppl will be linking to the root and not the home.aspx page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I add my html sitemap to Robots?
I have already added the .xml to Robots. But should I also add the html version?
Technical SEO | | Trazo0 -
301 redirect file question
Hi Everyone, I am creating a list of 301 redirects to give to a developer to put into Magento. I used Screaming Frog to crawl the site, but I have noticed that all of their urls 302 to another page. I am wondering if I should 301 the first URL to the url on the new site, or the second. I am thinking the first, but would love some confirmation. Thank you!
Technical SEO | | mrbobland0 -
301 redirect question
Hi Everyone When doing 301 redirects for a large site, if a page has 0 inbound links would you still redirect it or just leave it? Im just curious on the best practice for this Thanks in advance
Technical SEO | | TheZenAgency0 -
Google Indexing Development Site Despite Robots.txt Block
Hi, A development site that has been set-up has the following Robots.txt file: User-agent: * Disallow: / In an attempt to block Google indexing the site, however this isn't the case and the development site has since been indexed. Any clues why this is or what I could do to resolve it? Thanks!
Technical SEO | | CarlWint0 -
301 Redirect Best Practices
Hi SEOs, Question about ranking/redirects. If I have a particular page that is already ranking for a couple KWs in top SERPs, but know there are higher volume KWs I can optimize for should I just leave it as is or change the URL key and redirect for the time being until Google re-indexes. Example:
Technical SEO | | IceIcebaby
current URL: www.example.com/action/best-movies
new URL: www.example.com/action/best-action-movies
(the current would be ranking for "best action moves" whereas the new would include the actual "best action movies" KW) Let me know if I can clarify, thank you!0 -
Canonical tag or 301
Hi, Our crawl report is showing duplicate content. some of the report I am clear about what to do but on others I am not. Some of the duplicate content arises with a 'theme=default' on the end of the URL. Is this version of a page necessary for people to see when they visit the site (like a theme=print page is) in which case I think we should use a canonical tag, or is it not necessary in which case we should use a 301? Thanks
Technical SEO | | Houses0 -
Can't find mistake in robots.txt
Hi all, we recently filled our robots.txt file to prevent some directories from crawling. Looks like: User-agent: * Disallow: /Views/ Disallow: /login/ Disallow: /routing/ Disallow: /Profiler/ Disallow: /LILLYPROFILER/ Disallow: /EventRweKompaktProfiler/ Disallow: /AccessIntProfiler/ Disallow: /KellyIntProfiler/ Disallow: /lilly/ now, as Google Webmaster Tools hasn't updated our robots.txt yet, I checked our robots.txt in some ckeckers. They tell me that the User agent: * contains an error. **Example:** **Line 1: Syntax error! Expected <field>:</field> <value></value> 1: User-agent: *** **`I checked other robots.txt written the same way --> they work,`** accordign to the checkers... **`Where the .... is the mistake???`** ```
Technical SEO | | accessKellyOCG0 -
301 an old URL with a ? in the URL?
I am redoing a site and the URL's are changing structure. The client's site was in magento and in the store they would get two URLs, for example: /store/categoryname/productname and /store/categoryname/productname?SID=dslkajsfdoiu947598whouieht983hg98 Do I have to 301 redirect both of these URL's to their new counterpart? Both go to the same content but magento seemed to add these SIDs into the navigation and Google has both versions in the index.
Technical SEO | | DanDeceuster0