Robots.txt and 301
-
Hi Mozzers,
Can you answer something for me please.
I have a client and they have 301 re-directed the homepage '/' to '/home.aspx'.
Therefore all or most of the linkjuice is being passed which is great. They have also marked the '/' as nofollow / noindex in the Robots.txt file so its not being crawled.
My question is if the '/' is being denied access to the robots is it still passing on the authority for the links that go into this page? It is a 301 and not 302 so it would work under normal circumstances but as the page is not being crawled do I need to change the Robots.txt to crawl the '/'?
Thanks
Bush
-
Thanks Yannick - really helpful
cheers and have a good weekend
Bush
-
Yes. Remove the noindex nofollow for / from the robots.txt. It doesn't add anything. It can only confuse SE's and lose you rankings.
-
Hi Yannick.
Thanks for your reply. The / redirect to /home redirect has been done and the / is excluded.
I'm limited as to what changes I can make. I can amend the robots.txt to crawl / which is 301's to /home and that way the links will be attributed and the 301 remains.
Will that be beneficial do you think?
Bush
-
I would suggest not excluding the / in the robots.txt.
First you redirect the / to home.aspx but then you tell SE's to exclude the home page which you are redirecting anyway. Makes no sense to me, so I would remove it in the robots.txt. No use anyway, because the url is being redirected.
I wouldn't use the 301 redirect to home.aspx. Why not make some (technical?) modifications so that the home page is available on the root of the domain? It's much more beneficial to you site's SEO. Or the other way around, redirect home.aspx to /. Because way more ppl will be linking to the root and not the home.aspx page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt vs. meta noindex, follow
Hi guys, I wander what your opinion is concerning exclution via the robots.txt file.
Technical SEO | | AdenaSEO
Do you advise to keep using this? For example: User-agent: *
Disallow: /sale/*
Disallow: /cart/*
Disallow: /search/
Disallow: /account/
Disallow: /wishlist/* Or do you prefer using the meta tag 'noindex, follow' instead?
I keep hearing different suggestions.
I'm just curious what your opinion / suggestion is. Regards,
Tom Vledder0 -
Redirecting 301 or 302?
Hi, I think the part of this question has been already discussed, but not exactly the same, I think. My site requires authentication for member page. When a user try to go to member area, we redirect to 3rd party to do the authentication. 1. user clicks a link to www.mysite.com/member/contents.html
Technical SEO | | HypermediaSystems
2. www.myauthenticate.com/login?h=somehashuniquehash454859428778545 (enters id/pass)
3. login success => redirect back to www.mysite.com/member/contents.html We are doing it 302, temporary redirect. But moz crawler error seems to suggest we should do it 301.
So my question is:
A. Should we do it 301???
B. If we do 301, what happens to myauthenticate.com? since it has hashtag, I am afraid it could create a lot of duplicate contents on myauthenticate.com side... Thank you so much for your help in advance...0 -
IIS 7.5 - Duplicate Content and Totally Wrong robot.txt
Well here goes! My very first post to SEOmoz. I have two clients that are hosted by the same hosting company. Both sites have major duplicate content issues and appear to have no internal links. I have checked this both here with our awesome SEOmoz Tools and with the IIS SEO Tool Kit. After much waiting I have heard back from the hosting company and they say that they have "implemented redirects in IIS7.5 to avoid duplicate content" based on the following article: http://blog.whitesites.com/How-to-setup-301-Redirects-in-IIS-7-for-good-SEO__634569104292703828_blog.htm. In my mind this article covers things better: www.seomoz.org/blog/what-every-seo-should-know-about-iis. What do you guys think? Next issue, both clients (as well as other sites hosted by this company) have a robot.txt file that is not their own. It appears that they have taken one client's robot.txt file and used it as a template for other client sites. I could be wrong but I believe this is causing the internal links to not be indexed. There is also a site map, again not for each client, but rather for the client that the original robot.txt file was created for. Again any input on this would be great. I have asked that the files just be deleted but that has not occurred yet. Sorry for the messy post...I'm at the hospital waiting to pick up my bro and could be called to get him any minute. Thanks so much, Tiff
Technical SEO | | TiffenyPapuc0 -
How to know what pages are 301 redirecting to me?
Hi! It is easy to know if somebody is spam linking your website, looking i.e., looking at open site explorer to analyse the links profile. But, is it possible to know if a competitor of mine is redirecting a bad domain to main with a 301 redirect, thus transfering any bad SEO reputation to me? Best Regards, Daniel
Technical SEO | | te_c0 -
Best use of robots.txt for "garbage" links from Joomla!
I recently started out on Seomoz and is trying to make some cleanup according to the campaign report i received. One of my biggest gripes is the point of "Dublicate Page Content". Right now im having over 200 pages with dublicate page content. Now.. This is triggerede because Seomoz have snagged up auto generated links from my site. My site has a "send to freind" feature, and every time someone wants to send a article or a product to a friend via email a pop-up appears. Now it seems like the pop-up pages has been snagged by the seomoz spider,however these pages is something i would never want to index in Google. So i just want to get rid of them. Now to my question I guess the best solution is to make a general rule via robots.txt, so that these pages is not indexed and considered by google at all. But, how do i do this? what should my syntax be? A lof of the links looks like this, but has different id numbers according to the product that is being send: http://mywebshop.dk/index.php?option=com_redshop&view=send_friend&pid=39&tmpl=component&Itemid=167 I guess i need a rule that grabs the following and makes google ignore links that contains this: view=send_friend
Technical SEO | | teleman0 -
301 Redirecting weird URLs with % in them
I've been working on redirecting links reported as 404 in Google webmaster tools. I've stumbled upon 41 URLs that Google is reporting as 404 that include a '%' in the URL, but I don't know how to redirect. Here is an example: URL: bond_information.htm%20Surety%20Bond%20Information,%20with%20FAQ Attempted redirect: redirect 301 /bond_information.htm%20Surety%20Bond%20Information,%20with%20FAQ http://www.mysite.com/ Unfortunately, after implementing the redirect, http://www.mysite.com/bond_information.htm%20Surety%20Bond%20Information,%20with%20FAQ still resolves a 404 error. Anyone successfully fix these errors using Apache .htaccess?
Technical SEO | | TheDude0 -
Robots.txt
should I add anything else besides User-Agent: * to my robots.txt file? http://melo4.melotec.com:4010/
Technical SEO | | Romancing0 -
301 Redirect Issue
I'm having an issue with 301 redirects: Let's see if I can verbalize my thoughts on this one... So we just recently moved our site to Wordpress. One of our new 301 commands is redirecting oursite.com/news to oursite.com/blog . However there are other links from our previous site that look like oursite.com/news/XYZ and the issue is that, because wordpress structures its links differently, that URL is not equivalent to oursite.com/blog/XYZ. Instead, it might look something more like oursite.com/blog/yaddayadda/XYZ or something. Does that make sense? The issue is that when I find an old link of ours on google that looks something like "oursite.com/news/XYZ" or "oursite.com/news/ABC" it is automatically replacing "news" with "blog". When I try to go in manually and redirect anything that says "/news/XYZ" to "/blog/yaddayadda/XYZ" it still doesn't work. It still just replaces "news" with "blog." Wow I realize that might not make sense to anyone but if it does - please advise!! Thanks!!!!
Technical SEO | | EntrustSEO0