How best to fix 301 redirect problems
-
Hi all
Wondering if anyone could help out with this one. Roger Bot crawler has just performed it's weekly error crawl on my site and I appear to have 18,613 temp redirect problems!! Rather, the same 1 problem 18,613 times.
My site is a magento store and the errors it is giving me is due to the wishlist feature on the site. For example, it is trying to crawl links such as index.php/wishlist/index/add/product/29416/form_key/DBDSNAJOfP2YGgfW (which would normally add the item to one's wishlist). However, because Roger isn't logged into the website it means that all these requests are being sent to the login url with the page title of Please Enable Cookies.
Would the best way to fix this be to enable wishlists for guests? I would rather not do that but cannot think of another way of fixing it.
Any other Magento people come across this issue?
Thanks, Carl
-
thanks. I completely forgot about the robots.txt method of fixing it. Will give it a go. I will add comparison in there too just to be safe.
-
Hey Carl
I have this problem a lot on my Magento stores since theres a ton of ways to access the same product and it can be an SEO nightmare. If you have FTP access I would recommend adding this to the robots.txt file:
Disallow: /wishlist/
This will stop the bots from crawling everything passed /wishlist/. Here's a good resource if you need further assistance: http://www.magentocommerce.com/knowledge-base/entry/setting-up-a-robotstxt-file
Hopefully that helps!
Good luck
-
Thanks will try and work out how to do that. Magento is a pain to make minor changes to.
Hopefully the crawl stats will look a little healthier next week !!
-
Use Rel="no index, no follow" on the link and roger will ignore the link. Job done
You wouldn't get any advantage of seeing the added items in someones wishlist as it would be duplicate content I'm presuming.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Selective 301 redirections of pages within folders
Redirection Puzzle - it's got me puzzled anyhow! The finished website has just been converted from an old aspx affair to a wordpress site. Some directory structures have changed significantly; there appears to be a load of older medical articles that have not been added back in and it sounds unlikely that they will be. Therefore unmatched old news articles need to be pointed to the top news page to keep hold of any link value they may have accrued. The htaccess file starts with ithemes security's code, Followed by the main wordpress block and I have added the user redirects to the final section of the htaccess file . I have been through the redirects and rewrites line by line to verify them and the following sections are giving me problems. This is probably just my aging brain failing to grasp basic logic. If I can tap into anybody's wisdom for a bit of help I would appreciate it. My eyes and brain are gone to jelly. I have used htaccesscheck.com to check out the underlying syntax and ironed out the basic errors that I had previously missed. The bulk of the redirects are working correctly. #Here there are some very long media URLs which are absent on the new site and I am simply redirecting visiting spiders to the page that will hold media in future. Media items refuse to redirect
Technical SEO | | TomVolpe
Line 408 redirect 301 /Professionals/Biomedicalforum/Recordedfora/Rich%20Media%20http:/kplayer.kcl.ac.uk/ess/echo/presentation/15885525-ff02-4ab2-b0b9-9ba9d97ca266 http://www.SITENAME.ac.uk/biomedical-forum/recorded-fora/ Line 409 redirect 301 /Professionals/Biomedicalforum/Recordedfora/Quicktime%20http:/kplayer.kcl.ac.uk/ess/echo/presentation/15885525-ff02-4ab2-b0b9-9ba9d97ca266/media.m4v http://www.SITENAME.ac.uk/biomedical-forum/recorded-fora/ Line 410 redirect 301 /Professionals/Biomedicalforum/Recordedfora/Mp3%20http:/kplayer.kcl.ac.uk/ess/echo/presentation/15885525-ff02-4ab2-b0b9-9ba9d97ca266/media.mp3 http://www.SITENAME.ac.uk/biomedical-forum/recorded-fora/ #Old site pagination URLs redirected to new "news" top level page - Here I am simply pointing all the pagination URLs for the news section, that were indexed, to the main news page. These work but append the pagination code on to the new visible URL. Have I got the syntax correct in this version of the lines to suppress the appended garbage? RewriteRule ^/LatestNews.aspx(?:.*) http://www.SITENAME.ac.uk/news-events/latest-news/? [R=301,L] #On the old site many news directories (blog effectively) contained articles that are unmatched on the new site, have been redirected to new top level news (blog) page: In this section I became confused about whether to use Redirect Match or RewriteRule to point the articles in each year directory back to the top level news page. When I have added a redirectmatch command - it has been disabling the whole site! Despite my syntax check telling me it is syntactically correct. Currently I'm getting a 404 for any of the old URLs in these year by year directories, instead of a successful redirect. I suspect Regex lingo is not clicking for me 😉 My logic here was rewrite any aspx file in the directory to the latest news page at the top. This is my latest attempt to rectify the fault. Am I nearer with my syntax or my logic? The actual URLs and paths have been substituted, but the structure is the same). So what I believe I have set up is: in an earlier section; News posts that have been recreated in the new site are redirected 1 - 1 and they are working successfully. If a matching URL is not found, when the parsing of the file reaches the line for the 1934 directory it should read any remaining .aspx URL request and rewrite it to the latest news page as a 301 and stop processing this block of commands. The subsequent commands in this block repeat the process for the other year groups of posts. Clearly I am failing to comprehend something and illumination would be gratefully received. RewriteRule ^/Blab/Blabbitall/1934/(.*).aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] #------Old site 1933 unmatched articles redirected to new news top level page RewriteRule ^/Blab/Blabbitall/1933/(.*).aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] #------Old site 1932 unmatched articles redirected to new news top level page RewriteRule ^/Blab/Blabbitall/1932/(.*)/.aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] #------Old site 1931 unmatched articles redirected to new news top level page RewriteRule ^/Blab/Blabbitall/1931/(.*)/.aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] #------Old site 1930 unmatched articles redirected to new news top level page RewriteRule ^/Blab/Blabbitall/1930/(.*)/.aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] Many thanks if anyone can help me understand the logic at work here.0 -
301 Redirects
Hi, I have switched my site from a http .co.uk site to a https .com site. I have set a 301 redirect in the .htaccess file pointing all traffic going to the original .co.uk site to go to the new https: RewriteEngine on
Technical SEO | | imoprojects
RewriteCond %{HTTP_HOST} ^up-bus.co.uk$ [OR]
RewriteCond %{HTTP_HOST} ^www.up-bus.co.uk$
RewriteRule ^(.*)$ "https://www.up-bus.com/$1" [R=301,L] however when i search in google for keywords the original .co.uk site is still registering in search, is there something else I am required to do to tell google to use the new https site instead? Do i need to do redirects for every page, or is what I have done above sufficient? Hope you can help, I am struggling with getting our site to register on google search, any advice greatly welcome Thanks in advance, Ian0 -
301 Redirect Question
I am working with a website and I ran a Screaming Frog and noticed there are 4,600 301's on the website (www.srishoes.com). It seems like the issue is between the www. and without it and they aren't working together. Is this something that the website provider should update and what type of impact might this have on the site? Thanks!
Technical SEO | | ReunionMarketing
Matt0 -
HTacess 301 redirect with special characters
Hello moz community ! I would to make a special 301 redirection through my htaccess file. I am a total noob concerning regexp and 301 redirection. I would like to redirect(301) this url : http://www.legipermis.com/stages-points/">http://www.legipermis.com/stages-points/</a></p>; yes yes it's in the index of google, this strange url includes the last ; to http://www.legipermis.com/stages-points/ I have already include a canonical tag by security, i would like to remove url with a 301 redirection and by remove this url through GWT (but the removal tool can't "eat' this kind of URL) Please consider the fact that i am not an expert about 301 redirections and regexps. No 301 redirect generator works properly for such a strange URL (which triggers content duplication corrected anyway with canonical tag). Thanks for your help.
Technical SEO | | LegiPermis0 -
I need to know more clearance on rel=canonical usage than 301 redirects ?
Hi all SEOmozs, As we all know purposes of rel=canonical , I have a query to ask that If we don't have any possibility to use 301 redirects on a domain , can it be really right to use rel=canonical on an old domain to let search engine to treat those all pages should be not priority where the domain we are being promoted in the market to list up instead that. I found this interesting Matt Cutts video http://www.youtube.com/watch?v=gJK5Uloy76g where he has told or cleared the point very nicely, yes we can use it if there is no possibility in your older domain or pages. So here i am asking the same to know more detailed clarity on this so that i can be more confidence on it. I have been seeing issues in my domains where old one domain comes than new domain why with new domain contents, and can it be really very good to bring new domain with **rel=canonical without using 301 redirect :
Technical SEO | | Futura
Old : kanin.com (leaving) New : kangarokanin.com (promoting) Where i might have not used yet the rel=canonical in old domain, will be going to use it soon , after finishing this discussion.** Regards,
Teginder Ravi tcSnN.jpg tcSnN.jpg dGd34.jpg0 -
302 or 301 redirect to https ?
I am redirecting whole site to https. Is there a difference between 302 or 301 redirect for seo? Site never been indexed. Planning to do that with .htaccess command RewriteCond %{HTTPS} !=on
Technical SEO | | Kotkov
RewriteRule ^(.*) https://%{SERVER_NAME}/$1 [R,L] There are plenty of ways http://www.askapache.com/htaccess/ssl-example-usage-in-htaccess.html Which way would be the best? Thanks is advance0 -
301 redirect: Old pages to new
Hi, We have recently made some changes to our agency site. Looking in webmaster tools we have identified a number of old pages with existing link juice. Not a great deal mostly 32/100 PA. There are a mixture of URLs "meet the team" and people pages etc. The anchor text on the majority of pages is our brand name. Could we now 301 all these pages to one page, or is this a no no in the eyes of Google? Any help greatly appreciated. Best Regards Sean
Technical SEO | | Yozzer0
