Magento Core_URL_Rewrite Problems
-
Hi Everyone,
We are currently caught between a rock and a hard place with Magento and are wondering if anyone else had similar problems and could share their advice.
Our Core_URL_Rewrite now containt 1.3 million records for an account that has 12000 products on 4 different store views. This has ballooned past the point that we are no longer able to reindex our URL Management.
The option that is being suggested to us is to truncate the table and start over, though this will essentially kill our SEO for those pages.(Which as there are duplicates, I can only imagine how much they are going to be penalized by it)
Would anyone have any advice other than truncating and starting over?
Any advice would be greatly appreciated.
Thanks!
-
Hi,
I found the exact problem you are facing with a solution on this link
http://magento.stackexchange.com/questions/17553/magento-core-url-rewrite-table-excessively-large
There are patch codes available on this link, however do read this reply on this page
Bugs in earlier (and possibly current) versions of Magento is one. Another is there's logic in this table that tries to track changes to the URL key value so that 301/302 rewrites are setup for old products. Because of this, and complicating things, truncating the table and regenerating may make existing URL rewrites go away, and this will have an unknown effect on your search engine listing (not necessity bad, just hard to predict).
My general advice to clients who ask is
-
Leave the giant growing table as is if you don't have a good handle on your URL/SEO situation
-
Until the table size starts being a problem (generating site maps, for example). When that happens, get a handle on your URL/SEO situation.
-
Once you have a handle on your URL/SEO situation, backup the table, then truncate the table and regenerate. Address any URL/SEO problems caused by the truncating.
-
Automate step 3
Trying to fix this on the Magento code level is admirable, but you'll be swimming upstream. Sometimes it's better to accept that "That's just Magento being Magento", and to solve the problem with and external process.
I hope this helps, if you have further questions, then post a response, I will be happy to answer.
Regards,
Vijay
-
-
I'm not sure the answers previously presented are related to the issues you're having. Having worked with Magento for a long time, this can be an issue that occurs over and over again.
To answer your initial question, truncating your core_url_rewrite table will remove all of these URLs, but it'll only delay the problem until it reoccurs again in the future (unless you've had a problem in the past which has been rectified). You're also correct in that any rewrites in the system previously there will disappear, so you'll probably end up with a lot of crawl issues appearing in Search Console.
Your best move would be to find out why you have so many URLs in there in the first place. Do you have a huge product catalog with multiple stores? Or is this something to do with an issue in your Magento version or some setup issues. The most common time this usually occurs is if two products get added to your site with the same URL Key. Every time the reindex process runs, your core_url_rewrite table will grow. You could check this by looking at the number of rows in the table, reindexing the site and if it grows further, then it's likely to be the problem. The quickest way to fix this is to ensure all URL key are unique.
There's also an article here about duplicate keys - https://firebearstudio.com/blog/magento-url-reindex-core_url_rewrite-duplicates-patch.html - this should hopefully clear the issue.
I hope this helps! If it doesn't solve the problem, then sending over a little more information around the number of stores, catalog site and the split between system generated URL rewrites and custom URL rewrites would be great so we can try to help further!
Thanks,
Lewis -
This is an issue to to set-up. When you set up multiple ecommerce websites on Magento as 'Stores', then all SKUs will load on other domains. if they were set-up as 'Websites' then this would alleviate the issue. However, with Stores you are able to share shopping carts (i.e. Add a product from website A and checkout on website B).
What I did was turn off the XML cron jobs and set-up cross-domain canonicals. Also make sure your session IDs (/?SID=) are working properly. Not sure if this solves the technical issues, but should help clear up dupe content.
-
Is it creating a new url for each option (size, color, etc) as well as what page it shows up on or other various sort orders (by price, by size, etc.) and session id's that you could exclude? Are you sure they are truly duplicates?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Htaccess or url rewrite module for Magento 301 redirects?
I need to do about 6000 redirects for a Magento site. The pages no longer exist. I have tried the URL rewrite module but it isn't working for me and I don't want to do 6000 redirects in the htaccess files. Any suggestions?
Intermediate & Advanced SEO | | Tylerj0 -
Magento: Should we disable old URL's or delete the page altogether
Our developer tells us that we have a lot of 404 pages that are being included in our sitemap and the reason for this is because we have put 301 redirects on the old pages to new pages. We're using Magento and our current process is to simply disable, which then makes it a a 404. We then redirect this page using a 301 redirect to a new relevant page. The reason for redirecting these pages is because the old pages are still being indexed in Google. I understand 404 pages will eventually drop out of Google's index, but was wondering if we were somehow preventing them dropping out of the index by redirecting the URL's, causing the 404 pages to be added to the sitemap. My questions are: 1. Could we simply delete the entire unwanted page, so that it returns a 404 and drops out of Google's index altogether? 2. Because the 404 pages are in the sitemap, does this mean they will continue to be indexed by Google?
Intermediate & Advanced SEO | | andyheath0 -
What is true impact of permanent magento redirects?
Hi everyone, I got a tough technical SEO question, that is bugging almost everyone in the (ecommerce) company at the moment. Due to a very "unhealthy" structure of Magento folders, with different countries using same folders in different store views, many of our URL's do change almost on a weekly basis and this is terrifying us. What happens is, that there is a "-numberx"(ex. /category/product-1.html) added to hundreds of URLs so that we are more and more concerned about the impact on SEO. I checked the redirect information with the Moz Toolbar and saw, the following information: http://prntscr.com/81v23e So, even though we had URLs with /category/product-1.html, /category/product-2.html,... the redirect seems to go straight to the last number. My question? -Can this be interpreted as one redirect and therefore it is "less" painful from an SEO point of view?
Intermediate & Advanced SEO | | ennovators
-As we do not have a constant target URL, where does the link juice go if the target page constantly keeps changing (number goes still up) Any advice would be much appreciated. Thanks0 -
A Magento site driving me crazy... any suggestion is appreciated
Hi guys, this time it's me asking for help :D. I have a client with a Magento 1.7.0.0 version site: www.mybomboniere.it I audited it, and found out tons of issues, but the one that worries me more is the fault of canonicalization, which is causing serious duplicated content problems. I'm not new to Magento, hence, the first thing I did was going to: First: Going to System > Configuration > Catalog > Search Engine Optimization and setting on "No" the Use Categories Path for Product URLs voice. Doing so I quit all the duplicated product pages. System > Configuration > Catalog > Catalog > Search Engine Optimization and setting on "yes" the voices Use Canonical Link Meta Tag For Categories and Use Canonical Link Meta Tag For Products. Doing this I should see URLs with sort parameters having the URLs without them as canonical The BIG PROBLEM is that even if I did that, I am still not seeing any rel="canonical" tag added to the code. I've tried to figure out the reason of this, but - sincerely - I cannot find one. Secondly, the client created so many categories and subcategories that - honestly - the best thing would be to start cutting some of them. But one thing is what is correct in theory, another what the client desires, and she does not desire cutting any subcategory.
Intermediate & Advanced SEO | | gfiorelli1
That means that some risk to be a substantial duplicate of others. The correct choice should be to canonicalized the overly identical subcategories to a main one... but this is not possible using the default Magento functions. So, or using an SEO extension (but, which one is the best for Magento 1.7.0.0.? Yoast plugins seem outdated), or using a solution like the second option proposed in this post: http://www.adurolabs.com/blog/technical/how-to-add-rel-canonical-in-magento. The doubt is that the post is presenting it in case of products pages, not categories ones. Hence, is it correct also for them, or do you have others suggestions. Sorry for the long question, but any help will be much appreciated :). Ciao Gianluca0 -
We will be switching our shopping cart platform from volusion to magento and really cautious / nervous about our rankings / seo stuff. any advice for anyone that has migrated stores, etc. these urls are years old, etc.
shopping cart platform switch and SEO. What do you suggest? What's the best way to ensure we keep rankings.
Intermediate & Advanced SEO | | PaulDylan0 -
Are menu links with the same anchor text but different destinations a problem?
My website uses the same anchor text to go to different pages, with section headers indicating where they go. Is this a problem? f so, why? Here is an example of what I mean: Women Shoes - link to women shoes Clothes - link to women clothes Accessories - link to women accessories Men Shoes - link to men shoes Clothes - link to men clothes Accessories - link to men accessories Kids Shoes - link to kids shoes Clothes - link to kids clothes Accessories - link to kids accessories
Intermediate & Advanced SEO | | theLotter0 -
Is Affiliate masking a problem for Google?
Does Google consider affiliate masking as unethical? I have a offer website that has 1000's of affiliate links that are masked. I feel google does not like masking URL's which is why my ranking started dropping. Can any one explain the context of affiliate masking please?
Intermediate & Advanced SEO | | SEOMad0 -
Have a problem with our home page. Is temporary 301 redirect an option?
Hey Mozers, I discovered this morning that the home page for my website is rendering fine in Chrome and Firefox, but very poorly in IE. My analytics show that over 50% of my visitors are using IE. As a result of the problem, IE has a bounce rate 32% higher than other browsers. I'm not a web developer and I'm fairly new to SEO, so I'm guessing that it's going to take me at least a couple days to get it fixed. In the meantime, I was considering doing a 301 redirect from the home page to the largest category page in hopes of keeping some of the IE users from bouncing while I get the home page sorted out. Would there be any long term negative effects from this once I get the page sorted out and take the 301 off it? Are there any other solutions that would be better? Thanks for the help!
Intermediate & Advanced SEO | | matthewbyers0