Magento Dynamic Pages Being Indexed
-
Hi there, I have about 50k Moz medium priority errors in my Crawl Diagnostic report.
The bulk of them are classified as "Temporary Redirect" problems. Then if you drill into those further, I can see that the problem urls all kinda are center around:
mysite.com/catalogsearch/result..
mysite.com/wishlist..
mysite.com/catalog..
Is this something I should disallow in my Robstxt file? And if so how specific do I get with it..
Disallow /catalogsearch/result/?q=
Will listing the /catalogsearch be enough to cover anything after it?
thanks
-
Jeff is completely right on this one!
-
Thank you very much Jeff!
-
Yes, Magento does have a way of handling search results that are not always SEO-friendly.
To restrict Google from indexing anything in the /catalogsearch/ directory, I'd add this to your robots.txt file:
User-agent: *
Disallow: /catalogsearch/Here's a Moz blog about the robots.txt file:
http://moz.com/blog/interactive-guide-to-robots-txtAnd also, just in case the "killer" robots ever take over, Google has made sure that their two founders are not able to be indexed by both the T-1000 and T-800 models, with a special killer-robots.txt file:
http://www.google.com/killer-robots.txtHope this helps!
-- Jeff
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pages with URL Too Long
Hello Mozzers! MOZ keeps kindly telling me the URLs are too long. However, this is largely due to the structure of E-commerce site, which has to include 'brand' 'range' and 'products' keyword. For example -
Moz Pro | | tigersohelll
https://www.choicefurnituresuperstore.co.uk/Devonshire-Rustic-Oak-Bedside-Cabinet-1-Drawer-p40668.html MOZ recommends no more than 75 characters. This means we have 25-30 characters for both the brand name and product name. Questions:
If it is an issue, how to fix it on my site?
If it's not an issue, how can we turn off this alert from MOZ?
Anyone know how big an issue URLs are as a ranking factor? I thought pretty low.0 -
Moz is treating my pages as duplicate content but the pages have different content in reality
Attached here is a screenshot of links with duplicate content. Here are some links that is also based on the screenshot http://federalland.ph/construction_updates/paseo-de-roces-as-of-october-2015 http://federalland.ph/construction_updates/sixsenses-residences-tower-2-as-of-october-2015/ http://federalland.ph/construction_updates/sixsenses-residences-tower-3-as-of-october-2015 The links that I have placed here have different content. So I don't why they are treated as duplicates BWWJuvQ
Moz Pro | | clestcruz0 -
Pages to Optimise Issues
HI, Can someone tell me why 2 A pages are showing up in red in the F Grade page ? I see Page title missing from the output and just curious if I need to addrss some issue. xpOW4tY.jpg
Moz Pro | | ecrmeuro0 -
How can I deal with tag page duplicate issues
The Moz crawler reported some dupliated issues. Many of them have to do with tags.
Moz Pro | | IamKovacs
Each tag has a link, and as some articles are under several tags, these come up as duplicate content. I read Dr Peter's piece on Canonical stuff, but it's not clear to me if any of these are the solution. Perhaps the solution lies somewhere else? Maybe I need to block the robots from these urls (But that seems counter-SEO-productive) Thanks
Kovacs0 -
Crawl Diagnostics - Crawling way more pages than my site has?
Hello all, I'm fairly new here, more of a paid search guy dabbling in SEO on the side. I have a client that I have in SEOMoz and the Crawl Diagnostics report is showing 10,000+ pages crawled and I think the site has at most 800 pages (e-commerce site using freewebstore.org as the platform). Any reasons this would be happening?
Moz Pro | | LodestoneGen0 -
Wrong duplicated page content
I found out that some errors on my website are considered as "duplicated page content" while they are not, the content is different on each page. I wonder why ? Is it an issue from Seomoz ?
Moz Pro | | Amadeus_eBC0 -
4xx status code a page that cannot be accessed..
All the error is because the Danish letters "åøæ" in the url.. But i can access all pages, therefor the error isent a true error!!! Why is it an Issue, Google can read "øæå", why cant SEOMOZ??
Moz Pro | | seopeter290 -
Strange Ranking On 1st Page Of Google For Competitive Keyword
Hey all, So, I've noticed a site ranking on the first page in Google for a keyword that I have been targeting for years. The keyword is "mixtapes" and pretty competitive. The site, http://www.newyorkmixtapes.com is ranking around #4 in Google. It seemed odd to me so I looked into the site using some of the tools here. Open Explorer showed that they have minimal links. The actual site is pretty spammy also. Does anyone have any idea how this site ranked so high so fast? Is it ranking because I am located in New York? Does Google still put lots of weight on exact match domains? It's just been bothering me because it's a keyword that I put a lot of work into and then the site rolls around an ranks well for it with barely any backlinks. I just wanted to put it out there and see if I'm missing something obvious. Thanks everyone!
Moz Pro | | getrightmusic0