Magento Dynamic Pages Being Indexed
-
Hi there, I have about 50k Moz medium priority errors in my Crawl Diagnostic report.
The bulk of them are classified as "Temporary Redirect" problems. Then if you drill into those further, I can see that the problem urls all kinda are center around:
mysite.com/catalogsearch/result..
mysite.com/wishlist..
mysite.com/catalog..
Is this something I should disallow in my Robstxt file? And if so how specific do I get with it..
Disallow /catalogsearch/result/?q=
Will listing the /catalogsearch be enough to cover anything after it?
thanks
-
Jeff is completely right on this one!
-
Thank you very much Jeff!
-
Yes, Magento does have a way of handling search results that are not always SEO-friendly.
To restrict Google from indexing anything in the /catalogsearch/ directory, I'd add this to your robots.txt file:
User-agent: *
Disallow: /catalogsearch/Here's a Moz blog about the robots.txt file:
http://moz.com/blog/interactive-guide-to-robots-txtAnd also, just in case the "killer" robots ever take over, Google has made sure that their two founders are not able to be indexed by both the T-1000 and T-800 models, with a special killer-robots.txt file:
http://www.google.com/killer-robots.txtHope this helps!
-- Jeff
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which is the better keyword strategy? Specifying page or seeing what ranks?
I recently began using another third-party tool that requires a list of keywords and the URLs I want those keywords to rank on (for example, "vegetarian recipes" on www.123.com/vegetarian-recipes). Is this a better strategy than using the Moz ranking tool to see which pages on my domain my target keywords are already ranking for? I can see how the former might mean we miss opportunities but the latter doesn't necessarily show how my priority pages are doing for specific keywords.
Moz Pro | | RikkiAyers1 -
Is it normal for Moz to report on nofollow pages in crawl diagnostics?
I have a dev version of my website, for example, devwww.website.com. The htaccess page has a noindex and nofollow request, but I got crawl issues reported from these pages in my Moz report. Does this mean that I don't have the development site hidden from search like I thought I did?
Moz Pro | | houstonbrooke0 -
How can I deal with tag page duplicate issues
The Moz crawler reported some dupliated issues. Many of them have to do with tags.
Moz Pro | | IamKovacs
Each tag has a link, and as some articles are under several tags, these come up as duplicate content. I read Dr Peter's piece on Canonical stuff, but it's not clear to me if any of these are the solution. Perhaps the solution lies somewhere else? Maybe I need to block the robots from these urls (But that seems counter-SEO-productive) Thanks
Kovacs0 -
Only One page crawled..Need help
I have run a website in Seomoz which have many URLs with it. But when I saw the seomoz report that showing Pages Crawled: 1. Why this is happen my campaign limit is OK Tell me what to do for all page crawling in seomoz report. wV6fMWx
Moz Pro | | lucidsoftech0 -
Only few pages (308 pages of 1000 something pages) have been crawled and diagnosed in 4 days, how many days till the entire website is crawled complete?
Setup campaign about 4-5 days ago and yesterday rogerbot said 308 pages were crawled and the diagnostics were provided. This website has over 1000+ pages and would like to know how long it would take for roger to crawl the entire website and provide diagnostics. Thanks!
Moz Pro | | TejaswiNaidu0 -
What is more important, page authority or linking root domains?
I have 2 pages in my report that are identical: www.ferringway.com (more page authority......21 vs. 1) www.ferringway.com/index.php (more linking root domains...... 40 vs. 21) Which one should I delete or should I do something like a 301 redirect? Your help is greatly appreciated! Sheryl
Moz Pro | | TOMMarketingLtd.0 -
Is there a tool to show me exactly how a search engine sees my pages?
Our site is very graphics heavy and I know the search engines aren't seeing it as humans do. Is there a tool (almost like Wirify) that can show me exactly how Google sees my page(s)? Thanks in advance. 🙂
Moz Pro | | askotzko0 -
Page Authority vs Domain Authority
I'm using the site explorer to compare a potential clients site against 4 others, in an incredibly competitive market. Each of their competitiors has a higher page authority (on the home page) than their domain authority. This is untrue for the clients site. (which have much lower metrics all round) Any input as to what this means/says about their competitors who I would guess (looking at some of their backlink profiles) have done some failry widespread grey hat stuff in the past. (Though haven't we all 😉 )
Moz Pro | | FDC0