Magento Dynamic Pages Being Indexed
-
Hi there, I have about 50k Moz medium priority errors in my Crawl Diagnostic report.
The bulk of them are classified as "Temporary Redirect" problems. Then if you drill into those further, I can see that the problem urls all kinda are center around:
mysite.com/catalogsearch/result..
mysite.com/wishlist..
mysite.com/catalog..
Is this something I should disallow in my Robstxt file? And if so how specific do I get with it..
Disallow /catalogsearch/result/?q=
Will listing the /catalogsearch be enough to cover anything after it?
thanks
-
Jeff is completely right on this one!
-
Thank you very much Jeff!
-
Yes, Magento does have a way of handling search results that are not always SEO-friendly.
To restrict Google from indexing anything in the /catalogsearch/ directory, I'd add this to your robots.txt file:
User-agent: *
Disallow: /catalogsearch/Here's a Moz blog about the robots.txt file:
http://moz.com/blog/interactive-guide-to-robots-txtAnd also, just in case the "killer" robots ever take over, Google has made sure that their two founders are not able to be indexed by both the T-1000 and T-800 models, with a special killer-robots.txt file:
http://www.google.com/killer-robots.txtHope this helps!
-- Jeff
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Pages
Hello, we have an issue which I'm hoping someone can help with. Our Moz system is saying that this page http://www.indigolittle.com/fees/ Is a duplicate page. We use this page purely for mobiles and we have added code to say This has been on for over a month now however Moz is still picking the page us as a High Priority Issue.
Moz Pro | | popcreativeltd0 -
ERROR: Too Many on-page links!
User wise, my product menu @LEDSupply.com is user friendly, but I'm concerned that it might be seen by crawlers as bad because of the TOO MANY ON-PAGE LINKS error I am getting in my moz crawl report. Is it really counting all the links in every drop-down menu? If so, is there are resource on how to fix it????
Moz Pro | | saultienut0 -
On Page Reports for Long-Tail Keywords?
First Q&A here, so take it easy on me. Hopefully this is not a dumb question. 99% of the on page reports I look at are for local keywords. I'm finding that some of the page reports get and F where they should be receiving A's. I noticed if I manage a long-tail keyword like "books in houston tx" that I get a grade based on the exact phrase as opposed to a combination of the keywords. So when I have text in the body saying "books in the Houston area" or "Houston Books" for example, instead of using my exact managed keyword, it will tell me that the keyword is not mentioned in the body anywhere. When it is, it's just not in the exact order... I'm trying to write my pages for users and I don't think the users want to hear "books in houston tx" several times. If I do a report on the same page for the keyword "books" I get an A. So this is where it gets a little in the grey area for me. I need a solid on page report for local long-tail search terms. Anyone have any advice? Is there a way I could be using this tool to better suit my needs?
Moz Pro | | AmericomMarketing0 -
1 page crawled - again
Just had to let you know that it happend again. So right now we are at 2 out of the last 4 crawls. Uptime here is 99,8% for the last 30 days, with a small downtime due to an update process at the 18/5 from around 2:30 to 4:30 GMT In relation to: http://moz.com/community/q/1-page-crawled-and-other-errors
Moz Pro | | alsvik0 -
Why my site page don't appeare in the ranking ?
why my site page don't appear in the ranking, which type of domain tracking should i choose to track all my site pages ( my site is a .blogspot.com ) ? when i check it i found that they appear in the first 30 results although seomoz saying they are not in the first 50 result !
Moz Pro | | AmrKamal0 -
Duplicate Page Titles and Content
The SeoMoz crawler has found many pages like this on my site with /?Letter=Letter, e.g. http://www.johnsearles.com/metal-art-tiles/?D=A. I believe it is finding multiple caches of a page and identifying them as duplicates. Is there any way to screen out these multiple cache results?
Moz Pro | | johnsearles0 -
Strange Ranking On 1st Page Of Google For Competitive Keyword
Hey all, So, I've noticed a site ranking on the first page in Google for a keyword that I have been targeting for years. The keyword is "mixtapes" and pretty competitive. The site, http://www.newyorkmixtapes.com is ranking around #4 in Google. It seemed odd to me so I looked into the site using some of the tools here. Open Explorer showed that they have minimal links. The actual site is pretty spammy also. Does anyone have any idea how this site ranked so high so fast? Is it ranking because I am located in New York? Does Google still put lots of weight on exact match domains? It's just been bothering me because it's a keyword that I put a lot of work into and then the site rolls around an ranks well for it with barely any backlinks. I just wanted to put it out there and see if I'm missing something obvious. Thanks everyone!
Moz Pro | | getrightmusic0 -
On-Page Optimisation tool on intranet pages
Does anybody know if there's any easy way to use the On-Page Optimisation tool on intranet or not publicly accessible pages? Thanks!
Moz Pro | | neooptic0