Magento Dynamic Pages Being Indexed
-
Hi there, I have about 50k Moz medium priority errors in my Crawl Diagnostic report.
The bulk of them are classified as "Temporary Redirect" problems. Then if you drill into those further, I can see that the problem urls all kinda are center around:
mysite.com/catalogsearch/result..
mysite.com/wishlist..
mysite.com/catalog..
Is this something I should disallow in my Robstxt file? And if so how specific do I get with it..
Disallow /catalogsearch/result/?q=
Will listing the /catalogsearch be enough to cover anything after it?
thanks
-
Jeff is completely right on this one!
-
Thank you very much Jeff!
-
Yes, Magento does have a way of handling search results that are not always SEO-friendly.
To restrict Google from indexing anything in the /catalogsearch/ directory, I'd add this to your robots.txt file:
User-agent: *
Disallow: /catalogsearch/Here's a Moz blog about the robots.txt file:
http://moz.com/blog/interactive-guide-to-robots-txtAnd also, just in case the "killer" robots ever take over, Google has made sure that their two founders are not able to be indexed by both the T-1000 and T-800 models, with a special killer-robots.txt file:
http://www.google.com/killer-robots.txtHope this helps!
-- Jeff
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Block Moz (or any other robot) from crawling pages with specific URLs
Hello! Moz reports that my site has around 380 duplicate page content. Most of them come from dynamic generated URLs that have some specific parameters. I have sorted this out for Google in webmaster tools (the new Google Search Console) by blocking the pages with these parameters. However, Moz is still reporting the same amount of duplicate content pages and, to stop it, I know I must use robots.txt. The trick is that, I don't want to block every page, but just the pages with specific parameters. I want to do this because among these 380 pages there are some other pages with no parameters (or different parameters) that I need to take care of. Basically, I need to clean this list to be able to use the feature properly in the future. I have read through Moz forums and found a few topics related to this, but there is no clear answer on how to block only pages with specific URLs. Therefore, I have done my research and come up with these lines for robots.txt: User-agent: dotbot
Moz Pro | | Blacktie
Disallow: /*numberOfStars=0 User-agent: rogerbot
Disallow: /*numberOfStars=0 My questions: 1. Are the above lines correct and would block Moz (dotbot and rogerbot) from crawling only pages that have numberOfStars=0 parameter in their URLs, leaving other pages intact? 2. Do I need to have an empty line between the two groups? (I mean between "Disallow: /*numberOfStars=0" and "User-agent: rogerbot")? (or does it even matter?) I think this would help many people as there is no clear answer on how to block crawling only pages with specific URLs. Moreover, this should be valid for any robot out there. Thank you for your help!0 -
ERROR: Too Many on-page links!
User wise, my product menu @LEDSupply.com is user friendly, but I'm concerned that it might be seen by crawlers as bad because of the TOO MANY ON-PAGE LINKS error I am getting in my moz crawl report. Is it really counting all the links in every drop-down menu? If so, is there are resource on how to fix it????
Moz Pro | | saultienut0 -
Home Page Location Redirect
We have recently upgraded our Wordpress site to detect your local city and redirect to the proper location. Previously we had independent sites - for example, http://atlanta.styleblueprint.com is now http://styleblueprint.com/atlanta We've setup 301 redirects on all of the old site home pages. Now we have two issues: Moz will no longer crawl our domain. For two weeks now our campaign shows only four pages crawled None of our home pages show up in Google any longer for organic searches. We previously always ranked #1 for "styleblueprint" or "style blueprint" Does our new auto redirect mess things up? Or is this just a function of time until Google "learns" how to index our new site? All thoughts appreciated. Thanks in advance, Jay
Moz Pro | | SSBCI0 -
How long will it take for Page Rank (or Page Authority) to flow via a 301 redirect?
I've recently redeveloped a static site using WordPress and have created 301 redirects for the original urls to the new urls. I know I won't get all the value passed via the 301, but I'm hoping some will. Any idea how long this may take? It's been nearly a month since the changeover so wondering if it would be weeks, months or more?
Moz Pro | | annomd0 -
Why are not nofollowed links counted in On-Page Analysis Report?
When I run the On-Page Analysis on our homepage, the report says the page has 238 **"Internal followed links". ** Why are not nofollowed internal links counted as well? Nofollowed links have been leaking link juice for quite some time now. Martin
Moz Pro | | TalkInThePark0 -
Find pages containing broken links.
hi everyone, for each internal broken links I need to find all the pages that contain it. In the Seomoz report there is only a refferer link for each broken link, but google webmaster tools indicates that the dead link is present in many pages of the site. there is a way to have these data with SEOmoz or other software, in a csv report ? thanks
Moz Pro | | wwmind0 -
Pass Page LinkJuice? Or Pass Keyword LinkJuice?
I have a popular page that is not one of the three pages that I am hoping to raise awareness of (want to focus on). The dilemma I am trying to understand is that I really don't want to encourage all the flow from the popular to ONE of my hopeful pages (focus pages). Rather, I want to focus the keyword portions of that page to help the three hopeful pages. So I consider the rel=canonical tag.... err no. rel=canonical would pass ALL my popular page link juice to ONE of my three hopeful pages. What's the best way to pass the keyword link juice relevant to each of my three hopeful pages their, um, portion, of the popular page link juice. I'm white hat by preference. All four pages are good legitimate landing pages, and of course I dread sabotaging the popularity of what is working. Suggestions? Advice?
Moz Pro | | iansears0 -
Domain vs Page
I see a lot of different metrics pointing to domain or page. What is the difference between these two definitions?
Moz Pro | | Gfrink0