Magento Dynamic Pages Being Indexed
-
Hi there, I have about 50k Moz medium priority errors in my Crawl Diagnostic report.
The bulk of them are classified as "Temporary Redirect" problems. Then if you drill into those further, I can see that the problem urls all kinda are center around:
mysite.com/catalogsearch/result..
mysite.com/wishlist..
mysite.com/catalog..
Is this something I should disallow in my Robstxt file? And if so how specific do I get with it..
Disallow /catalogsearch/result/?q=
Will listing the /catalogsearch be enough to cover anything after it?
thanks
-
Jeff is completely right on this one!
-
Thank you very much Jeff!
-
Yes, Magento does have a way of handling search results that are not always SEO-friendly.
To restrict Google from indexing anything in the /catalogsearch/ directory, I'd add this to your robots.txt file:
User-agent: *
Disallow: /catalogsearch/Here's a Moz blog about the robots.txt file:
http://moz.com/blog/interactive-guide-to-robots-txtAnd also, just in case the "killer" robots ever take over, Google has made sure that their two founders are not able to be indexed by both the T-1000 and T-800 models, with a special killer-robots.txt file:
http://www.google.com/killer-robots.txtHope this helps!
-- Jeff
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
'Duplicate Page Content' for dissimilar pages
I'm using Moz's Crawl Diagnostics to try and clean up some SEO priorities for our website (http://www.craftcompany.co.uk) HOWEVER, virtually all of the pages that are being categorised as duplicate content are not the same, or indeed similar. For instance, these three pages have been deemed duplicated pages; http://www.craftcompany.co.uk/pme-rose-leaf-veined-plunger.html http://www.craftcompany.co.uk/double-faced-satin-ribbon-black-25mm-wide.html http://www.craftcompany.co.uk/double-faced-satin-maroon-10mm-wide-25mt.html Can anyone give me an insight into why this is? Many Thanks! http://www.craftcompany.co.uk/
Moz Pro | | The_Craft_Company0 -
Multiple Sites/Internal Pages Campaign
Can MOZ do reports/ranks/campaigns separately for each of our sites, then do separate keyword campaigns for specific internal content pages of each of the websites? For a law firm with both a defense and family law sites. We have multiple pages within each site, we will need to do separate and individual campaigns for assault, burglary, traffic tickets, ect. in our defense site. And we'll need other specific campaigns for other specific content in out family law site. Will Moz be able to accommodate our needs with their $99/mo plan or is that service only available for higher packages? (or is it even possible in ANY of their packages???) Thanks.
Moz Pro | | Wallin_Klarich0 -
How to solve duplicate page title & content error
I got lot of errors in Duplicate page title - 5000 Here the result page is same and content is also same,but it differs only with page no in meta title Title missing error In seomoz report i got empty msg - title,meta desc,meta robots,meta refresh But if i check the link which i got error it shows all meta tags..we have added all meta tags in our site..But i dont no why i got title missing error . 404 error In this report,if i click the link which i got error, it goes to main page of our site. But the url differs. eg: The error link is :www.example.com/buy/requirement-2-0-inmumbai-property it automatically goes to www.example.com page Let me know how to solve these issues.
Moz Pro | | Rajesh.Chandran0 -
Indexed sites
Hello I´m a newbie here on seomoz. My question is just simple but i need to be sure about that:Refering to the report which you can create in the "crawl diagnostic summary". Does the CSV export lists all and ONLY the URLS which are in the google index?If not: does a report is available where all and ONLY the URLS are listed which are in google index?Many thanks! Henrik
Moz Pro | | drgoodcat0 -
Find pages containing broken links.
hi everyone, for each internal broken links I need to find all the pages that contain it. In the Seomoz report there is only a refferer link for each broken link, but google webmaster tools indicates that the dead link is present in many pages of the site. there is a way to have these data with SEOmoz or other software, in a csv report ? thanks
Moz Pro | | wwmind0 -
Page Rank is different from google search
I use the ranking tools to check each page of our site. I use the keyword flora grand and exact url http://www.florahospitality.com/dubai-flora-grand-hotel.aspx . It return page rank 1 in US, UAE, and UK region but when I try to search it in google it doesn't appear.
Moz Pro | | shebinhassan0 -
HTTP 404 for 404-page?
Hi Mozzers! SEOmoz just finished crawling one of my websites and this crawl found 3 errors. One of these errors was the (custom) 404-page, because of the http-status 404. What's you suggestion about this? Should a 404-page have a status 404? Thanks in advance for your suggestions!
Moz Pro | | Partouter0 -
Only Crawling 1 page?
Hi Guys, Any advice much appreciated on this! Recently set up a new campaign on my dashboard with just 5 keywords. The domain is brammer.co.uk and a quick Google site:brammer.co.uk shows a good amount of indexed pages. However - first seomoz tool crawl has only crawled 1 url!! "Last Crawl Completed: Apr. 12th, 2011 Next Crawl Starts: Apr. 17th, 2011" Any ideas what's stopping the tool crawl anymore of the site?? Cheers in advance.. J
Moz Pro | | lovealbatross0