Need help with Robots.txt
-
An eCommerce site built with Modx CMS. I found lots of auto generated duplicate page issue on that site. Now I need to disallow some pages from that category. Here is the actual product page url looks like
product_listing.php?cat=6857And here is the auto generated url structure
product_listing.php?cat=6857&cPath=dropship&size=19Can any one suggest how to disallow this specific category through robots.txt. I am not so familiar with Modx and this kind of link structure.
Your help will be appreciated.
Thanks
-
I would actually add a canonical tag and then handle these using the Parameters section of Search Console. That's why it's there, for exactly this type of site with exactly this issue.
-
Nahid, before you use the robots.txt file's disallow for those URLs, you may want to reconsider. You may want to use the canonical tag instead. In the case where you have different sizes, colors, etc. we typically recommend using the Canonical Tag and not the disallow in robots.txt.
Anyhow, if you'd like to use the disallow you can use one of these:
Disallow: /?
or
Disallow: /?cat=
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do I need to worry about sub-domains?
Hi Moz commnity, Our website ranking was good and dropped for couple of recent months. We have around 10 sub-domains. I doubt them if they are hurting us. Being said all over in SEO industry like the sub-domains are completely different websites; will they hurt if they are not well optimised? And we have many links from our sub-domains to website top pages, is this wrong for Google? How to well maintain the sub-domains? Do I need to worry about them? Thanks
Intermediate & Advanced SEO | | vtmoz0 -
Magento Help - Server Reset
Good Morning, After rebooting a server, a magento based website reset itself going back to December 2013. All changes to the site and orders dating up until yesterday (6/19/14) have disappeared. There are several folders on the root of the server that have files with yesterday's date but we don't know how to bring everything back and restore. Any Magento or server experts out there ever face this issue or have any ideas or potential solutions? Thanks
Intermediate & Advanced SEO | | Prime850 -
Help with Effective Meta Description
Can someone give me a sample of an effective meta description tag? All of the best practice stuff I read doesn't talk about how to raise CTR. It seems to me that this is a neglected area of SEO, and we want to do this right. Obviously, we will need to test. For example, my main home page keyword is "IT Support" Things I might want to put in the tag: Free Network Assessments 100% Risk Free Trials "Relentless IT Support" (Major Theme) 30 Years of Experience (since 1984) Eliminate your IT Support Headaches Forever (Too long) Call to action? BTW, Thanks to everyone for your help. This is a great community. Solid advice from experts. Here's an example of what I would create Relentless IT Support Since 1984. Trust and Accountability. 100% Risk Free Trials. Contact us today for Free IT Assessment.
Intermediate & Advanced SEO | | CsmBill0 -
Help!!! Am I being Attacked???
Hello, I do not believe so much in spammy links attacks and I definitely do not believe my site is worth attacking. However, I'm seeing new links pointing to my site that I have no idea where they come from. I just spotted three articles on a poor crappy article site with exact match keywords point to me. The articles are completely unique (copyscaped them) and they were posted according to the site time stamp during Oct and Nov 2012. (And they Appear in the WMT recently discovered links from more or less the same time). What to do (besides for disavowing this domain)? Thanks
Intermediate & Advanced SEO | | BeytzNet0 -
Not using a robot command meta tag
Hi SEOmoz peeps. Was doing some research on robot commands and found a couple major sites that are not using them. If you check out the code for these: http://www.amazon.com http://www.zappos.com http://www.zappos.com/product/7787787/color/92100 http://www.altrec.com/ You fill not find a meta robot command line. Of course you need the line for any noindex, nofollow, noarchive pages. However for pages you want crawled and indexed, is there any benefit for not having the line at all? Thanks!
Intermediate & Advanced SEO | | STPseo0 -
Need advice re: selecting an Ecommerce platform
Hi. I am responsible for choosing an ecommerce platform and overseeing the implementation of a large ecommerce site. The site will have tens of thousands of products and will be fairly complex. Eventually the site will integrate with the suppliers back end inventory/order management system, which is some sort of custom Windows/.NET system. (I'm not very technical if you haven't noticed....) Primarily I want a platform that is SEO friendly, and I have to be sure that the site is developed properly from an SEO and usability perspective. I thought I would go with an asp.net solution (aspdotnetstorefront, specifically) to facilitate the future integration, but I am questioning this choice after reading some of the comments I have found here at SEOMoz. So is asp.net really a bad choice SEO-wise? I almost considered Magento, but was having trouble finding a good solution provider to work with. I also worried about integration issues down the road. I would appreciate any advice or input anyone may have. Thank you!
Intermediate & Advanced SEO | | sbg7810 -
Do i need different IP addresses for mini sites?
Hi everyone We are currently building some non-advertorial based mini sites the link to a main "money site", these mini sites are all run off wordpress or similar and have different designs, however all the WHOIS data remains under one company. So therefore I dont know if really you need different Class C IP's anymore as google et al will just look at the whois records and link the websites up that way? Is this tactic still worth doing? Thanks for any input!
Intermediate & Advanced SEO | | SEOwins0 -
Need advice on local search optimization
HI all, I've found myself in a puzzling position and not quite sure which direction to push my current SEO project so if anyone who's done this particular type of SEO can offer some suggestions I'd be eternaly grateful. I am currently working on a project for a Law Firm based in New Jersey. Lets say the town they are in is Garfield. What I really want to try and achieve is see them appearing in the number one spot whenever anyone within Garfield or the immediate area searches for a lawyer relating to the individuals need. E..g searches like "personal injury lawyers", "real estate lawyer". The problem is I can see how I can easily make it to the number one position if people are specific and enter garfield in the search term but in reality they wouldn't be doing that. An additional problem is that peoples ISP's in garfield aren't located in Garfield, in some cases they're as far away as Newark so when they're doing a search for 'real estate lawyer' google is bringing up results for the Newark based firms. It seems using tools like market samurai to look at the traffic and competition is proving useless as searches like the ones I'm doing for local business are so closely tied to the ISP location I don't really know whether to target broad range searches like "Real Estate Lawyer", or to be really specific and include the town name in my page titles, H1 tags etc... I hope I put across my dilemma and someone can help me chose which direction to go in.. Thanks
Intermediate & Advanced SEO | | davebrown19750