Need help with Robots.txt
-
An eCommerce site built with Modx CMS. I found lots of auto generated duplicate page issue on that site. Now I need to disallow some pages from that category. Here is the actual product page url looks like
product_listing.php?cat=6857And here is the auto generated url structure
product_listing.php?cat=6857&cPath=dropship&size=19Can any one suggest how to disallow this specific category through robots.txt. I am not so familiar with Modx and this kind of link structure.
Your help will be appreciated.
Thanks
-
I would actually add a canonical tag and then handle these using the Parameters section of Search Console. That's why it's there, for exactly this type of site with exactly this issue.
-
Nahid, before you use the robots.txt file's disallow for those URLs, you may want to reconsider. You may want to use the canonical tag instead. In the case where you have different sizes, colors, etc. we typically recommend using the Canonical Tag and not the disallow in robots.txt.
Anyhow, if you'd like to use the disallow you can use one of these:
Disallow: /?
or
Disallow: /?cat=
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Titles and Metas disappeared?? Help please!
Hi everyone, Had a bit of a concern today, my weekly report has come through and my crawl issues have skyrockets by over 400! It says my metas and titles are missing but when I check through the site manually they seem to all still be there, I'm getting the same problem when I use screaming frog to crawl the site. I would really appreciate an explanation from someone as to why this is happening as I am quite confused about the situation. Thank you people Charlie Our website is www.homelogic.co.uk 🙂
Intermediate & Advanced SEO | | MintySEO0 -
All of my blog titles have disappeared. In need of Wordpress help.
Not sure if this is the right place to ask this question but here it goes. All of the titles on my real estate website have disappeared. I have spent hours looking through different forums trying to figure out how to make them show up. Also whenever I hover the cursor over links they turn to white and disappear as well. This is the website: http://www.acolerealty.com/blog/ If this helps here is the custom CSS in worpress is the following: /* GREEN */ body {background: #eff3ec !important;} .header-membership {
Intermediate & Advanced SEO | | artscube.biz
background: #fff !important;
box-shadow: none !important;
border-bottom: 2px solid #e5e9e3 !important;
} .header-membership a {
color: #909090 !important;
text-shadow: none !important
} h1#site-title a {
color: #397249 !important;
} header nav#main-nav {
background: #7aad79 !important; /* Old browsers /
background: -moz-linear-gradient(top, #7aad79 0%, #397249 100%) !important; / FF3.6+ /
background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#7aad79), color-stop(100%,#397249)) !important; / Chrome,Safari4+ /
background: -webkit-linear-gradient(top, #7aad79 0%,#397249 100%); / Chrome10+,Safari5.1+ /
background: -o-linear-gradient(top, #7aad79 0%,#397249 100%) !important; / Opera 11.10+ /
background: -ms-linear-gradient(top, #7aad79 0%,#397249 100%) !important; / IE10+ /
background: linear-gradient(to bottom, #7aad79 0%,#397249 100%) !important; / W3C /
filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#7aad79', endColorstr='#397249',GradientType=0 ) !important; / IE6-9 */
} #t-header-container .home-search-container #header-top-search::before {
background: #7aad79 !important; /* Old browsers /
background: -moz-linear-gradient(top, #7aad79 0%, #397249 100%) !important; / FF3.6+ /
background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#7aad79), color-stop(100%,#397249)) !important; / Chrome,Safari4+ /
background: -webkit-linear-gradient(top, #7aad79 0%,#397249 100%); / Chrome10+,Safari5.1+ /
background: -o-linear-gradient(top, #7aad79 0%,#397249 100%) !important; / Opera 11.10+ /
background: -ms-linear-gradient(top, #7aad79 0%,#397249 100%) !important; / IE10+ /
background: linear-gradient(to bottom, #7aad79 0%,#397249 100%) !important; / W3C /
filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#7aad79', endColorstr='#397249',GradientType=0 ) !important; / IE6-9 */
} input.button-primary {
background: #7aad79 !important; /* Old browsers /
background: -moz-linear-gradient(top, #7aad79 0%, #397249 100%) !important; / FF3.6+ /
background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#7aad79), color-stop(100%,#397249)) !important; / Chrome,Safari4+ /
background: -webkit-linear-gradient(top, #7aad79 0%,#397249 100%); / Chrome10+,Safari5.1+ /
background: -o-linear-gradient(top, #7aad79 0%,#397249 100%) !important; / Opera 11.10+ /
background: -ms-linear-gradient(top, #7aad79 0%,#397249 100%) !important; / IE10+ /
background: linear-gradient(to bottom, #7aad79 0%,#397249 100%) !important; / W3C /
filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#7aad79', endColorstr='#397249',GradientType=0 ) !important; / IE6-9 */ border:1px solid #23472d !important;
} input.button-primary:hover {
background: #628b61 !important;
} footer {
background: #e4e8e1 !important;
}0 -
Canonicals for product pages - confused, anyone help?
I have an ecommerce website (built using Magento), and have just had the functionality extended to allow me to define my own canonical URLs. Currently the URLs are www. domainname.com/product-name.html but I can now change this to www.domainname. com/product-category/product-name.html. I was led to believe that this would be good for SEO. However, I have since had conflicting advice - it's been suggested that any links across the website that link to domain/category/sub-category/product will pass weight and authority through to the specified canonical anyway. Plus longer URLs are generally worse... I'm confused. Is it worth changing them? If so, would it be a bad thing to change all 700 canonical URLs at once?
Intermediate & Advanced SEO | | Coraltoes770 -
Google: How to See URLs Blocked by Robots?
Google Webmaster Tools says we have 17K out of 34K URLs that are blocked by our Robots.txt file. How can I see the URLs that are being blocked? Here's our Robots.txt file. User-agent: * Disallow: /swish.cgi Disallow: /demo Disallow: /reviews/review.php/new/ Disallow: /cgi-audiobooksonline/sb/order.cgi Disallow: /cgi-audiobooksonline/sb/productsearch.cgi Disallow: /cgi-audiobooksonline/sb/billing.cgi Disallow: /cgi-audiobooksonline/sb/inv.cgi Disallow: /cgi-audiobooksonline/sb/new_options.cgi Disallow: /cgi-audiobooksonline/sb/registration.cgi Disallow: /cgi-audiobooksonline/sb/tellfriend.cgi Disallow: /*?gdftrk Sitemap: http://www.audiobooksonline.com/google-sitemap.xml
Intermediate & Advanced SEO | | lbohen0 -
Need help or explanation on my site!
My site has suffered greatly since the recent Google update. I have done everything as suggested. I have had all bad links removed over 2 months ago. I have lowered keyword density (not easy since the keyword is in our company name!). I have rewritten various content and bolstered our existing content. What gives? What can I do? As an example the keyword, "maysville plumber" - I rank about 40th for this keyword. The first three pages are filled with websites with literally NO content or no added value. Maysville is a town of about 1k residents - there is no competition. Before the update I was #1 for years on this particular keyword. And this is the case with 35 other cities (mostly small cities, but a few larger ones). Please help me understand or suggest what I can possibly do at this point. We have hundreds of pages of unique content on each and every page. We have zero duplicate content (I have ran tests and crawlers). We have no fishy links. I have not gotten any messages from google on Webmasters. PLEASE HELP!! I asked a similar question a little while back and fixed all of the suggestions. My site is www.akinsplumbing.net.
Intermediate & Advanced SEO | | chuckakins0 -
Help! Why did Google remove my images from their index?
I've been scratching my head over this one for a while now and I can't seem to figure it out. I own a website that is user-generated content. Users submit images to my sites of graphic resources (for designers) that they have created to share with our community. I've been noticing over the past few months that I'm getting completely dominated in Google Images. I used to get a ton of traffic from Google Images, but now I can't find my images anywhere. After diving into Analytics I found this: http://cl.ly/140L2d14040Q1R0W161e and realized sometime about a year ago my image traffic took a dive. We've gone back through all the change logs and can't find where we made any changes to the site structure that could have caused this. We are stumped. Does anyone know of any historical Google updates that could have caused this last year around the end of April 2010? Any help or insight would be greatly appreciated!
Intermediate & Advanced SEO | | shawn810 -
New server update + wrong robots.txt = lost SERP rankings
Over the weekend, we updated our store to a new server. Before the switch, we had a robots.txt file on the new server that disallowed its contents from being indexed (we didn't want duplicate pages from both old and new servers). When we finally made the switch, we somehow forgot to remove that robots.txt file, so the new pages weren't indexed. We quickly put our good robots.txt in place, and we submitted a request for a re-crawl of the site. The problem is that many of our search rankings have changed. We were ranking #2 for some keywords, and now we're not showing up at all. Is there anything we can do? Google Webmaster Tools says that the next crawl could take up to weeks! Any suggestions will be much appreciated.
Intermediate & Advanced SEO | | 9Studios0 -
Does a 'Certified Domain' help SEO?
I see that GoDaddy offer a 'Certified Domain' option. Does this help SEO at all?
Intermediate & Advanced SEO | | Techboy0