Need help with Robots.txt
-
An eCommerce site built with Modx CMS. I found lots of auto generated duplicate page issue on that site. Now I need to disallow some pages from that category. Here is the actual product page url looks like
product_listing.php?cat=6857And here is the auto generated url structure
product_listing.php?cat=6857&cPath=dropship&size=19Can any one suggest how to disallow this specific category through robots.txt. I am not so familiar with Modx and this kind of link structure.
Your help will be appreciated.
Thanks
-
I would actually add a canonical tag and then handle these using the Parameters section of Search Console. That's why it's there, for exactly this type of site with exactly this issue.
-
Nahid, before you use the robots.txt file's disallow for those URLs, you may want to reconsider. You may want to use the canonical tag instead. In the case where you have different sizes, colors, etc. we typically recommend using the Canonical Tag and not the disallow in robots.txt.
Anyhow, if you'd like to use the disallow you can use one of these:
Disallow: /?
or
Disallow: /?cat=
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt was set to disallow for 14 days
We updated our website and accidentally overwrote our robots file with a version that prevented crawling ( "Disallow: /") We realized the issue 14 days later and replaced after our organic visits began to drop significantly and we quickly replace the robots file with the correct version to begin crawling again. With the impact to our organic visits, we have a few and any help would be greatly appreciated - Will the site get back to its original status/ranking ? If so .. how long would that take? Is there anything we can do to speed up the process ? Thanks
Intermediate & Advanced SEO | | jc42540 -
Help with facet URLs in Magento
Hi Guys, Wondering if I can get some technical help here... We have our site britishbraces.co.uk , built in Magento. As per eCommerce sites, we have paginated pages throughout. These have rel=next/prev implemented but not correctly ( as it is not in is it in ) - this fix is in process. Our canonicals are currently incorrect as far as I believe, as even when content is filtered, the canonical takes you back to the first page URL. For example, http://www.britishbraces.co.uk/braces/x-style.html?ajaxcatalog=true&brand=380&max=51.19&min=31.19 Canonical to... http://www.britishbraces.co.uk/braces/x-style.html Which I understand to be incorrect. As I want the coloured filtered pages to be indexed ( due to search volume for colour related queries ), but I don't want the price filtered pages to be indexed - I am unsure how to implement the solution? As I understand, because rel=next/prev implemented ( with no View All page ), the rel=canonical is not necessary as Google understands page 1 is the first page in the series. Therefore, once a user has filtered by colour, there should then be a canonical pointing to the coloured filter URL? ( e.g. /product/black ) But when a user filters by price, there should be noindex on those URLs ? Or can this be blocked in robots.txt prior? My head is a little confused here and I know we have an issue because our amount of indexed pages is increasing day by day but to no solution of the facet urls. Can anybody help - apologies in advance if I have confused the matter. Thanks
Intermediate & Advanced SEO | | HappyJackJr0 -
Help with https// redirects
Hey there
Intermediate & Advanced SEO | | Jay328
I have a client who just moved from a self hosted CMS to Adobe Catalyst (don't ask!)
The problem: Their url indexed with google is https://domain.com, Adobe Catalyst does not support third party SSL certificates or https domains. Now when people google them https://domain.com shows up in search, HOWEVER it does not have a trusted certificate and a pop up window blocks the site. They are a mortgage company so SSL is really not needed. What can I do to get google to recognize the site at http: vs. https? Would this be something in GWMT? Thanks!0 -
Help! Unnatural Linking Partial Manual Penalty
A friend was hit with a manual penalty for unnatural links-impacts links. (see attached) I'm thinking it may be because they copied their entire wordpress.com site over to site.org/blog. (without redirecting it, so they have duplicate content as well) Out of 76+k links, nearly 11,000 are from their wordpress.com blog. If that's the case is the problem solved by upgrading within wordpress.com to redirect to site.org/blog? (then making a reconsideration request?) Or do I risk negatively affecting their site somehow? They saw a significant increase in traffic when they moved the content over but I'm thinking that was more a matter of increasing content on their site than increasing backlinks. The .org site ranks relatively well, whereas the wordpress.com blog doesn't really rank at all.Worth noting: it's a partial match, not a sitewide match. Does that negate my theory about the wordpress.com blog being the cause in any way? Since many of the links from it are sitewide? The wordpress.com blog has a header link to the .org homepage, plus individual links to it in posts. There are also three links in the header to pages on their .com website which redirects to three corresponding pages on the main .org site (the whole .com redirects). There are 23 footer links from the blog to the targeted .org pages as well. In the attached screenshot of who links most from Google Webmaster Tools, note that martindale.com links most, but it's a lawyer's site so they naturally have referring content there. Could that be a problem?Thanks everyone! 🙂M8JVEI6.jpg?1 M6gYE90.jpg
Intermediate & Advanced SEO | | kimmiedawn0 -
Massive URL blockage by robots.txt
Hello people, In May there has been a dramatic increase in blocked URLs by robots.txt, even though we don't have so many URLs or crawl errors. You can view the attachment to see how it went up. The thing is the company hasn't touched the text file since 2012. What might be causing the problem? Can this result any penalties? Can indexation be lowered because of this? ?di=1113766463681
Intermediate & Advanced SEO | | moneywise_test0 -
What should I block with a robots.txt file?
Hi Mozzers, We're having a hard time getting our site indexed, and I have a feeling my dev team may be blocking too much of our site via our robots.txt file. They say they have disallowed php and smarty files. Is there any harm in allowing these pages? Thanks!
Intermediate & Advanced SEO | | Travis-W1 -
How a press release can help with Google serp?
Hi, After publishing a press release, if that press release is on top position on Google News for the keyword, how will it effect the SERP for that website?
Intermediate & Advanced SEO | | purplar0 -
Emergency Help...
Hello All, I'm trying to get a better handle on this, but any help would be hugely appreciated. Per my Pro account, i just found out that the keyword i was severely trying to rank for "Boston Wedding Phot*grapher" i just declined by over 40 positions. Just last week i was in the #3 position. Needless to say, this is extremely bad. I feel sick from it. This is my livelyhood. I recently hired a 'so-called' SEO expert to look at it, but i'm having my doubts. I'm using a php based site with a wordpress blog. He added a bunch of 301 redirects from pages that the crawler was complaining about to my .htaccess file. He also installed the following plugins: Link Juice Keeper NoFollow Free The SEO Rich Snippets Udinra All Image Sitemap WP Robots Txt WP-PageNavi Add Meta Tags These are essentially the only changes made. Does anyone see anything blaring and/or obvious? I could really really use some help. My blog link is : http://www.symbolphoto.com/blog/ I'm assuming it's the blog because that's where most of my site content is located. Any advice is hugely appreciated. TIA.
Intermediate & Advanced SEO | | symbolphoto0