Need help with Robots.txt
-
An eCommerce site built with Modx CMS. I found lots of auto generated duplicate page issue on that site. Now I need to disallow some pages from that category. Here is the actual product page url looks like
product_listing.php?cat=6857And here is the auto generated url structure
product_listing.php?cat=6857&cPath=dropship&size=19Can any one suggest how to disallow this specific category through robots.txt. I am not so familiar with Modx and this kind of link structure.
Your help will be appreciated.
Thanks
-
I would actually add a canonical tag and then handle these using the Parameters section of Search Console. That's why it's there, for exactly this type of site with exactly this issue.
-
Nahid, before you use the robots.txt file's disallow for those URLs, you may want to reconsider. You may want to use the canonical tag instead. In the case where you have different sizes, colors, etc. we typically recommend using the Canonical Tag and not the disallow in robots.txt.
Anyhow, if you'd like to use the disallow you can use one of these:
Disallow: /?
or
Disallow: /?cat=
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots blocked by pages webmasters tools
a mistake made in software. How can I solve the problem quickly? help me. XTRjH
Intermediate & Advanced SEO | | mihoreis0 -
Should I disallow all URL query strings/parameters in Robots.txt?
Webmaster Tools correctly identifies the query strings/parameters used in my URLs, but still reports duplicate title tags and meta descriptions for the original URL and the versions with parameters. For example, Webmaster Tools would report duplicates for the following URLs, despite it correctly identifying the "cat_id" and "kw" parameters: /Mulligan-Practitioner-CD-ROM
Intermediate & Advanced SEO | | jmorehouse
/Mulligan-Practitioner-CD-ROM?cat_id=87
/Mulligan-Practitioner-CD-ROM?kw=CROM Additionally, theses pages have self-referential canonical tags, so I would think I'd be covered, but I recently read that another Mozzer saw a great improvement after disallowing all query/parameter URLs, despite Webmaster Tools not reporting any errors. As I see it, I have two options: Manually tell Google that these parameters have no effect on page content via the URL Parameters section in Webmaster Tools (in case Google is unable to automatically detect this, and I am being penalized as a result). Add "Disallow: *?" to hide all query/parameter URLs from Google. My concern here is that most backlinks include the parameters, and in some cases these parameter URLs outrank the original. Any thoughts?0 -
Using folder blocked by robots.txt before uploaded to indexed folder - is that OK?
I have a folder "testing" within my domain which is a folder added to the robots.txt. My web developers use that folder "testing" when we are creating new content before uploading to an indexed folder. So the content is uploaded to the "testing" folder at first (which is blocked by robots.txt) and later uploaded to an indexed folder, yet permanently keeping the content in the "testing" folder. Actually, my entire website's content is located within the "testing" - so same URL structure for all pages as indexed pages, except it starts with the "testing/" folder. Question: even though the "testing" folder will not be indexed by search engines, is there a chance search engines notice that the content is at first uploaded to the "testing" folder and therefore the indexed folder is not guaranteed to get the content credit, since search engines see the content in the "testing" folder, despite the "testing" folder being blocked by robots.txt? Would it be better that I password protecting this "testing" folder? Thx
Intermediate & Advanced SEO | | khi50 -
Need onpage site audit and seo
i have a pretty old ecommerce website for home decor products. It has been experiencing some rank loss in the past year. No manual penalty but algo rank losses. I need someone to fix seo related issues on my site. It runs on magento with multistore configuration. please reply if you can offer any help nick
Intermediate & Advanced SEO | | orion680 -
Please, help me to understand these Google results
Hello here, I am eager to know your thoughts on this. If I search on Google for "fur elise violin sheet music", we are on the second page for our sheet music title of "Fur Elise for violin and piano" (look for "virtualsheetmusic.com"). Ok, that's not very good and I still have an hard time to figure out why there are many crappy and NOT really related websites listed before us, but here is the best (weird) part... .... search now for "fur elise violin and piano sheet music" which should narrow the query further down and so increase the chances for us to get on the first page results... and in fact we are on the first page with that query, but for a different page and a different music for a different instrument! If you scroll the first page of the results, you will find our site at the end of the 1st page for our version of "Fur Elise" for "viola and piano" and not for "violin and piano"... What the heck!??! Why's that??? Doesn't make any sense too me... why if the user search for "fur elise violin and piano" Google shows "Fur Elise for viola and piano"???!! I would really appreciate any thoughts on all this. Thank you in advance!
Intermediate & Advanced SEO | | fablau0 -
Approximate linking root domains we need based on these metrics
Our top 4 competitors for a single term we're targeting has the following metrics: PA 45, DA 89, 6 linking root domains to page, 40,000 linking root domains to domain PA 53, DA 100, 3 linking root domains to page, 1.6 million to domain PA 32 DA 37, 4 linking root domains to page, 200 to domain PA 55 DA 66, 6 linking root domains to page, 3300 to domain All other optimization is about the same, except in (2) they have half of the keyword phrase in the domain and the whole keyword phrase in the URL. Also everybody else has title and meta description with the plural form, and the singular is what I typed in. We have the whole keyword phrase in the domain. The above 4 sites were internal pages, ours is a home page rank. Our metrics: PA 33, DA 22, 30 linking root domains to page, 43 linking root domains to site How tough will it be for us to compete? How many strong linking root domains will it take?
Intermediate & Advanced SEO | | BobGW0 -
301 redirect help
Hey guys, I normally work in WordPress and just use a 301 redirect plugin. I bought a site and rather than maintain two similar ones have decided to redirect one to the other. I am having trouble with the .htaccess file. Here is an example. These are two redirects: redirect 301 /category/models/next/2
Intermediate & Advanced SEO | | DanDeceuster
redirect 301 /category/models I want both of these URLs to redirect to the same URL of the new site. However, the /category/models is the only one working. It redirects to the new page just fine. The /category/models/next/2 is redirecting to nearly the same URL on the new site, only it is adding /next/2 to the end and that is bringing up a 404. Why is it adding /next/2 to the new URL? How can I fix this? There are several doing this. Help appreciated!0 -
Robots.txt & url removal vs. noindex, follow?
When de-indexing pages from google, what are the pros & cons of each of the below two options: robots.txt & requesting url removal from google webmasters Use the noindex, follow meta tag on all doctor profile pages Keep the URLs in the Sitemap file so that Google will recrawl them and find the noindex meta tag make sure that they're not disallowed by the robots.txt file
Intermediate & Advanced SEO | | nicole.healthline0
