Need help with Robots.txt
-
An eCommerce site built with Modx CMS. I found lots of auto generated duplicate page issue on that site. Now I need to disallow some pages from that category. Here is the actual product page url looks like
product_listing.php?cat=6857And here is the auto generated url structure
product_listing.php?cat=6857&cPath=dropship&size=19Can any one suggest how to disallow this specific category through robots.txt. I am not so familiar with Modx and this kind of link structure.
Your help will be appreciated.
Thanks
-
I would actually add a canonical tag and then handle these using the Parameters section of Search Console. That's why it's there, for exactly this type of site with exactly this issue.
-
Nahid, before you use the robots.txt file's disallow for those URLs, you may want to reconsider. You may want to use the canonical tag instead. In the case where you have different sizes, colors, etc. we typically recommend using the Canonical Tag and not the disallow in robots.txt.
Anyhow, if you'd like to use the disallow you can use one of these:
Disallow: /?
or
Disallow: /?cat=
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Urgent help needed for site move with major ranking loss
URGENT HELP/ADVICE NEEDED I am so stressed and worried about my website domain change. I desperately need advice as soon as possible. I will try my best to keep this as brief as possible. I have owned and operated my punk clothing business online at the URL toofastonline.com for 15 years now. And for a long long time we ranked #1 for punk clothing on Google & life was good. However, thanks to the arrival of several cheap marketplaces and other unanticipated changes our ranking dropped considerably. The last few years have been extremely hard on us, to say the least, we came really close to losing the business altogether. But finally after lots of hard work & long hours, things started to improve. Ranking went back up, and we were busy again. I had been toying with the idea of buying the domain TooFast.com for about 10 years, but I never had the money to do it until this now, so I made the leap and as of Jan 9, toofastonline.com became toofast.com. Unfortunately, I now know that I set up the domain change hastily, without doing any of the pre-work Google suggests to do. I didn’t know it then but I did it wrong. And our site which wasranking #7 for punk clothing on Jan. 8th is now number 51 and today is only Jan 24th! I AM PANICKING. I have looked for help, posting jobs on Shopify Experts site several times now, opening accounts with MOZ and SEM Rush, spending countless hours on the phone with GoDaddy, Shopify and even long chats with Google. I have spent all day everyday for the past two weeks trying fix everything to no avail. No one can start on my site issues fast enough. And I have been given so much wrong information that I feel like I have done irreparable damage. I was (am) not qualified to make this kind of a site change alone. Too much was done too fast and without any real working knowledge Google SEO. My brother was the SEO guy and since he left the business I have just been struggling along with it, just trying to keep my head above water. So now for the big question: Should I temporarily change my Shopify stores domain back to toofastonline.com? This way I couldstart at the beginning, fix all the 404 redirects, fix the 301 redirects, clean up code, get the site in top working condition, and then, as Google suggests in theirGoogle Search Console Change of Address Toolstart to do the change of address in small sections, I can not afford to make any more reckless decisions. I have started and stopped, updated, fixed, changed and tried to fix again too many times now. I dont want Google to think I am trying something shady.. I’m not, I just don’t know what I’m doing, and I need help. Here is as much info as I can think of, I am more than willing to pay for help or do the work myself, as long as what I am doing is the right thing. Any and all help/advice/offers are welcome! Maureen CONTACT DETAILS: NAME: Maureen Keough, Owner EM:<a style="-webkit-text-size-adjust: 100%;">Maureen@TooFast.com</a> PH: 856-599-1675 (W) DETAILS OF OUR SET-UP THE APPS & SERVICES WE USE: Google Admin / G-Suite User Gmail for emails Godaddy holds our domains Shopify hosts our storefront. My Shopify store was located at TooFastOnline.com for about 5 years Our Domain Changed From toofastonline.com to toofast.com on Jan 9 In Godaddy both toofastonline.com is being forwarded to toofast.com In Shopify I added toofast.com, made it my primary domain, but left toofastonline.com in there but it is just redirecting to toofast.com. STEPS TAKEN TO CHANGE | ADD | VERIFY THE NEW DOMAIN GoDaddy DNS Records Both Sites - Updated Pointing to Shopify’s IP Address GoDaddy Subdomains For TooFastOnline.com - Redirected But Causing SSL/HTTPS/Privacy errors GoDaddy Subdomains For TooFast.com - Added But Causing SSL/HTTPS/Privacy errors Google Admin - Updated Gmail MX Records TooFast - Added and Updated Gmail MX Records TooFastOnline - Unchanged Google Merchant Center - Updated TooFastOnline is now TooFast Google Merchant Product Feed- Updated TooFastOnline is now TooFast Google Ads - Finally got the New Feed Approved and It is Working Google Search Console - Updated I Think Sitemaps - Added and Asked To Crawl Google Analytics Added TooFast As A Property Seems To Be Working Google Analytics Tag Updated in Shopify Admin Google Search Console - Requested to Move TooFastOnline.com to TooFast.com, still not done. No Redirects were made prior to the “Move” All Social Media Channels Links were Updated By Us Mailerlite MX Records For Bulk Emails - Updated/Verified
Intermediate & Advanced SEO | | TooFast130 -
Please help us undertsand the things we need to improve so that google crawler visit us more often to reindex pages from our domain
we are currently in the process of a massive project which involves us migrating our domain, we realised that Google crawlwer has not been crawling our pages Quiet often. i have observed some cases where google crawled these pages about 6 months back and then never visited the pages again
Intermediate & Advanced SEO | | bhaskaran
and we had to manually submit these pages for reindexing in some geographies. can you please help us undertsand the things we need to improve so that google crawler visit us more often to reindex pages from our domain0 -
How to make Google index your site? (Blocked with robots.txt for a long time)
The problem is the for the long time we had a website m.imones.lt but it was blocked with robots.txt.
Intermediate & Advanced SEO | | FCRMediaLietuva
But after a long time we want Google to index it. We unblocked it 1 week or 8 days ago. But Google still does not recognize it. I type site:m.imones.lt and it says it is still blocked with robots.txt What should be the process to make Google crawl this mobile version faster? Thanks!0 -
Robots.txt help
Hi Moz Community, Google is indexing some developer pages from a previous website where I currently work: ddcblog.dev.examplewebsite.com/categories/sub-categories Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types? As always, your expertise is greatly appreciated, -Reed
Intermediate & Advanced SEO | | IceIcebaby0 -
Robots Disallow Backslash - Is it right command
Bit skeptical, as due to dynamic url and some other linkage issue, google has crawled url with backslash and asterisk character ex - www.xyz.com/\/index.php?option=com_product www.xyz.com/\"/index.php?option=com_product Now %5c is the encoded version of \ - backslash & %22 is encoded version of asterisk Need to know for command :- User-agent: * Disallow: \As am disallowing all backslash url through this - will it only remove the backslash url which are duplicates or the entire site,
Intermediate & Advanced SEO | | Modi0 -
Website is not indexed in Google, please help with suggestions
Our client website was removed from Google index. Anybody could recommend how to speed up process of re index: Webmaster tools done SM done (Twitter, FB) sitemap.xml done backlinks in process PPC done Robots.txt is fine Guys any recommendations are welcome, client is very unhappy. Thank you
Intermediate & Advanced SEO | | ThinkBDW0 -
Certain Pages Not Being Indexed - Please Help
We are having trouble getting a bulk of our pages indexed in google. Any help would be greatly appreciated! The Following Page types are being indexed through escaped fragment: http://www.cbuy.tv/#! http://www.cbuy.tv/celebrity#!65-Ashley-Tisdale/fashion/4097-Casadei-BLADE-PUMP/Product/175199 <cite>www.cbuy.tv/celebrity/155-Sophia-Bush#!</cite> However, all our pages that look like this, are not being indexed: http://www.cbuy.tv/#!Type=Photo&id=b1d18759-5e52-4a1c-9491-6fb3cb9d4b95&Katie-Holmes-Hot-Pink-Pants-Isabel-Marant-DAVID-DOUBLE-BREASTED-Wool-COAT-Maison-Pumps-Black-Bag
Intermediate & Advanced SEO | | CBuy0 -
How much great targeted conent do we need to add?
Hi, I'm adding content to a client's website through textbroker. It's ecommerce and it's tough to find backlinks. We have decided to write 100 articles of at least 500 words so that we can say in our backlink campaign email that we have 100 helpful articles. We're thinking that people would like that. Also, we think that 100 good helpful articles will give us traffic and natural backlinks. How do we know if 100 is enough? Do we need 200? 500? Thanks.
Intermediate & Advanced SEO | | BobGW0