Ignore Urls with pattern.
-
I have 7000 warnings of urls because of a 302 redirect.
http://imageshack.us/photo/my-images/215/44060409.png/
I want to get rid of those, is it possible to get rid of the Urls with robots.txt.
For example that it does not crawl anything that has /product_compare/ in its url?
Thank you
-
in case they do not all start with /category
Disallow: product_compare
-
Then you simply add this to your robots.txt:
Disallow: /catalog/product_compare/
That should leave out all pages starting with:
https://www.theprinterdepo.com/catalog/product_compare/ -
-
Could you perhaps post a URL which has product_compare in it?
You could alter your robots.txt file to disallow robots to index pages in
http//www.domain.com/product_compare/ by adding this line to your robots.txt file: Disallow: /product_compare/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Clean URL vs. Parameter URL and Using Canonical URL...That's a Mouthfull!
Hi Everyone, I a currently migrating a Magento site over to Shopify Plus and have a question about best practices for using the canonical URL. There is a competitor that I believe is not doing it the correct way, so I want to make sure my way is the better choice. With 'Vendor Pages' in Shopify, they show up looking like: https://www.campusprotein.com/collections/vendors?q=Cellucor. Not as clean. Problem is that Shopify also creates https://www.campusprotein.com/collections/cellucor. Same products, same page, just a different more clean URL. I am seeing both indexed in Google. What I want to do is basically create a canonical URL from the URL with the parameter that points to the clean URL. The two pages are very similar. The only difference is that the clean URL page has some additional content at the top of the page. I would say the two pages are 90% the same. Do you see any issue with that?
Technical SEO | | vetofunk0 -
How important is keyword usage in the URL?
Hi,
Technical SEO | | Whebb
We have a client who has engaged us recently for some SEO work and most of their website looks pretty good seo-wise already. Many of their site pages rank at the top or middle of page two for their targeted keywords. In many cases they are not using the targeted keyword in the URL and most pages could use some additional on-page clean up. My question is, is it worth it to re-write the URLs to include the targeted keyword and then do 301 redirects to send the old pages to the new ones in order to improve the ranking? Or should we just do the minor on page work in hopes that this will be enough to improve the rankings and push them on to the first page. Thanks.0 -
Keyword in URL vs organization
I have a jobs site that currently has the following structure for jobs: www.site.com/jobs/openings/1234.html Categories used to be listed this way: www.site.com/jobs/openings/accounting But we changed it to: www.site.com/jobs/category/accounting Does it matter? Is one better than the other? The page title, heading, and description also have the word "openings" or "opening" in them.
Technical SEO | | cmp1010 -
Approved Word Separators in URLs
Hi There, We are in the process of revamping our URL structure and my devs tell me they have a technical problem using a hyphen as a word separator. There's a whole lot of competing recommendations out there and at this point I'm just confused. Does anyone have any idea what character would be next-best to the hyphen for separating words in a URL? Any reason to prefer one over another? Some links I've found discussing the topic: This page says that "__Google has confirmed that the point (.), the comma (,) and the hyphen (-) are valid word separators in URL’s.": http://www.internetofficer.com/seo/google-word-separator/ This page suggests the plus (+) symbol would be best: http://labs.phurix.net/posts/word-separators-in-urls This guy says he's tested and there's a whole bunch of symbols that will work as word separators: http://www.webproguide.com/articles/Symbols-as-word-separators-a-look-inside-the-search-engine-logic/ I'm leaning towards the tilde (~) or the plus (+) sign. Usage would be like so: http://www.domain.com/shop/sterling~silver OR /shop/sterling+silver etc... Thanks in advance for your help!
Technical SEO | | Richline_Digital1 -
Advice on strange URL problem
I'm considering doing some pro bono work for a local non-profit and upon initial review they have a number of serious issues but there is one in particular I'd like to check my thinking on. The developer who set up the site some years ago implemented a javascript redirect on their root domain so that it redirects to: http://domain.com/wordpress This is wrong for all kinds of reasons and I want to recommend eliminating this redirect and getting rid of the 'wordpress' part of the path altogether. However, the site is quite established with good PR and they would take a hit by changing the path. I'd do 301 redirects to the new URLs that would not have 'wordpress' in the path in addition to other remediation. My question - is my thinking here good? It's worth it, right? The other option is just get rid of the weird redirect and keep 'wordpress' in the path but this seems unacceptable to me. Any opinions?
Technical SEO | | friendlymachine0 -
Need Help with MAGENTO - URL rewrite
Hello... Hopefully a Magento expert will stumble across this question and help me out. I have noticed that my site is no longer as prominent as it once was for specific product pages... I am looking for help in rewriting the URL's for the product pages. I want it to have xyz.com/product (which exists if you hard code it into the site) If you wind up on the product by clicking throught the categories the url looks like: xyz.com/category/subcategory/product. Does anyone know how to make it so when you land on a product page it is just xyz.com/product ? My Site is : http://goo.gl/JgK1e Thanks for the help...
Technical SEO | | Prime850 -
How do I fix these duplicate URLs?
HI guys, I ran a report on my site and it shows some duplicate titles (example below). Do I need to add something to the htaccess file or another file to fix this? I understand that the search engines should only see 1 URL for the page. 2 pages have "Bikes for sale | used bikes | second hand bicycles" title pauslwebsite.com/bikes/ paulswebsite.com/bikes/index.asp Thanks
Technical SEO | | paulmund0 -
Wordpress URL weirdness - why is google registering non-pretty URLS?
I've noticed in my stats that google is indexing some non-pretty URLs from my wordpress-based blog.
Technical SEO | | peterdbaron
For instance, this URL is appearing google search: http://www.admissionsquest.com/onboardingschools/index.php?p=439 It should be: http://www.admissionsquest.com/onboardingschools/2009/01/do-american-boarding-schools-face-growing-international-competition.html Last week I added the plugin Redirection in order to consolidate categories & tags. Any chance that this has something to do with it? Recs on how to solve this? Fyi - I've been using pretty URLS with wordpress from the very beginning and this is the first time that I've seen this issue. Thanks in advance for your help!0