Ignore Urls with pattern.
-
I have 7000 warnings of urls because of a 302 redirect.
http://imageshack.us/photo/my-images/215/44060409.png/
I want to get rid of those, is it possible to get rid of the Urls with robots.txt.
For example that it does not crawl anything that has /product_compare/ in its url?
Thank you
-
in case they do not all start with /category
Disallow: product_compare
-
Then you simply add this to your robots.txt:
Disallow: /catalog/product_compare/
That should leave out all pages starting with:
https://www.theprinterdepo.com/catalog/product_compare/ -
-
Could you perhaps post a URL which has product_compare in it?
You could alter your robots.txt file to disallow robots to index pages in
http//www.domain.com/product_compare/ by adding this line to your robots.txt file: Disallow: /product_compare/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have multiple URLs that redirect to the same website. Is this an issue?
I have multiple URLs that all lead to the same website. Years ago they were purchased and were sitting dormant. Currently they are 301 redirects and each of the URLs feed to different areas of my website. Should I be worried about losing authority? And if so, is there a better way to do this?
Technical SEO | | undrdog990 -
Want to change URL for a page
Hey there Mozzers. I want to change the url of a certain page on my website. Example: www.example.com/poker-face I want to change this www.example.com/poker-faces Should I create a new page and make the old one 301? Does 301 pass all the link juice in the new page or do i have to make a rel=canonical also ?
Technical SEO | | Angelos_Savvaidis0 -
What is the best practice to seperate different locations and languages in an URL? At the moment the URL is www.abc.com/ch/de. Is there a better way to structure the URL from an SEO perspective?
I am looking for a solution for using a new URL structure without using www.abc.com**/ch/de** in the URL to deliver the right languages in specific countries where more than one language are spoken commonly. I am looking forward to your ideas!
Technical SEO | | eviom0 -
SEO friendly url strategy...
Hi guys i just wanted your expert opinion on keywords in urls. The example i'm giving you is in regards to a ecommerce website: Option 1: www.example.com/shop/coffee/coffee-beans/brand-coffee-beans-500gr Option 2: www.example.com/shop/coffee/beans/brand-coffee-beans-500gr We sell coffee so i'll keep the example relevant 🙂 Does it make a difference on how the keywords are stacked throughout? Would the search engine combine the two keywords eg. .../coffee/beans/... or would i be better of having .../coffee/coffee-beans/... and is there a penalty for having the same phrase more than once in the url? I hope my question makes sense... 😉 Looking forward to your opinions and ideas!
Technical SEO | | Immanuel0 -
Page URL Change
We're planning on rolling out a redesign of an existing page, and at the same time, we're looking to possibly changing the URL of the page. Currently, the URL is www.blah.com/phraseword1-phraseword2-phraseword3-phraseword4 and we're ranking top 3 in Google SERP for that 4-word phrase. The keyword phrase is something we have in our Page Title, Site Copy and the URL. Now, we are planning on simplifying the URL to below.. www.blah.com/phraseword1-phraseword2 The plan is to 301 redirect the original URL to this new URL and actually work the exact phrase into the copy a few more times. My understanding is that URL doesn't get as much weight as it does in the past, but it's still important. So my question is... How important is the URL in this case where we will continue to have it in our page title and also we'll be working more copy on to the page with the appropriate keyword? Will 301 redirect from the old URL address the issue of passing SEO value for that keyword phrase? Thanks,
Technical SEO | | JoeLin
Joe0 -
Spider Indexed Disallowed URLs
Hi there, In order to reduce the huge amount of duplicate content and titles for a cliënt, we have disallowed all spiders for some areas of the site in August via the robots.txt-file. This was followed by a huge decrease in errors in our SEOmoz crawl report, which, of course, made us satisfied. In the meanwhile, we haven't changed anything in the back-end, robots.txt-file, FTP, website or anything. But our crawl report came in this November and all of a sudden all the errors where back. We've checked the errors and noticed URLs that are definitly disallowed. The disallowment of these URLs is also verified by our Google Webmaster Tools, other robots.txt-checkers and when we search for a disallowed URL in Google, it says that it's blocked for spiders. Where did these errors came from? Was it the SEOmoz spider that broke our disallowment or something? You can see the drop and the increase in errors in the attached image. Thanks in advance. [
Technical SEO | | ooseoo
](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a> LAAFj.jpg
0 -
Blogger Blog URL Structure Questions
I'm starting to use my blog more and wanted to ask about an issue I've read about on SEOmoz in the past. I use blogger instead of wordpress. It's quick and simple - I have no interest in switching to wordpress for this particular blog. My blog is currently setup as blog.site.com. Is it still important (for seo reasons) to switch from blog.site.com to site.com/blog? If so, is there a way to do this in blogger? And if I do this, will my past posts lose their authority if their redirected to the new url structure? Rand mentions in this article: http://www.seomoz.org/blog/11-best-practices-for-urls "never use multiple subdomains" - This is an old article, but I've seen this mentioned several times. Does this still hold true? Am I losing out on links to my blog? Thanks in advance.
Technical SEO | | ChaseH0 -
URL rewriting from subcategory to category
Hello everybody! I have quite simple question about URL rewriting from subcategory to category, yet I can't find any solution to this problem (due to lack of my deeper apache programming knowledge). Here is my problem/question: we have two website url structures that causes dublicate problems: www.website.lt/language/category/ www.website.lt/language/category/1/ 1 and 2 pages are absolutely same (both also returns 200 OK). What we need is 301 redirect from 2 to 1 without any other deeper categories redirects (like www.website.com/language/category/1/169/ redirecting to .../category/1/ or .../category/). Here goes .htaccess URL rewrite rules: RewriteRule ^([^/]{1,3})/([^/]+)/([^/]+)/([^/]+)/([^/]+)/([^/]+)/$ /index.php?lang=$1&idr=$2&par1=$3&par2=$4&par3=$5&par4=$6&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/([^/]+)/([^/]+)/([^/]+)/([^/]+)/$ /index.php?lang=$1&idr=$2&par1=$3&par2=$4&par3=$5&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/([^/]+)/([^/]+)/([^/]+)/$ /index.php?lang=$1&idr=$2&par1=$3&par2=$4&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/([^/]+)/([^/]+)/$ /index.php?lang=$1&idr=$2&par1=$3&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/([^/]+)/$ /index.php?lang=$1&idr=$2&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/$ /index.php?lang=$1&%{QUERY_STRING} [L] There are other redirects that handles non-www to www and related issues: RedirectMatch 301 ^/lt/$ http://www.domain.lt/ RewriteCond %{HTTP_HOST} ^domain.lt RewriteRule (.*) http://www.domain.lt/$1 [R=301,L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_URI} !(.)/$RewriteRule ^(.)$ http://www.domain.lt/$1/ [R=301,L] At this moment we cannot solve this problem with rel canonical (due to our CMS limits). Thanks for your help guys! If You need any other details on our coding, just let me know.
Technical SEO | | jkundrotas0