Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When using long-tail keywords, should you exactly match for the url or delete "in" "to" etc.?
long-tail keyword - "seizures in adults with no history" Should you include "in and with" in the url?
On-Page Optimization | | Moleculera0 -
Ecommerce URLs with numbers
Hi everybody! I have to optimize an ecommerce where somebody has previously done the SEO optimization, although the URLs have numbers before the product's name They have told me that these numbers are useful to find the products, so I think it shouldn't be really bad if I don't redirect them to "clear" ones. For example: /colesterol-sobrepeso/2217-hc-grass-capsulas-duras-15-capsulas.html > /colesterol-sobrepeso/hc-grass-capsulas-duras-15-capsulas.html Am I right? After all, they contain the keywords and the subfolders are also ok. Or it would be better if I redirect the whole site? Thanks!
On-Page Optimization | | Estherpuntu0 -
Changing existing URL's to improve SEO
Here's a general question: At what PR/page rank (or Moz 'page authority') would you no longer change an existing URL that's cryptic to one that includes the related page keyword or at least relevant terms? Does using a 301 redirect to the new URL preserve the page rank? Thanks!
On-Page Optimization | | flyntime_tx0 -
Www.colourbanners.co.uk/ & colourbanners.co.uk showing up as two seperate URLs - is this going to be dupliacte content issue?
Hi Guys, I have just created a report in Moz and there appears to be 91 duplicate content issues with the site which i need to fix as i think it could be the reason why we are suffering from a penalty. One of the main questions i have is these 3 variations of the URL http://www.colourbanners.co.uk/ http://colourbanners.co.ukhttp://colourbanners.co.uk/Each have links pointing to them. My question is, could this be causing a dupe issue?regardsGerry
On-Page Optimization | | gezzagregz0 -
URL Length
I know a URL should "technically" shorter than 75 characters. Does that include the http://www.domainname.com ? Thank you 🙂
On-Page Optimization | | Libra0130 -
Keyword in URL?
I have a website that has been live for about 8yrs. I do not have any significant rankings for my main keywords but am now starting SEO on my site. I am contemplating changing the url to contain the main keyword prefixed by my brand name. Any views on the ranking benefits and or CTR benefits.
On-Page Optimization | | Johnnyh
Example:
Main Volume keyword - 'car leasing'
current url - www.bobleasing.co.uk (made up name) thinking of changing to - www.bobcarleasing.co.uk (made up name) Any advice would be much appreciated. John0 -
Search engine friendly URLs
I'm going to create some new content for my site, I'm trying to decide on the best search engine friendly format. Namely, is it ok to use a subdirectory or should I keep all content on root level? Is the SEO effect of either of these URLs superior to the other? domain.com/cooking/lasagna.php vs domain.com/lasagna.php
On-Page Optimization | | limens0 -
Redirecting to a keyword-rich domain URL
It's best practice to choose a domain that has keyword in it. But if someone has just launched a website and the domain name does not have keyword, is it better to purchase a new domain name that has a keyword in the name and redirect existing domain to the new domain? Will that help SEO? (This just launched website does not have any traffic or links yet.)
On-Page Optimization | | Amjath0