Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Tips on URL structure for a site re-design
Wanted to know what you would do with regards to urls – in an ideal world how would you structure them? Keen to know as me and dave are soon to have a meeting about this and were wondering about changing them from the current – http://www.looking4parking.com/airport/gatwick to something like - www.looking4parking.com/gatwick-airport-parking We will soon have pages for the specific parking types that will be a lot more engaging to users with some really useful content on benefits, features, how a certain type of parking works, images, video etc. Currently going to a type of parking, such as meet and greet just brings up a dropdown modal – I was thinking of having the url structure looking like this – www.looking4parking.com/gatwick-airport-meet-and-greet-parking www.looking4parking.com/gatwick-airport-on-site-parking www.looking4parking.com/gatwick-airport-park-and-ride We will then have specific pages for each parking product – in which this product will have unique content built around it – each will have an overview of the product, benefits, features, reviews, images, directions to the car park, find your route and eventually a video on each product So for example we currently have the product “Jet Parks 2” at Manchester airport – the current url is - http://www.looking4parking.com/airport/manchester/park-and-ride/jetparks-2 I would like to change this now we have the opportunity to refresh the whole system, to something along the lines of **domain/location/product title - **www.looking4parking.com/manchester-airport-parking/jetparks-2 or as we have some similar products at certain airports (mainly where the airport has multiple terminals) we would just change it to the following - www.looking4parking.com/manchester-airport-parking/jetparks-3 What are peoples thoughts/opinions on the above?
On-Page Optimization | | RyanCrawf19840 -
Question about url structure for large real estate website
I've been running a large real estate rental website for the past few years and on May 8, 2013 my Google traffic dropped by about 50%. I'm concerned that my current url structure might be causing thin content pages for certain rental type + location searches. My current directory structure is:
On-Page Optimization | | Amped
domain.com/home-rentals/california/
domain.com/home-rentals/california/beverly-hills/
domain.com/home-rentals/california/beverly-hills/90210/
domain.com/apartment-rentals/california/
domain.com/apartment-rentals/california/beverly-hills/
domain.com/apartment-rentals/california/beverly-hills/90210/
etc.. I was thinking of changing it to the following:
domain.com/rentals/california/
domain.com/rentals/california/beverly-hills/
domain.com/rentals/california/beverly-hills/90210/ ** Note: I'd provide users the ability to filter their results by rental type - by default all types would be displayed. Another question - my listing pages are currently displayed as:
domain.com/123456 And I've been thinking of changing it to:
domain.com/123456-123-Street-City-State-Zip Should I proceed with both changes - one or the one - neither - or something else I'm not thinking of? Thank you in advance!!0 -
Which URL Structure is best for New Website?
Hello All, I have one client and he want to develop his website but the he want the URL structure for his website page like below: http://www.example.com/category/xxproduct.html But I have suggest him below URL Structure http://www.example.com/category/xx-product.html So Can you people please suggest me that from above which URL structure is better from SEO side as well as from Visual side..? Is using - is better to separate the words in URL?
On-Page Optimization | | jemindesai0 -
What are the best practices for google indexing ajax response urls?
We just did a build to our site and our server erros went up to over 9,500. After looking into it, it seems like google is crawling the ajax urls and coming back with the errors. Here is one example http://www.rockymountainatvmc.com/productDetail.do?navType=type&vehicleId=1423&webTypeId=68&navTitle=Drive&webCatId=9&prodFamilyId=29660 If you know of any good articles on this please send them my way.
On-Page Optimization | | DoRM0 -
Replacing "_" with "-" in url, results in new url?
We ran SEOmoz's "On-Page Optimization" tool on a url which contains the character "_". According to the tool: "Characters which are less commonly used in URLs may cause problems with accessibility, interpretation and ranking in search engines. It is considered a best practice to stick to standard URL structures to avoid potential problems." "Rewrite the URL to contain only standard characters." Therefore we will rewrite the url, replacing "_" with "-". Will search engines consider the "-" url a different one? Do we need to 301 the old url to the new one? Thanks for your help!
On-Page Optimization | | gerardoH0 -
Is it better to embed my longtail article or give it a separate url?
Within my e-commerce site I have jewelers who using uncommon techniques and maetrials. I have a few long tail type article ready to publish about these niche topics. My site navigation has each jeweler as a category with their often changing products within their category. I am thinking I would add an article to the artist-category content. But in the past, I have put "how to" or "what is" content in an article section of the site. This way I could link to it from several places. With the long tail in mind, would I be better off adding the article to the jeweler's category page? If I have a 2nd jeweler using this same technique, I am thinking I would rewrite the 1st article including different long tail phrases. Thank You for Helping- Handcrafter
On-Page Optimization | | stephenfishman0 -
Url question for seo
Would it be beneficial to use url that is a match for my keyword to help with seo, then have my currently url forward to that one so I don't have to change any marketing materials? I was one of the feedback that I got when doing the on page keyword optimization tool on seo moz. Thanks J
On-Page Optimization | | fertilityhealth0 -
Is it worth changing urls with underscores?
A few pages on one of my sites have underscores linking keywords rather than hyphens (keywords_and_keyword rather than keyword-and-keyword). Possibly from a time before I knew hyphens were preferred... One of the pages ranks well, and drives a good amount of traffic. The others do not do so well, but are still within the top 10 landing pages for the site. Is it worth me changing the underscores to hyphens (setting up 301 redirects first of course) or doesn't it make that much difference?
On-Page Optimization | | Jingo010
