Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Query for paginated URLs - Shopify
Hi there, /collections/living-room-furniture?page=2
On-Page Optimization | | williamhuynh
/collections/living-room-furniture?page=3
/collections/living-room-furniture?page=4 Is that ok to make all the above paginated URLs canonicalised with their main category /collections/living-room-furniture Also, does it needs to be noindex, follow as well? Please advice, thank you!1 -
Newbie SEO ?: Does my About page URL have to contain the word About?
New to WordPress and SEO. Built and launched my website last week. The URL was originally domain/about. However, I installed Yoast plugin and it told me "about" was a stop word. So, without too much thought (my first problem), I changed the url (before Google crawled me) to clearwingcommunications.com/storytelling. Since then, I've noticed that sites I know are optimized have their URL with the word "about." So, is this considered a bad practice? My site HAS been crawled at this point. If I change it back to About and do a 301 redirect, does that hurt reporting? Thanks for your help! Christy
On-Page Optimization | | christyr0 -
Homepage canonical url with splash or not with splash? All other links are without but logo links with splash
Hello, There is so much contradicting information about the homepage canonical URL. Many websites have all the links without the trailing splash but their homepage URL still contains the splash. Now Moz is an example with this. Their urls don't have the splash, and their canonical does not have the splash. Why is it so and why so much different ways people have it?
On-Page Optimization | | advertisingcloud0 -
Hi - How do you get rid of duplicate content that was accidentally created on a tag url? For example, when I published a new article, the content was duplicated on: /posts/tag/lead-generation/
the original article was created with: /posts/shippers-looking-for-freight-brokers/ How can I fix this so a new URL is not created every time I add a tag to a new posting?
On-Page Optimization | | treetopgrowthstrategy0 -
How do I manage my organic SEO efforts with multiple product descriptions/specifications???
Good Evening MOZ community I am currently working on a website my company has just launched. I am in charge of the entire commercial side of the business from pricing to brand management including all elements of digital marketing. First of all I should make it clear I am by no means an SEO expert. However I appreciate the importance of organic search and particularly on page optimization and no I need to build it into our site from the onset. My problem lies in that our website stocks roughly 2000 individual products (Tools and Engineering products) and I am struggling to overcome the following; 1. I do not have the time or resource to re-write original product descriptions and specifications for 2000+ products 2. How can I create healthy on page SEO without having to do this? (we are not able to provide budget to outsource these efforts at the moment) Will copying manufacturer descriptions cripple our SEO 3. Many product specifications need to be displayed exactly as they would on the original manufacturer websites. How can I include exact copies of these without being penalized by google for duplicating content? Essentially I realise the importance of on page SEO and want to build it into the site for the future but need to find a way of optimizing it with minimal time. I understand much of this is a big ask but any hints or tips you may have would be greatly appreciated. Thanks Rob 🙂
On-Page Optimization | | robmarr7890 -
URL question
When we type in the URL of www.JustBunkBeds.com on firefox we end up with (S) in URL https://www.justbunkbeds.com/ When we type in the URL of www.JustBunkBeds.com on Explorer we end up with http://www.justbunkbeds.com/ Appreciate answer to this question Tony
On-Page Optimization | | OCFurniture0 -
URLs and folder structure for an E-commerce
Hi there !-) I´m helping a friend who has a e-commerce about nail polish in Brazil. I´m a little in doubt about the urls and folder structure. Two questions: 1. There are 10 products per category and 50 categories. Should I put them all in the root folder or creat 2 major categories ( 25 sub-categories each one)? 2. Whats the better product page url ( the store has around 500) nailpolish.com/IMPORT/BRAND/NAME-OF-THE-PRODUCT OR nailpolish.com/COMPLETE-NAME-OF-THE-PRODUCT Whats the best recomandation?
On-Page Optimization | | SeoMartin10 -
Does google treat all urls equal?
Sorry for the lame title, i couldn't think of a better one. I want to know if google treats this: http://www.domain.com/products/some-product-name the same as it would treat: http://www.domain.com/?products=some-product-name if not, could you tell me the differences?
On-Page Optimization | | adriandg0
