Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirect chains error on the home page URL
Hello Everyone, I'm getting redirect chains error on the home page URL:
On-Page Optimization | | Nikhil_Falcon
http://ebitdacatalyst.com in Wordpress. I've checked my redirection list in the plugin, and haven't found any redirections on http://ebitdacatalyst.com. Can anyone please help me in solving this issue? I don't know where from it's coming.0 -
Value of URL Changes
Hi Guys, I have a question. Each product listed on my webstie has product number like /product.php?id=3624. After I spent many hours with MOZ, I figured out that this approach is wrong and I should use the product name as URL to achieve better SEO performance. Now I am planing to change the URL generating algoritm but should I do it for existing products. Some of them have already been linked to external websites. I am thinking to create mirror URLs but this may cause rather damage on my website. Do you know what is the right answer? Best, Tony
On-Page Optimization | | Threeding.com0 -
Multi Keyword URL Ranking at Number 1
Here is part of a URL that takes the local number 1 spot for "implant dentist glasgow" [website] /implant-dentistry-glasgow-scotland/implant-dentistry-glasgow-scotland.html The first /implant-dentistry-glasgow-scotland/ directory or page is protected and presumably just exists for ranking reasons. I am tempted to copy that URL on a client's implant page to compete for the keyword (I believe I have better content). Given that it works well for the other site, can you think of any reason that would that be a bad idea? Thanks very much.
On-Page Optimization | | neilmac0 -
How do you handle URLs with slashes?
I asked this question before, but with a different scenario. I upgraded my plan to a more advanced cart and all of my URLs changed about 1.5 years ago. I knew nothing about redirects and such, so none of that was done. Basically, let's say my site was: http://www.abc.com, but when people actually visit my site, they are directed to https://www.abc.com/. I have asked my host about redirecting and she that it is not possible. In the past, the link shared has been just www.abc.com . Will this hurt my ranking? My second question is ...let's say I have a link http://www.abc.com/blog , but now, the link is http://www.abc.com/blog/ . Will I be affected, since all my old links omit the slash?
On-Page Optimization | | tiffany11030 -
Should a title tag and the URL be a 100 percent match?
My understanding is that a URL should be as short as possible and also match the title tag, but in order to keep the URL shorter, can you abbreviate it? For example: Title Tag: Eat Your Way to Beauty with Superfoods URL: websitename.com/sbeauty-with-superfoods
On-Page Optimization | | KimCalvert0 -
Hierarchy and consistency in ecommerce URLs
One of the first things I remember reading about SEO and URLs, a long time ago, is that keywords are important, and hierarchy is important, for search engines and for users. Hierarchy in URLs would give the search engines an idea of the structure of the site, and users would be able to edit the URLs to continue navigating. I'm wondering about URLs, hierarchy and usability lately, since I've seen that ASOS uses a new URL structure on their site. At first glance, I thought it was brilliant, so I would like to get all of your opinions as well. For those of you that haven't seen the URLs: for categories, ASOS uses a structure as you would expect it, but for products they don't insert the category in the URL. Instead they insert the brand name as the first part of the URL, followed by the product title. Some examples: Category:
On-Page Optimization | | DocdataCommerce
www.asos.com/women/dresses/... Product:
www.asos.com/french-connection/french-connection-tie-waist-pocket-stripe-dress/... I can see the importance of brand name for a site like ASOS, and like how they stressed this by inserting not the category but the brand for products. I don't know how much ASOS still relies on organic non-ASOS related keyword traffic, but still. Now, for hierarchy, I guess a good internal linking structure will tell the search engines about the hierarchy of a site as well, right? So perhaps hierarchy in the URL isn't that important? Perhaps something like this would be just as good as anything, given a good internal link structure? www.onlinestore.com/category/
www.onlinestore.com/subcategory/
www.onlinestore.com/brand/product-title/ Now, I understand that if you use this structure, you wouldn't be able to have men/shirts and women/shirts, but let's say that you don't have subcategories that use the same names. In this case, how important is hierarchy? And, what do you think about this URL structure for an ecommerce site for which brands are important?0 -
3 Different Home Page URL's Being Indexed?
Hello Everyone! I own a dog supplies eCom site on the x-cart platform. I recently upgraded to 4.4 version about 3 weeks ago and am noticing 3 different home page URL's getting indexed and ranked: /
On-Page Optimization | | k9byron
/home.php
/home.php?cat= I dont know why this is happening and I dont claim to be an expert SEO but know this cant be good! I am seeing high rankings on certain terms for all 3 URL's. Has anyone seen this before and can anyone give me any feedback on this and how it may be effecting my sites ranking in the future? Thanks in advance!
Byron-0 -
Would it be bad to change the canonical URL to the most recent page that has duplicate content, or should we just 301 redirect to the new page?
Is it bad to change the canonical URL in the tag, meaning does it lose it's stats? If we add a new page that may have duplicate content, but we want that page to be indexed over the older pages, should we just change the canonical page or redirect from the original canonical page? Thanks so much! -Amy
On-Page Optimization | | MeghanPrudencio0