Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disadvantages of Migrating Website to New URL
Hi There, I am currently struggling with the ranking of my website. No matter how many initiatives I try (backlinking, blog commenting, social posting, etc.) I can't seem to make any progression in Google Search. I've done competitive metrics through Open Site Explorer and can't seem to really find the reason why my site is not ranking as well as my competitors. The only one possible glaring element I've thought about is my website URL. This company is in the heating and cooling industry and majority of my competitors have either "heating" or "cooling" or both in their website URL's but mine does not. Does anyone have any thoughts or recommendations on if changing my URL and then redirecting my current URL would be a step in the right direction help me to climb the rankings in Google Search? Thanks!
On-Page Optimization | | MainstreamMktg0 -
Googlebot found an extremely high number of URLs on your site:
Website: www.gobol.in Although I have no indexed my search pages by adding /catalogsearch in robots.txt, still we are getting same error again and again Here's a list of sample URLs with potential problems. http://www.gobol.in/catalogsearch/result/index/?category=&mobile_feature=4575_4578&q=panasonic+NR-BU303LH1H+REFRIGERATOR+296+L+GREY&special_price=32%2C456&x=0&y=0 http://www.gobol.in/mobile-and-accessories/mobiles-and-brands.html?manufacturer=4753_3355_455_4435_4720_3407_2412_4728_4784_4790_2010_4789_4376_2469&operating_system_mobile=4612 Please help
On-Page Optimization | | Obbserv0 -
My home page URL http://seadwellers.com/ redirects to http://www.seadwellers.com/. Is this a problem?
"The URL http://seadwellers.com/ redirects to http://www.seadwellers.com/. Do you want to crawl http://www.seadwellers.com/ instead?" I was given this when I tried to crawl my home page using MOZ software. I was not aware of this, do not know if it could be a problem concerning any aspect of SEO, etc? :
On-Page Optimization | | sdwellers0 -
Redirects for new site new urls?
If redoing a site and updating some of the url's for SEO should you do permanent redirects for the old sites url's ? Using WordPress. I saw that the Yoast Pro plugin allows you to do this inside WordPress , is this the best way? Suggestions.? I know there are old articles written out on the web pointing back to the what will be soon old url's so just wondering what's the best way to go about this. Thanks Scott
On-Page Optimization | | scott3150 -
URL structure
Hello all, I am about to sort out my websites link structure, and was wondering which approach to our services page would be best. should we have: services/digital-marketing & services/website-design etc or: digital-marketing/website-design & digital-marketing/seo Basically I see digital marketing as the top level category that is the umbrella term for all of our digital services. But would it make more sense to have service to be the main category and digital marketing within that (along with all the other services from web design to seo)? all thoughts welcome!
On-Page Optimization | | wseabrook0 -
Why is my website not showing up in a specific, very uncompetitive niche?
My website just went live about 3-4 weeks ago and while it has been indexed by Google-it shows up when I do a search for my site's name, it does not show up for the keyword terms it has been optimized for. This is a very specific, non-competitive niche-so I just can't imagine why it would not be showing for these non-competitive keyword term searches? Any insight would be greatly appreciated!!! Thanks
On-Page Optimization | | lfrazer0 -
If i only want to rank for one specific keyword and use it in all my page titles, will it negatively affect my rankings?
If i want to rank highest for one specific keyword (virtualization management, for example) and use that keyword in all the titles on my website, will that negatively affect my search rankings? SEOmoz is telling me that i should use unique titles for my different pages to ensure that they describe each page uniquely and don't compete with each other for keyword relevance.
On-Page Optimization | | foonista0 -
Cononical URL problem
Hello, In the keywords difficulty tool, one of my client's sites has a different number for links to root domain and links to page even though they're both the home page. How do I tell if I have a cononical URL problem, and what do I do about it if I do? Thank you.
On-Page Optimization | | BobGW0