Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate home page URL on crawl test
Hi i just recently made a crawl test but before doing that i made sure that i have no more duplicates on my site i am using joomla and as of now i only have 11 links on my site but when my crawl test is done i saw duplicate url of my homepage the duplicate url has a trailing backslash so basically i have all the 11 links + 1 duplicate URL http://mangthomas.com http://mangthomas.com/ can you guys give advise how i can remove the duplicate i dont even know which one to retain. THANKS A LOT, cris
On-Page Optimization | | crisbasma0 -
My home page URL http://seadwellers.com/ redirects to http://www.seadwellers.com/. Is this a problem?
"The URL http://seadwellers.com/ redirects to http://www.seadwellers.com/. Do you want to crawl http://www.seadwellers.com/ instead?" I was given this when I tried to crawl my home page using MOZ software. I was not aware of this, do not know if it could be a problem concerning any aspect of SEO, etc? :
On-Page Optimization | | sdwellers0 -
Short URL's vs Optimised URL's
Howdy Mozzers! What are your thoughts on short URL's vs Optimised URL's. For example if a website currently sells wood furniture and wants to target the keyword "Wood Furniture For Sale", which URL would be preferable: Short URL: www.domain.com/wood-furniture Optimised URL: www.domain.com/wood-furniture-for-sale The website also uses facet navigation and selected attributes are added in a fixed order sequence after the category. For example if Cane is selected as wood type: Short URL: www.domain.com/wood-furniture/Cane Optimised URL: www.domain.com/wood-furniture-for-sale/Cane Which one do you prefer (between the short URL and optimised URL) and why? Cheers! MozAddict
On-Page Optimization | | MozAddict0 -
Changing URL Structure From Flat to Pyramid Theme
Hello Mozzers, I have an on-page SEO question regarding URL structure. A few months back we hired a full-time SEO person who is working on-page right now and she really wants us to completely re-due our URL restructure from a flat to pyramid style (example below). Current URL structure / page title is: Dog training Collars - K9electronics.com
On-Page Optimization | | k9byron
http://www.k9electronics.com/dog-training-collars/ Small Dog training Collars - K9electronics.com
http://www.k9electronics.com/small-dog-training-collars/ Einstein ET-300TS Mini Dog Training Collar - K9electronics.com
http://www.k9electronics.com/einstein-et-300ts-mini-remote-dog-trainer.html Suggested URL structure / page title change: Quality Dog Training Collars - Lowest Price Guarantee - K9electronics
http://www.k9electronics.com/dog-training-collars/ Dog Training Collars - Small Dog - K9electronics.com
http://www.k9electronics.com/dog-training-collars/small-dog/ Einstein ET-300TS Mini - Dog Training Collars - K9electronics.com
http://www.k9electronics.com/dog-training-collars/small-dog/einstein-et-300ts-mini I guess you could say we are the poster-boys for Google penalties and have received just about every penalty in the book. Panda, penguin algo penalties and a partial manual action for unnatural links. Several months ago we removed 1000's of in-bound links and had our manual action lifted a few months back and are now hovering around the top / mid 2nd page for all our big terms ...we used to be top 3 for everything. As we were removing bad links, we also completely redesigned the site and removed lots of categories and products and 95% of all our old, low quality content and replaced it with new, high quality content. The site was really slooooow, so we optimized it and moved it to a big dedicated server and tripled page load time. Added rich snippets, Google authorship, increased our FB and other social presences and much more ... I had also considered this URL structure change during the redesign because I had heard and read that it was good to do, but it required redirecting practically all our URL's which I know can hurt the site even more so then it already has been ... Our SEO says that as it sits now, our pages are competing with each other and really seems to think this is going to improve our rankings a lot ...after several weeks. My question is, at this stage in the game, is it really going to help a lot and give us more benefit compared to the 301 redirect link juice loss? Any comments and/or suggestions are very much appreciated!1 -
URL and SEO
How much weight do search engines give the URL? We're a medical call center provider and medicalcallcenter is part of our URL. Does that help us much? Thanks!!
On-Page Optimization | | THMCC0 -
The URL Inside
Howdy SEO'ers, I have a quick question for the SEO gurus out there. When constructing "better" search friendly URLs would one of these be better than the other? Example 1: http://Domain.com/Category/Sub-Category/Product-name Example 2: http://Category.Domain.com/Sub-Category/Product-name In this example the category could be phones and the sub-category brands of phones. Is either one of these URLs "better" than the other in terms of ranking? Thanks! I'll hang-up and listen to your answer. 🙂 Jonathan
On-Page Optimization | | creativedepartment0 -
Which method should I use for my URL structure?
I have an existing site that is currently utilizing a structure that is like this: http://www.mysite.com/Ohio/City-of-Cleveland-PRODUCT-NAME Should I restructure it like: http://www.mysite.com/Ohio/City-of-Cleveland/Product-Name We are doing very well with very specific searches already but are sometimes coming in 2nd and 3rd place. For example: If I search for CLEVELAND PRODUCT NAME I always come up in the top three and about 60% of the time I am #1. I want to make it better. We have only launched in 4 states but plan on launching an additional 4 states over the next few weeks and I want to make sure we are building things properly. Any feedback would be wonderful. As usual, thanks everyone!! -Alex
On-Page Optimization | | dbuckles0