Blocking Google from Crawling Parameters
-
Hi guys:
What is the best way to keep Google from crawling certain urls with parameters? I used the setting in Webmaster Tools, but that doesn't seem to be helping at all. Can I use robots.txt or some other method? Thanks!
Some examples are:
<colgroup><col width="797"></colgroup>
www.mayer-johnson.com/category/assistive-technology?manufacturer=179 www.mayer-johnson.com/category/assistive-technology?manufacturer=226 www.mayer-johnson.com/category/assistive-technology?manufacturer=227
<colgroup><col width="797"></colgroup>
www.mayer-johnson.com/category/english-language-learners?condition=212 www.mayer-johnson.com/category/english-language-learners?condition=213 www.mayer-johnson.com/category/english-language-learners?condition=214
<colgroup><col width="797"></colgroup>
| www.mayer-johnson.com/category/english-language-learners?roles=164 |
| www.mayer-johnson.com/category/english-language-learners?roles=165 |
| www.mayer-johnson.com/category/english-language-learners?roles=197 ||
|
-
anytime Dana
-
Thanks, Wissam!
-
No,
Disallow ?condition=
Disallow ?cat=
Disallow ?instructional_level=
-
So, for example, it would look like this?:
Disallow:?condition=
Disallow: ?cat=
-
Thanks. I didn't want to use rel="canonical" because there are thousands of variations of these parameters, and it would be time consuming, to say the least.
-
Yes you can block it thew robots.txt and also by Adding a rel="canonical" link in the page code itself will accomplish the task.
Disallow: /category/english-language-learners/*?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawled My Site. Now What?
Hey everyone! So Moz crawled my site and I passed it over to my dev team who's curious about what they should prioritize. Curious what everyone's thoughts are. Here are the issue types: Duplicate Content - Missing Title - Duplicate Title Tag - Redirect Chain - Title too long - Description too short - Missing Description - Missing h1 - Thin Content - URL Too Long - Has meta noindex Would love any assistance! Thank you!
Technical SEO | | inksoft_mm0 -
New website - not showing in Google?
This site was launched 3 days ago, bimcosupply.com and I'm trying to get it to show in Google just for a branded search for the moment (Bimco, Bimco Corporation, etc). The old site is still showing in search, bimcoplumbingsupplies.com instead. This site was taken down a while back. I set up a redirect for the domain in cPanel, and also set individual pages to redirect in WordPress on the bimcosupply.com site. I've verified the site in Google Search Console, submitted a sitemap and did URL inspection on each page. Each page is showing as indexed, though now when I search site:bimcosupply.com not all pages are there, and there are two results for the home page, one "https" and one "http." (Before today, all of the pages were showing so not sure what changed). I know this new domain does not have any (or very little) domain authority yet, but I would have thought that the site should display for branded search by now. So I'm concerned that something is wrong with the site, how the redirects are set up, etc. that is preventing it from displaying. Could anyone take a look and help me figure this out please?
Technical SEO | | browncreative0 -
Wrong page title in Google
Hi there, A while ago we took over the domain www.hoesjes.nl and forwarded it to our website www.telefoonhoesjesxl.nl. If you perform a search for the keyword 'hoesjes' in Google then we (www.telefoonhoesjesxl.nl) show up on an organic number 1 position. The problem is that the page title isn't correct. Google shows the page title of the website hoesjes.nl we took over and (correctly?) redirected to our domain www.telefoonhoesjesxl.nl. Does anybody have any idea how to get rid of this wrong page title in Google?
Technical SEO | | MarcelMoz
Here you can find a screenshot of what I mean. Thanks! Marcel0 -
When will all of Google Maps be the same again?
As many of you are aware that the pigeon update was only applied to the new Google maps resulting in very different search results for Google local business. When you search for a business on old Google maps then you get totally different results vs the new Google maps. Some businesses totally disappeared completely from the search results. I have done my research and found out that it's because the new Algo was only applied to the new maps. Also new algo does not apply to other countries. Well the reason I posted this topic is because I have noticed that all the new Google Business listings I am verifying for my clients are all being put under the old Google maps and not the new ones. They come up fine when searching from old maps but not the new ones. I understand Google has not rolled out the pigeon on all data centers but why? Will Google eventually roll out the update to old maps? Since Google is adding businesses to old google maps then what's the point of even adding new listings?
Technical SEO | | bajaseo0 -
Rel Canonical Crawl Notices
Hello, Within the Moz report from the crawl of my site, it shows that I had 89 Rel Canonical notices. I noticed that all the pages on my site have a rel canonical tag back to the same page the tag is on. Specific example from my site is as follows: http://www.automation-intl.com/resistance-welding-equipment has a Rel Canonical tag <link rel="<a class="attribute-value">canonical</a>" href="http://www.automation-intl.com/resistance-welding-equipment" />. Is this self reference harmless and if so why does it create a notice in the crawl? Thanks in advance.
Technical SEO | | TopFloor0 -
How to know which pages are indexed by Google?
So apparently we have some sites that are just duplicates of our original main site but aiming at different markets/cities. They have completely different urls but are the same content as our main site with different market/city changed. How do I know for sure which ones are indexed. I enter the url into Google and its not there. Even if I put in " around " it. Is there another way to query google for my site? Is there a website that will tell you which ones are indexed? This is probably a dumb question.
Technical SEO | | greenhornet770 -
What else: struggling with google position
Hi. I understand everyone is offering their time for free here so any advice or support is much appreciated. http://www.cytronex.com
Technical SEO | | AdamJamesCytronex
PA 44 || mR 4.6 || mT 5.73 || 986 links from 43 Root Domains
DA 33 || 3,942 links from 71 Domains We've dropped from position 25ish to position 70ish in keyword searches for 'electric bikes'. I've tried everything and I just don't understand! It's genuine content, the actual product is increasingly popular, we have several links from sites which are (well, to my mind) reasonable quality. I've only just been brought in to look at this and my lack of any SEO or web experience is not putting my boss off expecting an instant solution 😞 As I'm only just getting to grips with it, Analytics was only installed about a month ago so I can't pin point a moment when it dropped. We're consistently out-positioned by sites with lower PA/DA scores. Any insight anyone might have would be amazing! Thanks
Adam0 -
RSS Feed Errors in Google
We recently (2 months ago) launched RSS feeds for the category pages on our site. Last week we started seeing error pages in Webmaster Tools' Crawl Errors report pop up for feeds of old pages that have been deleted from the site, deleted from the sitemap, and not in Google's index since long before we launched the RSS feeds. Example: www.mysite.com/super-old-page/feed/ I checked and both the URL for the feed and the URL for the actual page are returning 404 statuses. www.mysite.com/super-old-page/ is also showing up in our Crawl Errors. Its been deleted for months but Webmaster Tools is very slow to remove the page from their Crawl Error report. Where is Google finding these feeds that never existed?
Technical SEO | | Hakkasan0