How to prevent Google from crawling our product filter?
-
Hi All,
We have a crawler problem on one of our sites www.sneakerskoopjeonline.nl.
On this site, visitors can specify criteria to filter available products. These filters are passed as http/get arguments. The number of possible filter urls is virtually limitless.
In order to prevent duplicate content, or an insane amount of pages in the search indices, our software automatically adds noindex, nofollow and noarchive directives to these filter result pages. However, we’re unable to explain to crawlers (Google in particular) to ignore these urls.
We’ve already changed the on page filter html to javascript, hoping this would cause the crawler to ignore it. However, it seems that Googlebot executes the javascript and crawls the generated urls anyway.
What can we do to prevent Google from crawling all the filter options?
Thanks in advance for the help.
Kind regards,
Gerwin
-
The following is added to our robots.txt .. now lets wait and see the results
User-agent: * Disallow: /admin/
Disallow: /?
Allow /?product_date=&product_date2=*
Disallow /?product_date=&product_date2=&To check the working of the robots.txt i found a handy website;
-
The url looks like this;
http://www.sneakerskoopjeonline.nl/herensneakers?product_brand=
So just adding;
User-agent: *
Disallow: /*?product_brandShould do the trick?
Most important is that herensneakers itself should be indexed, followed and crawled -
I would use your robots.txt file to prevent them from crawling the specific strings / pages. Go into your Google Webmaster Tools and you can see all the information Google has on your site and any issues, you can also specify robots.txt information in there. That would be the best route as Google is obedient with what is on the robots.txt file. If you want more information about robots.txt, go here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Homepage is deindexed in Google
Please help for some reason my website home page has disappeared, we have been working on the site but nothing that I can think of which would block it. There are no warnings in google console? Can anyone lend a hand in understanding what has gone wrong, I would really appreciate it. The site is: http://www.discountstickerprinting.co.uk/ Seems to be working again but I had to fetch the home page in google console, any idea why this has happened cannot afford a heat op at this age lol?
Intermediate & Advanced SEO | | BobAnderson0 -
I have never seen this in Google SERPs before?
What is this that I am seeing in the Google SERPs? It's a bank of 3 separate articles from one site, but they are grouped together like an organic listing. Checkout the link to see what I'm talking about http://imgur.com/xbUk1NG
Intermediate & Advanced SEO | | MJTrevens0 -
Google is not indexing an updated website
We just relaunched a website that has 5 years old, we maintain all the old URLs and articles but for some reason google is not picking up the new website https://www.navisyachts.com. In Google Webmaster Tools we can see the sitemap with over 1000 pages submitted but shows nothing as indexed. The site is loosing traffic rapidly and positions, from the SEO side all looks fine for me. What can be wrong? I’ll appreciate any help. The new website is built over Joomla 3.4, we have it here at MOZ and other than some minor details it doesn't show that something can be wrong with the website. Thank you.
Intermediate & Advanced SEO | | FWC_SEO0 -
Google News sitemap keywords
My company is a Theater news and reviews site. We're building a google news sitemap and Google suggests some recommended keywords we can use with their <keywords>tag: https://support.google.com/news/publisher/answer/116037</keywords> Our writers also tag their stories with relevant keywords. What should we populate the <keywords>tag with?</keywords> We were thinking we'd automatically populate it with author-added tags, in addition to one or more of the recommended ones suggested by Google, such as Theater, Arts, and Culture (all of our articles are related to these topics). Finally, many of our articles are about say, celebrities. An author may tag an article with 'Bryan Cranston,' and when this is the case we're considering also tagging it with the 'Celebrities' tag. Are all or any of these worthwhile?
Intermediate & Advanced SEO | | TheaterMania0 -
Why are some pages indexed but not cached by Google?
The question is simple but I don't understand the answer. I found a webpage that was linking to my personal site. The page was indexed in Google. However, there was no cache option and I received a 404 from Google when I tried using cache:www.thewebpage.com/link/. What exactly does this mean? Also, does it have any negative implication on the SEO value of the link that points to my personal website?
Intermediate & Advanced SEO | | mRELEVANCE0 -
Google Hummingbird Update - Any Changes ?
Google has update with the new alogrithm and did you see any effects and as they are not revelaing the techinicaly how they work ? What's your opinion ?
Intermediate & Advanced SEO | | Esaky0 -
Keywords under product listing pages
Hi guys, One of my main concerns when we start redesigning the site Trespass.co.uk, is the current pages like this one http://www.trespass.co.uk/snow-sports/clothing/ski-jackets/womens-ski-jackets are bordering over optimisation. Is this the case as each product listed in the url above has "womens ski jacket" under each product. If we have 50 products on each product listing page with the product name + type of product, ie. flora womens ski jacket, xyz mens waterproof jacket. Are we over optimising the page for the main keywords by having them under each product? Would that page be over optimised for womens ski jackets? Thanks guys
Intermediate & Advanced SEO | | Trespass0 -
Why am I not ranking in Google, but I am in Yahoo and Bing?
The website in question is: www.stbarthexclusives.com Our keywords are currently ranking for both Bing and Yahoo, but we're not appearing anywhere on Google. The website is being crawled successfully, but we still don't have any results. I hoping somebody can point me in the general right direction to fix/correct this problem. Additionally, there's a decent amount of "rel=canonical tags" on the website. If that helps your evaluation. Any advice would be greatly appreciated
Intermediate & Advanced SEO | | Endora0