How to stop crawls for product review pages? Volusion site
-
Hi guys, I have a new Volusion website. the template we are using has its own product review page for EVERY product i sell (1500+) When a customer purchases a product a week later they receive a link back to review the product. This link sends them to my site, but its own individual page strictly for reviewing the product. (As oppose to a page like amazon, where you review the product on the same page as the actual listing.)
**This is creating countless "duplicate content" and missing "title" errors. What is the most effective way to block a bot from crawling all these pages? Via robots txt.? a meta tag? **
Here's the catch, i do not have access to every individual review page, so i think it will need to be blocked by a robot txt file? What code will i need to implement? i need to do this on my admin side for the site? Do i also have to do something on the Google analytics side to tell google about the crawl block?
Note: the individual URLs for these pages end with: *****.com/ReviewNew.asp?ProductCode=458VB
Can i create a block for all url's that end with /ReviewNew.asp etc. etc.?
Thanks! Pardon my ignorance. Learning slowly, loving MOZ community
-
No you should be fine
-
thanks. you say "update on 4/21" you're talking about googles update requiring more mobile friendly sites? My volusion template has its own mobile version. It is not a responsive template. So i should not be affected correct?
-
Parameters are good for pages that are a result of a search or sort. I guess it isn't necessary really, I am just a little ocd about that kind of stuff. The parameters in WMT basically tell google that these things might appear in the URL, and then you can the bot to ignore it or let GoogleBot decide how to read the URL.
The mobile site is not the same as a responsive design and on of the main reasons I left Volusion. The mobile site will get you through the update on 4/21, but if possible you should ask them for a responsive site. Just call the support number, or your account manager and ask.
-
ive had the following in my robots.txt file. Do i need to add the astrisk like you have posted above?
Currently in my robot.txt:
User-agent:*
Disallow: /reviews.asp/User-agent:*
Disallow: /reviewnew.asp/ -
Thanks monica, can you elaborate a bit more on the webmaster tools parameter? what specifically does adding a parameter like that do? You did that as a backup in case the robot txt file was not working? we do have a mobile version enabled which came with our template. Ill keep an eye out for the 404's. where do i check for a responsive template? Ours is one of their premium templates so its possible we are already on a responsive one? Can you clarify what responsive template means?
thanks.
-
I did that in my Volusion store. I also added ReviewNew.asp?ProductCode= as a parameter in Google Webmaster Tools. Do you have an enabled mobile site as well? If you do there are several 404 errors that you will start to see from there. Make sure you are adding parameters accordingly. I am not sure if Volusion has started offering their responsive templates yet, but if they have I would see if you can implement that over the mobile site.
-
Hi ,
Yes you can block such URL by using below code in robots.txt file.
User-agent: *
Disallow: /*ReviewNew.aspThanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site redesign makes Moz Site Crawl go haywire
I work for an agency. Recently, one of our clients decided to do a complete site redesign without giving us notice. Shortly after this happened, Moz Site Crawl reported a massive spike of issues, including but not limited to 4xx errors. However, in the weeks that followed, it seemed these 4xx errors would disappear and then a large number of new ones would appear afterward, which makes me think they're phantom errors (and looking at the referring URLs, I suspect as much because I can't find the offending URLs). Is there any reason why this would happen? Like, something wrong with the sitemap or robots.txt?
Technical SEO | | YYSeanBrady1 -
Pages with Duplicate Page Content Crawl Diagnostics
I have Pages with Duplicate Page Content in my Crawl Diagnostics Tell Me How Can I solve it Or Suggest Me Some Helpful Tools. Thanks
Technical SEO | | nomyhot0 -
Rel canonical for partner sites - product pages only or also homepage and other key pages?
Hello there Our main site is www.arenaflowers.com. We also run a number of partner sites (eg: http://flowershop.cancerresearchuk.org/). We've relcanonical'd the products on the partner site back to the main (arenaflowers.com) site. eg: http://flowershop.cancerresearchuk.org/flowers/tutti_frutti_es_2013 rel canonicals back to: http://www.arenaflowers.com/flowers/tutti_frutti_es_2013). My question: Should we also relcanonical the homepage and other key pages on partner sites back to the main arenaflowers website too? The content is similar but not identical. We don't want our partner sites to be outranking the original (as is the case on kw flower delivery for example). (NB this situation may be complicated by the fact we appear to have an unnatural link penalty on af.com (and when we did an upgrade a while back, the af.com site fell out of the index altogether due to some issues with our move to AWS.) We're getting professional SEO advice on this but wondered what the Moz community's thoughts were.. Cheers, Will
Technical SEO | | ArenaFlowers.com0 -
Are aggregate sites penalised for duplicate page content?
Hi all,We're running a used car search engine (http://autouncle.dk/en/) in Denmark, Sweden and soon Germany. The site works in a conventional search engine way with a search form and pages of search results (car adverts).The nature of car searching entails that the same advert exists on a large number of different urls (because of the many different search criteria and pagination). From my understanding this is problematic because Google will penalize the site for having duplicated content. Since the order of search results is mixed, I assume SEOmoz cannot always identify almost identical pages so the problem is perhaps bigger than what SEOmoz can tell us. In your opinion, what is the best strategy to solve this? We currently use a very simple canonical solution.For the record, besides collecting car adverts AutoUncle provide a lot of value to our large user base (including valuations on all cars) . We're not just another leech adword site. In fact, we don't have a single banner.Thanks in advance!
Technical SEO | | JonasNielsen0 -
How is this site doing this?
http://www.meccabingo.com It shows a splash / promotion page yet you check the cache and it's the real homepage, they are doing this so they don't lose rankings but how are they redirecting users to that but Google is caching the real homepage? is it friendly? thanks!!
Technical SEO | | AdiRste0 -
Page crawling is only seeing a portion of the pages. Any Advice?
last couple of page crawls have returned 14 out of 35 pages. Is there any suggestions I can take.
Technical SEO | | cubetech0 -
Will frequently adding and frequently removing pages from my site hinder any SEO?
Hi Guys, Just looking through our crawl diagnositcs and we have a ton errors, well over 5000 actually, on 404 pages that cannot be accessed. Our website runs a lot of "Hot Offers" that are time bound, so they expire at the end of each month and we remove the page via our CMS. It's making the crawl diagnositcs loook bad, but will this be hindering our seo and Google 'stuff' because they are finding thousands of 404 errors? Any advice would be greatly appreciated! Website: www.vospers.com Lee Greenhill
Technical SEO | | lee_greenhill0 -
Non-www home page indexed, but www for rest of site
Hi there, grateful for any ideas on why this is happening: http://www.google.co.uk/search?q=site:www.vitispr.com vs http://www.google.co.uk/search?q=site:vitispr.com Google seems to be indexing and caching vitispr.com for our home page but the www. versions for everything else. As you can see the second query finds the home page. Any ideas why that might be? Other info that might be relevant: non-www etc. are all 301'd to www versions. moved domains/urls etc. around in March of this year and for a week or we were redirecting to the non-www version webmaster tools says 'www' preferred Thanks!
Technical SEO | | JaspalX0