How to stop crawls for product review pages? Volusion site
-
Hi guys, I have a new Volusion website. the template we are using has its own product review page for EVERY product i sell (1500+) When a customer purchases a product a week later they receive a link back to review the product. This link sends them to my site, but its own individual page strictly for reviewing the product. (As oppose to a page like amazon, where you review the product on the same page as the actual listing.)
**This is creating countless "duplicate content" and missing "title" errors. What is the most effective way to block a bot from crawling all these pages? Via robots txt.? a meta tag? **
Here's the catch, i do not have access to every individual review page, so i think it will need to be blocked by a robot txt file? What code will i need to implement? i need to do this on my admin side for the site? Do i also have to do something on the Google analytics side to tell google about the crawl block?
Note: the individual URLs for these pages end with: *****.com/ReviewNew.asp?ProductCode=458VB
Can i create a block for all url's that end with /ReviewNew.asp etc. etc.?
Thanks! Pardon my ignorance. Learning slowly, loving MOZ community
-
No you should be fine
-
thanks. you say "update on 4/21" you're talking about googles update requiring more mobile friendly sites? My volusion template has its own mobile version. It is not a responsive template. So i should not be affected correct?
-
Parameters are good for pages that are a result of a search or sort. I guess it isn't necessary really, I am just a little ocd about that kind of stuff. The parameters in WMT basically tell google that these things might appear in the URL, and then you can the bot to ignore it or let GoogleBot decide how to read the URL.
The mobile site is not the same as a responsive design and on of the main reasons I left Volusion. The mobile site will get you through the update on 4/21, but if possible you should ask them for a responsive site. Just call the support number, or your account manager and ask.
-
ive had the following in my robots.txt file. Do i need to add the astrisk like you have posted above?
Currently in my robot.txt:
User-agent:*
Disallow: /reviews.asp/User-agent:*
Disallow: /reviewnew.asp/ -
Thanks monica, can you elaborate a bit more on the webmaster tools parameter? what specifically does adding a parameter like that do? You did that as a backup in case the robot txt file was not working? we do have a mobile version enabled which came with our template. Ill keep an eye out for the 404's. where do i check for a responsive template? Ours is one of their premium templates so its possible we are already on a responsive one? Can you clarify what responsive template means?
thanks.
-
I did that in my Volusion store. I also added ReviewNew.asp?ProductCode= as a parameter in Google Webmaster Tools. Do you have an enabled mobile site as well? If you do there are several 404 errors that you will start to see from there. Make sure you are adding parameters accordingly. I am not sure if Volusion has started offering their responsive templates yet, but if they have I would see if you can implement that over the mobile site.
-
Hi ,
Yes you can block such URL by using below code in robots.txt file.
User-agent: *
Disallow: /*ReviewNew.aspThanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On our site by mistake some wrong links were entered and google crawled them. We have fixed those links. But they still show up in Not Found Errors. Should we just mark them as fixed? Or what is the best way to deal with them?
Some parameter was not sent. So the link was read as : null/city, null/country instead cityname/city
Technical SEO | | Lybrate06060 -
Why is my site crawled so much more in Bing then in Google?
I recently setup Cloudflare so I can see how much my site is being crawled. It looks like Bing is crawling me about 3 times as much as Google. Any ideas on why that would be or what I should check?
Technical SEO | | EcommerceSite0 -
Can Googlebot crawl the content on this page?
Hi all, I've read the posts in Google about Ajax and javascript (https://support.google.com/webmasters/answer/174992?hl=en) and also this post: http://moz.com/ugc/can-google-really-access-content-in-javascript-really. I am trying to evaluate if the content on this page, http://www.vwarcher.com/CustomerReviews, is crawlable by Googlebot? It appears not to be. I perused the sitemap and don't see any ugly Ajax URLs included as Google suggests doing. Also, the page is definitely indexed, but appears the content is only indexed via its original source (Yahoo!, Citysearch, Google+, etc.). I understand why they are using this dynamic content, because it looks nice to an end-user and requires little to no maintenance. But, is it providing them any SEO benefit? It appears to me that it would be far better to take these reviews and simply build them into HTML. Thoughts?
Technical SEO | | danatanseo0 -
Micro-sites for Landing Pages?
We are working with a site that is difficult at best to update. The client intends to re-do the site in 18 months or so but needs to start generating more traffic (and sales) now. What are thoughts on creating landing pages as micro-sites that point to the current site conversion page as a stop gap? Beyond not sharing authority is there any known penalty? By the way they don't have tremendous ranking right now - often bottom of page two - and the micro-site won't duplicate any content.
Technical SEO | | InformaticsInc0 -
What to do when you want the category page and landing page to be the same thing?
I'm working on structuring some of my content better and I have a dilemma. I'm using wordpress and I have a main category called "Therapy." Under therapy I want to have a few sub categories such as "physical therapy" "speech therapy" "occupational therapy" to separate the content. The url would end up being mysite/speech-therapy. However, those are also phrases I want to create a landing page for. So I'd like to have a page like mysite.com/speech-therapy that I could optimize and help people looking for those terms find some of the most helpful content on our site for those certain words. I know I can't have 2 urls that are the same, but I'm hoping someone can give me some feedback on the best way to about this. Thanks.
Technical SEO | | NoahsDad0 -
Same Video on Multiple Pages and Sites... Duplicate Issues?
We're rolling out quite a bit of pro video and hosting on a 3-party platform/player (likely BrightCove) that also allows us to have the URL reside on our domain. Here is a scenario for a particular video asset: A. It's on a product page that the video is relevant for. B. We have an entry on our blog with the video C. We have a separate section of our site "Video Library" that provides a centralized view of all videos. It's there too. D. We eventually give the video to other sites (bloggers, industry educational sites etc) for outreach and link-building. A through C on our domain are all for user experience as every page is very relevant, but are there any duplicate video issues here? We would likely only have the transcript on the product page (though we're open to suggestions). Any related feedback would be appreciated. We want to make this scalable and done properly from the beginning (will be rolling out 1000+ videos in 2010)
Technical SEO | | SEOPA0 -
Decreasing the size of a site to increase SEO value of remaining pages?
My website has thousands of pages and I have so many keywords on the bottom of page 1 and on page 2 of SERPs. I am considering making the site smaller to lessen the dilution of the overall domain authority and in theory the remainder pages should get pushed up in rank. Do you feel this theory is flawed? Is it better to 301 or remove the pages if they don't have backlinks directly to the internal page? These are pages I would re-enable down the road once overall domain authority is increased. thanks, David couponcactus.com
Technical SEO | | CouponCactus0 -
How do I set up a site review for a password protected site?
We need to conduct a SEO analysis for a website that is on a private, password protected development site -- is there anyway for SEOMoz tools to access and analyze a PW protected site? Thank you, Sara Merten
Technical SEO | | kev110