How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
-
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect...
Obviously have it in robots.txt but is their any other form of action that should be done?
-
Hi Shawn,
It is inadvisable to use more than one method of blocking bots from specific pages on your site.
There is a very good explanation of this in this blog post from Lindsay - Robot Access and Indexation Restriction Techniques: Avoiding Conflicts. Definitely worth reading!
Hope that helps,
Sha
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
On-Page Optimization | | absoauto0 -
Pages or Blog posts?
Hi, I am currently building content for a customer's website. There are approximately 50 new content pages I am building about the business, products they serve, how-tos and tips and advice. The website is built on Wordpress so my question is would it be best to post this content as a different blog posts or as separate pages in Wordpress and link them up to a 'hub page' as mentioned on this post about How to rank (point 16) Thanks for any advice.
On-Page Optimization | | btiffin0 -
Home Page Content
Hello. i'm optimizing this website, > home page for one keyword phrase and i was wondering how many words article do i need with that keyword?and if i need it at all? as you can see if i add some content on my home page before the slider, it will ruin the look of the website, What is the right way to do it? Thank you!
On-Page Optimization | | KentR0 -
Main page link reduction
I am in the process of reducing the number of internal links from the homepage with the assumption that the more links I have the more "juice" goes to internal pages I have two options since many of the links are costumer service related 1. create one link that leads to a costumer service page and place all the links their. 2. get all those links on the main page as nofollow links. what are your opinions
On-Page Optimization | | ciznerguy0 -
Autogenerated pages
My main product is database conversion software. As it supports tons of databases, it's fairly easy to generate thousands of landing pages simply by variating source/target database names, connection information etc. In fact, I autogenerated almost 25k pages that way. As I didn't want to jeopardize my main site, I placed all that content to a new microsite (www.fullconvert.com) which had no history and no inbound links. Results were nice - site is live two months and in second month already had 1300 visitors. Now, my question is - should I create the same thing on my (old and rather authoritative) main site www.spectralcore.com? I could use a different template to avoid duplicate content. Of course, my main concern is being penalized by Google. In my opinion, this autogenerated content is fine because it provides (tons of) laser-focused landing pages, so visitors will instantly recognize they found what they're looking for. But Google might disagree! What do you think? Is there a danger in trying to leverage authority of my main site in adding 20k+ autogenerated pages with inbound no links to them?
On-Page Optimization | | metadata0 -
Page without content
Hey Everyone, I've started an SEO On Page analysis for a web site and I've found a lot of duplicate content and useless pages. What do I have to do? Delete this useless page, redirect or do canonical tag? If I have to delete what is the best way to do? Should I use GWT to delete? or just delete from the server? This URL for example: http://www.sexshopone.com.br/?1.2.44.0,0,1,13,0,0,aneis-evolved-boss-cock's.html [admin note: NSFW page} There is no content and it is duplicate in reference of this: http://www.sexshopone.com.br/?1.2.44.0,0,1,12,0,0,aneis-evolved-boss-cock's.html [admin note: NSFW page} and the correct page of the product is: http://www.sexshopone.com.br/?1.2.44.0,423,anel-peniano-evolved-boss-cock's-pleasure-rings-collar-white-reutilizavel-e-a-prova-d'agua-colecao-evolved.html [admin note: NSFW page} What is happening is that we have 8.000 pages like this. Useless and without any content. How do I proceed? Thanks!
On-Page Optimization | | luf07090 -
Duplicate Page Title Elements
Hello Mozzers. My questions is below and I would like to thank everyone in advance for any feedback 😉 I own a dog supplies site (www.k9electronics.com). When I launched the site several years back I hired a guy for SEO and he optimized my home page for specific categories search terms such as "dog training collars", "dog shock collars:, ect instead of general search terms such as "dog supplies", "dog accessories", ect. I would like to start moving these home page title element terms (starting with "dog shock collars") over to the dog training collars category but have high rankings for this term on the home page. Current Home Page Title Element:
On-Page Optimization | | k9byron
dog training collars, dog shock collars, electric dog collar, dog supplies (recently added) Current Dog Training Collars Category Title Element:
dog training collars I was hoping to add "dog shock collars" to the dog training collars category page until I achieved higher ranking then delete if from the home page. ..or swap it out with "dog accessories". I am currently ranked #5 in Google for "dog shock collars" on the home page & dog training collars category page ...and I am a little concerned about changing these title elements. My question is; If I add 'dog shock collars" to the dog training collars category page title as well, how will it effect my ranking on both pages having this duplicate term in both page titles? Thank You,
Byron-0 -
Page title structure?
From an SEO and user perspective what structure do you recommend for page titles. For example (given that they shouldn't ideally be more than 70 characters) :- Keywords (maybe two or three) | Company Name | more keywords I understood the best place for the company name was about second place. Is this now the considered view taking into consideration 'branding' which has been flagged up as the way forward. Keywords are separated by the vertical bar | - any thoughts? For 'house keeping' pages such as Privacy Policy - should this be optmised or simply stated as 'privacy policy' Thanks
On-Page Optimization | | PH2920