How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
-
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect...
Obviously have it in robots.txt but is their any other form of action that should be done?
-
Hi Shawn,
It is inadvisable to use more than one method of blocking bots from specific pages on your site.
There is a very good explanation of this in this blog post from Lindsay - Robot Access and Indexation Restriction Techniques: Avoiding Conflicts. Definitely worth reading!
Hope that helps,
Sha
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Listing all services on one page vs separate pages per service
My company offers several generalized categories with more specific services underneath each category. Currently the way it's structured is if you click "Voice" you get a full description of each voice service we offer. I have a feeling this is shooting us in the foot. Would it be better to have a general overview of the services we offer on the "Voice" page that then links to the specified service? The blurb about the service on the overview page would be unique, not taken from the actual specific service's page.
On-Page Optimization | | AMATechTel0 -
On-page report tool
This is a question regarding the advise this tool offer to increase the ranking of a webpage with focusing with a particularly keyword we choose. I give an example: On-page Report Card am checking my keywords and I use. "cleanse london" my surprise is Report card give an "F" for my target landing page http://www.purifyne.com , but the issue is I am already in first place first position. I know SEOmoz know have the algorithm from Google to know how to rank better but my issue is should be a little more accurate! at least. I don't want to be misunderstood here, I just want more guidance, to rank much better using this tool that I am paying for. Any thoughts?
On-Page Optimization | | teksyte0 -
Pages vs Posts
What are your thoughts on pages vs posts? I am setting up a new blog for a client but not sure how to structure the content. I may just do posts or a whole bunch of page listed down the sidebar. It seems like my pages always rank better than my posts. Has anyone else noticed this? Could it be because of the dates tied posts?
On-Page Optimization | | SixTwoInteractive0 -
Faq page
We are redoing our faq page and we were trying to decide on the best format. 1. Create each question on a separate page 2. Create one page with all the question and have the questions expand 3. Create different faq category pages (like 4) and divide the questions between them From my perspective #1 seems the best ---. you can create hyper relevant content for the user and optimize each question really well Any experience with this?
On-Page Optimization | | Morris770 -
Ecommerce: content on category pages
I have to optimize some online Shops and after Panda I really don't know what to think about thin content on product overview pages anymore... used to be that we could improve our rankings easily just by adding 1-2 sentences on such a page. This always worked for non-overly competitive terms. Now It feels like it doesn't work any longer, but I couldn't put my finger on it and I don't have the resources to test. Here's an example of what I mean: http://www.geschenkidee.ch/wandtattoos/aus_aller_welt.html
On-Page Optimization | | zeepartner
I would add max. 3 lines of text directly over the product thumbnails. What do you think? Is it worth adding some text on a product overview page or do I not even have to bother post-Panda?0 -
Too many on-page links
Hi, My website - www.thepartyhouse.com.au is showing as having too many on-page links for over 4,000 pages. Take for example the homepage which is showing as 188 links, but I don't understand this because I've used SEO tools to display the links and I am showing around 90 links on this page. How can I see what all the links are? Thanks
On-Page Optimization | | Spyre0 -
If a site has https versions of every page, will the search engines view them as duplicate pages?
A client's site has HTTPS versions of every page for their site and it is possible to view both http and https versions of the page. Do the search engines view this as duplicate content?
On-Page Optimization | | harryholmes0070 -
Canonical home page
I have a site that shows duplicate page content for: www.autoserviceexpertsonline and www.autoserviceexpertsonline/index.html When looking at the files using the cms (intuit) file manager, I only see the /index.html version. I added the Caononical tag referencing/pointing to both the domain name only and then changed to .../index.html No matter how I code this, the seomoz On-Site SEO Grader still has a problem with it. Is this a bug with the Grading program or am I doing something wrong? Please help as I think this is causing me problems with Google and I'd like to get this right for future sites I will be working on. Thanks, Bill
On-Page Optimization | | Marvo0