How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
-
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect...
Obviously have it in robots.txt but is their any other form of action that should be done?
-
Hi Shawn,
It is inadvisable to use more than one method of blocking bots from specific pages on your site.
There is a very good explanation of this in this blog post from Lindsay - Robot Access and Indexation Restriction Techniques: Avoiding Conflicts. Definitely worth reading!
Hope that helps,
Sha
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I optimize my home-page or a sub-page for my most important keyword
Quick question: When choosing the most important keyword set that I would like to rank for, would I be better off optimizing my homepage, or a sub page for this keyword. My thinking goes as follows: The homepage (IE www.mysite.com) naturally has more backlinks and thus a better Google Page Rank. However, there are certain things I could do to a subpage (IE www.mysite.com/green-widgets-los-angeles ) that I wouldn't want to do to the homepage, which might be more "optimal" overall. Option C, I suppose, would be to optimize both the homepage, and a single sub-page, which is seeming like a pretty good solution, but I have been told that having multiple pages optimized for the same keywords might "confuse" search engines. Would love any insight on this!
On-Page Optimization | | Jacob_A2 -
Critical Page Problems.
As I design my new pages I place them into the page optimizer. When I place the keyword in that I am trying to rank for my page is coming up as an A. With that said, I am still receiving error messages with critical fix results. Here is a screenshot of this: http://screencast.com/t/DuUrP8xM Curious on how I should fix this? The pages are directly on my blog which makes it accessible to search engines ( So I think). Any input would be greatly appreciated.
On-Page Optimization | | Jasonalanmagic0 -
Should I consolidate pages to prevent "thin content"
I have a site based on downloadable images which tend to slow a site. In order to increase speed I divided certain pages up so that there will be less images on each page such as here: http://www.kindergartenteacherresources.com/2011/09/23/spongebob-alphabet-worksheets-uppercase-letters-a-h/ http://www.kindergartenteacherresources.com/2011/09/23/spongebob-alphabet-worksheets-uppercase-letters-i-q/ http://www.kindergartenteacherresources.com/2011/09/23/spongebob-alphabet-worksheets-uppercase-letters-r-z/ The problem is that I now have potential duplicate content and thin content. Should I consolidate them and put all of the content from the 3 pages on one page? or maybe keep them as they are but add a rel previous / next tag? or any other suggestion to prevent a duplicate/thin content penalty while not slowing down the site too much?
On-Page Optimization | | JillB20130 -
In-page Optimization check list....
Hey, Mozzers I have a question about optimizing a web-page. I was just having a conversation with our web developer about optimizing our website and which changes would make the most difference in affecting the SERPs as far as on-page optimization goes. I was explaining to her we wanted to optimize our pages in the following levels (ranked most important to least important): <title>: key word laden </p> <p>2) <H1> : key word in <strong>bold</strong></p> <p>3) page content: keyword laden as well</p> <p>The idea was to have three or four layers of keywords on the page. I work for a real estate brokerage and the context was the actual page of the property listing. So in this case, the keywords would be the address of the property. She explained to me that the the H1 tag is more important than the title tag.</p> <p>Does this ring true with you guys?</p></title>
On-Page Optimization | | AubbiefromAubenRealty0 -
Why don't all my pages have On Page Optimization Reports
Apologies if this question has been asked a million times, but I can't find it. I have 35 pages, yet only 5 of them have generated On Page Optimization Reports. I know I can create them manually, but wondered if I've done something incorrectly? Iain.
On-Page Optimization | | iainmoran0 -
Home Page
We are re-design our home page, one are of the current home page has a drop down window called "popular products" . We wrote short articles for our keywords and have them linked to product page. In the past, it has helped us rank. However, with new Google rules, our feeling is that such practice is no good. So, we lean towards to remove it. Still, we'd like to hear some opinions and ask some questions too: www.butterflycraze.com is it clear to you that this is not good in Google's eyes? how do I determine if these links serving any SEO purpose now after Panda? depend on the answer to 2), what should we do about these pages? shall be re-direct or shall we remove them from Google index?
On-Page Optimization | | ypl0 -
Faq page
We are redoing our faq page and we were trying to decide on the best format. 1. Create each question on a separate page 2. Create one page with all the question and have the questions expand 3. Create different faq category pages (like 4) and divide the questions between them From my perspective #1 seems the best ---. you can create hyper relevant content for the user and optimize each question really well Any experience with this?
On-Page Optimization | | Morris770 -
100 links on one page
we're recommended 100 links or less on one page. is the 100 links including header and footer links?
On-Page Optimization | | jallenyang0
