How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
-
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect...
Obviously have it in robots.txt but is their any other form of action that should be done?
-
Hi Shawn,
It is inadvisable to use more than one method of blocking bots from specific pages on your site.
There is a very good explanation of this in this blog post from Lindsay - Robot Access and Indexation Restriction Techniques: Avoiding Conflicts. Definitely worth reading!
Hope that helps,
Sha
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I optimize my home-page or a sub-page for my most important keyword
Quick question: When choosing the most important keyword set that I would like to rank for, would I be better off optimizing my homepage, or a sub page for this keyword. My thinking goes as follows: The homepage (IE www.mysite.com) naturally has more backlinks and thus a better Google Page Rank. However, there are certain things I could do to a subpage (IE www.mysite.com/green-widgets-los-angeles ) that I wouldn't want to do to the homepage, which might be more "optimal" overall. Option C, I suppose, would be to optimize both the homepage, and a single sub-page, which is seeming like a pretty good solution, but I have been told that having multiple pages optimized for the same keywords might "confuse" search engines. Would love any insight on this!
On-Page Optimization | | Jacob_A2 -
Spammy page titles
Over the last couple of weeks, I have noticed that Google aren't showing the page titles for my online shop anymore. They're set up with a third party plug-in piece of software, and while it's an old version of the software, the developer said it wouldn't be causing issues. They have suggested that I re-write my page titles to be less spammy. The thing is, Google haven't attacked just spammy looking titles, they're just taking a swoop through my whole site and not showing any of my page titles in their search results. I'm getting "Category Name - Shop Name" showing. Here's some of the page titles no longer appearing and I honestly have no idea how to rewrite these to not be spammy. Are there any good articles on what's spammy and what isn't? "Coconut oil - best tasting in Australia. Buy online from <my business="" name="">"</my> "Discount Vitamix Blender. Best deal in Australia. Buy online from <my business="" name="">."</my> "Natural & Organic skin care for the face | buy online in Australia from <my business="" name="">."</my> There are others that are showing the real page titles, but I think it's only a matter of re-indexing before they're all not showing. Any clue?
On-Page Optimization | | sparrowdog0 -
Why is this page not ranking?
Can you please tell me why this page is not ranking. http://goo.gl/BqoRT The page doesn't rank at all for keywords but even if I copy a line or 2 of text it still doesn't rank for that text. Any help will be much appreciated.
On-Page Optimization | | JillB20130 -
Imiges on own page???
Can somebody help me to explain why when I click on an image in a post it goes to its own page?? And why if I click on other image it does not go anywhere?? this is one of the posts: http://villasdiani.com/beach-wedding-at-a-luxury-boutique-hotel/ what is good or bad?? On most of the images SEOmoz is reporting that I am missing meta description!
On-Page Optimization | | VillasDiani0 -
Several Links in Some Pages
Dear all, Our main site is a bussiness directory, and following some SEO advices, we are creating landing pages for each category, in order to optimize them for the keywords. Those landing pages have links to the listings related to them. Using the same idea, we have created pages related to the regions, and those pages include links to the listings located in them. The only problem that I see with that, is the number of links that some categories or regions could have. Is there a limit of recomended number of links per page, from a SEO perspective? We also have a main category page, that includes a list of all categories, and this page could also have a relatively high number of links. The pages have around 300 to 500 words, some include also images, some include videos. Many thanks for your help, Daniel
On-Page Optimization | | te_c0 -
URL for location pages
Hello all We would like to create clean, easy URLs for our large list of Location pages. If there are a few URLs for each of the pages, am I right when I'm saying we would like this to be the canonical? Right now we would like the URL to be: For example
On-Page Optimization | | Ferguson
Domain.com/locations/Columbus I have found some instances where there might be 2,3 or more locations in the same city,zip. My conclusion for these would be: adding their Branch id's on to the URL
Domain.com/locations/Columbus/0304 Is this an okay approach? We are unsure if the URL should have city,State,zip for SEO purposes?
The pages will have all of this info in it's content
BUT what would be best for SEO and ranking for a given location? Thank you for any info!0 -
Duplicate Page Title
Hi Guys, First off, it's an honour to be a part of this awesome community. I'm using WordPress and getting top 3 rankings for great keywords and I'm very excited, however my page titles are in this format "keyword optimised title here - site name here" eg: "This is my keyword - this is the name of my blog", "This is another keyword - this is the name of my blog", "This is a longtail keyword - this is the name of my blog" SEOMoz is reporting errors because of duplicate page title tags due to the "this is the name of my blog" being in every page title. Will this hurt my rankings? Thanks in advance and keep up the great work! Cheers, Troy.
On-Page Optimization | | TroyDean710 -
Can I have a strong brand category page and a strong product page?
It seems Google base and other Comparison Shopping Engines like to see the brand in the product name. But, on my category page for that brand, website optimizer tells me including the brand name with each product is cannabilizes links. For example; I have a page for jewelerABC with 20 pieces of jewelry listed as well as original content about jewelerABC. I do not currently name these products as xyz by jewelerABC. This page comes up nicely in the serps. But in Google base The top listings for jewelry by jewelerABC seem to have every product named xyz by jewelerABC or JewelerABC xyzs. What is the best way to optimize.for both? Stephen
On-Page Optimization | | stephenfishman0