How can I prevent Google and other search engines to crawl my secured pages (https:)?
-
Let me know your thoughts guys. Thanks in advance!
-
Your best bet is to place a meta noindex tag on each secure page. If it's only a few pages, you could just add it by hand. If it's many, you should be able to access each pages protocol with whatever server-side language you're using, and dynamically add it on all secure pages.
If you use robots.txt to exclude the pages, Google can still show them in search results, with the description below them that reads, "A description for this result is not available because of this site's robots.txt – learn more." Personally, I don't care for that.
-
Hi there, blocking the HTTPS version of your pages from being crawled by the search engines is a bit tricky. You might need to come up with a separate robots.txt file to handle the HTTPS requests.
Here you go to know more about the process:
http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html
Hope it helps.
Best,
Devanur Rafi
-
Hi esiow
You have a choice of placing a robots.txt file in the root folder of your website or if blocking individual pages you could use the meta robots tag. See these page for more information: http://moz.com/learn/seo/robotstxt and https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1
I hope that helps,
Peter
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ecommerce Category Pages
First, let's define the terminology for the various types of ecommerce pages. The terminology differs from organization to organization: Product Description Pages (PDPs): These pages have a single product, pricing, an "add to cart" button, reviews, and a product description. Product Listing Pages (PLPs): These are product category/subcategory pages that have product image links and text links to Product Description Pages (PDPs). Category Pages: These pages have subcategory image and text links to subcategory pages. No product images are displayed Hybrid Category Pages: these pages combine sub-Category Images and text at the top of the page and product listings below. Our CMS currently does not allow us to create hybrids. This conversation revolves primarily around mobile. Our ecommerce team is having discussions around the appropriate use of PLPs vs Category pages. After doing a quick audit of the mobile sites of some top ecommerce players, there is definitely a trend to use Category Pages at the top of the category and sub-category hierarchy and use PLPs at the very bottom. The logic from a usability perspective is to allow visitors to navigate a site without ever using the hamburger navigation. ex: Baby (Category Page) => Car Seats (Category Page) => Convertible Car Seats (PLP) The sites I audited all had hamburger menus. A visitor would navigate from a home page image for "Baby," an image on the "Baby" page to "Car Seats", and an image on the "Car Seats" page to the Convertible Car Seats page. At that point, they would be able to shop for "Convertible Car Seats" on a PLP. This appears to be excellent UX and easy to use navigation. Theoretically, good for SEO as well. In short, category and subcategory pages are being used as navigation to allow visitors to easily navigate to the bottom of the hierarchy and shop on the most narrow page in the hierarchy. Much easier to use than a hamburger menu, but it does entail more clicks. The discussion revolves around allowing users to shop for product at a higher level in the taxonomy. For example, what if a visitor wants to shop all Car Seats? In the above taxonomy, we are precluding users from shopping in this manner. There is no "Car Seats" PLP. Our CMS has the ability to create both a Category Page and a PLP for "Car Seats". We could theoretically place an image on the "Car Seats" category page for "View All Car Seats", and allow users to click to a "Car Seats" PLP. None of the major ecommerce players I've audited are adding a PLP option higher up in the hierarchy. That doesn't mean that it's not good UX. Problems: From an SEO perspective, having a Category Page and a PLP for "Car Seats" would cause cannibalization - they would be competing for the same keywords. I am skeptical that canonicals would work. The pages are not near duplicate content. One page has category images, the other has product images. We could place content blocks on the page to make them more similar. We could noindex the PLP, but that's a waste of internal link juice. Need advice: Will canonicals work in this situation? Should we trash this idea entirely? Does adding a PLP add value or confusion? Is noindex a good idea? Is there an option to target keyword variations with the PLP? Is there another solution?
Web Design | | Satans_Apprentice0 -
Multiple sites using same text - how to avoid Google duplicate content penalty?
Hi Mozers, my client located in Colorado is opening a similar (but not identical) clinic in California. Will Google penalize the new California site if we use text from our website that features his Colorado office? He runs the clinic in CO and will be a partner of the clinic in CA, so the CA clinic has his "permission" to use his original text. Eventually he hopes to go national, with multiple sites utilizing essentially the same text. Will Google penalize the new CA site for plagiarism and/or duplicate content? Or is there a way to tell Google, "hey Google, this new clinic is not ripping off my text"?
Web Design | | CalamityJane770 -
Hiding content until user scrolls - Will Google penalize me?
I've used: "opacity:0;" to hide sections of my content, which are triggered to show (using Javascript) once the user scrolls over these sections. I remember reading a while back that Google essentially ignores content which is hidden from your page (it mentioned they don't index it, so it's close to impossible to rank for it). Is this still the case? Thanks, Sam
Web Design | | Sam.at.Moz0 -
Heres a puzzle for you... Htags on left hand nav for ecomm category pages
Hi Guys, So I am back asking more questions, but I am slowly learning 🙂 This next one, I have looked everywhere and I can't get my head around h tags on ecomm sites. I have looked at competitors and non competitors and still am not sure which is the right or wrong thing to do, specifically in this instance, category pages. Our I.T. dept is somewhat under resourced and I don't want to waste their time with test and trial on this particular issue. Category landing pages... There are shed loads of category listing pages on our site, at the moment the h1 tag for each sub-category is listed in the end path of the breadcrumb, there is no other spaces on the page accept the left hand navigation. Which is the better to use, breadcrumb or nav? We would have to totally recode our left hand nav which is pretty set up for the whole site. The reason I ask this question to you is because an SEO agency recommended that we... Add the H1 to the left hand navigation and make it customisable so that it is not the same as the breadcrumb keywords. I have attached an image of a competitor of ours that so the same thing currently, to show what I mean... Right now I am not sure what to tell the agency and what the right thing to do is. I read a post saying that we are actually doing the right thing under the circumstances. Does anyone have a best practise good example of generally what we should do for category pages? Your help is always muchly appreciated Kindest, Kay new
Web Design | | eLab_London0 -
Spaces at beginning of title tag - negatively affect the optimization of the page?
For some reason, our title tags have a long space after the beginning title tag and before the text appears. The beginning title tag is on one line, then a break, a tab and then the content of the title tag. I'm pretty sure this is not good and is affecting optimization of the page. Am I correct or is this not an issue and does not need to be fixed? Example: | <title></span></p> <p> </p> </td> </tr> <tr> <td class="line-number"> </td> <td class="line-content"> First keyword</td> </tr> </tbody> </table></title> |
Web Design | | CFSSEO0 -
Does Google count the domain name in its 115-character "ideal" URL length?
I've been following various threads having to do with URL length and Google's happiness therewith and have yet to find an answer to the question posed in the title. Some answers and discussions have come close, but none I've found have addressed this with any specificity. Here are four hypothetical URLs of varying lengths and configurations: EXAMPLE ONE:
Web Design | | RScime25
my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (115 characters) EXAMPLE TWO: sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (126 characters) EXAMPLE THREE: www.sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (130 characters) EXAMPLE FOUR: http://www.sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (137 characters) Assuming the examples contain appropriate keywords and are linked to appropriate anchor text (etc.,) how would Google look upon each? All I've been able to garner thus far is that URLs should be as short as possible while still containing and contextualizing keywords. I have 500+ URLs to review for the company I work for and could use some guidance; yes, I know I should test, but testing is problematical to the extreme; I look to the collective/accumulated wisdom of the MOZVerse for help. Thanks.1 -
What else should you call the Home page?
In the menu bar and footer the main page is called Home. Would it confuse people to rename it to Business Name Home or Business Name? How do you handle this?
Web Design | | CFSSEO0 -
Google indexing Quickview popups
Hi Guys I can't seem to find any info on this. Maybe you can help. We are using xcart as our shopping cart. When you land on a product page you have the option to "Quickview" the item. Google is picking up the quickview urls" and the vote on product urls. I have added the following to the robots.txt file but not sure if this will work. Any help on this would be great. Disallow: /?popup=Y Disallow: /?mode=add Undesired URL Examples: <colgroup><col width="735"></colgroup>
Web Design | | fasctimseo
| http://www.funlove.com/store/6_Pack_Shooter_Beer_Belt/?mode=add_vote&vote=60 | <colgroup><col width="735"></colgroup>
| http://www.funlove.com/store/6_pack_shooter_beer_belt/?popup=Y |0