How can I prevent Google and other search engines to crawl my secured pages (https:)?
-
Let me know your thoughts guys. Thanks in advance!
-
Your best bet is to place a meta noindex tag on each secure page. If it's only a few pages, you could just add it by hand. If it's many, you should be able to access each pages protocol with whatever server-side language you're using, and dynamically add it on all secure pages.
If you use robots.txt to exclude the pages, Google can still show them in search results, with the description below them that reads, "A description for this result is not available because of this site's robots.txt – learn more." Personally, I don't care for that.
-
Hi there, blocking the HTTPS version of your pages from being crawled by the search engines is a bit tricky. You might need to come up with a separate robots.txt file to handle the HTTPS requests.
Here you go to know more about the process:
http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html
Hope it helps.
Best,
Devanur Rafi
-
Hi esiow
You have a choice of placing a robots.txt file in the root folder of your website or if blocking individual pages you could use the meta robots tag. See these page for more information: http://moz.com/learn/seo/robotstxt and https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1
I hope that helps,
Peter
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Changing Platforms & HTTPS Same Time
Hi Everyone, We know this is a bad idea. Our owner purchased another website/company and we plan on moving it to Shopify. The site is a very old (not been updated in 4 years) Able Commerce site. My developer whom has been doing this for years with me couldn't get the current host to work with us in order to move the site to https. Looks like we would actually have to spend a lot of money to update the site in order to. So my question is this. If we move the site over to Shopify, they make it https right away. I am out of luck trying to redirect http pages to specific https pages? Jeff
Web Design | | vetofunk0 -
Google Indexing Multi-Store Best Practice
Hi Guys, We currently have a main store view and a uk store view setup with a Litespeed Redirect for our website, redirecting UK IP Customers to the UK Store. We recently noticed that we were running into some issues with Google indexing pages from the uk site as well as the main store view. With trying to avoid duplicate content, my question being: What is the best practice for google indexing the UK and Main store views? Any advice would be greatly appreciated. Thanks.
Web Design | | centurysafety0 -
What is your opinion in the use of jquery for a continuous scroll type of page layout?
So, I'm in 2 minds about this; let me start with a bit of background info. Context
Web Design | | ChrisAshton
We have a new client who is in the final days of their new site design and were when they first contacted us. Their design essentially uses 5 pages, each with several pages worth of content on each, separated with the use of jquery. What this means is a user can click a menu item from a drop-down in the nav and be taken directly to that section of content like using internal anchor links as if it were a separate page, or they can click the top-level nav item and scroll through each "sub-page" without having to click other links. Vaguely similar to Google's "How Search Works" page if each sector of that page had it's own URL, only without the heavy design elements and slow load time. In this process, scrolling down to each new "sub-page" changes the URL in the address bar and is treated as a new page as far as referencing the page, adding page titles, meta descriptions, backlinks etc. From my research this also means search engines don't see the entire page, they see each sub-page as their own separate item like a normal site. My Reservations I'm worried about this for several reasons, the largest of them being that you're essentially presenting the user with something different to the search engines. The other big one being that I just don't know if search engines really can render this type of formatting correctly or if there's anything I need to look out for here. Since they're so close to launching their new site, I don't have time to set up a test environment and I'm not going to gamble with a new corporate website but they're also going to be very resistant to the advice of "start the design over, it's too dangerous". The Positives
For this client in particular, the design actually works very well. Each of these long pages is essentially about a different service they offer and the continuous scrolling through the "sub-pages" acts as almost a workflow through the process, covering each step in order. It also looks fantastic, loads quickly and has a very simple nav so the overall user experience is great. Since the majority of my focus in SEO is on UX, this is my confusion. Part of me thinks that obscuring the other content on these pages and only showing each individual "sub-page" to search engines is an obvious no-no, the other part of me feels that this kind of user experience and the reasonable prevalence of AJAX/Paralax etc means search engines should be more capable of understanding what's going on here. Can anyone possibly shed some light on this with either some further reading or first-hand experience?0 -
Will changing product from Grouped to Simple on my magento category page affect my SEO?
Hi all, A category page on my site http://www.porcelainsuperstore.co.uk/wood-effect.html currently ranks number 3 on Google for the keyword "Wood Effect Tiles" We're currently reorganising some of our product and I would like to know if this is going to affect the SEO and ranking for the above page and keyword. The majority of products on that page are magento grouped products. I would like to change the page so that it displays only the different constituent simple products rather than the grouped products on the category page. My question is, will this have any impact on SEO? I intend on leaving all other data on the category page the same - so the metadata and the description/title etc. Any help/comments would be much appreciated! Ben
Web Design | | piazza0 -
Can the website pages have the site name like Title of the page | Sitename.com
Hi, Can the website pages have the site name like Title of the page | Sitename.com I have a site with 50K pages and all pages have | Sitename.com mentioned would that be a good practice or bad? Thanks Martin
Web Design | | mtthompsons0 -
Redirecting duplicate pages
For whatever reason, X-cart creates duplicates of our categories and articles so that we have URLs like this www.k9electronics.com/dog-training-collars
Web Design | | k9byron
www.k9electronics.com/dog-training-collars/ or http://www.k9electronics.com/articles/anti-bark-collar
http://www.k9electronics.com/articles/anti-bark-collar/ now our SEO guy says that we dont have to redirect these because google is "smart enough" to know they are the same, and that we should "leave it as-is". However, everything I have read online says that google sees this as dupe content and that we should redirect to one or the other / or no /, depending on which most of our internal links already point to, which is with a slash. What should we do? Redirect or leave it as is? Thanks!0 -
Page Title Optimization
I am reviewing the optimization on my site and it appears that my page titles follow this method: PAGE_NAME | KEYWORD in CITY ST - COMPANY_NAME I am pretty well optimized for "KEYWORD in CITY ST" but am wondering if I should drop it from all page titles except for the pages that actually deal with that keyword. What are your thoughts on optimizing?
Web Design | | nusani0 -
One big page vs. multi-step pages
Hi mozers! Brand new to SEO and LOVING it! Having several key questions that I don't see answered yet, but I'll start with one we've been very curious about. Consider this guide we have for Forming a Delaware Corp.
Web Design | | Mase
https://www.upcounsel.com/Free-Legal/Guide/17/Form-A-Delaware-Corporation This is our overview page, giving you a breakdown of what this process involves. We love this page, but (Question1:) does it lack better real "content" rather than lots of links to the guide process itself? Then, you can start to walk through the guide beginning with step one, where each step has crowd sourced answers to it. But as you see, the step pages are all very similar, except for the answers and step info. (Question 2) Would it be better to put all our answers into the one overview page and skip having separate pages for each step? We like the process and simplicity of seeing one step at a time, but then these pages don't seem to have enough unique content on them. Related, at what point (if any) is a page too big with too much content and considered bad for SEO? We're recovering from a big hit from Google, and slowly recovering by nailing down various SEO mistakes. We DO have great, unique and valueable content - now we just need it to rank!0
