Removing secure subdomain from google index
-
we've noticed over the last few months that Google is not honoring our main website's robots.txt file. We have added rules to disallow secure pages such as:
Disallow: /login.cgis Disallow: /logout.cgis Disallow: /password.cgis Disallow: /customer/* We have noticed that google is crawling these secure pages and then duplicating our complete ecommerce website across our secure subdomain in the google index (duplicate content) https://secure.domain.com/etc. Our webmaster recently implemented a specific robots.txt file for the secure subdomain disallow all however, these duplicated secure pages remain in the index.
User-agent: *
Disallow: /My question is should i request Google to remove these secure urls through Google Webmaster Tools? If so, is there any potential risk to my main ecommerce website? We have 8,700 pages currently indexed into google and would not want to risk any ill effects to our website. How would I submit this request in the URL Removal tools specifically? would inputting https://secure.domain.com/ cover all of the urls? We do not want any secure pages being indexed to the index and all secure pages are served on the secure.domain example. Please private message me for specific details if you'd like to see an example. Thank you,
-
I think you're saying you have
mainwebsitethatsellsstuff.com
securesubdomainof.mainwebsitethatsellsstuff.comand that you want to keep the main domain, and remove the subdomain, and that it's not a case of http vs https with the URL otherwise being the same, right?
You can verify a subdomain in Google Webmaster Tools and remove the entire subdomain. I've had to do this for a dev subdomain that accidentally got indexed. I was able to keep the main domain, and remove the subdomain. The key is to verify that subdomain, and leave the main domain alone, provided I'm understanding your question correctly.
-
Do you need 8700 pages served on https? Protocol should transition when a page is ok to serve unsecured. Generally you would only serve pages on https that contain confidential information and have general content on http. If you look at the site and ask how many of those pages can a no logged in user see? If they are not protected by authorization then they do not need https as the content is publically viewable.
-
URL Removal would not be a good action in this case. According to Google, when they remove the https version, they will also remove the http version along with it.
How long ago did you implement the robots.txt exclusion for the https pages? It will take Google some time to pull this from their index. To help you can add the following on your https pages which will keep the pages from continuing to be cached:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are we confusing Google with our internal linking?
Hi all, We decided to give importance to one of our top pages as it has "keyword" in it's slug like website.com/keyword. So we internally linked even from different sub-domain pages more than homepage to rank for that "keyword". But this page didn't show up in Google results for that "keyword"; neither homepage, but our login page is ranking. We wonder why login page is ranking. Has our internal linking plan confused Google to ignore homepage to rank for that primary keyword? And generally do we need to internally link homepage more than anyother page? Thanks
Algorithm Updates | | vtmoz0 -
Bing not indexing pages
We have taken all recommended steps to index our site sitegeek.com pages to Bing Bot but failed to index them. Bing bot crawled more than 5,000 pages every day but strange why pages are not getting index ? if we query site:sitegeek.com in Bing Bing Search Engine shows only 1,200 pages got indexed. but we query site:sitegeek.com in Google Google Search Engine show more 546,000 pages got indexed. For example : https://www.sitegeek.com/000webhost Above page crawled by Google but Bing. Can anyone suggest what we are missing on this page? what need to change to index such pages? Thanks! Rajiv
Algorithm Updates | | gamesecure0 -
Site not in Google top 50 for key terms
Dear Moz Community, Our site - http://www.sportsdirectnews.com publishes a high volume of daily sport stories and aims to follow Google's Webmaster Guidelines, yet our pages don't appear anywhere in Google's SERP's. We've looked in details at the issue and think it could be something to do with: a) Unusual links or b) High page loading time or c) Too many on-page links If you could have a look at the site - http://www.sportsdirectnews.com - and give your professional opinion as to why our website is not appearing in SERP's, we would be most appreciative. SDN
Algorithm Updates | | BoomDialogue690 -
Google webmaster tool content keywords Top URLs
GWT->Optimization->Content Keywords section... If we click on the keyword it will further shows the variants and Top URLs containing those variants. My problem is none of the important pages like product details pages, homepage or category pages are present in that Top URLs list. All the news, guides section url's are listed in the Top URLs section for most important keyword that is also present in my domain name. How to make google realize the important pages for the important keyword?
Algorithm Updates | | BipSum0 -
Google penalty for one keyword?
Is it possible to get penalized by Google for a specific keyword and essentially disappear from the SERPs for that keyword but keep position for the brand (#1) and some other keywords (#4 and #7)? And how would you find out that this is what happened if there is no GWT message?
Algorithm Updates | | gfiedel0 -
Google indexing my website's Search Results pages. Should I block this?
After running the SEOmoz crawl test, i have a spreadsheet of 11,000 urls of which 6381 urls are search results pages from our website that have been indexed. I know I've read that /search should be blocked from the engines, but can't seem to find that information at this point. Does anyone have facts behind why they should be blocked? Or not blocked?
Algorithm Updates | | Jenny10 -
Google above the fold update
Hi everyone, Ever since the Jan 19th Google 'above the fold update' I have noticed some strange ranking changes in some of my sites. 1. rankings increased dramatically (not in top 50 to page 2) on Jan 19th for about 5 days then dropped out completely from the top 50. 2. our rankings then did the same thing again around Feb 2nd for about 5 -6 days then has bottomed out ever since. We do not have any ads on the site but our pages are dominated by images for most of the 'above the fold' section then followed by the content down the page. Any insight into this would be much appreciated. Cheers, Andrew
Algorithm Updates | | jay.raman0 -
Google's reaction to site updates
Hi, Is it safe to assume as soon as Google indexes updates I've made to my site that any ranking changes the updates effected will happen at that same time, or is there ever a lag time before these changes ( if any ) take effect?
Algorithm Updates | | minutiae0