Site with 2 domains - 1 domain SEO opimised & 1 is not. How best to handle crawlers?
-
Situation:
I have a dual domain site:
Domain 1 - www.domain.com is SEO optimised with product pages and should of course be indexed.
Domain 2 - secure.domain.com is not SEO optimised and simply has checkout and payment gateway pages.I've discovered that Moz automatically crawls Domain 2 - the secure.domain.com site and consequently picks up hundreds of errors.
I have put an end to this by adding a robots.txt to stop rogerbot and dotbot (mozs crawlers) from crawling domain 2. This fixes my errors in Moz reports however after doing more research into 'Crawler Control' I figure this might be the best option.
My Question:
Instead of using robots.txt to stop moz from crawing all of Domain 2 should I use on each page of domain 2?
I believe this would then allow moz and google to crawl Domain 2 but also tell them both not to index it.
My understanding is that this would be best, and might even help my overall SEO by telling google not to give any SEO value to the Domain 2 pages? -
Hello!
I can answer this from a Google / SEO perspective (a non-moz tool perspective).
First you want to be sure the secure subdomain content is not indexed.
-
If the secure subdomain is NOT indexed, leave the robotos.txt crawl blocking in place. You don't want and don't need Google crawling secure pages and payment pages. Just be sure they truly all are private pages. If they are NOT indxed, the crawl block is best - this will prevent google from crawling, and if they can't crawl they can't index.
-
If the secure pages ARE indexed
-
remove the robots.txt crawl block.
-
Add meta noindex on all the pages
-
Wait for them to be noindexed (removed from google)
-
Then, block them from being crawled with robots.txt - which will prevent them from being crawled, and thus prevent them from being indexed as well.
-
-
Hey, Dave here from the Help Team!
Jumping in to answer the technical question, you can definitely use the meta robots tag instead of a disallow directive in your robots.txt file. I would like to point out that Meta Noindex is something we report in Site Crawl so you would see an influx in that issue category but you can mark them as "ignored" as you see fit.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Crawler was not able to access the robots.txt
I'm trying to setup a campaign for jessicamoraninteriors.com and I keep getting messages that Moz can't crawl the site because it can't access the robots.txt. Not sure why, other crawlers don't seem to have a problem and I can access the robots.txt file from my browser. For some additional info, it's a SquareSpace site and my DNS is handled through Cloudflare. Here's the contents of my robots.txt file: # Squarespace Robots Txt User-agent: GPTBot User-agent: ChatGPT-User User-agent: CCBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: FacebookBot User-agent: Claude-Web User-agent: cohere-ai User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps User-agent: * Disallow: /config Disallow: /search Disallow: /account$ Disallow: /account/ Disallow: /commerce/digital-download/ Disallow: /api/ Allow: /api/ui-extensions/ Disallow: /static/ Disallow:/*?author=* Disallow:/*&author=* Disallow:/*?tag=* Disallow:/*&tag=* Disallow:/*?month=* Disallow:/*&month=* Disallow:/*?view=* Disallow:/*&view=* Disallow:/*?format=json Disallow:/*&format=json Disallow:/*?format=page-context Disallow:/*&format=page-context Disallow:/*?format=main-content Disallow:/*&format=main-content Disallow:/*?format=json-pretty Disallow:/*&format=json-pretty Disallow:/*?format=ical Disallow:/*&format=ical Disallow:/*?reversePaginate=* Disallow:/*&reversePaginate=* Any ideas?
Getting Started | | andrewrench0 -
Can't Crawl Site - but deducting crawls.
Why am I being deducted crawls if MOZ keeps telling me that it can't crawl my site?
Getting Started | | BloggyMoms1 -
Moz Site Crawl can't index WIX sites
We've been attempting to work on some SEO for a new potential client however they are using a WIX site. We've noticed that Moz SEO tools will not index any WIX sites. e.g. https://www.sharonradisch.com/ (which is one of their case studies). Anyone seen this that can offer any advice? Thanks,
Getting Started | | monkeex
Mark2 -
Mozbot Can Not Crawl Entire Domain
I'm trying to crawl Redken.com in Moz Analytics and the Search Diagnostics is only crawling 4 pages. The domain uses a "select your country" the first time you visit, and it seems as though the bot is not getting beyond that (aka, not clicking on "USA") and is therefore not crawling the rest of the domain. There is no country specific URL other than redken.com. I've tried entering both "redken.com" and "www.redken.com" as the URL, but no luck. Any tips?
Getting Started | | LabeliumUSA0 -
Moz Pro for simple local seo status reporting
I have a new client on a small budget with a local yoga studio. There's been no SEO work on this site so far, so, after the basics, I plan to work through the items on this list: https://moz.com/blog/top-20-local-search-ranking-factors-an-illustrated-guide I'm a new Moz Pro user and am looking for advice on how to use Moz Pro tools with my client to review progress over time...a simple checklist of reports/metrics to share and track, and to use in weekly status reports. Or, in other words, how can I configure and use Moz Pro tools for basic progress tracking? This is not a sophisticated, SEO-savvy client, so I want to keep things very simple. Thanks for any ideas or pointers. Dave
Getting Started | | dgbruns0 -
Does the Moz crawler have a static ip address?
We block AWS from crawling our site. Does the moz crawler use a static ip address that we could allow? I'm currently not able to add a campaign because Moz can't connect to our site.
Getting Started | | uShip0 -
In Open site explorer the page title and Url show in the left hand column. Why do some of my pages have no data for page title?
I am a first time user. Newly updated site using Drupal and having lots of SEO problems. Under site explorer, several pages list NO DATA for the page title. This doesn't seem right. Any suggestions on what this means?
Getting Started | | IV-Debbie0
