Some bots excluded from crawling client's domain
-
Hi all!
My client is in healthcare in the US and for HIPAA reasons, blocks traffic from most international sources.
a. I don't think this is good for SEO
b. The site won't allow Moz bot or Screaming Frog bot to crawl it. It's so frustrating.
We can't figure out what mechanism they are utilizing to execute this. Any help as we start down the rabbit hole to remedy is much appreciated.
thank you!
-
The main reason it's not good is that Google crawl from different data-centers around the world. So one day they may think the site is up, then the next they may think the site is gone and down
Typically you use a user-agent lance to pierce these kinds of setups. Screaming Frog for example, you can pre-select from a variety of user-agents (including 'googlebot' and Chrome) but you can also author or write your own user-agent
Write a long one that looks like an encryption key. Tell your client the user agent you have defined, let them create and exemption for it within their spam-defense system. Insert the user-agent (which no one else has or uses) into Screaming Frog, use it to allow the crawler to pierce the defense grid
Typically you would want to exempt 'Googlebot' (as a user agent) from these defense systems, but it comes with a risk. Anyone with basic scripting knowledge or who knows how to install Chrome extensions, can alter the user-agent of their script (or web browser, it's under the user's control) with ease and it is widely known that many sites make an exception for 'Googlebot' - thus it becomes a common vulnerability. For example, lots of publishers create URLs which Google can access and index, yet if you are a bog standard user they ask you to turn off ad-blockers or pay a fee
Download the Chrome User-Agent extension, set your user-agent to "googlebot" and sail right through. Not ideal from a defense perspective
For this reason I have often wished (and I am really hoping someone from Google might be reading) that in Search Console, you could tell Google a custom user-agent string and give it to them. You could then exempt that, safe in the knowledge that no one else knows it, and Google would use your own custom string to identify themselves when accessing your site and content. Then everyone could be safe, indexable and happy
We're not there yet
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
If I'm using a compressed sitemap (sitemap.xml.gz) that's the URL that gets submitted to webmaster tools, correct?
I just want to verify that if a compressed sitemap file is being used, then the URL that gets submitted to Google, Bing, etc and the URL that's used in the robots.txt indicates that it's a compressed file. For example, "sitemap.xml.gz" -- thanks!
Technical SEO | | jgresalfi0 -
Google how deal with licensed content when this placed on vendor & client's website too. Will Google penalize the client's site for this ?
One of my client bought licensed content from top vendor of Health Industry. This same content is on the vendor's website & my client's site also but on my site there is a link back to vendor is placed which clearly tells to anyone that this is a licensed content & we bought from this vendor. My client bought paid top quality content for best source of industry but at this same this is placed on vendor's website also. Will Google penalize my client's website for this ? Niche is HEALTH
Technical SEO | | sourabhrana1 -
Should I use my competitor's name in my content to help my rankings?
If I have a competitor that ranks higher than me, would it be helpful to use their name in my content, or in my meta information?
Technical SEO | | greaterstudio0 -
How do I check that domain I'm considering buying has clean history?
I'm weighing up buying a niche-related 13-year-old domain that hasn't expired but hasn't been used in some time. I've read the very informative thread here: www.seomoz.org/q/buying-domains-with-prior-age-and-or-pr The backlink profile is clean, with only on-topic links. However, it doesn't rank in the top 50 positions for its own (hyphenated & fairly competitive) domain name. Does this indicate a Google penalty of some kind? What's the surefire way of checking?
Technical SEO | | Jeepster0 -
Old domain vs. New keyword domain - Thoughts?
Okay. I'd like to get opinions as to what everyone thinks about domains lately. Here is any example: The current domain is general in nature, in fact, it's a persons name because they are a real estate agent. So the domain is something like JohnDoe.com. Current stats: Has approx. 130 linking domains pointing to it. Has over 300 incoming links from these linking domains. The link profile is clean and not spammy (not to say there are not a few that might be here and there) Was bough in 1994 The new domain would have very little value except it would be keyword rich such as PortlandHomesForSale.com (just an example). What are your thoughts. Thank you.
Technical SEO | | JordanRussell0 -
301'ing domain to an addon domain
My googlefu failed me in finding this... How to 301 a domain to an addon domain? Domain structure is as follows: http://addondomain.maindomain.com/ http://www.maindomain.com/addondomain/ http://www.addondomain.com/ <--(addon domain has its own domain as well) I want main domain to all point to the addon domain like so: http://www.maindomain.com/ --> http://www.addondomain.com/
Technical SEO | | JasonJackson0 -
What does it mean by 'blocked by Meta Robot'? How do I fix this?
When i get my crawl diagnostics, I am getting a blocked by Meta Robot, which means that my page is not being indexed in the search engines... obviously this is a major issue for organic traffic!!! What does it actually mean, and how can i fix it?
Technical SEO | | rolls1230 -
What's the best way to eliminate duplicate page content caused by blog archives?
I (obviously) can't delete the archived pages regardless of how much traffic they do/don't receive. Would you recommend a meta robot or robot.txt file? I'm not sure I'll have access to the root directory so I could be stuck with utilizing a meta robot, correct? Any other suggestions to alleviate this pesky duplicate page content issue?
Technical SEO | | ICM0