Hide messenger for crawlers
-
At Magnet.me we are using Intercom to communicate with our users. This means that we are actively adding javascript code which will load the Intercom javascript on each page, and render the button afterwards.
However, this button has no value for crawlers, and slows the page down as the javascript is big and fairly slow. Therefore I considered to ship some code which disables this button, such that performance would improve. To give a ball pack estimate, the buttons javascript is around 3x bigger than the actual entire react application...
Unfortunately this would result in giving users and crawlers slightly different content on the page. I'm unsure about the possible SEO impact:
- Would Google mark the page as faster due to less resources to load?
- Or would it penalize the page for showing slightly different content to users and search engines?
-
In general, I don't think that this is a great idea. Although Google does meter out crawl-allowance, Google also wants a realistic view of the pages which it is crawling. Your attempt at easing the burden of Google's crawl-bots may be seen as an attempt to 'fake' good page-speed metrics, for example (by letting Google load the web-page much faster than end users). This could cause some issues with your rankings if uncovered by a 'dumb' algorithm (which won't factor in your good intentions)
Your efforts may also be unrequired. Although Google 'can' fire and crawl JavaScript generated elements, it doesn't always do so and it doesn't do that for everyone. If you read my (main) response to this question, you'll get a much better idea of what I'm talking about here. As such, the majority of the time - you may be taking on 'potential' risk for no reward
Would it be possible to code things slightly differently? Currently you state that this is your approach:
"This means that we are actively adding javascript code which will load the Intercom javascript on each page, and render the button afterwards"
Could you not add the button through HTML / CSS, and bind a smaller script to the button which then loads the "Intercom javascript"? I am assuming here that the "Intercom javascript" is the large script which is slowing the page(s) down. Why not load that script, only on request (seems logical, but also admit I am no dev - sorry)? It just seems as though more things are being initiated and loaded up-front than are really required
Google want to know which technologies are deployed on your page if they choose to look, they also don't want people going around faking higher page-speed loading scores
If you really want to stop Google wasting time on that script, your basic options would be:
- Code the site to refuse to serve the script to the "googlebot" user agent
- Block the script in robots.txt so that it is never crawled (directive only)
The first option is a little thermonuclear and may mean you get accused of cloaking (unlikely), or at the least 'faking' higher page-speed scores (more likely). The second option is only a directive which Google can disregard, so the risks are lower. The down-side is that Google will pick up on the blocked resource, and may not elevate your page-loading speed. Even if they do, they may say "since we can't view this script or know what it does, we don't know what the implication for end-users is so we'll dampen the rankings a little as a risk assessment factor"
Myself, I would look for an implementation that doesn't slow the site down so much (for users or search-bots). I get that it may be tricky, obviously re-coding the JS from Intercom would probably break the chat entirely. Maybe though, you could think about when that script has to be loaded. Is it really needed, on page-load, all the time for everyone? Or do people only need that functionality, when they choose to interact? How can you slot the loading of the code into that narrow trench, and get the best of both worlds?
Sorry it's not a super simple answer, hope it helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawler issues on subdomain - Need resolving?
Hey Guys, I'm fairly new to the world of SEO and have a ton of crawler issues with a friends website I'm doing some work on. After Moz did a site crawl I'm getting loads of errors (Total of 100+ for critical crawler, content and meta data). Most of these are due to broken social links on a subdomain - so my question is do I need to resolve all of the errors even if they are on a sub-domain? Will it affect the primary website? Thanks, Jack
Technical SEO | | Jack11660 -
What crawler do you recommend for finding orphaned pages on a website?
Is there a crawler that you guys recommend for finding all pages, including orphaned pages on a website? A data export is not feasible. I saw a question from back in 2013 and was wondering if anything has changed since then in regards to crawling orphaned pages. Do most enterprise systems already have this built into their crawler? Or is it best to get a crawler like Xenu or Screaming Frog or Deepcrawl?
Technical SEO | | DigitalMarketingSEO0 -
301 Redirects, Sitemaps and Indexing - How to hide redirected urls from search engines?
We have several pages in our site like this one, http://www.spectralink.com/solutions, which redirect to deeper page, http://www.spectralink.com/solutions/work-smarter-not-harder. Both urls are listed in the sitemap and both pages are being indexed. Should we remove those redirecting pages from the site map? Should we prevent the redirecting url from being indexed? If so, what's the best way to do that?
Technical SEO | | HeroDesignStudio0 -
Will Google hit me because I hide my H1 tag?
I read an article this morning regarding keywords on a web page. In the article it said that Google would hit anyone putting keywords on a web page but then hiding them from anyone visiting the website. This makes sense. What it did make me think about though is the technique I use when building a website. If, for example, I'm building a website for "Acme Cheap Products". In the header of the page, I will have a H1 tag as well as an image tag for the company's logo. As the logo has the company name on it, I would usually put the company name in a H1 tag as well, and then hide the H1 tag, so I wouldn't have a logo and then a title next to it saying the same thing as the logo. The question is though, would this sort of technique trigger Google in to hitting my site?
Technical SEO | | -Al-0 -
Duplicate Content Issue: Google/Moz Crawler recognize Chinese?
Hi! I am using Wordpress multisite and my Chinese version of the website is in www.mysite.com/cn Problem: I keep getting duplicate content errors within www.mysite.com/cn (NOT between www.mysite.com and www.mysite.com/cn) I have downloaded and checked the SEOmoz report and duplicate_page_content list in CSV file. I have no idea why it says they have the same content., they have nothing in common in content . www.mysite.com is the English version of the website,and the structure is the same for www.mysite.com/cn *I don't have any duplicate content issues within www.mysite.com Question: Does google Crawler properly recognizes chinese content??
Technical SEO | | joony20080 -
SeoMoz crawler giving false positives?
SeoMoz crawler indicated a few times that my site has a duplicate home page error (http://mysite.com and www.mysite.com) I eliminated the the couple remaining internal links that pointed to http://mysite on a couple pages (all other internal links point to http://www.mysite.com) I ran the crawl again and it said no errors this time. I naturally thought the duplicate page error problem was fixed. However this morning I got the regularly scheduled crawl report from SeoMoz that said again I have those duplicate error pages. No changes were made to any of my site's pages between the crawls. That makes me wonder if the crawler is providing false positives at times or was wrong when it said on the crawl a couple days ago that I don't have any errors (no duplicate page error). Now, I don't know what to think.
Technical SEO | | finalfrontier0 -
Does using parentheses affect the crawlers?
Quick question: if you using a parantheses around a keyword, do search bots still recognize the keyword? Fox ex: Welcome to a website about the National Basketball Association (NBA). Will the bots recognize that I'm trying to optimize to NBA and not (NBA)? Is this different for tags vs. actual body copy?
Technical SEO | | BPIAnalytics2 -
Hiding text with Javascript and a more button
I am considering putting a block of text on may pages, that initially appears as a snippet with a 'show more' button that expands to show the whole lot. Question: If the search engines can see the whole lot, but the visitor only sees the snippet until they click 'show more' then is this cloaking? Is it a really bad idea? Or can I get away with it because I am not being deceptive just improving the design? Help!
Technical SEO | | mascotmike0