Blocking Google from telemetry requests
-
At Magnet.me we track the items people are viewing in order to optimize our recommendations. As such we fire POST requests back to our backends every few seconds when enough user initiated actions have happened (think about scrolling for example). In order to eliminate bots from distorting statistics we ignore their values serverside.
Based on some internal logging, we see that Googlebot is also performing these POST requests in its javascript crawling. In a 7 day period, that amounts to around 800k POST requests. As we are ignoring that data anyhow, and it is quite a number, we considered reducing this for bots.
Though, we had several questions about this:
1. Do these requests count towards crawl budgets?
2. If they do, and we'd want to prevent this from happening: what would be the preferred option? Either preventing the request in the frontend code, or blocking the request using a robots.txt line?The latter question is given by the fact that a in-app block for the request could lead to different behaviour for users and bots, and may be Google could penalize that as cloaking. The latter is slightly less convenient from a development perspective, as all logic is spread throughout the application.
I'm aware one should not cloak, or makes pages appear differently to search engine crawlers. However these requests do not change anything in the pages behaviour, and purely send some anonymous data so we can improve future recommendations.
-
Hi Rogier,
- Yes, this is usually counting towards crawl budgets as Googlebot is doing this per request.
- It depends on how your request is being set up obviously, otherwise, I would advise going with the exclusion for the robots.txt that you're already heading towards.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google serp pagination issue
We are a local real estate company and have landing pages for different communities and cities around our area that display the most recent listings. For example: www.mysite.com/wa/tumwater is our landing page for the city of Tumwater homes for sale. Google has indexed most of our landing pages, but for whatever reason they are displaying either page 2, 3, 4 etc... instead of page 1. Our Roy, WA landing page is another example. www.mysite.com/wa/roy has recently been showing up on page 1 of Google for "Roy WA homes for sale", but now we are much further down and www.mysite.com/wa/roy?start=80 (page 5) is the only page in the serps. (coincidentally we no longer have 5 pages worth of listings for this city, so this link now redirects to www.mysite.com/wa/roy.) We haven't made any major recent changes to the site. Any help would be much appreciated! *You can see what my site is in the attached image... I just don't want this post to show up when someone google's the actual name of the business 🙂 nTTrSMx.jpg C4mhfgh.jpg
Technical SEO | | summithomes0 -
Wrong page title in Google
Hi there, A while ago we took over the domain www.hoesjes.nl and forwarded it to our website www.telefoonhoesjesxl.nl. If you perform a search for the keyword 'hoesjes' in Google then we (www.telefoonhoesjesxl.nl) show up on an organic number 1 position. The problem is that the page title isn't correct. Google shows the page title of the website hoesjes.nl we took over and (correctly?) redirected to our domain www.telefoonhoesjesxl.nl. Does anybody have any idea how to get rid of this wrong page title in Google?
Technical SEO | | MarcelMoz
Here you can find a screenshot of what I mean. Thanks! Marcel0 -
No google traffic for this site? Help?
Hi We have not done this web site http://climateacs.co.uk but have now picked it up and its getting no traffic what so ever from google do you think its been blacklisted? I have added it to my webmaster tools and there are no manual actions on it and most of the backlinks on google webmaster tools are from yell.com. However when I run it on opensiteexplorer I am seeing some chinese type links?? It is not really showing many search queries at all when you view them in webmaster tools under United Kingdom. I was going to start citation building for the address to help support the google places entry but just wanted to see what other peoples opinion was really on this site? Thanks Tracy
Technical SEO | | dashesndots0 -
Homepage disappeared from Google Serp
I redirected my domain using this code in .htaccess : RewriteCond %{HTTP_HOST} ^xxxx.com
Technical SEO | | digitalkiddie
RewriteRule (.*) http://www.xxxx.com/$1 [R=301,L]
<ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L] RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]/)index.(html?|php)(?[^\ ])?\ HTTP/
RewriteRule ^(([^/]/)*)index.(html?|php)$ http://www.xxxx.com/$1 [R=301,L]</ifmodule> A day after I did it, got an error in GWMT "Google can't find your site's robots.txt" and my homepage disappeared from the result pages. When I try to open Google cache of the homepage I got an error 404. I generated new robots.txt, uploaded it , now the error doesnt show but still my homepage is not in the serps. Its been 3 days. What should I do ? Thanks in advance "Google can't find your site's robots.txt" error? - Pro ...0 -
Google is ranking the wrong page
We are trying to figure out why google is ranking the wrong page for the key word motorcycle tires. We have a few ideas but are not sure yet. If you do a search for Motorcycle Tires you will see site on page 2 or top of 3; however, the page will be going to our dirt bike tires page (http://www.rockymountainatvmc.com/t/44/86/176/742/Dirt-Bike-Tires-All) not our Motorcycle page (http://www.rockymountainatvmc.com/t/49/-/181/750/Motorcycle-Tires-All) any thoughts? We think we know why but want others opinions too.
Technical SEO | | DoRM0 -
What else: struggling with google position
Hi. I understand everyone is offering their time for free here so any advice or support is much appreciated. http://www.cytronex.com
Technical SEO | | AdamJamesCytronex
PA 44 || mR 4.6 || mT 5.73 || 986 links from 43 Root Domains
DA 33 || 3,942 links from 71 Domains We've dropped from position 25ish to position 70ish in keyword searches for 'electric bikes'. I've tried everything and I just don't understand! It's genuine content, the actual product is increasingly popular, we have several links from sites which are (well, to my mind) reasonable quality. I've only just been brought in to look at this and my lack of any SEO or web experience is not putting my boss off expecting an instant solution 😞 As I'm only just getting to grips with it, Analytics was only installed about a month ago so I can't pin point a moment when it dropped. We're consistently out-positioned by sites with lower PA/DA scores. Any insight anyone might have would be amazing! Thanks
Adam0 -
Why is google ignoring my sitelinks demotions?
I'm referring to the sitelinks that appear in the SERPs when searching for my brand name. 6 subpages come up- some found in my main navigation, some not. 3 of the 6 sitelinks have been demoted in Webmaster Tools under "Site Configuration > Sitelinks." I realize that they say in their instructions: "Google doesn't guarantee that demoted URLs will never appear as a sitelink, but we do consider a demotion a strong hint that we'll try to honor when generating sitelinks." I am just surprised they would ignore 3 of the demotions and for so long. The pages that are demoted do not have very many internal links pointing to them (unlike other pages of my site that are targeted specifically). Also, the site has tens of thousands of pages to choose from. Why are they ignoring my request? What else can I do to fix this?
Technical SEO | | Hakkasan0 -
Look of google results
Can anyone tell me why some google results show the main page and then a listing of all subsequent pages (i.e. results for SEOMOZ) while others just show the main page with nothing under it. I have two different sites (one personal the other biz) and they both show their search results differently. Is it something in the site creation or how it is crawled by google? Thanks. bKs3C
Technical SEO | | STF0