Questions created by rogier_slag
-
Recovering from an open redirect
From a previous company we have inherited a domain which once contained an open redirect redirect.magnet.me. Even though this domain has been returning a 410 for every single request directed at it for the last few months, we continue to see new links popping up in Google which refer to this domain https://www.google.com/search?safe=off&q=%22redirect.magnet.me%22 Currently we just keep disavowing these links, but at the same time they keep appearing in our Moz.com tooling, being unable to disavow those links here. Does anybody have any successful tips how to deal with this scenario, compared to only disavow new links after the fact?
Intermediate & Advanced SEO | | rogier_slag0 -
Blocking Google from telemetry requests
At Magnet.me we track the items people are viewing in order to optimize our recommendations. As such we fire POST requests back to our backends every few seconds when enough user initiated actions have happened (think about scrolling for example). In order to eliminate bots from distorting statistics we ignore their values serverside. Based on some internal logging, we see that Googlebot is also performing these POST requests in its javascript crawling. In a 7 day period, that amounts to around 800k POST requests. As we are ignoring that data anyhow, and it is quite a number, we considered reducing this for bots. Though, we had several questions about this:
Technical SEO | | rogier_slag
1. Do these requests count towards crawl budgets?
2. If they do, and we'd want to prevent this from happening: what would be the preferred option? Either preventing the request in the frontend code, or blocking the request using a robots.txt line? The latter question is given by the fact that a in-app block for the request could lead to different behaviour for users and bots, and may be Google could penalize that as cloaking. The latter is slightly less convenient from a development perspective, as all logic is spread throughout the application. I'm aware one should not cloak, or makes pages appear differently to search engine crawlers. However these requests do not change anything in the pages behaviour, and purely send some anonymous data so we can improve future recommendations.0 -
Hide messenger for crawlers
At Magnet.me we are using Intercom to communicate with our users. This means that we are actively adding javascript code which will load the Intercom javascript on each page, and render the button afterwards. However, this button has no value for crawlers, and slows the page down as the javascript is big and fairly slow. Therefore I considered to ship some code which disables this button, such that performance would improve. To give a ball pack estimate, the buttons javascript is around 3x bigger than the actual entire react application... Unfortunately this would result in giving users and crawlers slightly different content on the page. I'm unsure about the possible SEO impact: Would Google mark the page as faster due to less resources to load? Or would it penalize the page for showing slightly different content to users and search engines?
Technical SEO | | rogier_slag0