Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Help Blocking Crawlers. Huge Spike in "Direct Visits" with 96% Bounce Rate & Low Pages/Visit.
-
Hello,
I'm hoping one of you search geniuses can help me.
We have a successful client who started seeing a HUGE spike in direct visits as reported by Google Analytics. This traffic now represents approximately 70% of all website traffic. These "direct visits" have a bounce rate of 96%+ and only 1-2 pages/visit. This is skewing our analytics in a big way and rendering them pretty much useless. I suspect this is some sort of crawler activity but we have no access to the server log files to verify this or identify the culprit. The client's site is on a GoDaddy Managed WordPress hosting account.
The way I see it, there are a couple of possibilities.
1.) Our client's competitors are scraping the site on a regular basis to stay on top of site modifications, keyword emphasis, etc. It seems like whenever we make meaningful changes to the site, one of their competitors does a knock-off a few days later. Hmmm.2.) Our client's competitors have this crawler hitting the site thousands of times a day to raise bounce rates and decrease the average time on site, which could like have an negative impact on SEO. Correct me if I'm wrong but I don't believe Google is going to reward sites with 90% bounce rates, 1-2 pages/visit and an 18 second average time on site.
The bottom line is that we need to identify these bogus "direct visits" and find a way to block them. I've seen several WordPress plugins that claim to help with this but I certainly don't want to block valid crawlers, especially Google, from accessing the site.
If someone out there could please weigh in on this and help us resolve the issue, I'd really appreciate it. Heck, I'll even name my third-born after you.
Thanks for your help.
Eric
-
Hi SirMax,
Thanks for your input. I appreciate it. We'll add Wordfence to our WordPress toolbox and see if that addresses the issue.
In response to previous posts, thanks to everyone for your input. We were able to apply some filters to remove the bogus bot traffic from the analytics and normalize the data, however, this did not actually resolve the issue and in my eyes is more of a BandAid fix. The evil crawlers are still there, we just can't see them.
Thanks again for all of your input.
Eric
-
Hostname filtering does not work any more. Unfortunately most of the spammers have adapted and are using your website as hostname.
For the WordPress I use Wordfence plugin( using paid version - not affiliated with them in any shape or form beyond paying for their services). In the advance blocking you can set limits on how fast and how many pages crawlers can request. You can also block by country or ip range. It can also show you live traffic with a lot of details ( a lot more then google analytic - more like server log ). It might not be the complete remedy but it can help.
-
I wish I had an answer for how to stop the bots from hitting your site at all - I don't think a good one exists, as any solutions that wouldn't also block real human traffic to your site are going to be easy for spam bots to get around. I think your best bet is just to do everything you can to keep your data as clean as possible.
-
Hi Ruth,
Thanks a bunch for taking the time to respond to my post. Great advice. This is reassuring on a number of levels, however, it doesn't address the underlying issue of how to stop these spam bots in the first place.
We've already started the process of filtering out some of this bogus data. We'll also be integrating some WordPress plugins to see if that helps. That said, if the spam bots are hitting Analytics directly, as opposed to the actual website, WP plugins won't do anything.
Anyway, I appreciate your input and advice. Thanks so much.
Eric
-
Hi Eric,
A few things to reassure you off the bat:
- For what it's worth, there is a huge, HUGE amount of crawler spam happening in the web today. Every site I work on is being hit hard with false referrals and direct visits. I know Google Analytics is working on a solution to better filter these visits out. So I wouldn't be too concerned that it is something a competitor is doing to your site, specifically - it's more likely that it's been caught up in the general wave of spam crawlers.
- It's important to note that when we talk about Google looking at bounce rate and dwell time as part of ranking your site, those numbers are specifically from clicks through from search - that's data that Google can get without using your private web analytics data as a ranking factor, which they've said repeatedly that they don't and won't do. So a bunch of direct visits with high bounce rates will NOT affect your rankings.
So, it's not dangerous, just annoying. On to how to get that data out of your reports:
- Make sure you're not filtering out spam referrers at a View level - this can cause those visits to incorrectly appear as direct traffic.
- You could set up an Advanced Segment in Google Analytics to filter out direct visits with visit times of, say, under 5 seconds. Some real traffic may get caught in that, but it will get the noise levels down.
- The best way to filter out spam bot traffic, in my opinion, is to set up hostname filtering. Here's a post on Megalytic on how to do that: https://megalytic.com/blog/how-to-filter-out-fake-referrals-and-other-google-analytics-spam. Make sure you've also got an "Unfiltered Data" View so you'll still have historic raw data if you need it.
Hope that helps! Good luck.
-
Check webserver log files, or log visits (ip address, user agent, __utma, __utmz, possibly browser fingerprint, etc...)
Analyzing those you can easily find out if the traffic is from scraping bot or humans.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Abnormally High Direct Traffic Volume
We have abnormally high amounts of direct traffic to our site. It's comprising over half of all web traffic while organic is second with considerably less. From there the volume decreases amongst other channels. I've never seen such a huge proportion of traffic being attributed the Direct. Does anyone know how to test this or see if there is an error in Google Analytics reporting?
Reporting & Analytics | | graceflack 01 -
Paid traffic or "Paid Search" is not showing in my Google Analytics
Hi, I have two campaigns running in Google Adwords or Google Ads now and I saw in Google Ads account that I had 5 clicks today (09/18/2018) but when I try to search for this clicks in my Google Analytics in ACQUISITION > All Traffic > Channels I don't find nothing about "Paid Search" or something like that. Bellow is a picture of my Google Analytics account to prove it. The accounts are linked and I can find the 2 campaigns in the Analytics. How can I interpret this picture? Where the paid traffic is showing? or not showing there? Thanks Leandro uvAtrsg
Reporting & Analytics | | lmoraes0 -
Using logical operators (AND / OR) in Google Analytics Goal Funnels
When setting up a Funnel within Google Analytics, is it possible to use logical operators (e.g. OR, AND) in the first (required) step of the funnel? For example, suppose I want to track users who visit page1.html AND page2.html before proceeding to the destination goal. I've entered two pages separated by the OR operator, and neither the "Verify this Goal" nor "Save" produces an error message - is it safe to assume that this is working as I intend? Thanks in advance!
Reporting & Analytics | | ahirai0 -
Difference Between Android Browser & Android Webview
Hello All, In google analytic I can see traffic from android browser & android webview so android webview is also a browser? Thanks!
Reporting & Analytics | | dhisman0 -
Should I use sessions or unique visitors to work out my ecommerce conversion rate?
Hi all First question here but I've been lingering in the shadows for a while. As part of my companies digital marketing plan for the next financial year we are looking at benchmarking against certain KPIs. At the moment I simply report our conversion rate as Google Analytics displays it. I was incorrectly under the impression that it was reported as unique visits / total orders but I've now realised it's sessions / total orders. At my company we have quite a few repeat purchasers. So, is it best that we stick to the sessions / total orders conversion rate? My understanding is multiple sessions from the same visitor would all count towards this conversion rate and because we have repeat purchasers these wouldn't be captured under the unique visits / total orders method? It's almost as if every session we would have to consider that we have an opportunity to convert. The flip side of this is that on some of our higher margin products customers may visit multiple times before making a purchase. I should probably add that I'll be benchmarking data based on averages from the 1st April - 31st of March which is a financial year in the UK. The other KPI we will be benchmarking against is visitors. Should we change this to sessions if we will be benchmarking conversion rate using the sessions formula? This could help with continuity and could also help to reveal whether our planned content marketing efforts are engaging users. I hope this makes sense and thanks for reading and offering advice in advance. Joe
Reporting & Analytics | | joe-ainswoth1 -
Google Analytics - Next Page Path is the Same URL?
Hey Everyone, I have a Google analytics question. I'm looking through a client's site and when I look at the next page path, I get the same URL as the next path. For example, on the homepage, the next page path I get is the homepage again? This happens for all URL's, is this an implementation error? Is there a way to fix this? Thanks!
Reporting & Analytics | | EvansHunt0 -
Cookies/Tracking/Code developement
So. I have an online application form, on the website. It's maked with php, html and css. What i like to place on the email, is a cookie tracking system. For example, when someone search on the google for " web design ", and then see my result on google organic, comes the visitor to my site, charge the fields of application forms and click send. Than, me like an admin, i receive an email with informations of a customer, i see name/surname/etc.. but i like to integrate is for example, like to see a: IP from Visitors, who charged the fields Referrerr, if google, like to see which query he typed. How can i do this please?
Reporting & Analytics | | leadsprofi0