Help Blocking Crawlers. Huge Spike in "Direct Visits" with 96% Bounce Rate & Low Pages/Visit.
-
Hello,
I'm hoping one of you search geniuses can help me.
We have a successful client who started seeing a HUGE spike in direct visits as reported by Google Analytics. This traffic now represents approximately 70% of all website traffic. These "direct visits" have a bounce rate of 96%+ and only 1-2 pages/visit. This is skewing our analytics in a big way and rendering them pretty much useless. I suspect this is some sort of crawler activity but we have no access to the server log files to verify this or identify the culprit. The client's site is on a GoDaddy Managed WordPress hosting account.
The way I see it, there are a couple of possibilities.
1.) Our client's competitors are scraping the site on a regular basis to stay on top of site modifications, keyword emphasis, etc. It seems like whenever we make meaningful changes to the site, one of their competitors does a knock-off a few days later. Hmmm.2.) Our client's competitors have this crawler hitting the site thousands of times a day to raise bounce rates and decrease the average time on site, which could like have an negative impact on SEO. Correct me if I'm wrong but I don't believe Google is going to reward sites with 90% bounce rates, 1-2 pages/visit and an 18 second average time on site.
The bottom line is that we need to identify these bogus "direct visits" and find a way to block them. I've seen several WordPress plugins that claim to help with this but I certainly don't want to block valid crawlers, especially Google, from accessing the site.
If someone out there could please weigh in on this and help us resolve the issue, I'd really appreciate it. Heck, I'll even name my third-born after you.
Thanks for your help.
Eric
-
Hi SirMax,
Thanks for your input. I appreciate it. We'll add Wordfence to our WordPress toolbox and see if that addresses the issue.
In response to previous posts, thanks to everyone for your input. We were able to apply some filters to remove the bogus bot traffic from the analytics and normalize the data, however, this did not actually resolve the issue and in my eyes is more of a BandAid fix. The evil crawlers are still there, we just can't see them.
Thanks again for all of your input.
Eric
-
Hostname filtering does not work any more. Unfortunately most of the spammers have adapted and are using your website as hostname.
For the WordPress I use Wordfence plugin( using paid version - not affiliated with them in any shape or form beyond paying for their services). In the advance blocking you can set limits on how fast and how many pages crawlers can request. You can also block by country or ip range. It can also show you live traffic with a lot of details ( a lot more then google analytic - more like server log ). It might not be the complete remedy but it can help.
-
I wish I had an answer for how to stop the bots from hitting your site at all - I don't think a good one exists, as any solutions that wouldn't also block real human traffic to your site are going to be easy for spam bots to get around. I think your best bet is just to do everything you can to keep your data as clean as possible.
-
Hi Ruth,
Thanks a bunch for taking the time to respond to my post. Great advice. This is reassuring on a number of levels, however, it doesn't address the underlying issue of how to stop these spam bots in the first place.
We've already started the process of filtering out some of this bogus data. We'll also be integrating some WordPress plugins to see if that helps. That said, if the spam bots are hitting Analytics directly, as opposed to the actual website, WP plugins won't do anything.
Anyway, I appreciate your input and advice. Thanks so much.
Eric
-
Hi Eric,
A few things to reassure you off the bat:
- For what it's worth, there is a huge, HUGE amount of crawler spam happening in the web today. Every site I work on is being hit hard with false referrals and direct visits. I know Google Analytics is working on a solution to better filter these visits out. So I wouldn't be too concerned that it is something a competitor is doing to your site, specifically - it's more likely that it's been caught up in the general wave of spam crawlers.
- It's important to note that when we talk about Google looking at bounce rate and dwell time as part of ranking your site, those numbers are specifically from clicks through from search - that's data that Google can get without using your private web analytics data as a ranking factor, which they've said repeatedly that they don't and won't do. So a bunch of direct visits with high bounce rates will NOT affect your rankings.
So, it's not dangerous, just annoying. On to how to get that data out of your reports:
- Make sure you're not filtering out spam referrers at a View level - this can cause those visits to incorrectly appear as direct traffic.
- You could set up an Advanced Segment in Google Analytics to filter out direct visits with visit times of, say, under 5 seconds. Some real traffic may get caught in that, but it will get the noise levels down.
- The best way to filter out spam bot traffic, in my opinion, is to set up hostname filtering. Here's a post on Megalytic on how to do that: https://megalytic.com/blog/how-to-filter-out-fake-referrals-and-other-google-analytics-spam. Make sure you've also got an "Unfiltered Data" View so you'll still have historic raw data if you need it.
Hope that helps! Good luck.
-
Check webserver log files, or log visits (ip address, user agent, __utma, __utmz, possibly browser fingerprint, etc...)
Analyzing those you can easily find out if the traffic is from scraping bot or humans.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to measure Bounce Rate via Timeout
Hi, I want to change the way Google Analytics measures my bounce rate: after 20 seconds timeout instead of next page criteria. I know that I may do it adding some content like: setTimeout(“_gaq.push([‘_trackEvent’, ‘NoBounce’, ’20 seconds’])”,20000); But I wonder if there is some way to do it via Google Analytics configuration. Thanks in advance,
Reporting & Analytics | | Webicultors0 -
Can a 100% bounce rate page hurt whole website?
Hello, So with trying to figure out what I can do to better my website, I noticed a post on here that mentioned bounce rates. So I went to my Google analytics pages and listed my bounce rates. My average is 44% But I have a picture page that is 100% and a contact page that is 80% Can pages like this cause an algo penalty that could hurt a whole site? Thank you for your valuable insights
Reporting & Analytics | | Berner0 -
Can underscore blanks trigger home page bounce rates
Buongiorno from 8 degrees C Wetherby UK 🙂 On this site the scrolling banner has been hyperlinked with underscore blank causing new pages to open when a user clicks on a banner. My question is please... "If a user clicks on a banner will Google measure this as a home page bounce" ( I dont think it will i just want to be 100% sure) Click here for illustration:
Reporting & Analytics | | Nightwing
http://i216.photobucket.com/albums/cc53/zymurgy_bucket/home-page-sw-banner_zps0fda6318.jpg Grazie Tanto,
David0 -
Identifying conversion rate for product
Hi, I need to identify the conversion rate for a product, lets call it a spanner. I have 100s of spanner product urls and I ensured that the url protocols must include the product name e.g /red_spanner so its easy for me to work out the conversion rate in analytics for all my spanner pages as I just add 'spanner' to the landing page filter, hit the ecommerce tab and bingo. What I cant figure out is how to work out the conversion rate for all spanner sales which includes alot of sales which didnt originate on spanner pages e.g. home page > search result > checkout. Theres 1000s of variations of this e.g. email > home > search > product page > checkout. How can I work out the total conversion rate for spanners which needs to include: people landing on a spanner page and people who didnt arrive at a spanner page and did a search with eventually got them to the spanner pages. Hopefully its not as complicated as I think! Thanks in Advance!
Reporting & Analytics | | AndyMacLean0 -
Webmaster tools help
Looking at some Google Webmaster tools data and im a bit stumped. Can anyone tell me if the "not selected" pages should be taken from the "total indexed pages" or are they a seperate entity? The way the data is displayed on the graph leads you to believe they should be taken from the total indexed pages, yet Googles own description of "not selected" pages says that these pages are not indexed. Confused.
Reporting & Analytics | | Silkstream0 -
Finding and removing "Bad" Back Links
In the process of trying to figure out where all of the “Bad” backlinks are coming from I used the SEOmoz Site Explorer. I can see the links that may be questionable but am not sure how to determine if these are the issue causing the loss of rank or could it be something else. On Google webmasters they list Siteloki.com as the one with the most links. The count is now at 13,005. (see attached WMT report)
Reporting & Analytics | | rdominey
I first noticed this a month ago, 6,742 links and have tried contacting them with no reply, no results, I have even posted on the site asking to be removed from their listing and not response. Website: www.getyourphotosoncanvas.com I do not understand why this site is not showing up in the Site Explorer link analysis report (See attached)? Could this be some sort of hack or hidden links that Site Explorer does not see? How do I determine if this is real or not, if it is the reason that Google is demoting us? Google says that we are not being manually penalized? 5zAQq Iz9ct0 -
How to filter pages in Analytics by multiple criteria
Hello, we have several pages with the same page title. Now out of all those pages I want to pick two. Let's call them "/page1" and "/page2". For those pages I want the following information (combined for both): Avg. time on page, Bounce rate, Navigation Summary Normally I get all the information under "Content" "Pages" and by choosing the "page title" as primary dimension and clicking on the respective page title. Let's call it "page | title". Choosing the filter for 1 page works fine (I just enter "/page1" in simple filter). But how can I filter for two pages ( entering " include page ends with /page1 and include page ends with /page2" in the advance filter will show 0 results). Thanks in advance
Reporting & Analytics | | guitarslinger0 -
Google Analytics: Difference Between Goal Conversions & Goal Completions
When using Google Analytics, what is the difference between total goal conversions and total goal completions? We have many goals set up in a lead generation environment. Therefore, the only element of conversion is submitted a lead and arriving on the "Thank You" page. THose thank you pages are tagged accordingly. When we run reports though, the number of "Total Goal Conversions" and "Total Goal Completions" never match up.
Reporting & Analytics | | eMagineSEO0