Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
-
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem.
To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board.
Context that might be helpful:
- I did check technical settings, and the sites are crawlable.
- The sites do appear in search but seem to be losing organic search traffic.
Thanks for any help you can provide!
-
Hey David,
I thought I'd jump in here, as it's our tool
We have more information on bot verification here, including a troubleshooting section with common issues for genuine events being marked as spoofed -
https://www.screamingfrog.co.uk/log-file-analyser/user-guide/configuration/#verify-bots
You can also reach us via our support here - https://www.screamingfrog.co.uk/log-file-analyser/support/
Cheers.
Dan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My site is showing indexed in search console but not appearing in Serps
hi, i have recently made sites.google site and submitted to search console but when I copy paste in google , its not appearing
Algorithm Updates | | alan-shultis0 -
Google not showing the recent cache info: How to know the last cached version of a page?
Hi, We couldn't able to see the last Google cached version of our homepage after March 29th. Just wondering why this is happening with other websites too. When we make some changes to the website, we will wait to our website indexed and cached, so the changes will have some ranking impact. Now we couldn't able to check if the website got indexed with changes. Is there any other way to check the latest cached version or time of last index? Thanks
Algorithm Updates | | vtmoz0 -
What happens when most of the website visitors end up at an "noindex" log-in page?
Hi all, As most of the users are visiting our website for log-in, we are planning to deindex login page. As they cannpt find it on SERP, they gonna visit our website and login; I just wonder what happens when most of the visitors just end up at homepage by browsing into an "noindex" page. Obviously it increases bounce rate and exit rate as they just gonna disappear. Is this going to push down us in rankings? What are the other concerns to check about? Thanks
Algorithm Updates | | vtmoz0 -
Google SERPs showing blog comments in Answer Box?
I was recently researching Schema markup for local businesses and I was presented with an Answer Box that used blog comments as answers (at least I feel that's what they were attempting to show). This is what is says currently when I search for "schema markup hours" (screenshot also attached): 12 thoughts on “How to Use Schema Markup for Local SEO” Lauren says: March 11, 2013 at 2:22 pm. ... souleye says: March 11, 2013 at 3:29 pm. ... Daniel Bennett says: March 11, 2013 at 8:51 pm. ... sammy. says: ... Nathan says: March 11, 2013 at 11:53 pm. ... Rishav says: March 12, 2013 at 5:51 am. ... Paul Sherland says: ... keyword removed says: Right now it shows the time and date of the comment, but is this something that's new or has it been around? Thanks in advance! tp5y1od.png
Algorithm Updates | | TomBinga11250 -
Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches. The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines. Is this a problem? How did we get so many pages reported? What can we do to bring the number of searched pages back to a "normal" level? BT
Algorithm Updates | | achituv0 -
How to you get a company details to appear on the right side of google searches? For example is you type dr martens in google , the search page shows links and then a snippet of the company on the right. Thank you.
For example is you type dr martens in google , the search page shows links and then a snippet of the company on the right. Thank you. https://www.google.com/search?q=dr+martens&oq=dr+martens&aqs=chrome..69i57j0l5.2176j0j7&sourceid=chrome&espv=210&es_sm=122&ie=UTF-8
Algorithm Updates | | vijayvasu0 -
The risk of semi-hidden text, which only shows-up when page viewer clicks button.
Hello Mozzers! I'm working on a holiday accommodation website and there's an accessibility statement at the bottom of each of the (50 odd) accommodation types on offer. This only comes up on the page (the text extends on the same page as the accommodation type) when you click the button (although it's there in the HTML at all times!). My other concern is might this "hidden until button pressed" semi-hidden text be seen as potentially manipulative by Googlebot, although it isn't!
Algorithm Updates | | McTaggart0 -
What do you think Google analyzes for SERP ranking?
I've been doing some research trying to figure out how the Google algorithm works. The one thing that is constant is that nothing is constant. This makes me believe that Google takes a variable that all sites have and divides it by that number. One example would be taking the load time in MS and dividing it by the total number or points the website scored. This would give all of the websites a random appearance since there that variable would throw off all the other constants. I'm going to continue doing research but I was wondering what you guys think matters in the Google Algorithm. -Shane
Algorithm Updates | | Seoperior0