Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
-
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem.
To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board.
Context that might be helpful:
- I did check technical settings, and the sites are crawlable.
- The sites do appear in search but seem to be losing organic search traffic.
Thanks for any help you can provide!
-
Hey David,
I thought I'd jump in here, as it's our tool
We have more information on bot verification here, including a troubleshooting section with common issues for genuine events being marked as spoofed -
https://www.screamingfrog.co.uk/log-file-analyser/user-guide/configuration/#verify-bots
You can also reach us via our support here - https://www.screamingfrog.co.uk/log-file-analyser/support/
Cheers.
Dan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site:www Issue - Homepage of the website is not showing in Google
Hello everyone, When I have manually search site:www.blinds4uk.co.uk in google.co.uk to know about webpages status, home page of the website is not showing in google search engine result pages. Please let me know, what is the reason behind this? because website crawling and indexing is good. Many Thanks.
Algorithm Updates | | Kuldeep-Sharma0 -
Why is the AMP tool saying I have invalid structured data when the structured data tool shows no errors?
Why is the AMP tool saying there's missing or invalid structured data on http://www.tasteofhome.com/recipes/flavorful-chicken-fajitas/amp when the structured data tool shows no errors? In addition, I'm not able to see a preview of the rich card in the AMP tool like I can for other recipes like https://allrecipes.com/recipe/19621/eggs-on-the-grill/amp/. If you check https://allrecipes.com/recipe/19621/eggs-on-the-grill/amp in the AMP tool, we get this message: "Page has valid structured data. This page is eligible for extended AMP features." Google has instructions on how to get rich cards for recipes (https://developers.google.com/search/docs/data-types/recipes), but i'm not sure if we're violating anything other than image aspect ratio. Thanks!
Algorithm Updates | | dianedragan0 -
The risk of semi-hidden text, which only shows-up when page viewer clicks button.
Hello Mozzers! I'm working on a holiday accommodation website and there's an accessibility statement at the bottom of each of the (50 odd) accommodation types on offer. This only comes up on the page (the text extends on the same page as the accommodation type) when you click the button (although it's there in the HTML at all times!). My other concern is might this "hidden until button pressed" semi-hidden text be seen as potentially manipulative by Googlebot, although it isn't!
Algorithm Updates | | McTaggart0 -
Two months - No Articles or Post Published in our blog. Moz shows less organic traffic.
Two months - No Articles or Post Published in our blog. Moz shows less organic traffic. i know i could not write - i was sick. organic search and keyword also. total pageviews dropped. DA increased by +3 and then -1 in last update. What should i do.
Algorithm Updates | | Esaky0 -
Struggling with Google Bot Blocks - Please help!
I own a site called www.wheretobuybeauty.com.au After months and months we still have a serious issue with all pages having blocked URLs according to Google Webmaster Tools. The 404 errors are returning a 200 header code according to the email below. Do you agree that the 404.php code should be changed? Can you do that please ? The current state: Google webmaster tools Index Status shows: 26,000 pages indexed 44,000 pages blocked by robots. In late March, we implemented a change recommended by an SEO expert and he provided a new robots.txt file, advised that we should amend sitemap.xml and other changes. We implemented those changes and then setup a re-index of the site by google. The no of blocked URLs eventually reduced in May and June to 1,000 for a few days â but now the problem has rapidly returned. The no of pages that are displayed in a google search request of www.google.com.au where the query was âsite:wheretobuybeauty.com.auâ is 37,000: This new site has been re-crawled over last 4 weeks. About the site This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster tools robots.txt file has been modified several times: Firstly we had none Then we created one but were advised that it needed to have this current content: User-agent: * Disallow: Sitemap: http://www.wheretobuybeauty.com.au/sitemap.xml
Algorithm Updates | | socialgrowth0 -
Does a KML file have to be indexed by Google?
I'm currently using the Yoast Local SEO plugin for WordPress to generate my KML file which is linked to from the GeoSitemap. Â Check it out http://www.holycitycatering.com/sitemap_index.xml. A competitor of mine just told me that this isn't correct and that the link to the KML should be a downloadable file that's indexed in Google. Â This is the opposite of what Yoast is saying... Â "He's wrong. đ Â And the KML isn't a file, it's being rendered. You wouldn't want it to be indexed anyway, you just want Google to find the information in there. What is the best way to create a KML? Â Should it be indexed?
Algorithm Updates | | projectassistant1 -
What do you think Google analyzes for SERP ranking?
I've been doing some research trying to figure out how the Google algorithm works. The one thing that is constant is that nothing is constant. This makes me believe that Google takes a variable that all sites have and divides it by that number. One example would be taking the load time in MS and dividing it by the total number or points the website scored. This would give all of the websites a random appearance since there that variable would throw off all the other constants. I'm going to continue doing research but I was wondering what you guys think matters in the Google Algorithm. -Shane
Algorithm Updates | | Seoperior0 -
Anyone have stats on numbers of Google users searching while logged in?
In light of Google's recent "social search update", I am curious to know how many Google users perform searches while logged into their Google account thereby showing "social results".
Algorithm Updates | | Gyi0