Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
-
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem.
To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board.
Context that might be helpful:
- I did check technical settings, and the sites are crawlable.
- The sites do appear in search but seem to be losing organic search traffic.
Thanks for any help you can provide!
-
Hey David,
I thought I'd jump in here, as it's our tool
We have more information on bot verification here, including a troubleshooting section with common issues for genuine events being marked as spoofed -
https://www.screamingfrog.co.uk/log-file-analyser/user-guide/configuration/#verify-bots
You can also reach us via our support here - https://www.screamingfrog.co.uk/log-file-analyser/support/
Cheers.
Dan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Bot Crawling issues
Dear all, Is this cache:www.subhavaastu.com is not working now with Google. Why it is not showing when the site is crawled. What were the new algorithms which Google is adopted? in my searches of all of my site internal links of www.SubhaVaastu.com, I observed only 404 instead of Google visiting time and date. After observing this 404 of each of my site links, I understand Google stopped crawling my site. Some more examples are shown below with other websites: cache:www.vastuwebsite.com ("NOT" showing when Google visited this site) cache:www.vastuconsultantusa.com ("NOT" showing when Google visited this site) cache:www.shubhavaastu.com ("NOT" showing when Google visited this site) cache:www.subhavastu.com (Showing when Google visited this site) To my surprise, I noticed that Google crawled latest links in my site, which I added a new link (https://www.subhavaastu.com/remove-negativity.html) just 10 days back, this new link was clearly crawled by Google. I typed "remove negativity subhavaastu", I saw the results with this new page in the SERP, but on the same way when I typed "cache:www.subhavaastu.com/remove-negativity.html", it is showing again 404. what is happening with Google, is Google is following any new algorithms now. Is google changed any new concept? or, is my site is penalized in any case, I think it may not be, because if my site is penalized, then Google should not visit the new links and should not show the results with my site. Coming with my site, its pure from viruses, no malicious codes. Indeed it's an article based site, which has a good reputation. This domain is taken in the year 2003. We never spam anywhere we never did any wrong methods. If my site is penalized, then is it manual penalized or bot penalized. I thoroughly checked webmaster tools and google console, I never found any notices or any note from Google. Require experts analyzation on this doubt. Thanks in Advance.
Algorithm Updates | | SubhaVaastu0 -
Category Containing a Product searched shows up higher in google then the product page itself?
Hello Moz Wizards, We have recently launched a new eCommerce website www.memoky.com and think we did a pretty good job with the markup structure for feeding the hungry google bot all information available about a the products. However google doesn't like us very much : ( It seems every time you google a product that we carry; the category pages that contain that product will show up, but the product page itself does not. Below are two examples, however this seems to be site-wide which makes me feel like there is an underlying issue that we are missing. Examples
Algorithm Updates | | Memoky
when searched for "Eduardo floor lamp - matt black/matt yellow shade"
Shows ups - http://www.memoky.com/lighting/floor-lamps.html
Does not - http://www.memoky.com/eduardo-floor-lamp-matt-black-matt-yellow-shade.html when searched for "Derrick arm chair - white leather/ walnut"
Shows ups - http://www.memoky.com/living/lounge-chairs.html_
Does not - http://www.memoky.com/derrick-arm-chair-white-leather.html_ that is the pattern for almost all the products on this site. Any thoughts on why this could be the case?0 -
Studies showing that social sharing does/doesn't affect rankings?
I'm currently researching this area in order to show to a client that social shares aren't as valuable for SEO as they might think. Can anyone point me in the direction of the best studies done on this topic? Thanks in advance!
Algorithm Updates | | QubaSEO0 -
Time taken for Google Algorithm updates to show affect in Middle East?
Hello everyone, Just a quick question. Can anyone give me a safe estimate of how much time it could take for a Google Algorithm Update to show its effect in the Middle East after roll out? Maybe you guys can direct me to a post to read through and learn more about it myself. Your input will be highly appreciated. Regards, Talha
Algorithm Updates | | MTalhaImtiaz0 -
Does a KML file have to be indexed by Google?
I'm currently using the Yoast Local SEO plugin for WordPress to generate my KML file which is linked to from the GeoSitemap. Check it out http://www.holycitycatering.com/sitemap_index.xml. A competitor of mine just told me that this isn't correct and that the link to the KML should be a downloadable file that's indexed in Google. This is the opposite of what Yoast is saying... "He's wrong. 🙂 And the KML isn't a file, it's being rendered. You wouldn't want it to be indexed anyway, you just want Google to find the information in there. What is the best way to create a KML? Should it be indexed?
Algorithm Updates | | projectassistant1 -
Author Photo No Longer Showing Up in Search
My author photo has been showing up in search for quite some time but starting today I noticed that it no longer does even though the structured data testing tool still has it showing up. It only seems to show up on my google+ profile. Anyone notice the same thing?
Algorithm Updates | | casper4340 -
How long does it take for a new website to start showing in the SERP'S
I launched my website about 6 weeks ago. It was indexed fairly quickly. But it is not showing up in the Google SERP. I did do the on page SEO and followed the best practise's for my website. I have also been checking webmaster tools and it tells me that there is no errors with my site. I also ran it through the seomoz on page seo analyzer and again no real big issues. According to seomoz I had 1 duplicate content issue with my blog posts, which i corrected. I understand it takes some time, but any ideas of how much time? And f.y.i it's a Canadian website. So it should be a lot easier to rank as well. Could my site be caught in the Google 'sandbox effect' ? Any thoughts on this would be greatly appreciated.
Algorithm Updates | | CurtCarroll0 -
Wrong result showing on seomoz
I have created 1 campaign for my site, www.giftbig.com The campaign ID is - 206474 I added 7 keywords on my campaign out of 7 there is a one keyword "gift vouchers" as per pro.seomoz the google rank is 4. buy when i search on google its not coming on 4th rank. can anybody tell me why its showing wrong result on my campaign.
Algorithm Updates | | Joydeep_das0