Can you confirm legitimate Google Bot traffic?
-
We use Cloudflare as a firewall. I noticed a significant number of blocks of bot traffic. One of the things they do is try to block bad bot traffic. But it seems they are mistakenly blocking Google Bot traffic. If you use Cloudflare, you may want to look into this as well. Also, can you confirm if the following IPs are for legitimate Google Bots?
66.249.79.88
66.249.79.65
66.249.79.8066.249.79.76
Thanks,
-
Thank you very much for all of your feedback folks. I really appreciate it. Cloudflare has acknowledged that they are aware of the issue and working to correct it. It also affected Bing Bots. I ended up white listing the IP ranges, and our Google crawl has increased as a result.
-
Dammit, my mistake.
Was proabaly still asleep. I've checked with my log files and updated itThanks Alex!
-
10.0.0.0/8 is a private subnet and therefore unroutable, so I don't think you would have seen traffic from that subnet from Googlebot!
-
What Alex has pointed out is what you're needing.
Also, I'd like to remember you that Googlebot crawls from several IP ranges. Not only the 66.xxx.xxx.xx.
EDIT:
Alex pointed me out that I've made a mistake. So I've checked my data in server logs.
In some of the sites I'm working, most of the traffic comes from the range: [66.249.6x.xxx-66.249.7x.xxx]Always, when in doubt, with a reverse DNSlookup you'll find out whether it's an official GoogleBot or not.
Hope it helps.
Best luck.
Gaston -
Following the advice here: https://support.google.com/webmasters/answer/80553?hl=en I can confirm that they are indeed Googlebot.
If you use Fetch as Google in Search Console, are you seeing errors?
Cloudflare has an article here on the subject:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Not Indexing Pages (Wordpress)
Hello, recently I started noticing that google is not indexing our new pages or our new blog posts. We are simply getting a "Discovered - Currently Not Indexed" message on all new pages. When I click "Request Indexing" is takes a few days, but eventually it does get indexed and is on Google. This is very strange, as our website has been around since the late 90's and the quality of the new content is neither duplicate nor "low quality". We started noticing this happening around February. We also do not have many pages - maybe 500 maximum? I have looked at all the obvious answers (allowing for indexing, etc.), but just can't seem to pinpoint a reason why. Has anyone had this happen recently? It is getting very annoying having to manually go in and request indexing for every page and makes me think there may be some underlying issues with the website that should be fixed.
Technical SEO | | Hasanovic1 -
Deindexed homepage by Google
I just noticed that my homepage was de-indexed by google. Any thoughts would be appreciated.
Technical SEO | | Jenny_H0 -
Using Google Adwords is good?
I heard about that if you using adwords, google drops your ranking a little bit. Because of you already pay money for results. I think that is reasonable.
Technical SEO | | umutege0 -
Problems with google cache
Hi Can you please advise if the following website is corrupted in the eyes of Google, it has been written in umbraco and I have taken over it from another developer and I am confused to why it is behaving the way it is. cache:www.tangoholidaysolutions.com When I run this all I see is the header, the start of the main content and then the footer. If I view text view all the content is visible. The 2nd issue I have with this site is as follows: Main Page: http://www.tangoholidaysolutions.com/holiday-lettings-spain/ This page is made up of widgets i.e. locations, featured villas, content However the widgets are their own webpages in their own right http://www.tangoholidaysolutions.com/holiday-lettings-spain/location-picker/ My concern is that this part pages will affect the performance of the seo on the site. In an ideal world I would have the CMS setup so these widgets are not classed as pages, but I am working on this. Thanks Andy
Technical SEO | | iprosoftware0 -
Google not visiting my site
Hi my site www.in2town.co.uk which is a lifestyle magazine has gone under a major refit. I am still working on it but it should be ready by the end of this week or sooner but one problem i have is, google is not visiting the site. I took a huge gamble to redo the site, even though before the refit i was getting a few thousand visitors a day, i wanted to make the site better as i was getting google webmaster errors. But now it seems google is not visiting the site. for example i am using sh404sef and i have put friendly url in the site and on the home page it has its name and meta tag but when you look at google it is not giving the site a name. Also it has not visited the site since october 13th Can anyone advise how to encourage google to visit the site please.
Technical SEO | | ClaireH-1848860 -
Will Google Continue to Index the Page with NoIndex Tag Upon Google +1 Button Impression or Click?
The FAQs for Google +1 button suggests as follows: "+1 is a public action, so you should add the button only to public, crawlable pages on your site. Once you add the button, Google may crawl or recrawl the page, and store the page title and other content, in response to a +1 button impression or click." If my page has NoIndex tag, while at the same time inserted with Google +1 button on the page, will Google recognise the NoIndex Tag on the page (and will not index the page) despite the +1 button's impression or clicks send signals to Google spiders?
Technical SEO | | globalsources.com0 -
My site has vanished from google
Hi my site has vanished from google. We have been for a very long time. for example if you put in gastric band hypnotherapy then we would be first page number two and also lots of other keywords but now we have vanished from google and i do not know why or how to solve this. can anyone please help me and help me understand what i need to do to solve this please My site is http://www.clairehegarty.co.uk I am not sure if i have been banned or why i have dropped out of google
Technical SEO | | ClaireH-1848860 -
Are Google now indexing iFrames?
A client is pulling content through an iFrame, and when searching for a snippet of that exact content the page that is pulling the data is being indexed and not the iFrame page. Seen this before?
Technical SEO | | White.net0