Can you confirm legitimate Google Bot traffic?
-
We use Cloudflare as a firewall. I noticed a significant number of blocks of bot traffic. One of the things they do is try to block bad bot traffic. But it seems they are mistakenly blocking Google Bot traffic. If you use Cloudflare, you may want to look into this as well. Also, can you confirm if the following IPs are for legitimate Google Bots?
66.249.79.88
66.249.79.65
66.249.79.8066.249.79.76
Thanks,
-
Thank you very much for all of your feedback folks. I really appreciate it. Cloudflare has acknowledged that they are aware of the issue and working to correct it. It also affected Bing Bots. I ended up white listing the IP ranges, and our Google crawl has increased as a result.
-
Dammit, my mistake.
Was proabaly still asleep. I've checked with my log files and updated itThanks Alex!
-
10.0.0.0/8 is a private subnet and therefore unroutable, so I don't think you would have seen traffic from that subnet from Googlebot!
-
What Alex has pointed out is what you're needing.
Also, I'd like to remember you that Googlebot crawls from several IP ranges. Not only the 66.xxx.xxx.xx.
EDIT:
Alex pointed me out that I've made a mistake. So I've checked my data in server logs.
In some of the sites I'm working, most of the traffic comes from the range: [66.249.6x.xxx-66.249.7x.xxx]Always, when in doubt, with a reverse DNSlookup you'll find out whether it's an official GoogleBot or not.
Hope it helps.
Best luck.
Gaston -
Following the advice here: https://support.google.com/webmasters/answer/80553?hl=en I can confirm that they are indeed Googlebot.
If you use Fetch as Google in Search Console, are you seeing errors?
Cloudflare has an article here on the subject:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can someone evaluate this page so I can continue adding others?
Hi, I am adding a bunch of similar category stickers and I am not looking into that good SEO for these since there will be hundreds of them coming but I just want to include the relevant keywords that people perhaps use in the Google image search to take them to our site. They are all related to JDM (Japanese Domestic Motors) so I decided to include JDM at the end of all the SEO titles. I am writing totally different short descriptions for all of these stickers and the Related Products are changing as well. I just want to achieve something like Amazon or eBay listings do - not the perfect SEO since I cannot spend too much time with each sticker optimizing it but I don't want to NOINDEX, FOLLOW them either - hence the different related products for all items and also unique short descriptions. If you check one of the pages: http://www.redrockdecals.com/rising-sun-wakaba-leaf-sticker-red-black-jdm Do you think I should be in the safe side so I don't hurt my overall SEO? Thanks!!
Technical SEO | | speedbird12290 -
Google Penalty Investigation
Hi All, I've recently had a google penalty, and have spent a long while working through tying up any loose ends with my site (I'm building a new one, so you'll still see some probs on my current one). My search referrals do slowly seem to be recovering, but only for variations of terms that are similar to 'bike repair'. Now, my site does primarily offer bike repair advice, so that's a good thing, but I'm not yet ranking for any of my specific keywords. One example is that I used to rank quite high for the term 'schrader valve'. Is this a signal of anything in particular? Or can I not read anything specific into this? Thanks!
Technical SEO | | madegood0 -
Google Places, Google Plus, Oh my!
Ok - So I am in the position to try and clean up the current Google places nightmare for a company. Right now there is about 3 or 4 different google places listings for them that they have no control over. So here is what I did: 1. I took control of them all by verifying via phone and confirmed all of them. 2. I suspended all the listings but 1 3. I edited the one listing to be accurate and complete.
Technical SEO | | DylanPKI
Then I waited, and waited... A month later, the old listings are still up and none of the changes to the one listing have been made. Today it gets a bit more complicated. Today I created a Google+ page for the business which seems like it may end up adding yet ANOTHER Google Places listing, is that correct? They are sending a post card to verify, but I have the page all set up ready to go and plan on tying it to the website. I am not exactly sure what my specific question is, but I am looking for any advice anyone has on the best way to go about this situation. Thank you in advance!0 -
Google Custom Site Search
I am an admin on a google custom site search account. I am also the owner of a verified webmaster tools account for the same site. The Custom Search control panel will not let me add URL's or a Site map for on demand indexing, but says "you must submit a sitemap of your own verified sites". Has anyone else has this issue? Does the Owner of the custom search account have to be the owner of the webmaster account, or can the logged in admin be? Thanks
Technical SEO | | SEMPassion0 -
How a google bot sees your site
So I have stumbled across various websites like this: http://www.smart-it-consulting.com/internet/google/googlebot-spoofer/ The concept here is to be able to view your site as a googlebot sees it. However, the results are a little puzzling. Google is reading the text on my page but not the title tags according to the results. Are websites like this accurate OR does Google not read title tags and H1 tags anymore? Also on a slighly related note. I noticed the results show the navigation bar is being read first by google, is this bad and should the navigation bar be optimized for keywords as well? If it did, it would read a bit funny and the "humans" would be confused.
Technical SEO | | StreetwiseReports0 -
Google Places - need to update
I've got a client who verified their Google Places listing years ago, and noone knows who so I can't access it. The business is now moving and I need to update the address. What should I do? thanks
Technical SEO | | garymeld0 -
Blocking Google from Crawling Parameters
Hi guys: What is the best way to keep Google from crawling certain urls with parameters? I used the setting in Webmaster Tools, but that doesn't seem to be helping at all. Can I use robots.txt or some other method? Thanks! Some examples are: <colgroup><col width="797"></colgroup> www.mayer-johnson.com/category/assistive-technology?manufacturer=179 www.mayer-johnson.com/category/assistive-technology?manufacturer=226 www.mayer-johnson.com/category/assistive-technology?manufacturer=227 <colgroup><col width="797"></colgroup> www.mayer-johnson.com/category/english-language-learners?condition=212 www.mayer-johnson.com/category/english-language-learners?condition=213 www.mayer-johnson.com/category/english-language-learners?condition=214 <colgroup><col width="797"></colgroup>
Technical SEO | | DanaDV
| www.mayer-johnson.com/category/english-language-learners?roles=164 |
| www.mayer-johnson.com/category/english-language-learners?roles=165 |
| www.mayer-johnson.com/category/english-language-learners?roles=197 | | |0