Bingbot appears to be crawling a large site extremely frequently?
-
Hi All! What constitutes a normal crawl rate for daily bingbot server requests for large sites? Are any of you noticing spikes in Bingbot crawl activity?
I did find a "mildly" useful thread at Black Hat World containing this quote: "The reason BingBot seems to be terrorizing your site is because of your site's architecture; it has to be misaligned. If you are like most people, you paid no attention to setting up your website to avoid this glitch. In the article referenced by Oxonbeef, the author's issue was that he was engaging in dynamic linking, which pretty much put the BingBot in a constant loop.
You may have the same type or similar issue particularly if you set up a WP blog without setting the parameters for noindex from the get go."
However, my gut instinct says this isn't it and that it's more likely that someone or something is spoofing bingbot.
I'd love to hear what you guys think!
Dana
-
Thanks Lesley. Yes, I agree. I think the only way we are going to get a definitive answer is to look at the logs. We are working on getting access.
-
I have recently had Bingbot crawl a site until it almost locked the database up, so it is possible. If you have doubts whether it is Bing bot or not, take to the logs and start extracting the ip addresses. You can verify them here, http://www.bing.com/webmaster/help/how-to-verify-bingbot-3905dc26
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will putting a one page site up for all other countries stop Googlebot from crawling my UK website?
I have a client that only wants UK users to be able to purchase from the UK site. Currently, there are customers from the US and other countries purchasing from the UK site. They want to have a single webpage that is displayed to users trying to access the UK site that are outside the UK. This is fine but what impact would this have on Google bots trying to crawl the UK website? I have scoured the web for an answer but can't find one. Any help will be greatly appreciated. Thanks 🙂
Technical SEO | | lbagley0 -
Strange Crawl Report
Hey Moz Squad, So I have kind of strange case. My website locksmithplusinc.com has been around for a couple years. I have had all sorts of pages and blogs that have maybe ranked for a certain location a longtime ago and got deleted so I could speed up the site and consolidate my efforts. I said that because I think that might be part of the problem. When I was crawl reporting my site just three weeks ago on moz I had over 23 crawl report issues. Duplicate pages, missing meta tags the regular stuff. But now all of a sudden when I crawl report on MOZ it comes up with Zero issues. So I did another crawl On google analytic and this is what came up. SO im very confused because none of these url's are even url's on my site. So maybe people are searching for this stuff and clicking on broken links that are still indexed and getting this 404 error? What do you guys think? Thank you guys so much for taking a shot at this one. siS44ug
Technical SEO | | Meier0 -
Why my site is not ranking for any of the keywords?
We have a site for Property management software, we have done everything like set proper Title and descriptions, heading tags, robots tag is also ok, set schema and its ok with Google webmaster too also we are doing Social media promotion. can you please check our website and tell me what is the problem??
Technical SEO | | rootwaysinc0 -
No google traffic for this site? Help?
Hi We have not done this web site http://climateacs.co.uk but have now picked it up and its getting no traffic what so ever from google do you think its been blacklisted? I have added it to my webmaster tools and there are no manual actions on it and most of the backlinks on google webmaster tools are from yell.com. However when I run it on opensiteexplorer I am seeing some chinese type links?? It is not really showing many search queries at all when you view them in webmaster tools under United Kingdom. I was going to start citation building for the address to help support the google places entry but just wanted to see what other peoples opinion was really on this site? Thanks Tracy
Technical SEO | | dashesndots0 -
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Please advise.
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Are there any other precautions I should be taking? Please advise.
Technical SEO | | BVREID0 -
I have 3500 pages crawled by Google, - why is SEOMOZ only able to crawl 400 of these ?
I added my site almost two weeks ago to the PRO DashBoard, and so far only 404 pages has been crawled, - but I know for a fact that there is 3500 pages that should be crawled. Other search engines has no problem in crawling and indexing these pages, so what can be wrong here ?
Technical SEO | | haybob270 -
Is any know if seomoz update for site crawl.
i belive my site www.breeze-air.com hit by penguin; i found that i had un-natural anchors text and able to remove around 1200 from the 1900 seomoz found. seomoz still shows those anchors - but when i check the link its not there. i removed them 3-4 weeks ago any idea?
Technical SEO | | eoberlender0 -
Adding Google +1 To My Site
This may seem like a silly question, but I had my site designed by a third party and I don't know how to find this info. I want to add Google +1 button to my site. I've already added the script that makes it work in my , but don't know how to add the to the part of my site I'd like the icon to appear. If you take a look at my site, http://www.youdrivethesuccess.com/, you will see the twitter and facebook icons that follow you around the site in the top right of the page. I'd like to drop in the +1 right next to them, but I don't know how to get to it. How do I access this on the server? What section of my FTP do these lil icons live? A folder? Any direction is greatly appreciated...thanks.
Technical SEO | | brentmitchell0