Up-to-date list of search engine bot user agents
-
Does anyone know of an up-to-date-list of search engine bot user agents?
Thanks.
-
We just found this, which may be helpful. It was exactly what I was looking for.
-
Thanks, Keri. I found http://www.botsvsbrowsers.com/ to be very helpful. I'm looking for all the bots from the "major", U.S. search engines (Google, Bing, Ask, AOL, Blekko).
-
I haven't found any list that's newer than a year old in the brief search I did. I did find these comprehensive user agent lists at http://www.botsvsbrowsers.com/ and http://www.user-agents.org/.
Are you more interested in every variation of bot there is, or more after the top 20 bots?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SERPs started showing the incorrect date next to my pages
Hi Moz friends, I've noticed since Tuesday, November 9, half of my post's meta dates have changed in regards to what appears next to the post in the search results. Although published this year, I'm getting some saying a random date in 2010! (The domain was born in 2013; which makes this even more odd). This is harming the CTR of my posts and traffic is decreasing. Some posts have gone from 200 hits a day to merely 30. As far as on our end of the website, we have not made any changes in regards to schema markup, rich snippets, etc. We have not edited any post dates. We have actually not added new content since about a week ago, and these incorrect dates have just started to appear on Tuesday. Only changes have been updating certain plugins in terms of maintenance. This is occurring on four of our websites now, so it is not just specific to one. All websites use Wordpress and Genesis theme. It looks like only half of the posts are showing weird dates we've never seen before (far off from the original published date as well as last updated date -- again, dates like 2010, 2011, and 2012 when none of our websites were even created until 2013). We cannot think of a correlation as to why certain posts are showing weird dates and others the correct. The only change we can think of that's related is back in June we changed our posts to show Last Updated date to give our readers an insight into when we changed it last (since it's evergreen content). Google started to use that date for the SERPs which was great, it actually increased traffic. I'm hoping it's a glitch and a recrawl soon may help sift it around. Anybody have experience with this? I've noticed Google fluctuates between showing our last updated date or not even showing a date at all sometimes at random. We're super confused here. Thank you in advance!
Technical SEO | | smmour2 -
Duplicate content on Places to Stay listings pages
Hello, I've just crawled our website https://www.i-escape.com/ to find we have a duplicate content issue. Every places to stay listing page has identical content (over 1,500 places) due to the fact it's based on user searches or selections. If we hide this pages using canonical tags, will we lose our visibility for each country and/or region we promote hotels? Any help on this would be hugely appreciated! Thanks so much Clair
Technical SEO | | iescape0 -
International Targeting - Google Search Console not recognizing the tags
Hi, We are facing a problem with international targeting not being recognized by the Google Search Console. This is the URL to which we added the following tags: URL: http://kilgray.com/memoq/2015-100/help-en/index.html TAGS: Flang tool Result: http://screencast.com/t/rrBgcr1X Search Console result: http://screencast.com/t/fP45ZR2c I am a bit lost here, as the tags were validated also from different members of the community. Is this because of the frames? (Yes, the site is built in frames). Thanks for your help!
Technical SEO | | Kilgray0 -
Search Operator Link: How accurate is the data?
How often do you guys use the search operator "link:"? How accurate is the data? Why the numbers of links when we use the parameter is way lower than the number on Google webmaster Tool or open site explorer? It only show the most powerful links?
Technical SEO | | Felip30 -
Can I promote a business page from Kuduz or City Search?
Can I promote a business page from Kuduz or City Search? Will other website link to these pages?
Technical SEO | | KristopherWho0 -
Google not showing profile photo in search
I seen on google snippet review and it was showing profile photo in search but when i see directly on google.com it is not showing there.. what's the problem ?
Technical SEO | | xplodeguru0 -
About Bot's IP
Hi, one of my customers had probably block the IP of SEOMOZ's bot. Could you give me : IP User-agent's name thks for helping me 😉
Technical SEO | | dawa1
