Up-to-date list of search engine bot user agents
-
Does anyone know of an up-to-date-list of search engine bot user agents?
Thanks.
-
We just found this, which may be helpful. It was exactly what I was looking for.
-
Thanks, Keri. I found http://www.botsvsbrowsers.com/ to be very helpful. I'm looking for all the bots from the "major", U.S. search engines (Google, Bing, Ask, AOL, Blekko).
-
I haven't found any list that's newer than a year old in the brief search I did. I did find these comprehensive user agent lists at http://www.botsvsbrowsers.com/ and http://www.user-agents.org/.
Are you more interested in every variation of bot there is, or more after the top 20 bots?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Landing Pages showing up in search results
Hey Guys, I recently noticed that our Christmas Gifts landing page was ranking twice in the Google serps for the query "Christmas Gifts." One of these pages is an old url that has already been 301 redirected to the new url page which is also showing up in the search results. In the results, the following shows up in position 2 & 3 for the Christmas Gifts query: <cite class="_Rm">www.uncommongoods.com/gifts/christmas/christmas-gifts</cite> <cite class="_Rm">www.uncommongoods.com/occasions/christmas-gifts/christmas-gifts</cite>The url with "occasions" in it has already been 301 redirected to the url above it. Not sure why this is still showing up. I know it takes Google some time to index 301s and sometimes they show old urls, but it's been a few months since the old "occasions" url was redirected.The title tags for these pages are different but they are actually the same page. The new "gifts" version of the url was made live in the Navigation of our site just last week and before that it was hidden from our Navigation. Would this be the reason it's now showing up in search?Any ideas on why this might be happening? ThanksExplanations?
Technical SEO | | znotes0 -
Blog posts outranked for Title a String searches in content...why?
Site Pages: When I wrap a page title, or a string of several words in quotes, and GG search, my client's page shows up first. My understanding is that this shows general health of site, and acknowledgement as the original source of the content. Blog Posts: When I wrap a blog page, or post, title or string of words in quotes, and GG search, Feedblitz, Facebook, and other scraper sites appear before the blog home page, and also the actual blog post. The blog is in a separate directory. Does this suggest that the /blog/ is being penalized or demoted in any way? Does it indicate the /blog/ directory does not have authority? Both the static site pages, and the blog pages, are using rel=canonical tags. What causes this, what does it indicate, and how can I fix it? Thanks,
Technical SEO | | seagreen
Greg0 -
Does server (host) location effect local search results?
Hey I was wondering if the location of your server (host) effects your local search engine results?Suppose I have an e-commerce website in the Netherlands and I want to host my website in the USA or UK, does this effect my search engine results in the Netherlands?
Technical SEO | | kevba0 -
Is Google caching date same as crawling/indexing date?
If a site is cached on say 9 oct 2012 doesn't that also mean that Google crawled it on same date ? And indexed it on same date?
Technical SEO | | Personnel_Concept0 -
Website disappeared from Google organic keyword searches.
We have an auto repair company as a client www.autorepairauroratilden.com who for the better part of a year their website had ruled the 1st page organic Google search results. Their website, Blogs, Facebook, and Twitter all came up on page one for their keyword searches. On May 13th, it all came to a screeching halt. The website is nowhere to be found for any of their keywords (example: brake repair Aurora.) There are a couple of blogs on page 2 but it’s nothing like it was prior to May 13th. On May 12th we published 5 branded websites for this client – Chrysler, Ford, Honda, Jeep, and Toyota, all on separate URL’s. All the page titles, keywords, and descriptions were specifically branded to the individual websites as were all the keywords. Since the beginning of June we’ve taken down the 5 branded websites and we’ve gone through our keywords on the auto repair website. The website was last crawled on June 11th. We still do not have any page 1 placement or for that matter any page placement. I checked 10 pages out. We have a 2nd auto repair client that has been running their website as well as their 5 branded websites a couple of months longer than this client and we’ve had no problems with any of their websites and keyword search results. How do we fix this?
Technical SEO | | markindenver0 -
My domain does not come in the search results, what do I do?
Hi, I have a website called www.bollykings.com It had a pretty solid rank on google for a number of keywords but 4-5 months back, it was badly affected by the Panda update. Now it comes nowhere. I have started updating and posting new articles on it since the last two months. When I search for "bollykings" on Google.com, website does not come only in the first 40 results. What could this mean?
Technical SEO | | modifyed0 -
CSS for SEO - can search engine see the text in the body?
We use CSS to arranging (absolute positioning) our content to makes it easier to crawl. I am using your On-Page Keyword Optimization tool and other tools to check our pages (i.e. http://www.psprint.com/gallery/invitation-cards), to make sure it works. For the “On-Page Keyword Optimization” tool, it gives a petty good grade (I guest it sees the text in the body). However, when I am using other tool to test the page (e.g. http://tools.seobook.com/general/spider-test/) it could not see the text in the body. Did we do something wrong? Thanks Tom
Technical SEO | | tomchu0
