Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Up-to-date list of search engine bot user agents

Technical SEO

7964

JoeAmadon last edited by

Does anyone know of an up-to-date-list of search engine bot user agents?

Thanks.
1 Reply Last reply
Reply Quote 0
westernspirit last edited by

We just found this, which may be helpful. It was exactly what I was looking for.

http://www.monperrus.net/martin/list+of+robot+user+agents
1 Reply Last reply
Reply Quote 0
JoeAmadon @KeriMorgret last edited by

Thanks, Keri. I found http://www.botsvsbrowsers.com/ to be very helpful. I'm looking for all the bots from the "major", U.S. search engines (Google, Bing, Ask, AOL, Blekko).
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

I haven't found any list that's newer than a year old in the brief search I did. I did find these comprehensive user agent lists at http://www.botsvsbrowsers.com/ and http://www.user-agents.org/.

Are you more interested in every variation of bot there is, or more after the top 20 bots?
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Nofollow/Noindex Category Listing Pages with Filters

Our e-commerce site currently has thousands of duplicate pages indexed because category listing pages with all the different filters selected are indexed. So, for example, you would see indexed: example.com/boots example.com/boots/black example.com/boots/black-size-small etc. There is a logic in place that when more than one filter is selected all the links on the page are nofollowed, but Googlebot is still getting to them, and the variations are being indexed. At this point I'd like to add 'noindex' or canonical tags to the filtered versions of the category pages, but many of these filtered pages are driving traffic. Any suggestions? Thanks!
Technical SEO | | fayfr

0
How to handle Friendly URLs together with internal filters search?

I've been trying to handle URLs from a unfrendly folder format to a semantic one, the thing is by doing so I end up with a longer URL and therefore a longer Title. Right now the format of my classified site for job seeking looks like this (folders): site.com/search/category/sales/level/supervisor/location/seattle/company/acme/published/2-days-ago/type/full-time/q/salesman/page/2 format: Filter/Content where at the end q is the query people are writting My suggestion is the following: Mixing Jobs with location, mixing category and level, and puting the rest of the filters at the end adding "--" between them. And adding 2 parameters, query (q) and pagination (pag) site.com/jobs-at-seattle/sales-supervisor/company/acme/full-time--published-2-days-ago?q=salesman&pag=2 Any thoughts on how to handle URLs over 100 chararcters and titles that go over 65?, or maybe is ok to have "friendly" long URLs and long titles when it comes to classified ad sites since they are based on internal filters to help people find what they are looking for. Sidenote: Is itok to have 2 parameters in the URL (Query and Pagination) Thanks a lot.
Technical SEO | | JoaoCJ

0
301 redirect and search engines

How long until 301 redirects get recognized by search engines? I noticed my link on Google isn't forwarding over to my new domain even after the 301 redirect. If I go to the site directly, the 301 redirect works. Anyone know how long it takes for search engines to pick it up? Thanks!
Technical SEO | | timeintopixels

0
Should I index my search result pages?

I have a job site and I am planning to introduce a search feature. The question I have is, is it a good idea to index search results even if the query parameters are not there? Example: A user searches for "marketing jobs in New York that pay more than 50000$". A random page will be generated like example.com/job-result/marketing-jobs-in-new-york-that-pay-more-than-50000/ For any search that gets executed, the same procedure would be followed. This would result in a large number of search result pages automatically set up for long tail keywords. Do you think this is a good idea? Or is it a bad idea based on all the recent Google algorithm updates?
Technical SEO | | jombay

0
Need advice on search listings and link building

Search results on my keyword (engraved wedding glasses) produces several pages of linked domains. (My domain is giftthings.net) Some are good. And admittedly, some are not so good. My question then is simply, why does seomoz link analysis show such a small number of links? And the second part of my question is, "Is there some sort of "magic number", some sort of thresh hold that triggers Google's interest? With a link list that is small but growing, am I missing something in my concern that I'm not moving up in the search listings? I've written a few articles, continuing my work on link building but I remain buried in the search results.
Technical SEO | | AhmadS

1
Getting a link removed from brand search - please help!

Hello all you mozzers! Ive just come into work with an established company who have one major problem when you google "palicomp" the second link that comes up is to consumeractiongroup with a thread that has been damaging the business for over 2 years, this thread is absolutely not representative of the business today. Strangely stronger links in search have better authority but google has ranked this post as being highly relevant to the business, does anybody know of any strategies we can do to get this removed, we have contacted consumeractiongroup directly but they are not prepared to move it. Does anyone have any idea of removal ideas or what we can do its crippling our business, we cant work out as to why its ranking better! Chris
Technical SEO | | palicomp

0
Bots gone from site for 1-2 hours

Hi everyone, From the beginning of March bots and crawlers begin to disappear from site completely for 1-2 hours and then come back again. It never happened earlier, they constantly crawled our site without stopping. We didn't make any changes on the site. After that pages and images start to disappear from G index. Does anyone have the same trouble?
Technical SEO | | bubliki

0
Yahoo/Bing Search

Are there current articles that describe the differences in Yahoo/Bing vs Google SERPs? We have solid top 10 rankings in Google for a bunch of competitive keywords but aren't ranking well in Yahoo.
Technical SEO | | blueroom

0