Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Up-to-date list of search engine bot user agents

Technical SEO

7965

JoeAmadon last edited by

Does anyone know of an up-to-date-list of search engine bot user agents?

Thanks.
1 Reply Last reply
Reply Quote 0
westernspirit last edited by

We just found this, which may be helpful. It was exactly what I was looking for.

http://www.monperrus.net/martin/list+of+robot+user+agents
1 Reply Last reply
Reply Quote 0
JoeAmadon @KeriMorgret last edited by

Thanks, Keri. I found http://www.botsvsbrowsers.com/ to be very helpful. I'm looking for all the bots from the "major", U.S. search engines (Google, Bing, Ask, AOL, Blekko).
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

I haven't found any list that's newer than a year old in the brief search I did. I did find these comprehensive user agent lists at http://www.botsvsbrowsers.com/ and http://www.user-agents.org/.

Are you more interested in every variation of bot there is, or more after the top 20 bots?
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Google Search Console Not Sending Messages

One of our sites received a Manual Penalty for unnatural links by Google. However, we never received a message in Google Search Console or an email about the manual action. The only reason we knew about the penalty is by the obvious drop in rankings, then signing into search console to look for any manual actions, which we found. Since then, we have submitted a disavow file and a reconsideration request. However, once again we did not receive an email or message in search console that shows confirmation of the disavow or that they received the reconsideration request. The disavow file does show up after I upload it, and it says it was successfully uploaded... but no messages or emails. After many hours of investigating the various canonical versions of our website on Search Console, we found out that there were several “owners” of the various canonical versions of our site that had “could not find the email address” as a site owner. We found out that these were previous employees who no longer worked with the company and their email address was deleted. After unverifying these site owners, (all the ones that had “could not find the email address” as the site owner), the notifications, emails and messages in Search Console started to appear. However, the only place they did not appear, is the main canonical version of our site. Of course, the main canonical version of our site (https://www) is the version that we uploaded the disavow and reconsideration request. This is the canonical version of the site that we need to receive these messages to know if our reconsideration request was granted! We’ve just reuploaded the disavow file and reconsideration request to all of the other canonical versions (2 of the 3 received the message about the penalty)…. and we are currently awaiting a response. Has anybody else had problems with not receiving notifications in search console due to deleted email addresses?
Technical SEO | | Fiyyazp

0
Best directories for a new(ish) hospital to be listed on?

Just thought I'd throw this question out to the Moz community. Anyone have leads for good, credible directories where I could get a smaller, local hospital listed on? (besides the obvious ones like HealthGrades, etc.,) Thanks!
Technical SEO | | TaylorRHawkins

1
Google not indexing /showing my site in search results...

Hi there, I know there are answers all over the web to this type of question (and in Webmaster tools) however, I think I have a specific problem that I can't really find an answer to online. site is: www.lizlinkleter.com Firstly, the site has been live for over 2 weeks... I have done everything from adding analytics, to submitting a sitemap, to adding to webmaster tools, to fetching each individual page as googlebot and then submitting to index via webmaster tools. I've checked my robot files and code elsewhere on the site and the site is not blocking search engines (as far as I can see) There are no security issues in webmaster tools or MOZ. Google says it has indexed 31 pages in the 'Index Status' section, but on the site dashboard it says only 2 URLS are indexed. When I do a site:www.lizlinketer.com search the only results I get are pages that are excluded in the robots file: /xmlrpc.php & /admin-ajax.php. Now, here's where I think the issue stems from - I developed the site myself for my wife and I am new to doing this, so I developed it on the live URL (I now know this was silly) - I did block the content from search engines and have the site passworded, but I think Google must have crawled the site before I did this - the issue with this was that I had pulled in the Wordpress theme's dummy content to make the site easier to build - so lots of nasty dupe content. The site took me a couple of months to construct (working on it on and off) and I eventually pushed it live and submitted to Analytics and webmaster tools (obviously it was all original content at this stage)... But this is where I made another mistake - I submitted an old site map that had quite a few old dummy content URLs in there... I corrected this almost immediately, but it probably did not look good to Google... My guess is that Google is punishing me for having the dummy content on the site when it first went live - fair enough - I was stupid - but how can I get it to index the real site?! My question is, with no tech issues to clear up (I can't resubmit site through webmaster tools) how can I get Google to take notice of the site and have it show up in search results? Your help would be massively appreciated! Regards, Fraser
Technical SEO | | valdarama

0
How to recover search volume after domain name change?

On the 3rd of November we changed our company name and domain. The new site was not changed at all so the 301 process was quite straightforward. The change over was successful, no downtime, all pages redirected correctly (with a few minor exceptions). However, after a few days we started to see more and more links into the new site from the old site. They now stand at over 3 million. And links from the new site to the old site of over 200K. Links from the new site back to the old, were due to us having left a lot of links tucked away on various pages which were possibly causing loops with the 301 redirects on the old site. We fixed these and now there are no remaining links back to the old site, though we are still showing just over 200K links back to the old site. We are also seeing a LOT more back-links on the new site from old junk sites, which are not showing for the old site. A couple of years ago we went through about a year of trying to track down and remove thousands of spam backlinks. We did what we could, got a lot removed, showed Google the evidence, then Google lifted the penalty and said they had made some changes that meant the links were no longer causing the penalty. I added the old disavow file to the new site, but it doesn't cover a fraction of the sites which are being displayed as providing backlinks... many of which are clearly spammy. Is it possible that Google made some manual actions to lift the penalties but failed to associate these changes with the new domain? Changes that were not included in the disavow file? All help appreciated.
Technical SEO | | Exotissimo

0
Do other search engines use meta keywords

Just want to know even those Google states it doesn't use them is there any benefit from using them for the other search engines?
Technical SEO | | ReSEOlve

0
Schema showing up in product listing?

I have added schema to my product page as follows: Example product description ..... and so forth. However, the schema is showing up on the product page itself. How do I keep the schema on the SERP page, but not on the product page? My client doesn't want it to affect the site design. Thanks!
Technical SEO | | DWallace

0
Any chance for my listing to rank in top 7 Google maps?

Was wondering if it is possible to position a clients listing in one of the top 7 local listings on Google. All of the listings in the top 7 spots have pr2,3, and even 4.The clients site is new and pr0. his listing is: Dallas Foundation Repair 2770 Live Oak St, Dallas, TX ‎(214) 770-8975 ‎ · dallasfoundation-repair.comand we are trying to rank for keywords: dallas foundation repair,foundation repair in dallasalso please give me your advice as far as what approach I should be taking, thanks guys
Technical SEO | | jesse1341

0
Blog article URL - with or without date?

Quick question to all you folks: does including the date in a blog article's permalink affect rankings? For example, here's an article with the month and year, as well as the blog title: http://www.ayzanyc.com/blog/2012/12/difference-between-hot-chocolate-hot-cocoa/ Is it better to omit the date and just put the blog title? Also, if is better to avoid using the date, is it worth it to change the link structure of our previous articles (given that the URL will now be different), or should we just focus on future articles? Thanks ahead of time for your advice.
Technical SEO | | onurkiyak

0