Up-to-date list of search engine bot user agents
-
Does anyone know of an up-to-date-list of search engine bot user agents?
Thanks.
-
We just found this, which may be helpful. It was exactly what I was looking for.
-
Thanks, Keri. I found http://www.botsvsbrowsers.com/ to be very helpful. I'm looking for all the bots from the "major", U.S. search engines (Google, Bing, Ask, AOL, Blekko).
-
I haven't found any list that's newer than a year old in the brief search I did. I did find these comprehensive user agent lists at http://www.botsvsbrowsers.com/ and http://www.user-agents.org/.
Are you more interested in every variation of bot there is, or more after the top 20 bots?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question about Unpredictability with the Knowledge panel showing up for the same search
The people in my client's office get different results when they search for their company name in Google. For example one person ALWAYS gets the right rail knowledge panel with full details about the company while her boss NEVER sees it. They are both on desktop search. Rosemary
Technical SEO | | RosemaryB0 -
Google Site Search
I'm considering to implement google site search bar into my site.
Technical SEO | | JonsonSwartz
I think I probably choose the version without the ads (I'll pay for it). does anyone use Google Site Search and can tell if it's a good thing? does it affects in any way on seo? thank you0 -
Salvaging links from WMT “Crawl Errors” list?
When someone links to your website, but makes a typo while doing it, those broken inbound links will show up in Google Webmaster Tools in the Crawl Errors section as “Not Found”. Often they are easy to salvage by just adding a 301 redirect in the htaccess file. But sometimes the typo is really weird, or the link source looks a little scary, and that's what I need your help with. First, let's look at the weird typo problem. If it is something easy, like they just lost the last part of the URL, ( such as www.mydomain.com/pagenam ) then I fix it in htaccess this way: RewriteCond %{HTTP_HOST} ^mydomain.com$ [OR] RewriteCond %{HTTP_HOST} ^www.mydomain.com$ RewriteRule ^pagenam$ "http://www.mydomain.com/pagename.html" [R=301,L] But what about when the last part of the URL is really screwed up? Especially with non-text characters, like these: www.mydomain.com/pagename1.htmlsale www.mydomain.com/pagename2.htmlhttp:// www.mydomain.com/pagename3.html" www.mydomain.com/pagename4.html/ How is the htaccess Rewrite Rule typed up to send these oddballs to individual pages they were supposed to go to without the typo? Second, is there a quick and easy method or tool to tell us if a linking domain is good or spammy? I have incoming broken links from sites like these: www.webutation.net titlesaurus.com www.webstatsdomain.com www.ericksontribune.com www.addondashboard.com search.wiki.gov.cn www.mixeet.com dinasdesignsgraphics.com Your help is greatly appreciated. Thanks! Greg
Technical SEO | | GregB1230 -
Local search engine optimization
Hello, Im doing local search engine optimization for the hair salon client, im ranking number one for 70% of my keywords but im nowhere to be found in map, im trying to rank it for 3 months now, no results. I have added my business to over 100 local directories with same name, address , phone #, and have over 30 reviews, my local listings is optimized, (keywords, images with tags, video, categories and more) but again no results, Can you please advise me, how can i improve that? what can be a problem ? let me know if you can help me and ill be glad to send you business details. Thank you
Technical SEO | | vladraush990 -
Do the search engines penalise you for images being WATERMARKED?
Our site contains a library of thousands of images which we are thinking of watermarking. Does anyone know if Google penalise sites for this or is it best practice in order to protect revenues? As watermarking these images makes them less shareable (but protects revenues) i was thinking Google might then penalise us - which might affect traffic Any ideas?
Technical SEO | | KevinDunne0 -
Does Google Bot accept Cookies
I am working with a per page results refinement that stores a cookie on the users computer and then keeps that same per page as the user goes around the site. I was just wondering if that was true for Google bot or Bing bot as well. Will they keep the cookie or would they not be able to accept it. I just want to know as I dont want different urls created if they can keep the cookie. Thanks!
Technical SEO | | Gordian0 -
301 technical issue / sitemap being found in search
Hi I recently did some 301 redirects. But now when I type into the search results 'Kyocera black and white printers' my site map is coming up in the listings - does any have any advise as to why this may be happening and how to fix it? Please see image attached, thanks. 1XqHW.jpg
Technical SEO | | Socialdude0 -
How can I get a listing of just the URLs that are indexed in Google
I know I can use the site: query to see all the pages I have indexed in Google, but I need a listing of just the URLs. We are doing a site re-platform and I want to make sure every URL in Google has a 301. Is there an easy way to just see the URLs that Google has indexed for a domain?
Technical SEO | | EvergladesDirect0