Up-to-date list of search engine bot user agents
-
Does anyone know of an up-to-date-list of search engine bot user agents?
Thanks.
-
We just found this, which may be helpful. It was exactly what I was looking for.
-
Thanks, Keri. I found http://www.botsvsbrowsers.com/ to be very helpful. I'm looking for all the bots from the "major", U.S. search engines (Google, Bing, Ask, AOL, Blekko).
-
I haven't found any list that's newer than a year old in the brief search I did. I did find these comprehensive user agent lists at http://www.botsvsbrowsers.com/ and http://www.user-agents.org/.
Are you more interested in every variation of bot there is, or more after the top 20 bots?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best directories for a new(ish) hospital to be listed on?
Just thought I'd throw this question out to the Moz community. Anyone have leads for good, credible directories where I could get a smaller, local hospital listed on? (besides the obvious ones like HealthGrades, etc.,) Thanks!
Technical SEO | | TaylorRHawkins1 -
Soft 404 in Search Console
Search console is showing quite a lot of soft 404 pages on my site, but when I click on the links, the pages are all there. Is there a reason for this? It's a pretty big site - I'm getting 141 soft 404s from about 20,000 pages
Technical SEO | | abisti20 -
Is site: a reliable method for getting full list of indexed pages?
The site:domain.com search seems to show less pages than it used to (Google and Bing). It doesn't relate to a specific site but all sites. For example, I will get "page 1 of about 3,000 results" but by the time I've paged through the results it will end and change to "page 24 of 201 results". In that example If I look in GSC it shows 1,932 indexed. Should I now accept the "pages" listed in site: is an unreliable metric?
Technical SEO | | bjalc20112 -
Google not indexing /showing my site in search results...
Hi there, I know there are answers all over the web to this type of question (and in Webmaster tools) however, I think I have a specific problem that I can't really find an answer to online. site is: www.lizlinkleter.com Firstly, the site has been live for over 2 weeks... I have done everything from adding analytics, to submitting a sitemap, to adding to webmaster tools, to fetching each individual page as googlebot and then submitting to index via webmaster tools. I've checked my robot files and code elsewhere on the site and the site is not blocking search engines (as far as I can see) There are no security issues in webmaster tools or MOZ. Google says it has indexed 31 pages in the 'Index Status' section, but on the site dashboard it says only 2 URLS are indexed. When I do a site:www.lizlinketer.com search the only results I get are pages that are excluded in the robots file: /xmlrpc.php & /admin-ajax.php. Now, here's where I think the issue stems from - I developed the site myself for my wife and I am new to doing this, so I developed it on the live URL (I now know this was silly) - I did block the content from search engines and have the site passworded, but I think Google must have crawled the site before I did this - the issue with this was that I had pulled in the Wordpress theme's dummy content to make the site easier to build - so lots of nasty dupe content. The site took me a couple of months to construct (working on it on and off) and I eventually pushed it live and submitted to Analytics and webmaster tools (obviously it was all original content at this stage)... But this is where I made another mistake - I submitted an old site map that had quite a few old dummy content URLs in there... I corrected this almost immediately, but it probably did not look good to Google... My guess is that Google is punishing me for having the dummy content on the site when it first went live - fair enough - I was stupid - but how can I get it to index the real site?! My question is, with no tech issues to clear up (I can't resubmit site through webmaster tools) how can I get Google to take notice of the site and have it show up in search results? Your help would be massively appreciated! Regards, Fraser
Technical SEO | | valdarama0 -
Domain Name appear as title no description in search result
Hello Everyone, I have an site suppose tatadocomo.com( imaginary name to just explain my query) .Two weeksa back everything was fine but now if i search any query that site ranks in SERPs I got domain name as title and url (homepage) beneath of title and after that 'click here to proceed'. .Tata Docomo tatadocomo.com Click here to Proceed Why I am seeing this result in SERPs? I checked robots.txt , everything is fine there. Please help me out. Thanks
Technical SEO | | Alick3000 -
Google appending keyword to local search result(s)?
I noticed an interesting change today in how one of my clients appears in the SERPs. Google seems to be appending a keyword to his listing title. Client website: www.mycalgarydentist.com Keyword: Calgary dentist Rank: #2 or #1 lately Title tag: Calgary Dentist | Ambiance Dental Google+ Local listing title: Ambiance Dental Link title in SERP: Ambiance Dental: Calgary Dentist That last point is what's interesting, and new. As of a couple weeks ago (before I went on holidays) his link would simply show "Ambiance Dental", which makes sense because that's the title of his Google+ Local listing. Given the above information, I can't see why his link in Google's SERP is "Ambiance Dental: Calgary Dentist" when doing a search for that keyword. When I do a search for "Calgary dentists" or other similar searches, he simply shows as "Ambiance Dental", not "Ambiance Dental: Calgary Dentists" To test yourself, use the Google AdWords Preview Tool (https://adwords.google.com/d/AdPreview/), change locality to "Calgary, AB, Canada" and search. I suspect this doesn't mean he's violating Google's guidelines for business listings (i.e. businesses aren't supposed to add keywords to their business title). I'm certainly curious why this is happening though. Can anyone provide any insight? Has anyone seen anything similar? calgary-dentist-search.png
Technical SEO | | Kenoshi0 -
Search result pages - noindex but auto follow?
Hi guys, I don't index my search pages, and currently my pages are tagged name="robots" content="noindex"> Do I need to specify follow or will it automatically be done? Thanks Cyto
Technical SEO | | Bio-RadAbs0 -
Optimising multiple pages for the same search term
We were having a discussion on title tags and optimising multiple pages for the same term. We rank well for the phrase 'chanel glasses' which points to our Chanel brand page. The Chanel brand page is optimised for this term, and has the phrase 'Chanel glasses' at the front of its title tag. Previously, the title tag on our home page had the words 'Chanel glasses' at the start in an attempt to rank twice for the term (as one of our competitors has managed). This never worked (though at the time, our DA/PA was lower than it is now). For this reason I switched the title tag on the homepage to try and rank for 'designer glasses'. My belief is, given we already rank highly for the term on a more relevant landing page, trying to rank for it again on the home page is not the best use of a title tag on our highest PA page. We may as well use it for something more generic like 'designer glasses' (though this term does not convert nearly as well, nor does it currently rank as well for us as we've not been attempting to get 'designer glasses' as anchor text. Plus it's more competitive. Another generic term maybe be preferable). My colleague's view is we should attempt to do what our competitor has done and try and rank twice on page one for this term. I like the idea of dominating the top results, but I feel that since attempting to get double-listed hasn't worked for us so far, we should use the homepage for optimising for a different term ( ideally something that we don't already rank for elsewhere on the site). I see his point of view - if we were ranking nowhere for the search term then, yes we should concentrate on getting one page to rank, not two. But since we already rank well for the term, perhaps his strategy is preferable? Just for clarity, the title tags are not duplicate, but the idea was to share many of the same keywords between the two title tags. What are your thoughts SEOmoz?
Technical SEO | | seanmccauley0