Seomoz Spider/Bot Details
-
Hi All
Our website identifies a list of search engine spiders so that it does not show them the session ID's when they come to crawl, preventing the search engines thinking there is duplicate content all over the place.
The Seomoz has bought a over 20k crawl errors on the dashboard due to session ID's. Could someone please give the details for the Seomoz bot so that we can add it to the list on the website so when it does come to crawl it won't show it session ID's and give all these crawl errors.
Thanks
-
An old answer to an old question, but one which has just helped me out tremendously. A true credit to the Q&A system!
-
Thanks Alan. much appreciated.
-
3 dots placed before last bracket, need to be removed,
-
Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to recover from penguin hit for my site http://rndegrees.net
My site http://rndegrees.net hit by penguin and after that I worked on cloud tag and one duplicate page content which was present on my site, also I worked on back links though my site hasn't recovered yet. Please guide me if I missed on something to recover from penguin hit.
Moz Pro | | HQP0 -
SEOMOZ Crawler unicode bug
for the last couple of weeks the SEOMOZ crawls my homepage only and gets 4xx error for most of the URL's. the crawler have no issues with English url's only with the unicode(Hebrew) ones. this is what is see in the csv export for the crawl (one sample) : http://www.funstuff.co.il/׳ž׳¡׳™׳‘׳×-׳¨׳•׳•׳§׳•׳× 404 text/html; charset=utf-8 you can see that the URL is Gibberish please help.
Moz Pro | | AsafY0 -
Crawl test tool from SEOmoz - which URLs does it actually crawl?
I am using for the first time the crawl test tool from SEOmoz and I do not really understand which URLs the tool is going to crawl. First, it says "enter any subdomain" --> why can´t I do the crawl for the root domain? Second it says "we'll crawl up to 3,000 linked-to pages" --> does that mean that the tool crawls all internal links that it can find on the given domain? Thanks for your help!
Moz Pro | | Elke.GetApp0 -
May not have a /path after the host
how to enter the competitor domain? on feedback i get: may not have a /path after the host. what is to do? Thanks Christian
Moz Pro | | cnort0 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0 -
Too many pages indexed in SEOMoz
I am running a campaign for a client that has 86 pages via Google and SEmoz is up to almost 10K pages. I am really confused. Any ideas?
Moz Pro | | LaurieK130 -
What's Your SEOmoz Favorite?
Obviously, SEOmoz is full of awesomeness. But I'm curious as to what sections, tools, etc. of SEOmoz are the users' favorites? For me, it's a close race between the blog, Linkscape/Open Site Explorer, and the search overlay tool on the Mozbar. What are your favorites?
Moz Pro | | Gyi0
