Moz crawling bot
-
Hi guys,
in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".
I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
However, my robots.txt doesn't have such entry. Take a look:
Automatically banned scanners and crawlers section
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
Section end
What could be the problem here, then? Why does the Moz bot think I'm blocking it?
-
Thank you Keri, problem solved.
-
If you pull out dotbot from your exclusions, you should be fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My company has about 70 live pages. Moz crawled 6k. Why?
My company has about 70 live, published pages on our website, which is managed through HubSpot. When Moz crawled the URL, it found 6K+. I noticed several were URLs with a campaign tag. Is this normal? It seems excessive to me. If there's a problem, how do I fix it?
Link Explorer | | ActionableResearch1 -
Moz tool to estimate a website's organic traffic?
Moz has tons of great tools, so I was quite suprised there is nothing similar to Alexa's siteinfo tool, or SimilarWeb's site comparison tool. Am I missing something? Or does such a tool exists at Moz? Or is there a different way of estimating my competitor's traffic? It would be great if Moz would have something like that.
Link Explorer | | GrandRapids0 -
Moz is not indexing all my backlinks
Why is there a difference in the backlinks of my website when checked in moz.com and SEMrush.com.
Link Explorer | | mzakaria0 -
Angular SPA & MOZ Crawl Issues
Website: https://www.exambazaar.com/ Issue: Domain Authority & Page Authority 1/100 I am using Prerender to cache/render static pages to crawl agents but MOZ is not able to crawl through my website (https://www.exambazaar.com/). Hence I think it has a domain authority of 1/100. I have been in touch with Prerender support to find a fix for the same and have also added dotbot to the list of crawler agents in addition to Prerender default list which includes rogerbot. Do you have any suggestions to fix this? List: https://github.com/prerender/prerender-node/commit/5e9044e3f5c7a3bad536d86d26666c0d868bdfff Adding dotbot to Express Server:
Link Explorer | | gparashar
prerender.crawlerUserAgents.push('dotbot');0 -
Moz Spam Score 9/17 when there are no links
Hi I was just looking at a competitors new site in OSE and it had a DA and PA of 1 which I would expect and also no links, which again I would expect, basically the guy used to design stuff for me and thinks he can do a better job so he is giving it ago. But he has a spam score of 9/17 but no links - how is this even possible. Thanks Andy
Link Explorer | | Andy-Halliday0 -
6 Months and still no PA/DA or links by Moz Explorer?
Hi, It's been 6 months and my site is still showing no PA/DA or backlinks according to moz opensite explorer. I know that it has backlinks as I created them myself, and I know its crawlable and the links is there as GWT tells me so. Also you can check for yourself on ahrefs.com. Is Moz indexed pages that much smaller then the other crawlers? The site in question is amanandvan247.com. Thanks.
Link Explorer | | Marvellous1 -
Why is OSE showing a higher Domain Authority than Moz?
Moz has today run monthly reports for 2 of my clients' sites. For both of these, the Domain Authority reported on the Moz dashboard is several points lower than that shown by OSE today for the same sites. Why would that be?
Link Explorer | | mfrgolfgti0 -
Is there a preferred Browser for Moz?
I having difficulties using Firefox on the Mac. Crawl Diagnostics results never show up. I see the numbers of errors but can't see what the errors are. Appreciate any help. I'm a newbie.
Link Explorer | | pompanoartist0