Moz crawling bot
-
Hi guys,
in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".
I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
However, my robots.txt doesn't have such entry. Take a look:
Automatically banned scanners and crawlers section
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
Section end
What could be the problem here, then? Why does the Moz bot think I'm blocking it?
-
Thank you Keri, problem solved.
-
If you pull out dotbot from your exclusions, you should be fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Moz Pro provide a list of external (vs internal) links on our website?
Is there a tool (in MOZ?) that lists all the links on our website that point to external links (vs internal links)?
Link Explorer | | analytics-moz0 -
Moz DA is not improving my site DA why?
Hello Moz community, I have created a lot of high domain authority backlinks for my official service website but my DA still less than 15, why my site DA does not increase as I have created huge backlinks.
Link Explorer | | Ijiubu7680 -
Hello There I have a site the problem with this site is I have taken countless backlinks but it is not reflecting on Moz. Is there any way to help Moz find my links?
Hello There I have a site health related..the problem with this site is I have taken countless backlinks but it is not reflecting on Moz. Is there any way to help Moz find my links? herpescurecare.com
Link Explorer | | thelancethealth0 -
Moz bot transforms special characters into fake 404 Pages in Reports
I have a lot of 404 links in my exported Moz Pro reports which are not actually Not Found pages when checking the Referring URL. For example: Referring URL:
Link Explorer | | ProspectoGroup
https://ofertolino.ro/oferte/barn%C3%A4ngen 404:
https://ofertolino.ro/barn%C3%83%C2%A4ngen What happens is that the referring URL leads to a redirect (https://ofertolino.ro/barn%C3%A4ngen) but it never leads to the specified 404 Page (which has even a more weird symbol in the URL when it is url decoded) in the report. What could be the issue here and is the Moz bot responsible for such misbehaviour?0 -
Error Message on Moz Crawler
Hi all, Just ran into this issue, when analysing this site. Just got this message when using MOZ "Page Optimisation Error". Anyone know why? It seems to be working fine on other SEO analyser tools. Website is: www.sbpcreativemedia.com.au Thanks in advance! luXS8V5
Link Explorer | | Dushala0 -
Doubt Regarding Moz Spam Score
My website is https://www.whatiscbd.com/ . When I check spam score of whatiscbd.com, moz show that its spam score is 9. And when i search spam score of www.whatiscbd.com, spam score is 0. Although both are same website. Why i am getting such issue. Is there any issue in my website's URL redirection. If yes then what's the issue?
Link Explorer | | HuptechWebseo0 -
Why Moz Back links check tool is very Slow compared with other tools?
I've created a new website 50 days old with a vision of sharing high-quality content. I created some pages which are useful and helpful, what make other people link to my site. Early days, two weeks ago I've started checking my backlinks starting with Ahref and majestic. three days ago I decided to join Moz too and use Moz explorer. But, I discovered that Moz tool is too slow compared with other tools. Why? Starting with Ahref. They discovered more than 100 links and updating their data for my site as a new website every 6 hours as I know. Also majestic for the first time discovered six links then every two days they updating their data. Even Google discovered some of my backlinks. Specially links from high-quality sites such as Wikipedia with high-quality pages. Moz tool discovered zero of my backlinks even after subscribing. I don't think Moz need to discover websites like Wikipedia which already indexed by Moz. So, I wonder Why Moz tools are very very very slow like that? Update: (This is a question, not a discussion. So, please don't edit it) Can any Moz support answer please? Thanks.
Link Explorer | | Eslam-yosef0 -
How Does Moz Assign Domain Authority?
When I use Site Explorer to test the following URL, http://jennathuening.results.net/site/index.php, it renders a domain authority of 47. (Clients site on real estate host) There are no page social metrics: no Facebook share no Facebook likes no Twitter tweets no Google+ plus 1's no "just discovered" in 60 days no one has made a post in 3 weeks much or most of the content is duplicate content across all affiliate sites inbound links - only one link - a 301 redirect http://jennathuening.results.net/ Images - 18 (Alt tags missing: 18) Created - 1998-10-15 Registered Org - REMAX RESULTS is associated with ~416 other domains Whois SEO Score 68% AND YET IT HAS A ROOT DOMAIN OF "47" and page authority of 33 More confusing, its page authority of 33 is at that top on blue background and then page 30 just listed below. When I use Site Explorer to test the following URL, http://www.homedestination.com, it renders a domain authority of 26. (Site personally owned and continually update with highly optimized fresh, relevant, quality content with grade "A" moz ranked pages and blog posts) 92 Facebook share 68 Facebook likes 13 Twitter tweets 420 Google+ plus 1's 2 "just discovered" in 60 days Images 25 (Alt tags missing: 0) inbound links - on 48 Root Domains Created - 2002-08-15 9 new indexed blog post so far in August / 19 new indexed blog post in July Whois SEO Score 100% AND YET IT HAS A ROOT DOMAIN OF "26" and page authority of 36 While I am sure there are many other factors as well; it still is hard to see where a "smaller guy" could ever be given an even opportunity to play in the same SEO arena. Could you help me understand better? I like to build a stronger marketing and optimization strategy as we are just beginning AdWord campaigns. | SEO Score | 100% |
Link Explorer | | jessential1