Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawling bot
-
Hi guys,
in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".
I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
However, my robots.txt doesn't have such entry. Take a look:
Automatically banned scanners and crawlers section
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
Section end
What could be the problem here, then? Why does the Moz bot think I'm blocking it?
-
Thank you Keri, problem solved.
-
If you pull out dotbot from your exclusions, you should be fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved My DA does not increase and Moz does not recognize my links
Hello everyone. My website (I am attaching the link to it, if you cannot attach the link remove it but please give me an answer) https://mejorhora.com/ 3 MONTHS ago I made different redirections from other websites that I have with high authority in order to increase both the Da and the DR since I sell links to that website and it suits me. Ahref showed me all the links after a few days, also in search console. However, in Moz it does not recognize more than 500 new domains that give me links. Because? It's been 3 months... What should I do?
Link Explorer | | SoyGabi0 -
Problem with third party cookies while using MOZ?
Hello, I am using Moz Pro and I am happy to have this tool for my SEO work. But I am having a little problem while working on my blog site https://thepoolparadise.com/ some times the Moz does not work and asks to allow third-party cookies, wherein the settings third-party cookies are always allowed. Please guide me in this matter. Thank you.
Link Explorer | | Niujbuj760 -
Why my website Backlinks not getting Crawl by Moz?
Hi, I've the query related to backlinks of my website. Some of the high authority sites give backlinks to my website but these links are still not showing in Moz. My website is 6 months old and also continuously getting backlinks from high authority sites but still these are not showing on Moz and also not improving DA and PA of my website. I've also attached the screenshot of Moz link explorer results please check and guide me what to do with website so Moz will consider it and give some authority. And also guide me How much Moz takes time to crawl website backlinks and shown in their link explorer. Website URL: https://www.welderexpert.com Moz Expert Suggestions needed. Thanks, overview?site=welderexpert.com⌖=domain
Link Explorer | | KOidue0 -
Does Moz register backlinks from Twitter/Facebook/LinkedIn?
I've noticed that on sites with decent social presence and regular content posts that Moz isn't showing/recording any inbound links from Twitter, Facebook or LinkedIn. Posts are regularly shared via these three channels but no inbound links ever seem to show up. Does Moz not record such inbound links, or are they all disguised using JavaScript or some other trick?
Link Explorer | | JCN-SBWD0 -
If I have a MOZ PRO account, do I still need Screaming Frog?
Hi, I have a MOZ PRO account which is brilliant for me. I am about to move my site from HTTP to HTTPS and was looking for a smooth way to list every page and its details so I can make sure I do the correct redirects etc.. before the switch to SSL. Do I need to get Screaming Frog, or can the tools with MOZ Pro ease my path just as well? Thanks
Link Explorer | | SeoSheikh0 -
Is Moz's backlink checker.... just... not good?
Hey everyone! Can somebody explain to me why this keeps happening: Whenever I'm trying to backlink my competitors, I typically use RavenTools. Every time, without fail, if I put that same URL into Moz's Open Site Explorer - It gives me about 1/20th of what RavenTools shows me. Sometimes it literally comes up with 2 or 3 links total. Unfortunately, RavenTools has a cap on how many backlink checks you can perform in a month - so once I've used those up, I have to start using OSE... But, it just doesn't work. Does anyone else have this issue? Thanks!
Link Explorer | | TaylorRHawkins1 -
Fastest way to get new backlinks recognised by Moz?
Hi all, I have seen a few discussions / advice pieces around this, but wondered what the current view is on the fastest way to get new backlinks recognised by Moz? I have seen people say that the content needs to be shared (e.g. on Twitter), and advice around creating Bitly links. I am not convinced that these techniques are working. On a related note, my website is very large. I have around 2.5m pages. Most of these pages are 3-4 clicks from the home page, but I have a small amount of important content much higher up in the site (1-2 clicks from the home page). Does Moz crawl pages nearer the home page more frequently, or does it make a pass of all pages on the site before it starts at the beginning again? If it makes a complete pass of all pages, then presumably making my site smaller will help to get Moz to see new content more quickly? Is there any other value to be gained in making a large site smaller? Many thanks in advance, Neil.
Link Explorer | | NHARRIS10 -
Does Moz have anything similar to ahrefs batch analysis tool?
I am wanting to do analysis of a whole bunch of URLs at once - i know ahrefs already has a very good tool for this - but I don't really want to have to pay the $79 a month if moz has something similar? I know it has the OSE. Can I do batch analysis in this? Thanks
Link Explorer | | SWD.Advertising1