Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
What is the IP address of the MOZ crawlbot?
-
Our preproduction site is restricted to a specific IP range.
Prelaunch I'd like to use MOZ to check the site but this obviously isn't currently possible. Is there an IP range I can whitelist ?
-
Hi,
Moz is hosted on Amazon. See this sample log entry from today:
54.146.235.41 - - [17/Sep/2017:09:48:13 +0200] "GET /best-deals.html/page/12/ HTTP/1.0" 200 15803 "-" "rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-97@moz.com)"[XF 54.146.235.41]
That IP is part of Amazon's ASN.
whois -h whois.radb.net -- '-i origin AS16509' | grep ^route
You can either use something like mod_authz to allow the domain amazonaws.com to connect or you can get each IP from AS16509 and allow that.
Hope it helps.
Regards
Chris
-
Just to expand on this, I found the following quote from a Moz member of staff:
"I wish I could give you IP addresses, but they change all the time since we host Roger in the cloud. There's not even a reliable range of IPs to give you. You can totally whitelist the useragent rogerbot, but that's the only reliable information about the crawler you can go off of. I hope that helps but let me know if there's any other solution you can think of. Thank you! "
From a previous question: https://moz.com/community/q/is-there-a-whitelist-of-the-rogerbot-ip-addresses
-
Unfortunately this is all I can find: https://moz.com/help/guides/moz-procedures/what-is-rogerbot with no IP range shown.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz tried to charge our credit card
Hello, We noticed that MOZ tried to charge our credit card, while we are on Trial period until today. We decided to cancel the subscription (until our budget grows and come back). The tool is great, but it currently is too expensive for our business. We enjoyed the trial period and will surely come back when our business can afford it!! Thanks!
Getting Started | | intenseed0 -
How to rank website on google with help of Moz ?
I am new here and my question is how rank a website with moz because my website was not ranking in any keywords so plz guide me
Getting Started | | RoyDigitalWorld0 -
Moz not able to crawl our site - any advice?
When I try and crawl our site through Moz it gives this message: Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt. Anyone got any advice on where I could go to get this sorted?
Getting Started | | MyFamilClubLtd1 -
What does Location National means in Moz Pro Campaigns?
I understand the concept of locations in Moz Pro Campaigns. What does National exactly means? Is it like globally for a United States Google Search Rank or what? Please explain.
Getting Started | | ds9.tech1 -
Moz can't crawl my site.
Moz cannot carry out the site crawl on my online shop. Not really sure what the issue is, it has no problem getting onto my site when you use www. before the address, but it needs to be able to access bluerinsevintage.co.uk Stuck as what to do, we are a shopify store. Anyone else had this problem, or know what i need to change so they can crawl the site? thjis is the page they are getting when trying to get on bluerinsevintage.co.uk but if they use www.bluerinsevintage.co.uk the site comes up.
Getting Started | | bluerinsevintage
Adam
0 -
MOZ for initial keyword research?
Can I use MOZ for initial keyword research? Can't create an AdWords campaign (no website yet to register) to get use of Google's Keyword Planner & my subscription to Wordtracker is showing me too many unrelated keywords... Peter
Getting Started | | nodiffrei0 -
Moz's official stance on Subdomain vs Subfolder - does it need updating?
Hi, I am drawing your attention to Moz's Domain basics here: http://moz.com/learn/seo/domain It reads: "Since search engines keep different metrics for domains than they do subdomains, it is recommended that webmasters place link-worthy content like blogs in subfolders rather than subdomains. (i.e. www.example.com/blog/ rather than blog.example.com) The notable exceptions to this are language-specific websites. (i.e., en.example.com for the English version of the website)." I am wondering if this is still Moz's current recommendation on the subfolders vs subdomains debate, given that the above (sort of) implies that SE's may not combine ranking factors to the domain as a whole if subdomains are used - which (sort of) contradicts Matt Cutts last video on the matter ( http://www.youtube.com/watch?v=_MswMYk05tk ) which implies that this is not the case and there is so little difference that their recommendation is to use whatever is easiest. It would also seem to me that if you were looking through the eyes of Google, it would be silly to treat them differently if there were no difference at all other than subdomain vs subfolder as one of the main reasons a user would use a sud-domain is a technical on for which it would not make sense for Google to treat differently in terms of its algorithm. I notice that in terms of Moz, while most of the site uses subfolders, you do have http://devblog.moz.com/ - and I was wondering if this is due to a technical reason or conscious decision, as it would seem to me that the content within this section is indeed linkworthy (as it has external links pointing to it from external sources), therefore it would seem to not be following the initial advice that is posted in Moz's basics on domains. Therefore I am assuming it is due to a technical reason - or that Moz's adive is out of date with current Moz thinking, and is indeed in line with Matt C in that it doesn't matter. Cheers
Getting Started | | James773
