How to authenticate Moz crawler so that others don't use Rogerbot useragent to scrape data from our site?
-
Is there any way to authenticate genuine Moz crawler. Because, our website keeps getting scrapping attacks and if there is no way to authenticate Moz crawler, then, any scraper can just set user agent as Rogerbot and scrape all our pages.
Is there a fixed IP that can be used or any other customization that will help us authenticate and allow only Moz crawler to crawl our site.
Looking forward to a solution to this problem. We haven't been able to use Moz crawler due to this issue.
-
Hi There,
Thanks for writing us so there seems to be a few things going on here so if you need any additional clarification please let me know. So Moz will use a dynamic IP, so there is not just one IP we can provide for authentication.
Unfortunately, your best course of action in this case would be to authorize Mozilla/5.0 (compatible; rogerBot/1.0) This would need to be conducted on the hosting level so you would need to work with your current hosting provider for a viable solutions.
Also, you could set up your robots.txt file to disallow all robots except for Google and Rogerbot, unfortunately malicious robots will often ignore robots.txt files, so any long term solutions would need to go through your hosting provider.
I am sorry that we could not provide more assistance on this matter and hopefully the attacks on your site do not last.
Have a great day!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I access old data/keyword research if I cancel my Moz Pro account?
I'm currently on the free month trial period for Moz Pro and I will probably cancel the account before the free period ends, but if I want to renew my subscription later, what happens to all the previous data? And does all the keyword research I've done disappear when I cancel it, or is it restored when I renew the subscription? Any insight is helpful! Thank you!
Getting Started | | TeamOneRep0 -
What does Location National means in Moz Pro Campaigns?
I understand the concept of locations in Moz Pro Campaigns. What does National exactly means? Is it like globally for a United States Google Search Rank or what? Please explain.
Getting Started | | ds9.tech1 -
I have a client with a wordpress.com site.
Is it possible to manage a campaign for such a site on Moz? It looks like in order to be able to add an independent Google Analytics tracking id, he has to upgrade to a business account. Does anybody have any experience with this?
Getting Started | | chill9860 -
Moz could not crawl my httpS website
Hi, we have a website with HTTPS, moz could not crawl it and we get "902 : Network errors prevented crawler from contacting server for page" while in logs we see moz robot access but fail after some seconds, what could be the problem, while moz can access site when it is without httpS | 902 : Network errors prevented crawler from contacting server for page. |
Getting Started | | Hamedkhorasani10 -
What is 'domain authority'?
IN seomoz, it mentions domain authority but it doesnt define it. what does domain authority mean?
Getting Started | | torbett0 -
So the page-grader is giving my site an A, but it is ranking below some websites that the grader gives an F to. What is the point of the page grader?
Basically, I am new to this..very new. In fact, my field was neuroscience, and now I work in marketing.. I just started using seomoz, and for one website that I build. I followed all of the guidelines moz has to offer. On the grade section, I got an for a specific keyword. However, when I rate my site for that keyword, it does not even rank in the top 50. The keyword is even in the domain! Also, some sites that rank in the top 25 have an F rating for the same keyword. Why do they rank higher?
Getting Started | | Meier0 -
Moz Analytics Beta Question
I've been invited to the Moz Analytics Beta; however, I am not seeing any data for any of my campaigns. Do I have to do something specific to enable the beta, or does it take a while for information to appear?
Getting Started | | Schwaab1 -
'Domain Does Not Respond To Web Requests'
Hi everyone, This seems to be a fairly common query on the Q&A section, but I haven't been able to find a solution by reading through previous threads. When I try to set up an SEOmoz campaign for spryz.co.nz, I get that ol' favourite error message: 'We have detected that the domain spryz.co.nz does not respond to web requests. Using this domain, we will be unable to crawl your site or present accurate SERP information.' The problem isn't being caused by a Robots.txt file, and the site has experienced 99% uptime since it was launched.Traffic stats show that visits are coming through to the site via the search engines, suggesting that it may not be all crawlers that fail to access the site. I've tried to set up this campaign several times throughout the day, since I've read that sometimes Roger goes on the blink, but I've still not been successful. Any suggestions as to why Roger might be unable to crawl my website would be great.
Getting Started | | e8creative0