How to authenticate Moz crawler so that others don't use Rogerbot useragent to scrape data from our site?
-
Is there any way to authenticate genuine Moz crawler. Because, our website keeps getting scrapping attacks and if there is no way to authenticate Moz crawler, then, any scraper can just set user agent as Rogerbot and scrape all our pages.
Is there a fixed IP that can be used or any other customization that will help us authenticate and allow only Moz crawler to crawl our site.
Looking forward to a solution to this problem. We haven't been able to use Moz crawler due to this issue.
-
Hi There,
Thanks for writing us so there seems to be a few things going on here so if you need any additional clarification please let me know. So Moz will use a dynamic IP, so there is not just one IP we can provide for authentication.
Unfortunately, your best course of action in this case would be to authorize Mozilla/5.0 (compatible; rogerBot/1.0) This would need to be conducted on the hosting level so you would need to work with your current hosting provider for a viable solutions.
Also, you could set up your robots.txt file to disallow all robots except for Google and Rogerbot, unfortunately malicious robots will often ignore robots.txt files, so any long term solutions would need to go through your hosting provider.
I am sorry that we could not provide more assistance on this matter and hopefully the attacks on your site do not last.
Have a great day!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword difficulty with Moz Pro
Hello, I want to know how moz pro calculates the keyword difficulty? What are the basic things which moz pro consider while calculating keyword difficulty? Whenever i check my site(Sehhaty app) it shows different keyword difficulty. why so? Thanks.
Getting Started | | emmaava60 -
How to rank website on google with help of Moz ?
I am new here and my question is how rank a website with moz because my website was not ranking in any keywords so plz guide me
Getting Started | | RoyDigitalWorld0 -
Changing the Moz Crawl Date
Hello, I am wondering where I can change the date of Crawl by Moz. I would like to change this crawl period from one week to 2 or even 3 weeks for Moz to crawl my website. Hope to hear from anyone soon. Kind regards, Koen.
Getting Started | | Koenniiee1 -
Do you have a tool to track blogger's product reviews?
Can I enter a blogger's product review URL and see what other bloggers have either shared that review on social media or written their own reviews on the product?
Getting Started | | PerfectTD0 -
"Gathering data" How long does it take?
Hi, I've only signed up recently and set up my first campaign this morning. I get this message which I presume is pretty standard _We are gathering data for your campaign. It may take a minute or two for us to bring back some initial data. In the meantime feel free to grab some coffee, we should be ready when you get back. _ Except, that was over an hour ago, and I've had several cups of coffee. Has something gone horribly wrong or does it normally take this long?
Getting Started | | bimjo0 -
Can't setup new campaign
Hi everyone, I'm trying to set up a new campaign for a website which has Cloudflare installed. After I enter the campaign name and URL the loading circle comes up and spins for a while, but then it just stays on the same page. No error message is given. I can't get to the next page of the setup campaign form sequence so that I can set up this campaign. Has anyone else had this problem and is there any fix? Thanks in advance
Getting Started | | _jrmo
James0 -
Lost since the change over to Moz
I was recently lead to a series of videos which looked to be quite helpful here http://moz.com/help/pro/introduction But is seems with the change over from SEO Moz to Moz, nothing is in the same place. Should I bother watching them or wait for new ones? I'm particularly interested in SEO to raise our rank.
Getting Started | | AliciaMarie0