How to authenticate Moz crawler so that others don't use Rogerbot useragent to scrape data from our site?
-
Is there any way to authenticate genuine Moz crawler. Because, our website keeps getting scrapping attacks and if there is no way to authenticate Moz crawler, then, any scraper can just set user agent as Rogerbot and scrape all our pages.
Is there a fixed IP that can be used or any other customization that will help us authenticate and allow only Moz crawler to crawl our site.
Looking forward to a solution to this problem. We haven't been able to use Moz crawler due to this issue.
-
Hi There,
Thanks for writing us so there seems to be a few things going on here so if you need any additional clarification please let me know. So Moz will use a dynamic IP, so there is not just one IP we can provide for authentication.
Unfortunately, your best course of action in this case would be to authorize Mozilla/5.0 (compatible; rogerBot/1.0) This would need to be conducted on the hosting level so you would need to work with your current hosting provider for a viable solutions.
Also, you could set up your robots.txt file to disallow all robots except for Google and Rogerbot, unfortunately malicious robots will often ignore robots.txt files, so any long term solutions would need to go through your hosting provider.
I am sorry that we could not provide more assistance on this matter and hopefully the attacks on your site do not last.
Have a great day!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the quickest and easiest way to run an SEO audit on a Wordpress site that at least shows all the mechanical problems?
What is the quickest and easiest way to run an SEO audit on a Wordpress site that at least shows all the mechanical problems?
Getting Started | | integratedproperty0 -
MOZ point
I don't know why, but since a week ago I'm not receiving moz point for my activity on moz forum. Example
Getting Started | | Roman-Delcarmen
Today I posted 3 answer in the Question section but in my moz profile does not show the 3 moz point that normally I receive for that. I week ago suddenly I received 20 moz point, why I dont have any idea, maybe someone mark one of my asnwer as good answer. So my point is where I cant found the exact tracking record of my activity1 -
Moz could not crawl my httpS website
Hi, we have a website with HTTPS, moz could not crawl it and we get "902 : Network errors prevented crawler from contacting server for page" while in logs we see moz robot access but fail after some seconds, what could be the problem, while moz can access site when it is without httpS | 902 : Network errors prevented crawler from contacting server for page. |
Getting Started | | Hamedkhorasani10 -
Error Selecting Search Engine During Moz Campaign Set-Up
Hi, I'm setting up a Moz campaign for a new client, and when I get to the third step, "Choose the search engines for which you would like to track rankings," there's nothing to select. No drop down, or anything, just a blank field under the heading. If I try to advance, I get an error pop-up that says I must select a search engine. Has anyone else run into this problem? What should I do? GlWPAsc zAmzqFf
Getting Started | | tommyKPseo0 -
What Moz tool is best to find reasons google has not spidered by site
I just joined Moz and am trying to use the tools however, when I attempt to do so every link comes to a that only allows me access to post questions here. If anyone can tell me what tool is best to find reasons google has not indexed my site, I would greatly appreciate the help. Also if anyone knows why I am keep getting routed to this forum when I try to use any of the tools, I would also appreciate help with this. So far Moz is very frustrating.
Getting Started | | Johndeeray19640 -
Moz Not Pulling Monthly Data
So I set up my Moz campaigns on the 8th of this month. I can see data for the week, however, I can't see data from the previous week or for the month. So my question has two parts: a.) Is there a way to view data for the previous week, and b.) when can I expect to see data for the entire month. Seeing data for the entire month is absolutely critical for me, since this is what I'm going to use to prepare my monthly client reports. Has anyone experienced anything like this? PLEASE HELP!!!
Getting Started | | maxcarnage0 -
Moz Private Message Restriction
Why does Moz only let you send 2 private messages a day? I'm a Moz Pro subscriber! Sorry Message Not Sent You are currently over your quota of 2 threads per day.
Getting Started | | deelo5550 -
Moz Pro won't stay connected to my Google Analytics profile
I just started a trial of Moz Pro today and it seems to have a problem staying connected to my Google Analytics account. Evenr time I refresh a page or click to a different page, I get a notification that reads "Our connection to your Google Analytics account was lost. Don't worry, you won't lose any data. Please reauthorize now." I click the link to reauthorize and it says it was successful, But then as soon as I refresh the page or click to a new page withing Moz Pro, the message appears again. I'm afraid that none of my GA data is being collected by Moz Pro. What should I do?
Getting Started | | InterCall0