How to authenticate Moz crawler so that others don't use Rogerbot useragent to scrape data from our site?
-
Is there any way to authenticate genuine Moz crawler. Because, our website keeps getting scrapping attacks and if there is no way to authenticate Moz crawler, then, any scraper can just set user agent as Rogerbot and scrape all our pages.
Is there a fixed IP that can be used or any other customization that will help us authenticate and allow only Moz crawler to crawl our site.
Looking forward to a solution to this problem. We haven't been able to use Moz crawler due to this issue.
-
Hi There,
Thanks for writing us so there seems to be a few things going on here so if you need any additional clarification please let me know. So Moz will use a dynamic IP, so there is not just one IP we can provide for authentication.
Unfortunately, your best course of action in this case would be to authorize Mozilla/5.0 (compatible; rogerBot/1.0) This would need to be conducted on the hosting level so you would need to work with your current hosting provider for a viable solutions.
Also, you could set up your robots.txt file to disallow all robots except for Google and Rogerbot, unfortunately malicious robots will often ignore robots.txt files, so any long term solutions would need to go through your hosting provider.
I am sorry that we could not provide more assistance on this matter and hopefully the attacks on your site do not last.
Have a great day!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Risk of moz on website
Does Moz collect information of customers accessing my website https://tructiepbongda.site/?
Getting Started | | gogoanimetp
Thank you!0 -
Do I have to manually mark as 'fixed' or 'Ignore' manually on Metadata Issues when completed?
When I have amended the missing description Metadata issue on an individual page do I have to manually mark as 'fixed' or 'Ignore'? I have attended to several pages regards Missing Description Metadata Issues and not manually marked them as fixed. However they still appear in metadata missing description issues after a second on demand crawl? Will they continue to appear until I manually mark them as fixed?
Getting Started | | LM_Marketing_Solutions_Ltd0 -
How do i stop Moz from indexing my dev site?
Hey, I am new to MOZ, and I am seeing in my page reports that I have duplicate content, but this duplicate content is mostly my development site. dev.domain.com. I have this blocked in robots.txt for google. How do i stop MOZ including it in its reports? Cheers everyone.
Getting Started | | Tholomew1
Bart0 -
My question is, when you translate your website to another language, does moz crawl both or do i have to add another campaign to moz so that they can crawl it seperately?
Hello, i recently translated my website to spanish, keywords,meta tags, content etc. All of our URL for the translated pages start "ep", which symbolizes espanol. My question is, does moz crawl those pages along with my english pages? or do i have add another campaign for moz to crawl my spanish pages seperately?
Getting Started | | prestigeluxuryrentals.com0 -
How do get Moz to spider a Development site PRE LAUNCH?
Hi, Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it? Thanks
Getting Started | | bjs20100 -
Need to Add Moz code?
Do i need to add tracking code in my website, for Moz tool to analyze my website. Please someone let me know.
Getting Started | | TOBOC0 -
Custom reports in Moz Analytics?
I am trying to find where to configure whitelabled custom reports in moz analytics. Where can I find them? If they are only accessible through the old version, how do I switch back?
Getting Started | | keybroker1
