Welcome to the Q&A Forum

Hi Federico,

Thanks for the prompt.  Yes, this solution worked.  I'm hopeful that this thread helps others too because when I was troubleshooting the problem, the other threads were not helpful for my particular situation.

Cheers

Hi Federico,

I think I have found the solution for this problem and am hopeful the crawl will be successful this time around.  Based on further digging and speaking to the team at StackPath CDN, I have done the following:

1. I added the following to my robots.txt file

User-agent: rogerbot
Disallow:

User-agent: dotbot
Disallow:

2. I added a custom robots.txt file in my CDN which includes the above and then created a rule in my Web Application Firewall which allows user agents rogerbot and dotbot.

I'll let you know if the crawl was successful or not.

Kind regards

Thanks for your response Federico.  I have checked my robots.txt tester in my Google Search Console and it said "allowed."

Oddly, it also happened on another site of mine that I'm also running through StackPath CDN with a web application firewall in place. This makes me wonder if perhaps the CDN/WAF are the culprits (?).

I'll keep poking around to see what I find.
Cheers

Hi all,
I ran a crawl on my site https://www.drbillsukala.com.au   and received the following error  "612 : Page banned by error response for robots.txt."

Before anyone mentions it, yes, I have been through all the other threads but they did not help me resolve this issue.

I am able to view my robots.txt file in a browser https://www.drbillsukala.com.au/robots.txt. 
The permissions are set to 644 on the robots.txt file so it should be accessible
My Google Search Console does not show any issues with my robots.txt file
I am running my site through StackPath CDN but I'm not inclined to think that's the culprit

One thing I did find odd is that even though I put in my website with https protocol (I double checked), on the Moz spreadsheet it listed my site with http protocol.

I'd welcome any feedback you might have.  Thanks in advance for your help.
Kind regards

Hi Federico,

I think I have found the solution for this problem and am hopeful the crawl will be successful this time around.  Based on further digging and speaking to the team at StackPath CDN, I have done the following:

1. I added the following to my robots.txt file

User-agent: rogerbot
Disallow:

User-agent: dotbot
Disallow:

2. I added a custom robots.txt file in my CDN which includes the above and then created a rule in my Web Application Firewall which allows user agents rogerbot and dotbot.

I'll let you know if the crawl was successful or not.

Kind regards