612 : Page banned by error response for robots.txt
-
Hi all,
I ran a crawl on my site https://www.drbillsukala.com.au and received the following error "612 : Page banned by error response for robots.txt."Before anyone mentions it, yes, I have been through all the other threads but they did not help me resolve this issue.
I am able to view my robots.txt file in a browser https://www.drbillsukala.com.au/robots.txt.
The permissions are set to 644 on the robots.txt file so it should be accessible
My Google Search Console does not show any issues with my robots.txt file
I am running my site through StackPath CDN but I'm not inclined to think that's the culpritOne thing I did find odd is that even though I put in my website with https protocol (I double checked), on the Moz spreadsheet it listed my site with http protocol.
I'd welcome any feedback you might have. Thanks in advance for your help.
Kind regards -
Hey there! Tawny from Moz's Help Team here.
After doing some quick searching, it looks like how you configure the rules for WAFs depends on what service you're using to host those firewalls. You may need to speak to their support team to ask how to configure things to allow our user-agents.
Sorry I can't be more help here! If you still have questions we can help with, feel free to reach out to us at help@moz.com and we'll do our best to assist you.
-
Hi, I am having the same issue.
Can you please tell me how you have created rule in Web Application Firewall to allow user agents rogerbot and dotbot.
Thanks!!
-
Hi Federico,
Thanks for the prompt. Yes, this solution worked. I'm hopeful that this thread helps others too because when I was troubleshooting the problem, the other threads were not helpful for my particular situation.
Cheers
-
Hi, did the solution work?
-
Hi Federico,
I think I have found the solution for this problem and am hopeful the crawl will be successful this time around. Based on further digging and speaking to the team at StackPath CDN, I have done the following:
- I added the following to my robots.txt file
User-agent: rogerbot
Disallow:User-agent: dotbot
Disallow:- I added a custom robots.txt file in my CDN which includes the above and then created a rule in my Web Application Firewall which allows user agents rogerbot and dotbot.
I'll let you know if the crawl was successful or not.
Kind regards
-
Thanks for your response Federico. I have checked my robots.txt tester in my Google Search Console and it said "allowed."
Oddly, it also happened on another site of mine that I'm also running through StackPath CDN with a web application firewall in place. This makes me wonder if perhaps the CDN/WAF are the culprits (?).
I'll keep poking around to see what I find.
Cheers -
Seems like an issue with the Moz crawler, as the robots.txt has no issues and the site loads just fine.
If you already tested your robots.txt using the Google Webmaster Tools "robots.txt Tester" just to be sure, then you should contact Moz here: https://moz.com/help/contact/pro
Hope it helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My company has about 70 live pages. Moz crawled 6k. Why?
My company has about 70 live, published pages on our website, which is managed through HubSpot. When Moz crawled the URL, it found 6K+. I noticed several were URLs with a campaign tag. Is this normal? It seems excessive to me. If there's a problem, how do I fix it?
Link Explorer | | ActionableResearch1 -
Why doesn't Moz crawler follow robots.txt?
It is crawling the entire site, and there is stuff we do not want it to. Please advise.
Link Explorer | | Tylerj0 -
OSE error?
Hi, I just started using moz pro, but if i try to check ose, I get this error: There was an error getting your data What's wrong?
Link Explorer | | NielsPNO0 -
Open Site Explorer: Error receiving data
Hi, When searching many URLs in Open Site Explorer, we seem to be getting the same "There was an error getting your data" message come up instead of the URL's metrics and inbound links. We have tried on different browsers as well, and the same issue results. Is there a reason for this or a way to fix this? Thank you! voPdqF5.png?1
Link Explorer | | MarriottDS0 -
Internal Pages marked as Spam by Moz
Hello,
Link Explorer | | Umesh-Chandra
We use Moz for our website,Open site explorer marks many of the internal pages of the website as spam. Even pages such as careers, privacy policy etc. Please let us know why is this the case ? Also does these pages have impact on overall rankings of the website ?, If yes, What should we do about it ? Please clarify. Thanks.0 -
Open site explorer: error message "There was an error getting your data"
I"m trying out MOZ, wanting to make an informed decision (for myself and my blogging students) whether this is a useful program to buy and use. I've had no luck so far with the Open Site Explorer. Each time I try, I get the same error message, "There was an error getting your data". I used the URL: http://writetodone.com I asked about the problem in the chat, but my question was posed over an hour ago and is still marked 'unread'... I"m keen to continue studying the program as I'm creating a video about MOZ. How can I make the Open Site Explorer work? I've tried the Analytics module but it seems to take some days to get results. Mary "There was an error getting your data"
Link Explorer | | MaryJaksch0 -
Error message coming up for Open Site Explorer
When in Open Site Explorer there seems to be an error getting the data for http://vagabondtoursofireland.ie/ or www.vagabondtoursofireland.ie I have used this with other websites and have never had a problem. Thanks.
Link Explorer | | Johnny_AppleSeed0 -
Does the inbound links report include links to all pages of the domain being researched?
If I enter 'abc.com' am I only getting results for 'abc.com' or will I get results for the internal pages of 'abc.com' as well (i.e. 'abc.com/page1.html)? There is a bit of a discrepancy between these results and inbound link results in semrush for example. Then again it seems whenever you use different tools to measure the same thing you get wildly varying results. How do you all deal with that?
Link Explorer | | AISEO0