612 : Page banned by error response for robots.txt
-
Hi all,
I ran a crawl on my site https://www.drbillsukala.com.au and received the following error "612 : Page banned by error response for robots.txt."Before anyone mentions it, yes, I have been through all the other threads but they did not help me resolve this issue.
I am able to view my robots.txt file in a browser https://www.drbillsukala.com.au/robots.txt.
The permissions are set to 644 on the robots.txt file so it should be accessible
My Google Search Console does not show any issues with my robots.txt file
I am running my site through StackPath CDN but I'm not inclined to think that's the culpritOne thing I did find odd is that even though I put in my website with https protocol (I double checked), on the Moz spreadsheet it listed my site with http protocol.
I'd welcome any feedback you might have. Thanks in advance for your help.
Kind regards -
Hey there! Tawny from Moz's Help Team here.
After doing some quick searching, it looks like how you configure the rules for WAFs depends on what service you're using to host those firewalls. You may need to speak to their support team to ask how to configure things to allow our user-agents.
Sorry I can't be more help here! If you still have questions we can help with, feel free to reach out to us at help@moz.com and we'll do our best to assist you.
-
Hi, I am having the same issue.
Can you please tell me how you have created rule in Web Application Firewall to allow user agents rogerbot and dotbot.
Thanks!!
-
Hi Federico,
Thanks for the prompt. Yes, this solution worked. I'm hopeful that this thread helps others too because when I was troubleshooting the problem, the other threads were not helpful for my particular situation.
Cheers
-
Hi, did the solution work?
-
Hi Federico,
I think I have found the solution for this problem and am hopeful the crawl will be successful this time around. Based on further digging and speaking to the team at StackPath CDN, I have done the following:
- I added the following to my robots.txt file
User-agent: rogerbot
Disallow:User-agent: dotbot
Disallow:- I added a custom robots.txt file in my CDN which includes the above and then created a rule in my Web Application Firewall which allows user agents rogerbot and dotbot.
I'll let you know if the crawl was successful or not.
Kind regards
-
Thanks for your response Federico. I have checked my robots.txt tester in my Google Search Console and it said "allowed."
Oddly, it also happened on another site of mine that I'm also running through StackPath CDN with a web application firewall in place. This makes me wonder if perhaps the CDN/WAF are the culprits (?).
I'll keep poking around to see what I find.
Cheers -
Seems like an issue with the Moz crawler, as the robots.txt has no issues and the site loads just fine.
If you already tested your robots.txt using the Google Webmaster Tools "robots.txt Tester" just to be sure, then you should contact Moz here: https://moz.com/help/contact/pro
Hope it helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When will Dotbot see changes to my robots.txt file?
Dotbot doesn't want to know anything about my site, my Page Authotize is still 1 and my DA is the same. How long does it take to start getting results? In all the websites we have been working on, we have no problems, but in ours it has come up.
Link Explorer | | Expansyon1 -
404 Error - Please Help us
When we checked valuable Top pages, we noticed two types of 404 pages listed in our domain Example 1 : www.test.com/www.test.com/ecommerce.html Example 2 : www.test.com/test/ecommerce.html But we do not see any such 404 page errors in the Google Webmaster tool. Moz Top Pages Section only shows these as errors So please advise, if these are major errors or not? If these are errors, please help us to fix this as we do not have such URLs in our domain Awaiting your urgent help
Link Explorer | | Intellect0 -
Why Page Authority = 1
Hi everyone! Long time reader, first time-submitter. I'm working with a client who has an old, established domain and brand name, decent DA score (43 overall, 52 for their home page), but almost all the rest of the pages of their small-ish website have a PA of 1. The domain itself has been around since 2000. The pages ranking at a 1 are not new URLs, they have a sitemap submitted to Google, are not being blocked by robots.txt, there's no auto-redirects or meta-refresh, and the response code for all pages is 200. What else am I missing? What could be causing this? TIA for the help!
Link Explorer | | camarin_w0 -
How is Moz Page Authority Calculated?
Can anyone refer me to a document that explains the measures that go into the calculation of Moz Page Authority? I'm chiefly interested in what counts and what doesn't -- follow links, no-follow links, internal and external, what (if any) on-site factors, etc.
Link Explorer | | GlennFerrell0 -
When moz will index my page?
HI. I use moz pro for 1.5 month and yet my site is not indexed and i do not see Competitive Link Metrics. Can you please include meko.lv in index?
Link Explorer | | Mekounko0 -
Do other sites linking to my page with a low domain authority affect my rankings?
Hello, my page authority 30, I see other page with an authority of 21 for example linking to us. Is this good or bad, does this affect our ranking? A lot of these don't have anchor text, I'm guessing we should reach out to these webmaster to include anchor text with the keywords we want to drive traffic to this page or make as descriptive as possible? Thanks! Laura
Link Explorer | | lauramrobinson320 -
Open Site Explorer Top Pages https Title tag issue
Hello, everybody. I've noticed this strange thing in Top Pages reporting of Open Site Explorer. If i do a report for any website which doesn't have security certificate installed, such as http://www.hyperlinksmedia.com I get normal results, with title tag shown for every page, but, if i do a report for website with Security certificate, such as https://www.hyperlinksmedia.com, report is coming back for non-secure URL version (http:...), so, it says "No Title" for any pages. I wonder, if it influence PA and DA scores. Although, if i run full crawl test, then i shows all meta and title tags for SSL version of a website's url. Thanks!
Link Explorer | | seomozinator0 -
Page Authority Migration
PageA.html has a lot of links and has a Page Authority score. It is 301 redirected to PageB.html and eventually in OSE this also accumulates a Page Authority score. Is there a reason why PageA.html many months/years after continues to have a Page Authority when it is expected that its Authority signals have migrated (with any dampening effect etc) across to PageB.html.
Link Explorer | | Infotex_UK0