Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
IT is blocking AWS via a proxy in front of our server. We've tried allowing the "roberbot" user-agent but crawling functionality still isn't working in my Moz Pro account. Is there a more specific user-agent we can allow in our proxy software? Thank you.
Thank you for your response but this doesn't answer my question. We aren't blocking rogerbot using robots.txt. We need to allow it through the proxy in front of our web server by using the exact user-agent (case sensitive) that is being sent by rogerbot. We've tried "rogerbot" but that isn't working. Based on the 3rd party documentation you linked to there seem to be a variety of possibilities:
roberbot/1.0
roberBot/1.0
RogerBot/1.0
rogerBot
RogerBot
It would be great if Moz provided clear documentation on this.