Why is Moz unable to crawl my site?

[jbarinaga]

Was hoping someone could advise why Moz is unable to crawl my site at https://www.oceaniacruises.com

**Moz was unable to crawl your site on Oct 5, 2017. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.

Any help would be appreciated. Thanks!

[jbarinaga]

Hi Dave,

Any thoughts on the above question?

[jbarinaga]

Hi Dave,

Unfortunately, our IT team has not been able to replicate the issue and can access the robots.txt file on the server side.

Is it possible to get more information (Moz logs, traces, etc…) from the error as a 503 error could be a result of multiple issues?

[EarlyS]

Hello I'm getting the same error as seen above on my website http://www.hadcoltd.com/. Can you please advise on what might be causing it or what I can do to fix it?

Thank You

[dave.kudera]

Hey!  Dave here from the Help Team.

It looks like we are getting a 503 status code when trying to access your robots.txt file.  You will want to check your server logs for the response from User-agent: rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+aardwolf@moz.com) and make sure you are not blocking us from accessing your site!  Hope that helps