Setting up MOZ to run on Staging

[kriskunisch]

Hi Moz,

We would like to setup Moz to run on our Staging Server. This would be extremely valuable as it alerts us to new SEO issues/risks in a controlled and secure environment that is not exposed to production.

Our internal team has recommended potentially setting up a reverse proxy server that will validate either via Moz's/Rogerbot's http header or IP and allow Moz access to our Staging environment to crawl.

Is this something that we can setup with Moz? Are there other ideas to enable Moz to crawl our Staging server?

[hcaruso4]

I know this is old - but I was also wondering this same thing.

Our specific use case is that we would love to be able to run a on-page grader via Moz on our staging environment before going live to make sure we are not degrading on-page seo for any specific keywords when changing things around to new designs.

[A Former User]

No worries!

You can add a directive to your site's robots.txt file as such:

User-agent: rogerbot 
Disallow:

User-agent:* 
Disallow: /

This will block all bots from accessing your site aside from rogerbot.

There may be other methods as well, which I am personally not aware of. Perhaps other users on the forum may be able to provide tips on what you can potentially try out.

Let me know if you have any other questions!

Eli

[kriskunisch]

Hi Eli,

Thanks for the fast response.

Creating a public facing 'staging' instance does create a certain degree of risk as it can potentially be discovered and indexed.

Are other clients currently using this method? If so what best practices do you recommend we follow if we were to go this route to ensure that our 'Staging' site is ONLY crawled by Rogerbot and not by any other bot or service.

[A Former User]

Hey!

Thanks for reaching out to us!

In order to track a new site you would indeed need to create a new campaign. You can do that here https://analytics.moz.com/campaigns/new . Each campaign tracks the exact URL entered when it's created.

Unfortunately you would not be able to validate us by IP address alone as we do not use a static IP address or range of IP addresses, as we have designed our crawler to have a dynamic approach. This means we use thousands of dynamic IP addresses which will change each time we run a crawl. We believe that this approach gives us the best dynamic view of the web!

However, one alternative I'd be able to suggest would be to identify our crawler by User-agent: rogerbot. You can read more about rogerbot in our guide. It's also worth pointing out that your site would need to be publicly available for our crawler to reach and crawl it.

I hope this helps, please do let me know if you would like more clarification or have any other queries.

You're welcome to reach out to help@moz.com if it's easier!

Eli