Setting up MOZ to run on Staging
-
Hi Moz,
We would like to setup Moz to run on our Staging Server. This would be extremely valuable as it alerts us to new SEO issues/risks in a controlled and secure environment that is not exposed to production.
Our internal team has recommended potentially setting up a reverse proxy server that will validate either via Moz's/Rogerbot's http header or IP and allow Moz access to our Staging environment to crawl.
Is this something that we can setup with Moz? Are there other ideas to enable Moz to crawl our Staging server?
-
I know this is old - but I was also wondering this same thing.
Our specific use case is that we would love to be able to run a on-page grader via Moz on our staging environment before going live to make sure we are not degrading on-page seo for any specific keywords when changing things around to new designs.
-
No worries!
You can add a directive to your site's robots.txt file as such:
User-agent: rogerbot
Disallow:User-agent:*
Disallow: /This will block all bots from accessing your site aside from rogerbot.
There may be other methods as well, which I am personally not aware of. Perhaps other users on the forum may be able to provide tips on what you can potentially try out.
Let me know if you have any other questions!
Eli
-
Hi Eli,
Thanks for the fast response.
Creating a public facing 'staging' instance does create a certain degree of risk as it can potentially be discovered and indexed.
Are other clients currently using this method? If so what best practices do you recommend we follow if we were to go this route to ensure that our 'Staging' site is ONLY crawled by Rogerbot and not by any other bot or service.
-
Hey!
Thanks for reaching out to us!
In order to track a new site you would indeed need to create a new campaign. You can do that here https://analytics.moz.com/campaigns/new . Each campaign tracks the exact URL entered when it's created.
Unfortunately you would not be able to validate us by IP address alone as we do not use a static IP address or range of IP addresses, as we have designed our crawler to have a dynamic approach. This means we use thousands of dynamic IP addresses which will change each time we run a crawl. We believe that this approach gives us the best dynamic view of the web!
However, one alternative I'd be able to suggest would be to identify our crawler by User-agent: rogerbot. You can read more about rogerbot in our guide. It's also worth pointing out that your site would need to be publicly available for our crawler to reach and crawl it.
I hope this helps, please do let me know if you would like more clarification or have any other queries.
You're welcome to reach out to help@moz.com if it's easier!
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No Ranking Keywords in Moz
I am using the link explorer to search my website https://bestsportsmask.com/ in Moz. But it is showing no ranked keywords but in other tools it's showing some keywords. What could be the issue.
Moz Pro | | DanielMaier0 -
What Moz reports would you suggest running when pitching SEO services to a client?
What do you think the best Moz reports are when presenting SEO services to a client? I need to share reports that show where the opportunity is and where we will be prioritizing our effort. I'd like to hear what you would suggest given your experience presenting SEO to clients. Thanks in advance for your help!
Moz Pro | | mrupert1 -
Any tool built into MOZ that can help tell who the owner of a URL is?
I'd like to know if there's any tool which would let us know who the owner of a web domain is.
Moz Pro | | daleseppie0 -
Is the keyword ranking number used for MOZ Pro an average position taken across the time period?
Hi there I am just trying to confirm how the ranking given in the 'rankings' report is calculated (an average etc). Thanks Sarah
Moz Pro | | Laurie-Tomahawk0 -
Moz campaign works around my robots.txt settings
My robots.txt file looks like this: User-agent: * Disallow: /*? Disallow: /search So, it should block (deindex) all dynamic URLs. If I check this url in Google: site:http://www.webdesign.org/search/page-1.html?author=47 Google tells me: A description for this result is not available because of this site's robots.txt – learn more. So far so good. Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors. One of the links is this one: http://www.webdesign.org/search/page-1.html?author=47 (the same I tested in Google and it told me that the page is blocked by robots.txt which I want) So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
Moz Pro | | VinceWicks0 -
Moz Trust
I saw a severe drop in MozTrust for multiple sites I am working on. We have added some links this month, but would be considered 'spammy' or to be low quality. No other links appear to have been lost / removed. If we had some very high quality links before and have added some mid-level links can this be what is causing the drop? From the description on SEOmoz, it seemed to me that trust could only go up as links were added. It appears that I am missing something.
Moz Pro | | DigitalDiameter0 -
How can I set up user account for differnt people within one account?
We have multiple team members and would like to setup unique profiles for each instead of one generic profile. Is that possible?
Moz Pro | | Tune_Up0 -
Trouble setting up a campaign
Hi Im brand new here - so not sure what im doing wrong. When I try to add a campaign it comes back and says - may not have a /path after the host . It doesnt seem to matter whether I put a path in or not it still gives same answer, so I cant launch my campaign. Help?? Thanx Kadira
Moz Pro | | Dragonfly70