Setting up MOZ to run on Staging
-
Hi Moz,
We would like to setup Moz to run on our Staging Server. This would be extremely valuable as it alerts us to new SEO issues/risks in a controlled and secure environment that is not exposed to production.
Our internal team has recommended potentially setting up a reverse proxy server that will validate either via Moz's/Rogerbot's http header or IP and allow Moz access to our Staging environment to crawl.
Is this something that we can setup with Moz? Are there other ideas to enable Moz to crawl our Staging server?
-
I know this is old - but I was also wondering this same thing.
Our specific use case is that we would love to be able to run a on-page grader via Moz on our staging environment before going live to make sure we are not degrading on-page seo for any specific keywords when changing things around to new designs.
-
No worries!
You can add a directive to your site's robots.txt file as such:
User-agent: rogerbot
Disallow:User-agent:*
Disallow: /This will block all bots from accessing your site aside from rogerbot.
There may be other methods as well, which I am personally not aware of. Perhaps other users on the forum may be able to provide tips on what you can potentially try out.
Let me know if you have any other questions!
Eli
-
Hi Eli,
Thanks for the fast response.
Creating a public facing 'staging' instance does create a certain degree of risk as it can potentially be discovered and indexed.
Are other clients currently using this method? If so what best practices do you recommend we follow if we were to go this route to ensure that our 'Staging' site is ONLY crawled by Rogerbot and not by any other bot or service.
-
Hey!
Thanks for reaching out to us!
In order to track a new site you would indeed need to create a new campaign. You can do that here https://analytics.moz.com/campaigns/new . Each campaign tracks the exact URL entered when it's created.
Unfortunately you would not be able to validate us by IP address alone as we do not use a static IP address or range of IP addresses, as we have designed our crawler to have a dynamic approach. This means we use thousands of dynamic IP addresses which will change each time we run a crawl. We believe that this approach gives us the best dynamic view of the web!
However, one alternative I'd be able to suggest would be to identify our crawler by User-agent: rogerbot. You can read more about rogerbot in our guide. It's also worth pointing out that your site would need to be publicly available for our crawler to reach and crawl it.
I hope this helps, please do let me know if you would like more clarification or have any other queries.
You're welcome to reach out to help@moz.com if it's easier!
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz isn't recognizing content on our website hosted on bubble.is
Hi all. We have a bubble.is site and it's difficult to get Moz to recognize the text on the site (even though Google seems to recognize it most of the time). The pages are recognized...just not the text. **In case you want to play around: ** https:getmarlow.com/about https://getmarlow.com/article/gather-data-on-where-your-time-and-energy-go-1579415891021x868278876529492000
Moz Pro | | Marlow20 -
Moz is treating my pages as duplicate content but the pages have different content in reality
Attached here is a screenshot of links with duplicate content. Here are some links that is also based on the screenshot http://federalland.ph/construction_updates/paseo-de-roces-as-of-october-2015 http://federalland.ph/construction_updates/sixsenses-residences-tower-2-as-of-october-2015/ http://federalland.ph/construction_updates/sixsenses-residences-tower-3-as-of-october-2015 The links that I have placed here have different content. So I don't why they are treated as duplicates BWWJuvQ
Moz Pro | | clestcruz0 -
Any tool built into MOZ that can help tell who the owner of a URL is?
I'd like to know if there's any tool which would let us know who the owner of a web domain is.
Moz Pro | | daleseppie0 -
Moz crawl only shows 2 pages, but we have more than 1000 pages.
Hi Guys Is there anyway we can test Moz crawler ? it showing only 2 pages crawls. We are running website on HTTPS ? Is HTTPS is issues for Moz ?
Moz Pro | | dotlineseo0 -
SEO moz Report Card
I just ran some on page report cards. As I was playing around with the tool I noticed that I would get different results if I used my primary domain vs a 2nd domain. The main difference was in how the tool was counting keywords on the page. The keyword used was 'vehicle inventory' Primary domain: www.brand-state.com/inventory.htm Title = 1, URL = 0, Meta = 1, H1 = 1, H2-4 = 1 Body =1, Strong = 1, IMG Alt = 1 Total = 7 2nd domain: www.company-name-brand.com/inventory.htm Title = 1, URL = 0, Meta = 1, H1 = 1, H2-4 = 2 Body =5, Strong = 4, IMG Alt = 2 Total = 13 I can understand if the keyword was in the domain, but it's not. So I'm wondering what is going on here - any help or suggestions on what to research would be a great help. Thank you!
Moz Pro | | gormaniavt0 -
How to set the crawler or reports to ignore
I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?
Moz Pro | | Str82u0 -
How Long Does a SERP Analysis Report Normally Take to Run?
I started running a SERP analysis report two days ago, and it still says it's "in progress." It is 5:40 PM Pacific on August 7 as I type this, and I started running the report on: August 05, 2012 10:03 (copied and pasted from the report). Is this normal? Thanks!
Moz Pro | | ScottShrum0 -
Cant understand Seo moz analysis
Did a semoz analysis of site www.1stclassdriving.co.uk and it came up with 484 pages with duplicate titles and 484 with duplicate content. Also gave me a 531st ranking for search term "Driving Lessons" (with many smaller schools with better rankings)- so I think that I may have a problem 1-Duplicate titles - I have many area pages - one for each of the post codes that we cover (some 40 odd) Each page is titled "Driving Lessons in "xxx" where xxx is the area.
Moz Pro | | Brian_Worger
Plus each Instructor has their own page which is tiled "Driving Lessons in xxx" where xxx is the more local towns/areas covered.
Is this the use of the term Driving Lessons xxx on each page the problem? The report doesnt seem to make this clear -it simply lists each page on the site 2- Duplicate content puzzles me as the content does differ although each instructor page has probably 80% content derived from a master page. Attach screenshot - hope someone can help Many thanx Brian0