How do get Moz to spider a Development site PRE LAUNCH?
-
Hi,
Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it?
Thanks
-
That would do it! Try this in robots.txt:
User-agent: Rogerbot
Disallow:User-agent: *
Disallow: /Keep in mind that this will only block crawlers that obey robots.txt. Once Roger crawls your site, it may be a good idea to lock the dev site down entirely.
-
Well, you could block Google and Bing from crawling your website (via Robots.txt) and run a Crawl via Moz, that's possible I think.
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What are the top 5 things you recommend you should do when you first subscribe to Moz?
Hi
Getting Started | | djnlowe
Im new to Moz and would like to get the most out of it as quickly as possible. so like the subject say - What are the top 5 things you would recommend I should to get the most out of Moz? #getstarted0 -
KD according to MOZ.
Hello, I want to know how moz calculates the keyword difficulty? What are the basic things which moz pro consider while calculating keyword difficulty? I wanna Calculate the KD of JB Battery Keywords,
Getting Started | | Shafqat_mehmood0 -
I want to increase the DA of my Site
Dear all i have established a new site and want to increase its DA fast. So i need your suggestions in this regard. This newly established blog is my first independant project so i want to make it perfect.
Getting Started | | hamza52520 -
Moz Free
hello guys, is there any way to use Moz free or any well-known company who would like to give me Moz login at a very cheap price. if anyone knows then please guide me shortly. Thanks With Regards, Abert sin.
Getting Started | | seochris172 -
How long will it take to get google Analytics results back so I can get rolling?
I just signed up for Moz. I'm wondering how long it will take until Google Analytics results will enter.
Getting Started | | John-Francis0 -
Need to Add Moz code?
Do i need to add tracking code in my website, for Moz tool to analyze my website. Please someone let me know.
Getting Started | | TOBOC0 -
Is the Moz Rank Tracker Broken?
Hi, I've waited a few days for data to appear in the account but all of the keywords show 'not in top 50'. When I know this isn't the case since I can see half of them myself either on page one or two (yes I have cleared cache / history and have tried an incognito window to see the results for myself; I have even tried another PC. Also I'm really not keen on the new layout of the pro account, everything is so much smaller event the text. Personally I think it doesn't look half as clean and simple as it used to. Another thing, the link analysis tab isn't showing all of my links - it's only showing 2! Which is way off, even when I check with Open Site Explorer there are a lot more than that. Thanks Ricky
Getting Started | | Thirsty-Media0 -
Why Moz Pro Campaign overview display not properly?
when i browse my campaign, i see like this on my computer http://i.imgur.com/7G9IRT8.jpg then when i see source code of http://pro.cdn.seomoz.org/stylesheets/campaigns/overview.css?1378244858 <html><head><title>404 Not Foundtitle>head><body bgcolor="<a class="attribute-value">white</a>"><center><h1>404 Not Foundh1>center><hr><center>nginxcenter>body>html> what happen??
Getting Started | | dimazm0