Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How do get Moz to spider a Development site PRE LAUNCH?

Getting Started

771

bjs2010 last edited by

Hi,

Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it?

Thanks
1 Reply Last reply
Reply Quote 0
MattRoney @Keszi last edited by

That would do it! Try this in robots.txt:

User-agent: Rogerbot
Disallow:

User-agent: *
Disallow: /

Keep in mind that this will only block crawlers that obey robots.txt. Once Roger crawls your site, it may be a good idea to lock the dev site down entirely.
1 Reply Last reply
Reply Quote 1
Keszi last edited by

Well, you could block Google and Bing from crawling your website (via Robots.txt) and run a Crawl via Moz, that's possible I think.

Istvan
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

How I can increase DA of my site?

Hi, I have my 4 month old blog, PA of site is 17 but DA is still 5. I don't know how to increase DA of site. Please suggest me how to increase DA of Site https://myeasygrader.com/ . Thanks
Getting Started | | markwillson

0
Why does the moz bar link count differ from the moz link analysis page

Hi all, Why does the Moz Bar show a different link count from the Moz link analysis page? For example, when I check the SERP below, for the first result, the bar shows 936 page links from 4 RDs. But when I check out the link analysis page, it tells me there are just 141 page links from 4 RDs. What gives? For the second entry, the bar shows 6 page links from 0 RDs. Not sure how that's possible. Can anyone explain these things. Thanks! Andy Reviewed SERP: https://www.google.com/search?source=hp&ei=hhRVW5yyH5C60PEP-_-isAw&q=mountain+bike+trails+near+me&oq=mountain+bike+trails+near+me&gs_l=psy-ab.3..0l7j0i22i30k1l3.645.4654.0.5322.28.19.0.8.8.0.243.2920.3j11j5.19.0....0...1c.1.64.psy-ab..2.26.2961...0i131k1.0.C4lAxLkGgH0
Getting Started | | AndyKubrin

0
Moz can't crawl my site.

Moz cannot carry out the site crawl on my online shop. Not really sure what the issue is, it has no problem getting onto my site when you use www. before the address, but it needs to be able to access bluerinsevintage.co.uk Stuck as what to do, we are a shopify store. Anyone else had this problem, or know what i need to change so they can crawl the site? thjis is the page they are getting when trying to get on bluerinsevintage.co.uk but if they use www.bluerinsevintage.co.uk the site comes up. Adam
Getting Started | | bluerinsevintage

0
Standard Syntax in robots.txt doesn't prevent Moz bot from crawling

A client is getting many false positive site crawl errors for things like duplicate titles and duplicate content on pages that include /tag/ in the URL. An example is https://needquest.com/place_tag/autism-spectrum-disorder/page/4/ To resolve this we have set up a disallow statement in the robots.txt file that says
Disallow: /page/ For some reason this appears not to work, as the site crawl errors continue to list pages like this. Does anyone understand why that would be and what we need to do to properly disallow crawling these pages?
Getting Started | | btreloar

0
How do i stop Moz from indexing my dev site?

Hey, I am new to MOZ, and I am seeing in my page reports that I have duplicate content, but this duplicate content is mostly my development site. dev.domain.com. I have this blocked in robots.txt for google. How do i stop MOZ including it in its reports? Cheers everyone.
Bart
Getting Started | | Tholomew1

0
New to Moz Pro? Join our upcoming free webinar this Friday!

Hello everyone! We'll be holding a webinar on Friday to help new members learn about what all Pro has to offer, show some off our most popular tools, and get you comfortable with the dashboard. Register here**: https://www3.gotomeeting.com/register/132016958** Date: Friday, October 17th Time: 10:00 AM - 11:00 AM PDT Hope to see you all there! If you can't make it, you can watch a previous webinar here: http://moz.com/help/guides/getting-started
Getting Started | | jennita

4
Moz Pro won't stay connected to my Google Analytics profile

I just started a trial of Moz Pro today and it seems to have a problem staying connected to my Google Analytics account. Evenr time I refresh a page or click to a different page, I get a notification that reads "Our connection to your Google Analytics account was lost. Don't worry, you won't lose any data. Please reauthorize now." I click the link to reauthorize and it says it was successful, But then as soon as I refresh the page or click to a new page withing Moz Pro, the message appears again. I'm afraid that none of my GA data is being collected by Moz Pro. What should I do?
Getting Started | | InterCall

0
Getting Redirect Loops in MOZ using Chrome

Been getting bizarre Redirect Loops from Chrome after I log-in to MOZ. Has anyone had something like this happen? I've tried clearing cache, rebooting, etc. but no luck. Thanks in advance!
Getting Started | | danny.wood

1