Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How do get Moz to spider a Development site PRE LAUNCH?

Getting Started

778

bjs2010 last edited by

Hi,

Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it?

Thanks
1 Reply Last reply
Reply Quote 0
MattRoney @Keszi last edited by

That would do it! Try this in robots.txt:

User-agent: Rogerbot
Disallow:

User-agent: *
Disallow: /

Keep in mind that this will only block crawlers that obey robots.txt. Once Roger crawls your site, it may be a good idea to lock the dev site down entirely.
1 Reply Last reply
Reply Quote 1
Keszi last edited by

Well, you could block Google and Bing from crawling your website (via Robots.txt) and run a Crawl via Moz, that's possible I think.

Istvan
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Has anyone purchased the MOZ SEO courses? Are they good?

moz academy

I'm looking to learn more about SEO. Has anyone purchased the courses available through MOZ? If so, are they useful?
Getting Started | | Stevepair

0
Moz not recognising url to setup a new account

I am attempting to setup a new account for my client but Moz is having a problem and not registering the url http://www.esc.uk.net/. I get the "oops" message & read the info on the FAQ page it refers to. Why do I keep getting an "Oops" message when I type and submit my campaign URL? To create a campaign, we need to receive a valid HTTP response from the web server hosting your site. While the site may be accessible from a standard web browser, other user-agents and search engine bots may receive a different response. If you receive this error, we recommend running a test by sending an HTTP response using any online HTTP tool. Using an online HTTP tool I believe I received an HTTP response. What else can I do? Regards Terry
Getting Started | | Buzzin

0
Moz is grading pages based on keywords that are not the correct keywords for the page. I'm getting F grades when I should be getting As because of this. how do I fix the keywords moz is using?

Help with page grades.
Getting Started | | GMacT

1
Getting errors -- Pages with Title Missing or Empty

I am getting a high priority warning that I have pages with title missing or empty, however, when I check the page, there is a title tag <title>Long Term Care Planning</title> . Anyone have any idea what could be doing on? I check the CMS (using AEM - adobe experience manager CQ5) and the title field is filled out as well.
Getting Started | | Laura-Genworth

0
I don't believe moz is seeing everything that is on my webpage

I used the page key word grader and got an "F" Moz said that my keyword employee handbook was not in my title nor was it found in the body of my page. But when I look at the page and double check everything it is there all over the place. I am not blaming moz this is a wiz site and while I am a beginner and very well could be wrong could anyone just take a look and tell me if I am nuts or what. The web page is http://www.cestoday.com/#!employee-handbook/co0h I now have the font so big I will have to fix that. Thank you
Getting Started | | redsman944

0
Moz profile

Hi, For some reason when I try and add my site's link in my Moz profile I keep getting an error. My site an SSL so I used https instead of http. I tried again without the 's' as my domain redirects to the https version, but this didn't work. Then I tried without http or https and this didn't work either. Why does Moz hate my site? It's actually really good!!! Thanks, Amelia
Getting Started | | CommT

0
New to MOZ and working with Web Mentions. Can I use operators?

Our name is HostDime but often put as Host Dime (2 words) by news sources and other sites. How do I set up my brand mention so I only get a notice when both words appear, in order, together. I don't want "That host is a dime" and such. Can I use a +Host +Dime?"Host Dime"? Do these operators work in MOZ?
Getting Started | | hostdime

0
Why does moz show "not in top 50" for all my keywords???

Hello, I signed up to moz pro 4 days ago. And so far it seems to be tracking visits etc. But all my keywords say "not in top 50" . Why is this? Is this normal? Just to confirm most of the keywords i pasted in from my webmaster tools and i only chose the ones that were in top 50
Getting Started | | casper0903

0