Blocked Production Site from Search Engines - How to get it Crawled by Moz Crawler
-
I have an 'under development' site hosted, (which is an exact replica of live site as working on to add new functionalities & modules) - but its password protected, excluded from robots.txt (Disallow) & also marked noindex on all pages in the index - so that Googlebot & other Search Engines can not crawl the site
At present the development work is almost 95% completed., Now - feel like to crawl the site through SEOMOZ Roger Bot - to know the errors and all indexed urls by Rogerbot.
What's the best way to get Moz Bot crawl the site - but simultaneously continue it blocking its access to Search Engines
I have gone through - https://support.google.com/webmasters/answer/93708?hl=en, it says
a) Save it in a password-protected directory. Googlebot and other spiders won't be able to access the content- But this way Moz will also not be able to crawl the site
b) Use a robots.txt to control access to files and directories on your server - However it also says - It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index.
c) Use a
noindex
meta tag to prevent content from appearing in our search results - It also says that a link to the page can still appear in their search results. Because we have to crawl your page in order to see the noindex tag, there's a small chance that Googlebot won't see and respect the noindex meta tagPassword Protected thus seems the best way to continue blocking. However, continuing with it will also block Moz bot to crawl the site. Any suggestions
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are we actually getting accurate data on keyword volumes from Moz (or other sources)?
I have a client who does patio furniture repair and restoration. When performing keyword research in Moz for terms like "patio furniture repair" I see that only 11-50 people in the entire US are searching for this term according to the Moz data. However, running an Adwords campaign currently and our top keyword is the phrase match for "patio furniture repair" which has generated over 100 clicks in just a couple of months in ONE county. Is there a better way to research more accurate results on search volume estimates? This makes organic SEO and keyword targeting hard! Thanks, Ricky
Moz Bar | | RickyShockley1 -
Crawling Problem : No-data Of My Site
Hey, Folks!! So It's Been More than a Month.And Moz is not crawling my website http://www.trickypedia.com/.I have seen Moz Last update I thought they will crawl my site but No results Disappointing. My Sites Domain Authority and Page Authority is still 1. But In other Seo Tools, they are Perfectly Crawling My Website. What Would Be the reason? Can anyone Please Explain.
Moz Bar | | seothatworks010 -
Moz Pro Question: Does the amount of keywords you are allowed to search reset each month?
I am a Moz Pro subscriber and I really love the new Keyword Explorer tool. One question I have that I couldn't find a clear answer was regarding the number of monthly keyword queries. Do they reset each month? I hope they do.
Moz Bar | | joemaclean0 -
My crawl report only shows 1 link
Hello, I've tried a crawl for the site www.doctify.co.uk and it's only returned 1 link in the report which is the homepage. Do you know what the issue could be? Thanks, Nina
Moz Bar | | Global_Blue0 -
Link to hotels on http://moz.com/mozcon doesn't work
Hi The link to the hotel for Mozcon 2015 doesn't work - seems like its the 2014 link still in place. Thanks Andy
Moz Bar | | Andy-Halliday0 -
I have been unable to use the export button on any of the Moz pages.
Every time I click the export button on the upper right portion of the page, it says "preparing to export", but either nothing is exported or a blank document is downloaded. I was specifically working with the Fresh Web Explorer but have noticed this on all pages. Is anyone getting this export function to work?
Moz Bar | | Manseo0 -
Did the Crawl Test tool go away or was it replaced
I loved that tool as it provided me with all of my URLs and it was easy to catch all errors at once. I had it booked marked but now I am just going to the regular tools page.
Moz Bar | | KJ-Rodgers0