Blocked Production Site from Search Engines - How to get it Crawled by Moz Crawler
-
I have an 'under development' site hosted, (which is an exact replica of live site as working on to add new functionalities & modules) - but its password protected, excluded from robots.txt (Disallow) & also marked noindex on all pages in the index - so that Googlebot & other Search Engines can not crawl the site
At present the development work is almost 95% completed., Now - feel like to crawl the site through SEOMOZ Roger Bot - to know the errors and all indexed urls by Rogerbot.
What's the best way to get Moz Bot crawl the site - but simultaneously continue it blocking its access to Search Engines
I have gone through - https://support.google.com/webmasters/answer/93708?hl=en, it says
a) Save it in a password-protected directory. Googlebot and other spiders won't be able to access the content- But this way Moz will also not be able to crawl the site
b) Use a robots.txt to control access to files and directories on your server - However it also says - It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index.
c) Use a
noindex
meta tag to prevent content from appearing in our search results - It also says that a link to the page can still appear in their search results. Because we have to crawl your page in order to see the noindex tag, there's a small chance that Googlebot won't see and respect the noindex meta tagPassword Protected thus seems the best way to continue blocking. However, continuing with it will also block Moz bot to crawl the site. Any suggestions
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I 'sign in' to the Moz Bar?
It's installed, I can see links etc as highlighted - but it won' t let me "sign in". This 20 second video explains: https://www.screencast.com/t/3kEjQFkTHZv Suggestions? Or shall I just ignore? Paul Barrs
Moz Bar | | PaulBarrs0 -
Why RogerBot can't crawl site https://unplag.com
Hello Please help me to solve the problem. The on-page grader and Crawl Test are not working for Unplag.com website. Both said that they can't access the url. Yes, I've tried different variants like unplag.com, http://unplag.com One more thing - RogerBot was disallowed in robots.txt file. I deleted it from the file a week ago so maybe moz index haven't been renewed.
Moz Bar | | Targeras0 -
Keyword Difficulty tool showing Google search volume or Bing?
Hi, I plugged in a bunch of keywords to know their search volume, chose Google from the drop down and then chose English from the next dropdown and clicked on Check dificulty. It shoes me "Bing Search Volume (Exact Match)" on top but shows the name of the keyword and "for Google CA" on the left. Is the search volume for Google? or Bing? and the difficulty as well? Google or Bing? "
Moz Bar | | SpaMedica0 -
Moz Crawl Showing Duplicate Content But It's Not?!
Unfortunately I can't give out the URL, but here's the deal... I have two URL's which have completely different content on them but are being crawled as duplicate content. Any Idea how that would happen? I'm not seeing any errors in WMT's. Has anyone seen this before? Is the duplicate content reporting based on a % of the page content matching as the same?
Moz Bar | | Swarm-SEO0 -
Moz Crawl Test: Referrer is sitemap.gz?
Hi,
Moz Bar | | Titan552
I'm looking at a crawl test report, and I'm seeing that most of the pages have the sitemamp.gz file listed as the referrer. As I recall in my other reports the referrer is usually the root domain - unless of course there's a redirect. Does having sitemap.gz as the referrer indicate a problem? If so, what problem does it indicate? Thanks!0 -
On Page Grade - Moz Analytics
When I go to search overview I can see my on page optimization grades, but when I try to use the on page grade optimizer for different keywords and url's I get the error. "The URL you entered does not appear to be returning a page successfully. Please make sure that you've entered the URL of valid, working page." However, I know it is a valid url and I get the error even if I use the same url and same keyword I can see the grade for from the search overview. Also, if I try to regrade the page nothing happens and where my previous grade and analysis was located I now have nothing. Any information about why this is happening? Thanks for the help.
Moz Bar | | ZeroWing0 -
How do I get Moz to crawl my site?
If I'm not mistaken, Moz will crawl my site once a week and let me know of any errors, notices etc.. But can I ask them to crawl once I have updated all the errors? or do I have to wait for them to do this automatically?
Moz Bar | | david.smith.segarra0 -
Duplicate Page Content Report on MOZ
Hi, I am just wondering as to the accuracy of this report - does it pick up all the duplicate on page content? Or is there a limit? We have an ecommerce store with a lot of copied and pasted descriptions - just wondering if there is a limit on how much the moz crawler picks up? In other words, once we fix what MOZ has detected, will there be more detected because it is limited to display say up to 200?? Hope you understand what I mean. Thanks
Moz Bar | | bjs20100