Why did Moz crawl our development site?
-
In our Moz Pro account we have one campaign set up to track our main domain. This week Moz threw up around 400 new crawl errors, 99% of which were meta noindex issues.
What happened was that somehow Moz found the development/staging site and decided to crawl that. I have no idea how it was able to do this - the robots.txt is set to disallow all and there is password protection on the site. It looks like Moz ignored the robots.txt, but I still don't have any idea how it was able to do a crawl - it should have received a 401 Forbidden and not gone any further.
How do I a) clean this up without going through and manually ignoring each issue, and b) stop this from happening again?
Thanks!
-
@multitimemachine a noindex tag only really applied to Bing/Google other crawlers etc.. You said you blocked (via wildcard) all robots, are you sure you've not gotten e.g. meta robots that might be different?
help@moz.com might be your best bet for a quick resolution for 'cleaning' the report though I'm still slightly lost as to how your main domain and dev/staging were confused as normally there is a subdomain in the way from my experience, even stranger as bots can't by-pass passwords unless it's your sitemap.xml?sorry I can't get you a direct response but without seeing the site or similar it's hard to diagnose though I'm sure the team at Moz can point you in the right direction .
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do I need a new moz campaign for a subdomain?
Will moz automatically track my new subdomain or do I need to set up a new campaign for it?
Moz Pro | | SamCUK0 -
SEO on-demand crawl
what happened to the on-demand crawl you could do in PRO when they switched to the new MOZ site?
Moz Pro | | Vertz-Marketing0 -
Where do I post this list of hacked sites?
Hey guys, Fairly new to SEOmoz but loving it so far. I was working on a new clients site a noticed some spammy links added right before the tag. Used Open site explorer to list the domains linking to the url and found nearly 300 unsuspecting domains. Some like heartresearch.com.au which just drives me craaazy, I have already emailed them. Below is the list. http://www.opensiteexplorer.org/links.html?group=0&page=3&site=www.rhcie.com Short of emailing every single person can anyone suggest a forum or such that would be helpful for posting this information ? I know it's just a few links but it is frustrating to me and If I can do something about it I would like to. Thanks in advance. Jason
Moz Pro | | RedshiftWebDesign0 -
Open Site Explorer and Escaped Fragments
Does OSE have the ability to crawl AJAX pages utilizing Google's escaped fragment directive? I ask because I'm seeing all our AJAX built pages returning HTTP status codes of 404 when I run OSE reports. See for yourself
Moz Pro | | RyanOD0 -
Crawl Diagnostics - Canonical Question
On one of my sites I have 61 notices for Rel Canonical. Is it bad to have these or is this just something that's informative?
Moz Pro | | kadesmith0 -
How can I get a report of the top 500 national sites using Domain Authority and Page Authority? Does someone have to type sites in one by one on Site Explorer or is there another way?
I want to know what sites and blogs have the to rankings overall based on their Domain Authority and Page Authority - Top 500 - 1000 in nation - I want to know which ones are follow and no-follow too - Does anyone know? Has anyone run such a report yet? Thanks for help - BD
Moz Pro | | creativeguy0 -
Errors on my Crawl Diagnostics
I have 51 errors on my Crawl Diagnostics tool.46 are 4xx Client Error.Those 4xx errors are links to products (or categories) that we are not selling them any more so there are inactive on the website but Google still have the links. How can I tell Google not to index them?. Can those errors (and warnings) could be harming my rankings (they went down from position 1 to 4 for the most important keywords) thanks,
Moz Pro | | cardif0