Why did Moz crawl our development site?
-
In our Moz Pro account we have one campaign set up to track our main domain. This week Moz threw up around 400 new crawl errors, 99% of which were meta noindex issues.
What happened was that somehow Moz found the development/staging site and decided to crawl that. I have no idea how it was able to do this - the robots.txt is set to disallow all and there is password protection on the site. It looks like Moz ignored the robots.txt, but I still don't have any idea how it was able to do a crawl - it should have received a 401 Forbidden and not gone any further.
How do I a) clean this up without going through and manually ignoring each issue, and b) stop this from happening again?
Thanks!
-
@multitimemachine a noindex tag only really applied to Bing/Google other crawlers etc.. You said you blocked (via wildcard) all robots, are you sure you've not gotten e.g. meta robots that might be different?
help@moz.com might be your best bet for a quick resolution for 'cleaning' the report though I'm still slightly lost as to how your main domain and dev/staging were confused as normally there is a subdomain in the way from my experience, even stranger as bots can't by-pass passwords unless it's your sitemap.xml?sorry I can't get you a direct response but without seeing the site or similar it's hard to diagnose though I'm sure the team at Moz can point you in the right direction .
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rookie question re Moz Crawl errors after deleting a property from console.
Hi all, I stupidly removed the "http" url of my one website a few days back (it is one of three, the other two being the https), then re-added it around a day later and, while google console isn't reporting back any errors, Moz Crawl is going to town on this now for one critical "4xx" issues, canonicals and various other content issues that I addressed days previously...last Moz crawl performed an hour ago, url deleted and re-added two days ago. I have resubmitted a sitemap, will this smooth itself out or shall I go and make changes? Many thanks in advance.
Moz Pro | | UkPod0 -
How to utilize Moz during website relaunch?
Hi folks - we're relaunching our website and have been sure to make it SEO optimized -- descriptive copy, new readable URLs, keyword-laden page metadata, and the like. However, I'm new to Moz Pro and unsure how to utilize it during this transition. My questions for those of you -- assuming many -- who have gone through a redesign: **How did you use Moz to track the success of your redesigned website? What do you wish you'd done? ** Any campaigns or resources you can suggest will be helpful, as I am new to the functionality!
Moz Pro | | meghanennes0 -
Moz Crawl Report more urls?
Hi. I have used Moz Crawl Test and get my 3,000 urls crawled no issue. However, my site has more than that, is it possible to crawl the entire website? Alot of the crawl urls in the Moz test are search string urls and filters so Ive probably wasted about 2,500 urls on filter urls. Any advise or alternative software that wont cost a fortune?
Moz Pro | | YNWA
Thanks0 -
Mass Moz Grading
I want to know if there's a workaround to grade my individual pages for multiple keywords in a more efficient manner. Let's say for example (this is an extreme hypothetical) that I'm targeting these keywords: "Red Castles" "Blue Cars" "Green Bottles" on one page. I want to know if there is a way to run three grade reports at once for the single page that they're on with the Moz tools. Thanks a bunch!
Moz Pro | | OOMDODigital0 -
SEO Moz analysing US site when I want it to analyse UK site
Hi there, I am trying to analyse the UK version of my website. However as the website is set to redirect people to the most relevant site and SEO Moz is based in the US, it is redirecting all the pages your crawling and therefore analysing US pages. So I am trying to look at the UK homepage but all the factors relate to the US site... Is there a way around this?? Thanks
Moz Pro | | MyTights1 -
Seomoz crawling filtered pages
Hi, I just checked an seo campaign we started last week, so I opened seomoz to see the crawl diagnostics. Lot's of duplicate content & duplicate titles showing up, but that's because Rogerbot is crawling all of the filtered pages as well. How do I exclude these pages from being crawled? /product/brand-x/3969?order=brand&sortorder=ASC
Moz Pro | | nvs.nim
/product/brand-x/3969?order=popular&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=DESC&page=10
/product/brand-x/3969?order=popular&sortorder=DESC&page=110 -
Open Site Explorer Update
What is taking OSE so long to update? The update schedule said the next update was going to be on Dec 28th.
Moz Pro | | Robbie8299
If you open OSE it says "Last Index Update: November 28th, 2011" Today in January 1st. Any thoughts as to why the delay?0 -
Crawling One Page
I set up a profile for a site with many pages, opting for setting up as a root directory. When SEOMoz crawled, they only found one page. Any ideas for why this would be? Thanks!
Moz Pro | | Group160