How to get moz to crawl a staging domain that is blocked by robots.txt
-
Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
-
Hi there!
I'm going to go ahead and marks this as answered since we took care of this in the support queue. For any others that are curious though, You can always add allow/disallow directives for Roger in your robots.txt. I recommend taking a look at our Help Hub for more info
Thanks!
Sam
Moz Helpster
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is Moz unable to crawl my site?
Was hoping someone could advise why Moz is unable to crawl my site at https://www.oceaniacruises.com **Moz was unable to crawl your site on Oct 5, 2017. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. Any help would be appreciated. Thanks!
Getting Started | | jbarinaga0 -
901 error code showing url back to back in crawl
Hi Everyone, I'm absolutely dumbfounded about this 901 issue (showing pages with our url back to back). Our site is hosted on Big Commerce: https://www.santabarbarachocolate.com When I look for these pages being crawled I don't find them. I've called BC for help and I can't seem to find a solution or where to turn as to how to fix the issue at hand or even if it matters. Please see below what the Moz crawl shows. Could this be related to Yotpo or some app we have running? Or does this even matter and does it have any influence on rank? Do you have recommendations or ideas? Thanks so much. Pages with Crawl Attempt Error as of Mar 3 URL Page Authority Linking Root Domains Status Code | Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/100-percent-pure-cacao-unsweetened-baking-chocolate -- -- 901 Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/buy-wholesale-bulk-chocolate -- -- 901 Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/organic-chocolate-wholesale | -- | -- | 901 |
Getting Started | | santabarbarachocolate0 -
Rel Canonical Notice in Moz Analytics
Hello all. I just received my first report fro Moz Analytics. Signed up here to see what problems my site might have and find out how to fix them. On all of my issue pages I see notice with black dot saying that my urls are Rel Canonical Using rel=canonical suggests to search engines which URL should be seen as canonical. I know that recently my web developer set up so that every url on my website has a rel=canonical as I think there was an issue with trailing slash. Is this to be considered an issue or I can just leave it? Thank you guys. Regards, Armands
Getting Started | | A_Fotografy0 -
How long will it take to get google Analytics results back so I can get rolling?
I just signed up for Moz. I'm wondering how long it will take until Google Analytics results will enter.
Getting Started | | John-Francis0 -
Anyone use On-Page Grades in Moz? Please Give us Feedback!
Jackie from Moz here. We're looking to get some feedback on the letter grades we use for On-Page Grades (in the application and in the Research Tools). We made this super easy survey to collect our Community's thoughts. Tell us what you think! Thanks for all your help!
Getting Started | | JackieRae2 -
New to Moz Pro? Join our free webinar this Friday!
Hello everyone! We'll be holding a webinar on Friday to help new members learn about what all Pro has to offer, show some off our most popular tools, and get you comfortable with the dashboard. Register here: https://www3.gotomeeting.com/register/589105390 Date: Friday, August 29th (this Friday!) Time: 10:00 AM - 11:00 AM PDT Hope to see you all there!
Getting Started | | jennita6 -
Moz Analytics - Custom Report Question
Hello Moz! I haven't been here since the days of SEOMoz, but it's great to be back and i'm really pleased to see lot of shiny new tools at my disposal, but I do have a question in regards to functionality. If I have, for example, 15 clients in my dashboard and would like to run the same report for all of them. Assuming that I just finished creating a custom report for my first client, how can I duplicate this report without having to re-create this report 15 times? Additionally, while I am here. I have noticed that when I add a campaign, the last screen never finishes loading and the window seems to freeze. A quick refresh fixes this, but I thought i'd flag it 🙂
Getting Started | | Paul_Tovey0 -
Why does moz show "not in top 50" for all my keywords???
Hello, I signed up to moz pro 4 days ago. And so far it seems to be tracking visits etc. But all my keywords say "not in top 50" . Why is this? Is this normal? Just to confirm most of the keywords i pasted in from my webmaster tools and i only chose the ones that were in top 50
Getting Started | | casper09030