It is important to keep in mind that the Search Visibility metric relates only to the keywords you are tracking, so it isn't exactly correct to say that the Search Visibility score is skewed if you are tracking branded keywords. This is an accurate visibility score for the keywords you are tracking.
You can definitely see the Search Visibility for all of your non-branded keywords by using the filters as you suggested, which will give you a different perspective of your Search Visibility versus your competitors.
However, you could say that, in and of itself, the score is somewhat skewed just by the fact that these are the keywords you have chosen to target, rather than the total Search Visibility for all of the keywords you and your competitors may be ranking for. If your competitors aren't specifically targeting the same keywords you are tracking, you may be ranking better for those keywords, but you may not necessarily be as competitive in the overall rankings for every keyword. Our metrics can only be calculated for the terms you choose to track, so it is really just a view of the keywords you are targeting and not necessarily the full picture of your visibility as a site overall.