I think the underlying assumption of people who have this type of opinion (if in good faith) is that you can filter for a higher-quality voter set and if you do that, the gains from having more competent governance (which lifts all boats) outweigh the risks that (i) if a subset of voters is unrepresented the others will gang up on them and face no resistance, (ii) without "everyone votes" as a schelling point you get a spiral of disenfranchising more and more people. I personally find this strategy quite risky. As I understand, it's already a standard opinion among "wise advocates of democracy" that the goal of democracy is to provide alignment, not competence; the competence comes from other layers of the social stack like entrepreneurs, intrinsically-motivated academic types, and other various structures. And so compromising the "alignment" property of democracy is exactly the road you do not want to go down.