So there's no essential unity because the only Catholics principle of unity is the papacy. That's no limited to SV, but also afflicts R&R.
In addition, there are some groups that work together just fine but are simply operationally distinct.
Where there's a problem is where some groups incorrectly elevate certain conclusions and propositions to being de fide when they're not quite at that level (though one or two are) ... but, then, that was a problem at times before in Church history too, such as when the Thomists and Molinists accused one another of heresy ... until the Holy See intervened and told them to knock it off. At other times, some groups refused to give in, but when there was a split, you always knew that the group which split off from the pope was the false one and was in schism.
Whether you believe the the See is vacant or simply crippled somehow (as R&R do), without that functioning Christ-established principle of unity, there can be no real unity, and if there is, it's either due to cult of personality of something weird like the Palmar de Toya group (and even they have groups that split off from time to time).
Bishop Williamson explained this shortly after +Lefebvre died and predicted SSPX would fragment, since the Archbishop had personally provided the closest thing to some principle of unity.