Pride and egoism come into play in regards to both and other things IMO. This is regarding the clergy and the laity.
Supposedly the only people in a state of sanctifying grace appear to be the least charitable.
Some people here who are ardent anti-sedevacantists or who are liberals seem to define charity as "being nice and agreeing with me," while charity actually means truly willing your neighbor's good.
I think the biggest source of division with traditional Catholic circles is cultural liberalism. That's what's tearing apart the SSPX and what causes most of the conflicts on this forum, besides the Feeneyites, but in the end not very many people follow them or will follow them.
The biggest divisions are between those who are consciously opposed to liberalism and those who, objectively speaking, think and act like liberals.