I've been saying for years that we're dealing with problem of mistaking the forest for the trees. There can be debate about the precise limits of infallibility in the strict sense, in any particular case of a papal teaching (apart from the obvious solemnly-defined ones). We're not going to resolve that here.
But, as 2Vermont indicated also, that is not the core problem here. It's about the larger picture of the Church, the nature of the Church, the nature of the Papacy, and the overall promises of Christ for the Church and for the papacy.
See, Vatican I explained the roots in Tradition for the doctrine of infallibility, namely, the constant tradition that the Papacy was intended by Our Lord to be the source of unity in faith (in addition to mere governance) as well as the rock upon which the Church's faith rests as if on a solid foudation.
One can debate the strict infallibility of any given papal pronouncement, but when the Vatican II and post-V2 Conciliar Magisterium and the New Mass have become so unacceptable to Catholics and so contrary to the Catholic faith that we are required to sever communion with and subjection to the Holy See, clearly the line has been crossed, and this degree of destruction ("systematic destruction", as Archbishop Lefebvre rightly characterized it) would be tantamount to a defection of the Church in her core mission to preserve / safeguard the faith and to sanctify souls.
Unity is one of the core notes of the Catholic Church, unity in faith and governance. But if you posit that groups of Catholics can both be in the same Church and yet completely split off from faith and governance because we hold the faith of the other group to be incompatible with Catholicism, then that's essentially to say that the Unity is not an essential note of the Catholic Church. It's almost like the same error that V2 teaches regarding Ecuмenism and the divided/separated Churches within the Church of Christ and consistent with the errors condemned in Mortalium Animos.
Here's another way to look at the core question: difference of degree vs. difference of kind.
If we just have a bunch of individual errors in the Conciliar Church (because, after all, nothing they taught was taught infallibly ... granting this now for the sake of argument), then there's only a difference of degree and there just happens to be more individual errors now than there may have been in the past with regard to the fallible Magisterium.
Or is the Conciliar Church so infested with error that it represents a difference in kind? Is it something other than the Catholic Church. Archbishop Lefebvre (IMO rightly) held that the Conciliar Church lacks the notes of the Catholic Church, which means that it's something other, a difference in kind, since notes define the essence of something. And one aspect of the Church's indefectibility is that it cannot transform essentially into something else. Cf. the Catholic Encyclopedia article on indefectibility. Would St. Pius X, if he had been time-warped forward to our day, recognize the Conciliar Church as the Catholic Church? No, he would most certainly not.
If the Conciliar Church just represented a collection of discrete errors (more than before), then we should rightly remain within the Church while working to help correct these errors. But once this collection of errors become tantamount to making this effectively a non-Catholic religion, so much so that we're required to sever Communion from the Church, the line has been crossed.
Let's say I'm a priest during the 1940s and feel that Pius XII erred regarding evolution, regarding NFP, and got it wrong with the 1955 Holy Week changes. Would I have been entitled to separate from the Catholic Church, and start my own chapel, calling myself the "Orthodox Roman Catholic Movement"? Absolutely not. I'd be rightly regarded as a schismatic. No, the proper approach there would be to respectfully disagree (internal assent does not require absolute philosophical acceptance) and work to have the error corrected through the appropriate channels.