I will start by saying that I do not have all the answers. I will also admit that I do not possess the authority to make any definitive declarations and disagreeing with the conclusions I have come to does not put one outside the Church (an invalid claim that has been made by some sedevacantists and many, not not most, anti-sedevacantists). I will do my best to answer your question.
First, your logic is not unassailable, though you ask a valid question. The question as to whether a pope can fall into heresy is still an open question. Though Bellarmine found the idea so distasteful as to suggest in would never happen, he admitted the idea could, indeed, happen.
Many sedevacantist believe John 23 was not a valid pope. Others, such as John Lane, believe he probably was. This is one reason many sedevacantists do not have a problem with the 1962 Missal. Calling the Council into existence does not, in itself, indicate that he fell into heresy. As for the claim that he was a Freemason, which would have invalidated his election, it has not been absolutely proven to be true. The fact that his alleged membership was virtually unknown at the time and is only a matter of conjecture today indicates that the issue was not manifest. I don't know enough of his public pronouncements to be sure of anything when it comes to John 23 and so I keep an open mind.
Much of the same can be said about Paul 6. He could have been validly elected even if he were an occult heretic. I have read arguments that he had become a public heretic prior to his election, though the evidence seems lacking. The problem is when he promulgated the heretical docuмents of Vatican 2 (and there are at least two docuмents that contain outright heresy that have been discussed on this and other forums many time--I will not debate them here). When he did so, even if his heresy was not manifest before, it was at that point and he was no longer pope.
Additionally, when he took action to change all of the sacraments, something the theologians had argued would be a sign that the pope had fallen from the Church and from his office (I think the actual comments I've read were that if the pope were to attempt to excommunicate the whole Church or change all of the sacraments it would be a sign that the pope had fallen).
Why weren't these actions immediately noted and a vacancy declared? Actually, these things were. A great many people commented on what was happening, the problem with what had been done, etc. I have a copy of a 1970 Life magazine (March 20 1970 issue) that contains an article entitled, "The Pope's Unruly Flock", which discusses "the rebellious demonstrations by preists and seminarians and the growth of an 'underground church' defying ecclesiastical laws." The article was not talking about Archbishop Lefebvre. However, Archbishop Lefebvre was a very early critic on some of the docuмents of Vatican 2, going so far as to not sign (at least, he believed that he had not signed) two of the Council docuмents. Yes, the problem was manifest, being noticed even by the secular press and non-Catholics. And it was around this time that some theologians began to put two and two together to discover that the answer was four and not, as the Conciliar church was insisting, five.
It was not that the problems were not noticed, it is that, as Bellarmine said, the idea was so distasteful that people wanted to believe there was another explanation. They grasped for anything that would keep the Rock of the Church stable because they could not imagine the Truth that was so terrible. Most people either left the Church altogether or remained in the Conciliar Church just as a battered wife either leaves the batterer completely or keeps returning because she believe she has nowhere to go. In the case of the Church, however, it was not that the husband was the batterer, it was that an imposter had taken the husband's place.
Consider what Catholics had to endure during the Arian Crisis. Most Catholics simply accepted whatever their bishop and priests said. The very few (i.e., the "handful" of) Catholics who rejected the heresy were expelled from their Churches. Arians, who said that they were the true Catholics, would roam the cities and assault Catholics who gathered in cemeteries on the Lord's Day to worship.
With the elections of John Paul 2 and Benedict 16, we found that we now had a professional class of laymen who specialize in parcing their words to demonstrate how everything they say can, with great effort and with technicalities, be in accord with Catholic teaching even though the great masses of people (including the rest of the clergy) seem to misunderstand everything.
Modernism, like all other heresies before it, is seductive. Even faithful Catholics may, for a time succuмb to its allure. But when a cleric succuмbs, he loses his office even if, and there is no evidence that many have, they repent and return to the True Faith.
So what happened first? First, the pope fell from office or he was never the pope to begin with. It does not matter which. Then the docuмents of the Council was the manifest evidence of the fall. Finally, some of the faithful recognized the truth and decided to remain or return to the True Church and condemn the false Conciliar church.