I have a point because there is no church teaching that allows for laymen deciding the legitimacy of a pope.
Yes, this is generally true. But in the scenario of a heretical pope, someone has to be the first to call it out. You don't go from zero to universal rejection of a papal claimant at the flip of a switch. There is a progression that goes something like this:
1) Pope manifests heresy in some way.
2) One or more Catholics call it out.
3) Gradually more and more Catholics come to the realization that the man is a heretic.
4) Universal rejection of the papal claimant.
We're at step #3 (somewhere along that path).
But that's the case of an ordinary heretic pope. And, as I try to point out over and over again, this isn't so much about the heresy or lack thereof of a Pope. We're confronted with this Conciliar Church. We find that we don't recognize it as the Catholic Church and feel oblige to sever communion with it. So the question is whether it is possible for the Catholic Church (via the authority of a legitimate pope), to become so corrupt as to basically require Catholics to sever communion with it and to submission to the Catholic hierarchy, in order to save our souls. At the end of the day, whether Bergoglio is a heretic or not, it really isn't our problem. What's our problem is the Conciliar Church.
What's at issue here is whether R&R actually undermine and destroy Catholic doctrine regarding the Magisterium and the Papacy in order to save Bergoglio, just so the can have some guy prancing around in white vestments in Rome. If you want a guy in white vestments, you would have been better off with David Bawden. Both of them in your eyes have the same authority, and in fact Bawden had more because his "teaching" was in conformity with Tradition. This is the R&R ecclesiology, and it's horrible. There's a real risk of people losing the actual Catholic faith by adhering to false R&R principles.
So, I ask you, why don't you guys jump on board with Father Chazal's sede-impoundism? This way you can have your guy walking around in white without attributing the evils of the Conciliar Church to the authority of the Pope. What's your aversion to it?