...and I will shew thee, by works, my faith - James 2:18
The issue of Papal Heresy can be more clearly understood in terms of behavior. There are popes doing what popes should not be doing. And a Church which is acting in such a way that would alarm even non-believers.
It is like abusive parents. One can try to cling to a bad parent, but they will simply keep hurting you. That is why we have Godparents.
For a Pope, when they simply do not act like a shepherd but a wolf, one is left looking for alternatives.