Unless the Roman Pontiff promulgated an Ex Cathedra heresy, then he would be a formal heretic.
Nobody said anything about
depriving him of his office. That is way far beyond, even if the alledged formal heresy can be proved, which cannot.
Formal heresy is when the Pope binds us to his heretical teaching. Most theologians do not believe this would actually ever happen. But even if it did, it is just not up to the layman to judge such a thing, it would be up to another Pope.
The pope is only infallible when speaking ex-cathedra, meaning that it is possible and indeed, history attests to this, that he can err in actions, opinions, and even teachings, which remain outside of the realm of infallibility. Ex Cathedra is speaking God's will, if it is heresy, then we would not for certainty that it is man's will because it has not link with Divine revelation or tradition.