To "kick someone out of office" is a legal process. Because it has to do with their human/govt office. Which is a canon law process. God has already judged them (spiritually) if they are guilty of heresy. But no layman can kick anyone out of office; this is the Church's job, because it's a human office/process.
Judging that another has committed the public sin of manifest formal heresy is not to say that the one making the judgement has the authority to enforce his judgement by kicking the heretic out of office. The enforcement is under the direction of the competent Church authorities. However, the one making the judgement simply refuses to recognize the heretic as still retaining office.
The question is: do you hold that Jorge Bergoglio has committed the public sin of manifest formal heresy?
If your answer is "yes", then you must refuse him as pope because the public sin of manifest formal heresy per se separates the heretic from the Church.