If the Chair is empty, that means that the visible head of the Church is no longer exists. He is gone. Why can't one reasonably speculate that if the head is gone, the body, effectively speaking, has disappeared, as well? We have no pope, so we have no visible, viable Church structure either?
This is not far from Protestant thinking, in my opinion. Luther called the pope a fraud and an "anti-Christ." Following on that He reasoned that the entire Roman Church had no legitimacy either. The sedes refer to the last four popes, if I am not mistaken, as "anti-popes." "(A)nti-Christ," "anti-popes," take your pick. These two pejoratives differ from one another but little.
I came out of a division of Protestant fundamentalism which argued that the church is invisible. That is, it doesn't have a visible, organized structure. Our sect's spiritual forefather, J.N. Darby, argued that the visible church had ceased to exist centuries earlier.
The sede thinks much the same way, in my opinion, though there may some qualitative differences, one being that, unlike many Protestants, the former does not overthrow and abandon, wholesale, basic tenets of the Catholic Faith.