Wow that is disheartening. He seems a bit misguided. Has he read all that the Soicety (and especially the Archbishop) have said about supplied jurisdiction? Or does he think that your chapel is diocesan?
the jurisdiction... is not territorial but personal, as becomes also the jurisdiction of the priests.
Your thinking is correct -- "lay investiture" would only apply if Fr. Pfeiffer were the Ordinary who SHOULD have jurisdiction over the chapel in question -- the cleric who should get to make the decision.
But any sane Catholic knows that's not the case. Fr. Pfeiffer (or any other trad priest) has ZERO jurisdiction, properly speaking. The only way to get jurisdiction is to get it from Rome. Anything else is operating under supplied jurisdiction.
Supplied jurisdiction comes from the needs of the Faithful, who are free to adhere to any priest or bishop they feel will help them preserve their faith during this Crisis. A group of 20 faithful approach Fr. Smith, begging him to say Mass for their group. He agrees. The Church supplies
the missing jurisdiction to Fr. Smith, even to hear confessions and witness marriages, because of the state of necessity (the Crisis in the Church). But if those 20 faithful decide to go with another priest 3 years later -- it's up to them.
That's why the SSPX wanted to own all the buildings. That tends to discourage MOST people from leaving. But even then, the faithful are always free to leave, split, etc. and start over from scratch somewhere else, if they feel it is needed to preserve their Faith.
The first priest working there for 3 years doesn't thereby "earn" a more permanent supplied jurisdiction, like some kind of supplied-tenure.
Obviously the dictates of charity would require a group to consider a priest's material needs. If he took care of your souls for 3 years, you should really take his needs into consideration.
But let's keep in mind that Fr. Voigt only said Mass here 7 or 8 times, once a month. It was a part-time job, as it were. And our goal was to find someone available "full time". I don't think it has gravely inconvenienced him or made it difficult for him to survive on account of our no longer having a need for his ministrations.
Many places around the country will benefit by Fr. Voigt's consistency. He arranges his own schedule, and he's quite consistent and predictable. He gives you several weeks notice when the next Mass will be.