Actually, secular priests have traditionally not taken the vow of poverty.
That's "secular", as opposed to "religious". Not secular, as in "worldly".
The vow of poverty entails not having any money or owning any material things.
This would be most inconvenient for a parish priest. Imagine not having any money to help someone on the street who asks for it, or not having transportation fare?
It makes perfect sense. The secular priest works out in the world, among the people. He might have a rectory to come home to, but he doesn't stay there all day. He is out and about, visiting and bringing the Sacraments to sick people, hearing confessions, travelling, etc. He needs to have some means to take care of the necessities of life, not just for himself but for those around him. If he collects too much "means", of course he should give the excess away. "Freely received, freely given."
It is most fitting that a priest should have the SPIRIT of poverty, not living luxuriously and wearing the finest cassock. People are not edified by wealthy priests -- on the contrary.
Some priests are called to the religious life, and others are not. It's that simple. It would appear that being a secular priest is a harder calling, at least as far as sanctity is concerned: how many parish priests have been canonized? St. John Marie Vianney.
Those in religious life own things "in common" and no one owns anything. In fact, some religious houses move the monks/brothers/nuns around to different quarters on a regular basis, so they don't even get attached to their humble cell.
Source: Seminary training.