I imagine there are a good number of trads attending and supporting SSPX chapels who have not gone on a vacation for years. Some, perhaps, have never gone on one and some because of financial considerations or otherwise have never even seriously thought about going on one.
Just because 200 years ago priests did not routinely get a rest from parish duties does not mean it is wrong to do so. Even Christ rested. I think it is a good thing that priests do have the chance to visit friends, family, and fellow priests and would not begrudge a priest a few weeks a year away from the parish.
I think it is a matter of perspective.
If you have 10 children, have to home school because you can't afford private school, have to work 60 hours a week just to pay rent, the car, and food, then you see the priests life as luxurious. Luxurious, when see the priest being fed gourmet 7 course meals with wine and after dinner cognac and cigars. Luxurious, when you see the priest does not lift a finger to mow the lawn or even wash his own underwear. Luxurious, when it looks to anyone with eyes to see, that the priest treats the parishioners like his servants and maids. Luxurious, when you see the priests taking a 1 month vacation every year. Luxurious, when you see the priests living better than the rich in the parish.
If you have no children, or had the typical 2 children and they are now gone off on their own, and you own your own home, have two cars, a boat or others toys, work 5 or less days a week, and take one month a year vacation. Then you do not see anything wrong with a priest doing the same thing.
It is a matter of perspective.
To the man with 12 children, the priests use of his donation money to live luxuriously, borders on mortal sin. To him the priest is no different than the shyster Protestant evangelists on TV.
To the well off with no children, the priests taking vacations is not a big deal.