Definitely let it go! One shouldn't complain unless he is willingly to help with the solution. Where we used to hear Mass, the choir was terrible. People complained. But when the director begged for volunteers, he got none. It does seem to me a chapel of 200 persons should be able to find people for basics like cleaning, lawn mowing, etc. If not, it's not right to expect the priest to personally do all the grunt work. You want a holy priest, right? One given to prayer and hours before Our Lord in the Blessed Sacrament. This is not to say the priest should be a lazy bum and have people waiting on him hand and foot, but his main calling is to priestly duties, not janitorial.
If people don't sign up for retreats, volunteer for, or attend parish events, there must be a reason other than the fact that the priest is frustrated. Some of the men of the chapel should meet with him to discuss the reasons for non-participation. I don't know the Sanford area. Is a place very expensive and not conducive to people being able hot afford retreats? Or maybe the distances people come from to hear Mass are prohibitive of events like parish picnics? Perhaps there are no women living close by who can come and clean on a regular basis? Men with the same problem?
Personally, I have not been on a retreat in over a decade and I get to Mass once every six weeks, if so blessed. My health doesn't allow me to make long road trips or to do heavy cleaning. I'm on a fixed income and cannot afford airfare much less the cost of the retreat. Many Catholic families are raising lots of children without the support of extended family or of a surrounding Catholic community life. These parents have all they can do to keep a roof over their heads, food on the table, and educate their children properly. For them, a retreat is out of the question. How many men can take an entire week off from work and how many women can go the whole week without adult help from a husband?
Maybe expense is a problem. Are chapel events not well matched to the members? A chapel populated by young families with many children should not count on crowds at a $100 a plate formal dinner and recital, as at an SSPX chapel that shall remain unnamed. Conversely, well-to-do seniors probably have little interest in attending an outdoor picnic in the park at the ballfield!
If people don't sign up for the priest's activities, no wonder he's frustrated. It could also be the sad truth that most who attend there are selfish, wanting to get something out of Mass, but not give. In which case, if the priest gets upset, one can hardly blame him. If nothing changes, the people can count upon eventually losing the priest, their Mass, and their Sacraments as punishment for lack of appreciation.
The people should take time to speak with their priest, and he with them. Maybe the chapel isn't in a good locale any longer. Circuмstances and social dynamics change.
Unless the OP is willing to do something other than complain, he should, indeed, let it go!