This logic could also apply to cruises, golf club memberships, jet sky, and other things of the sort, right?
Of course. "It is easier for a camel to go through the eye of a needle" and so on.
I don't think any of the things you listed are sinful in and of themselves, but naturally, the more money one has, the more they should be putting towards charitable causes. So someone wasting lots of money on luxuries, of which gambling is one, while not aiding his fellow man, would be sinning.
On the other hand, you can't be expected to spend every cent after necessary expenses/savings on charity. I recall another part of the Gospel where Christ was given an expensive gift and, when a disciple said it could be sold for charity, said "the poor will always be with us" - meaning that no matter how much money you said aside for charity, it will not solve poverty, and that as such there are other uses for money. In that case, it was to show love and appreciation.
Now, gambling doesn't do show love or anything. But - take for example the Wedding at Cana. Not only did Our Lord tolerate that they were drinking wine to be merry - he even topped them up. And our Church similar has not deemed drinking alcohol for merriment as sinful, but rather has only taught that it's sinful in excess. So, a man who meets friends on a Friday and has a few beers, but who does not get overly drunk or spend beyond his means commits no sin. But then, suppose he has one beer less and buys a lotto ticket instead (these things are of roughly equivalent prices, in my country at least) - why would that then suddenly be a sin? He spent the same money either way, and at least with the ticket he may win something.
Based on all that, I can only conclude that gambling is not inherently sinful but only becomes sinful when it either prevents one from using that money for a real need (i.e. supporting their family) or when it becomes excessively wasteful or decadent.