There is such a thing as the law which is written on the heart of every
human being. Sometimes it is called Natural Law.
We also have common laws which are laid down by societies in order to make them workable.
It is of course true that we need God's help to remain chaste. Maybe you are confusing the word "chastity" which is a Christian virtue, with what the mores of particular society implement in order to have an orderly society, by keeping family units intact etc.
In short, members of society generally conform to the mores of their particular society and from time immemorial there have been restrictions on what is allowed and what is not.
Where did you get the idea that
Buddhists .... despise all pleasures