It is interesting. I've read a little on this topic, and there are three terms, which seem to have been conflated in the modern world, but actually mean very different things:
To be "celibate" is to be unmarried.
To be "continent" is to refrain from any sɛҳuąƖ activity, regardless of one's state in life (married, single, or religious).
To be "chaste" is to only engage is such sɛҳuąƖ activity as is permitted by one's state in life.
So, when the Church says that all are called to chastity, it doesn't mean that all people should refrain from the marital act. Rather, it means that they should only engage in the marital act as the Church teaches, i.e., only with one's spouse and without contraception.