In today's culture, people focus on sex way too much.

When people hear "celibacy" they think "no sex".

But actually, celibacy just means "unmarried" or "singleness" which normally implies the same thing.

But today, being single IN NO WAY means "no sex" (on the contrary! Fornication is so popular and accepted nowadays that one's youth is when worldly people get the most "action"), hence the confusion.

Even Wikipedia acknowledges this:

Celibacy (from Latin, cælibatus) is the state of being unmarried and/or sɛҳuąƖly abstinent, usually for religious reasons.[1][2][3][4] Historically, it has simply been defined as the state of being unmarried.[5] A 1990 book that focuses on celibacy in Catholicism states that "the most commonly assumed definition of celibate is simply an unmarried or single person, and celibacy is perceived as synonymous with sɛҳuąƖ abstinence or restraint."[6] The book adds that even in the relatively uniform milieu of Catholic priests in the United States "there is simply no clear operational definition of celibacy".[7]
According to Garner's Modern American Usage (2009), the Oxford English Dictionary still gives only the traditional definition involving marriage, whereas, in contemporary usage, celibacy is almost universally understood to mean abstinence from sɛҳuąƖ activity.[8] Some authors define celibacy as necessarily voluntary,[9] while others consider the notion to encompass involuntary contexts, such as duress.[10]

It is interesting.  I've read a little on this topic, and there are three terms, which seem to have been conflated in the modern world, but actually mean very different things:

To be "celibate" is to be unmarried.

To be "continent" is to refrain from any sɛҳuąƖ activity, regardless of one's state in life (married, single, or religious).

To be "chaste" is to only engage is such sɛҳuąƖ activity as is permitted by one's state in life.

So, when the Church says that all are called to chastity, it doesn't mean that all people should refrain from the marital act.  Rather, it means that they should only engage in the marital act as the Church teaches, i.e., only with one's spouse and without contraception.