He's right. Marriage is for a purpose. God designed it primarily for the continuation of the human race. He also designed both spouses to desire the normal activities of marriage, especially if you become comfortable with each other, become good friends, and love each other.
Even if you have no "drive", your wife probably would, especially when she is fertile.
If you feel a calling to stay celibate, you should consider the religious life, priesthood, or the single life.