As for the OP, the only people who really need to think about this are those who are directly in the situation.  Passing generic opinions give rules to follow, but a good Catholic will discuss the matter with their priest and their doctor. The problem is the secular society and secular doctors and trying to find Catholic help is next to impossible.

I'm guessing that the OP wasn't the only one who heard the priest say this. If so, then there are others right now walking around with a dangerously compromised understanding of the truth. Dangerously compromised because they'll share that fallacy with others with whom they interact. And that's how the toxin spreads, with people not realizing what they're repeating. "Well, my priest said such-and-such so it must be right."

Quiet discretion has its appropriate place in private matters discussed with a priest or doctor. But the general issue of abortion has become a very public matter, and therefore errors need to be called out publicly. The cynical fake-concern canard repeated by crypto-feminists of "between a woman and her doctor" is how (((they've))) misled good people who should know better. 

OP, if that priest did say this in the presence of others besides you, please let us know whether/when he does restate and correct it.