I agree -- the priest is the guilty one here. He is getting his "social time with women" fix -- a temptation most priests face. There is a certain aspect where it is normal, humanly speaking, for men and women to complete each other and enjoy each other's company.
I am not talking about anything sɛҳuąƖ here, though such indulgences (talking too much to women, especially certain women at length) can lead to problems down the line. Most affairs start out as EMOTIONAL affairs.
The priest and the women are willing to indulge their mutual human desires in this matter, because they both figure it's "innocent enough", being in the confessional and all. There are plenty of people around. But if they dally with this temptation, they will eventually arrange meetings OUTSIDE of the confessional, with a believable pretext of course ("Father, can you bless a statue sometime during the week, if I stop by with it?", "Can you stop by to talk to my son about...")
There are also TONS of women that deal with this temptation -- to get attached to a priest, be blown away by his learning, piety, charm, even good looks -- and they want to talk to him as much as possible. Women are naturally vain and self-centered, just as men incline easily to pride and anger. Again, it starts out as human self-indulgence (pandering to mostly innocent desires springing from human nature) but Original Sin often takes over at some point, and things can get dangerous for both the (usually married) woman and the priest himself.
History is full of cases where this type of thing ended in disaster -- a married woman leaving her husband and family, and running off with a priest (who of course leaves his priesthood as well). This has even happened with SSPX priests! It also happens more often in Latin America. SSPX priests in the USA usually leave to become Sede or Novus Ordo. At least that's what I was told during my seminary days.
And the woman in question doesn't have to be "hot", to use modern parlance. When you are single, you naturally lower your "standards" to match whatever you have available in your daily life. This is human nature, I believe. In the end, a woman is a woman -- they all have feminine nature (as well as pheromones). So the fact that a woman is less than attractive, or over 35, is of no consequence to this particular discussion. Besides, some individuals are more attracted to mental or spiritual attributes anyhow. Some people simply aren't concerned with the physical at all. It's true (even if that doesn't describe you!) The priest is still, humanly speaking, missing that "feminine element" which normally completes a man. Not that a man can't live without a woman, but I believe it would be a lie to say that a women does not complete a man. A woman does a lot more for a man than merely A) give him an outlet for his sɛҳuąƖ desire, B) bear him children, and C) order the domestic household. It goes beyond that.
Just for starters, a priest is a man so he will naturally enjoy being looked up to and followed/loved by a woman. Any man, including a priest, has that same male ego deep down. Of course good priests will constantly be on their guard in this matter. But this is a temptation, like many others a priest will experience. There's a reason they spend 6 or 7 years in Seminary practicing and growing in virtue.