Yes, the retreats were always for ONE or the OTHER. How are they going to keep them sufficiently separated for decency sake? Usually the retreat building is too united to permit decent separation of the sexes.
As a male, I'm here to tell you that it would be distracting to be stuck in one place -- with no distractions, mind you -- with any number of women in the classroom for the various conferences. Human nature is human nature.
I had no problems in the seminary, because it was all men. When you go out into the world, you just go into "custody of the eyes" mode, put up barriers of sorts, stay busy, etc.
But when you are meant to cut out all distractions, even innocent ones like study, you really can't afford to have THOSE kind of distractions around!