Oh sure, of course men should imitate St. Joseph! As far as I know, Catholic men are highly encouraged to do this- I hear it from priests all the time. I'm sorry if I misunderstood you, I thought you were referring to imitating him just in dress.
I agree that Our Lady was very content to stay in the background, to remain unseen. However, that does not excuse us from our duty to not wear men's clothing. Avoiding pants does not make us stick out generally.
When I made the switch (though it was slow) from pants most of the time to skirts, my main and sincere objection and obstacle to it was exactly what many other trad women do/did say, and that is that to wear skirts all the time makes you stick out, and I don't want to draw attention to myself. After I finally bit the bullet and learned to get past that feeling I came to understand the reality- that my fears were largely unfounded, that I stick out A LOT less than I thought I did. Women who have no religious affiliation at all often wear skirts most of the time out of preference. You see women wearing skirts everywhere. I think a lot of it was the Devil tempting me to believe that I was drawing attention to myself, but it wasn't true.
Now to address the fact that skirts do indeed make you stick out at certain times. This is unavoidable, not immodest. I whole-heartedly believe that Our Lady would continue to dress modestly even if nobody else was. She remained in the background, being meek and humble because she is a woman, and she is the perfection of womanly nature. There are times that Our Lady stood out.
When Our Lord hung upon the cross, nearly everyone left- but she remained and everyone knew it. To this day we remember her doing this and use it as example for ourselves to remain with Our Lord. She did what most others would not do because it was right and good. She stuck out, and it led people to God.
So you see, sticking out is ok when it is necessary. Our Lady did it, Our Lord did it. Why then can't we do it?