I don't think one can say that there's anything wrong with wearing a rosary unless the purpose in the person's mind is to be fashionable, or for vanity's sake, and they don't pray it.
As I heard someone comment once "God is not going to hear your prayers better because you're saying them on an expensive set of beads." You can pray the rosary on your fingers just the same.
If you're wearing a rosary and someone assumes that you're a gang member, that is their error.
I personally don't ever put it around my neck (it's not a necklace) nor think of it like a charm (some people keep it under their pillow or in their pocket and touch it as a charm). It's good of course to carry your rosary in your pocket to remind yourself to pray and of the mysteries contained therein. Yet, you can't accuse someone for having a devotion. Certain people express their devotion with more externals - kisses, bows, hand motions, wearing objects, etc. I don't think we should judge them harshly. The Italian and the Irish (Spanish also) are very external about their devotions, and have to be careful about superstitions. Women are devotional more than men and show their devotion with emotion. We are frail humans with bodies that distract us, and we need images and trinkets often as aids to fix our mind on the prayers.
If a person is troubled by evil spirits or temptations, by all means, put on or hold the rosary. Especially if you have had it blessed, all the better. See attached page from St Louis de Montfort's book.