This can be taken to extremes, guilty consciences, anyone? But yes, there are people, mainly women, but men, too, who look for opportunities to be "scandalized" and spread juicy gossip to their like-minded friends. Innocent souls have had their lives ruined by the dirty minds, itching ears, and flapping jaws of such persons.
One cannot live in a state of constant paranoia over such matters. (Wouldn't the devil love that?) Such situations should be dealt with on a case by case basis. Use prudence, that's all. If there is a person whom you know will gossip or makes rash judgments, who may fall into this sin, make an alternate arrangement, or, better yet, invite her and one other trustworthy person along for the ride!
If you find yourself the victim of someone falsely scandalized and spreading gossip, think about what she's saying and to whom. If the gossip is nothing reasonable persons believe, let it die a natural death. If it is potentially harmful to the innocent, including yourself, follow Our Lord's directions to St. Matthew, chapter 18. Go to the person and correct her. If the situation is too far gone for that, take a friend or two with you. Last resort, go to your priest and let him handle it.
I was once accused of impropriety with two young men 35 and 29 years my junior because I drove them to a BJs store to get food for a conference. The person who spread the rumors was a man---go figure! The reason I drove the young men was because the vehicle designated for the food run was a standard shift and I was the only licensed driver on the premises able to drive it.
It turned out the purpose of the gossip was to mock the young men for being unmanly for not driving a stick! In the meantime, the rumor flew out to other people including the mother of one of the men that I wanted to get cozy with her son! Needless to say, I went to the man, told him what had happened, and demanded he correct the the situation. He refused, so I corrected it myself with the mother, a woman about 10 years my junior. Once she met me, her fears were allayed. I also reported it to the priest who did nothing.
I didn't attend the conference and decided if anyone actually believed the rumors, God could deal with it because I had more important things to do.
In another incident, a gentleman called me to inquire if I would be able to pick up a bishop at the airport and drive him about 50 miles to a hotel. The man's wife had gone into labor for the birth of their 6th child, so he wasn't able to do it. I accepted to do the favor, so took my car to the car wash for a thorough cleaning and next day picked up the bishop, and drove him to his hotel. His Excellency sat diagonally opposite me in the back. I asked for and received his blessing before returning home. End of story. For all anyone knew, I was a merely a hired limo driver.