It never sat well with me either that he sometimes preferred not to wear a cassock when he travelled.
I was taught that if you need to take your cassock off to go somewhere, it is a sign that you should not be there.
Archbishop Lefebvre wrote at length in "A Bishop Speaks" about the importance of the cassock and how a business suit ain't the same.
The parts of the cassock have theological significance, symbolism, and protection a mere business suit does not.
Archbishop Lefebvre said that everywhere the devil persecuted the church, his first attack is against the cassock.
Bishop Williamson is my hero, and the great man of our generation, but he is wrong not to wear the cassock.
None of us is infallible.