Matthew, you sound pretty defensive.
Has it ever been said by anyone that it does harm to listen to liturgical music outside of Church? I doubt it. Otherwise just reading scores of polyphonic music would be wrong. It isn't a holy activity, mind you, but it's at least positive. I don't see how it could be harmful unless you have nine kids and are in your den with headphones on all day.
That's pretty much all I listen to, and my collection is rather awesome frankly. I'm sure I've spent too much money, but there are worse habits. To me, bathing myself in liturgical music is a way to help replace the nagging pop melodies in my head with those more fitting for my reborn soul. I listen to almost exclusively liturgical music, and almost exclusively Catholic liturgical music, and I feel it has helped rewire my brain.
The proof is that before conversion, Catholic liturgical music was the only kind of music that I DIDN'T like, that I thought was boring. I had Couperin's Tenebrae Lessons and thought it was the most tedious CD in my collection. With my new more ascetic tastes, music from that country and period ( France in the 17th century ) now sounds almost decadent
I have another Couperin CD of motets and find the singing really overdone and operatic, though in Monteverdi I still sometimes like that style.
Learning to accustom myself to liturgical music paid off with great rewards. At first it was a chore, but now it's a pleasure, like so much involving our faith. Now it is secular symphonies by the likes of Brahms and Beethoven that are not only boring but sound like empty, pointless bluster to me. They always sound like they're reaching for some kind of grandeur and uplift, but based on WHAT? Then when you get to Mahler, the sound of exalted Jєωιѕн neurosis, the music becomes almost a religion in itself, with conductors who look as if they are on a drug trip at the podium... Revolting.
People who think this is good music need a full year of Palestrina detox. Palestrina is the greatest composer ever, full stop, no argument possible, those who don't agree are those who just don't understand. It sounds dry and drab at first, but if you listen dozens of times, it is like there is a flood of sublime, glorious melodies -- melodies that also reflect the spirit of God, in my opinion, grand and powerful and yet modest and sweet. It just takes time and patience to hear them.
I have thought about giving up music, and have given it up for Lent, but I didn't feel any spiritual benefit. If I had more to do, and didn't live in a time when it's impossible to accomplish anything, then music would be more of a distraction. If I'm ever a priest under the Great Monarch, I doubt I'll have much time to listen to music. But for now I'm in a defensive rather than offensive posture i.e. the main task for Catholics today is to fend off the encroaching worldly attitudes, rather than go out like a wandering preacher to convert people. Music does not take any time that would be better spent elsewhere, from what I so far have seen. I'm always looking around for a place to use my talents but apparently such a place doesn't exist, so until it does, back to my CDs.