As others have noted, I believe a good deal of it comes down to what you are using to listen to your music with. The artifacts of lossy compression can be particularly apparent when monitoring with high quality speakers or headphones.

I keep my collection of music in WAV format for the most part. It surely takes up more hard drive space, and there is a price premium when purchasing digital music online in that format, but it's worth it for me.