How much compression you'll "hear" from a 128 mp3 file also varies a lot. mp3 works on the idea that certain frequencies are less important than others, so the more it compresses the more it attacks those frequencies. If you stay within a certain range it may sound fine, but on higher frequencies for instance (a hi hat or even a voice with a naturally high pitch) you'll hear the compression a lot more.

The compressed mp3 file can even stand out more once you start mixing it with other sourced files (say 48k 24 bit uncompressed or better, which should pretty much be your only consideration for serious use). If you convert an mp3 back to wav it of course makes no difference - the compression (or "harm") has already been done