The loudness/compression wars also came along about the same time as mp3s becoming common. A good studio producer takes into account how the final music is going to be played back. Not everyone has a 1k+ stereo system they listen to everything on. You have to mix/master for cheap speakers, car radios, at one time you had to account for the limited bandwidth and snr of FM or AM analog broadcasts, and all of a sudden you had to account for the high likelihood that your music would be compressed into a 128 kbps mp3 and listened to on an audio player on the subway. So of course they turned to compression to even out the listening experience across all the formats. That's bad for us hi-fi people, but it was good for the much larger group of people listening to their ipods on the way to work or at the gym.
The popularity of the lossy CODEC (mp3) came about primarily because of illegal and legal file sharing and portable devices. Smaller size for the same music = less heavy on network connections (this was pre- broadband), faster downloads, more songs to store per unit of space (flash storage was in MiB not GiB).
Dynamic compression for the sake of making the song louder was around in the disco era. There was a loudness war of sorts, but the limitations of what vinyl could do prevented levels reaching what we have today on digital mediums. It's funny really, because radio processing made things a general homogeneous loudness and DJ mixers would have allowed some loudness equalisation when cueing up the next single. So I don't really understand why 'cutting a hot record' was even necessary.
Hot clipped garbage was being put on CD in 1999 ion the form of the Red Hot Chili Peppers
Californication. Charles Dye admits to shoving a Waves Limiter on the master bus of the ProTools session and just adding something like 6dB of makeup gain to Ricky Martin's 1999 hit
Livin' la Vida Loca.
Levels were steadily increasing all the way through the 90's and could be plotted on a graph. It's no surprise that limiter plugins because more available and transparent towards the end of the 90's, and virtually all albums were being recorded and mixed digitally.
..but we are digressing.
With the metering available, quality of converters and such, there are zero reasons why 3 or 6 dB of headroom cannot be baked into the digital file when it is distributed to the consumer, either as a CD or HiRes download. Yes, that reduces the dynamic range by some, but it's already fantastic what is available with just 16 bits.
Turning the digital audio down yourself, either destructively (in a DAW and rendering a new file) or on-the-fly with playback gain will, in most cases create enough headroom to avoid ISP clipping.
However, when we look at something I recently posted somewhere else on this forum, we see that a track looks to have been digitally clipped (flat top waveforms) and then reduced in level, after the fact, to fit on the album. There is clear clear flat-top clipping but now at a reduced gain. I suppose the sound will be more consistent across DACs as they won't necessarily clip internally, but how is that wave being created with the samples in the shape they are? Urgh, yuk.
It's just like the sample provided by
@bennetng above. Turn it down 6 dB and it still sounds strange. It's still clipped and sounds like it, it's now just not clipping the DAC.