Are there any long-form listening tests?

[dscottj]

I'm figuring most people here will be very familiar with ABX testing of streaming bit rates and how many people struggle to hear the difference between music encoded with lossy, lower bit rates and music encoded losslessly with much higher bit rates. I myself have experience this.

But that's just a single track. Sometimes it's not even that much.

Has anyone done any research on the effect of bit rates and codecs over long listening sessions? I'm talking album-length or longer, 45mins to 1.5 hrs. I've looked around, on and off, over the years and have never found any.

The reason I ask is because of my own personal, admittedly highly subjective, experience. I've been mucking around with streaming audio for about a decade now, when I got a receiver that was capable of it. Back then the only service I knew about was Pandora and, a little later, Google Play. Both services topped out at comparatively low bit rates.

I tried them both. What I consistently noticed was that, at first, they sounded fine. Quite nice. Whee! Lots of free music! Turn it up! But then as my listening session went on, I noticed I was turning the volume down. What started out sounding fine had me reaching for the volume knob about a half hour later to *make it stop*.

This is a complete reversal of my experience with CDs and (later) Tidal streaming. With those sources, the longer I listened the more likely I was to turn it up. This is a very consistent behavior for me (it drives my family nuts, especially on weekend mornings).

Please note that I am making no claims, just the following observations:

• The only ABX listening tests in the era of variable bit rates I've ever found have been short-term sessions. One track or less.
• In the thirty-plus years I've spent listening to high bit rate digital audio, my habit is to turn up the music as the album progresses. This is an extremely consistent behavior.
• When I streamed low bit rate music I at first could hear no real difference, but instead of turning the volume up as it progressed, I turned it down. Again it was an extremely consistent behavior.

So how about it. Is there any research out there that shows this to be a subjective behavior on my part? I can't find any.

 

[Blumlein 88]

I don't know of any research for your specific situation. The ability to hear differences is better short term in every parameter tested. The question might be if you were to do quick switched short term ABX could you hear the compressed stream as different? Even though in casual listening it seems fine at first you might find you can hear a difference were you to test yourself.

I've had the same subjective uncontrolled experience as you. Listening to compressed music seems okay at first, but seems to annoy over time. But this is usually when it sounds merely ok and not good initially.

If you wanted to try it yourself, easy enough to get a full data file and an MP3 at moderate bit rate. Put them into Foobar and listen however long you wish. You could use an entire album perhaps. Listen 30 minutes and choose what you think is what and over a few days you'd have enough results to see if you are hearing a difference or not.

 

[GGroch]

There has been speculation that distortion causes listener fatigue for quite a while. This 1989 article by a Hans Fantel, a founding editor of stereo review says it is definitely so. I am not sure his examples support his claim. For decades people happily listened to music on A.M. radio all day long.

 

[dscottj]

For decades people happily listened to music on A.M. radio all day long.

Well, yes, but those same people nowadays are listening to low bit rate music all day long, almost always IME as background music. I'm not sure I've ever read about anyone JUST listening to music on AM radio for any length of time. That's something likely before my time .

 

[rdenney]

A similar question is whether distortions that can’t be consciously identified (using short-term memory, objective analysis, and intense intellectual focus) are detected at a deeper level, when there is an emotional connection (as music will often create, with me at least).

The emotional part is in a different part of the brain, and bypasses the objective analysis parts of my thinking, at least. When I listen to music to analyze it, I can’t connect with it the same is when I listen for simple enjoyment. It’s either one or the other. But if I listen for enjoyment, anything that intrudes on that non-analytical pathway becomes an annoyance. I wonder if the perceptual mechanism work differently there.

I’m sure somebody knows of any careful research done in that vein, if it exists.

Rick “thinking this is the same question as the OP’s, just worded differently” Denney

 

[amirm]

You can learn to hear compression artifacts over time and have them become more annoying. Whether this has happened to you or not, is hard to say. Compress a CD to progressively lower it rates and perform some blind tests and see at what level you can reliably tell the difference.

 

[GGroch]

I think your questions are interesting, but am very skeptical that bit rates or compression artifacts impact listener fatigue. Skeptical because:

- Your fatigue seems to happen when you know you are listening to low bit rate streams. If you did not know the bitrate we could eliminate the effect of unconscious bias
- In decades past listeners absolutely did seriously listen to and enjoy music sources much worse than Spotify streams. From my own experience, my longest listening sessions were in the 70s on LPs that objectively are not close to Spotify in fidelity. Our parents/grandparents listened seriously for long periods to music on A.M. radio and spring driven Victrolas. Classical music...with its huge (live) dynamic range, was much bigger then than now. Classical music is where lack of fidelity should be most annoying.
- Live concerts/all day festivals seldom match the fidelity of well recorded streamed music, yet, it has strong emotional connection.

A google search on listener fatigue shows lots of charlatans with answers. Paul McGowan (an ASR favorite ) posted a video saying upgrading your electronics will solve it. A hearing aid vendor says hearing aids will solve it. From Amir's response, it appears not knowing much about compression artifacts may solve it (absolutely not calling Amir a charlatan)

 

[pozz]

Listening training is like a long-term listening test.

Also, there is a clear progression in acceptal standards for digital formats over the last 30 years. Thomas Lund of Genelec wrote about this. Initially, new media and playback systems seem amazing and lifelike, but as time goes on and the action and perceptual experience is better understood, the deficiencies stand out.

 

[dscottj]

- Your fatigue seems to happen when you know you are listening to low bit rate streams. If you did not know the bitrate we could eliminate the effect of unconscious bias
- In decades past listeners absolutely did seriously listen to and enjoy music sources much worse than Spotify streams. From my own experience, my longest listening sessions were in the 70s on LPs that objectively are not close to Spotify in fidelity. Our parents/grandparents listened seriously for long periods to music on A.M. radio and spring driven Victrolas. Classical music...with its huge (live) dynamic range, was much bigger then than now. Classical music is where lack of fidelity should be most annoying.
- Live concerts/all day festivals seldom match the fidelity of well recorded streamed music, yet, it has strong emotional connection.

• Which is why I was wondering if anyone has done any serious long format ABX testing. It seems that, forty years after digital music was introduced, nobody has.
• As I recall, properly cared for (and recorded) vinyl is measurably better than low bit rate digital. There probably is academic literature studying this. Citations welcome!
• Classical music converted to digital first--when vinyl was still the playback standard--and has never looked back. AFAIK classical music does not appear on vinyl to this day, at least IME. As far as relative popularity then versus now, again citations welcome!
• Live music has different goals than hi-fi IMO. Our hobby is recreating a performance as accurately as possible. Live music is ensuring everyone can hear what's going on. Apples vs. oranges IMO.

 

[dscottj]

You can learn to hear compression artifacts over time and have them become more annoying. Whether this has happened to you or not, is hard to say. Compress a CD to progressively lower it rates and perform some blind tests and see at what level you can reliably tell the difference.

And if I had any idea how to do that I might try. I was just wondering if anyone better equipped than me had tried. It seems nobody has.

 

[rdenney]

Yes, but in the 70’s (or the 50’s) that was the best we had experienced, and we had little better to compare it to. But sound still mattered—Kirsten Flagstad would not agree to record Wagner’s Ring... for Decca’s John Culshaw in 1964 because she had never heard anything but monophonic 78’s and AM radio, and didn’t want anything to do with such annoying sound. She relented when Culshaw bought her a good stereo and played her some good recordings.

But Amir’s suggestion is a good one. I think the way to do it would be to choose a favorite CD (something I’m willing to listen to 30 times over the course of a month or two). Then rip and append all the tracks into a single uncompressed file. Make a copy using MP3, compressed to, say, 256 kbps—too good to hear in a focused ABX but still compressed. Convert it back to lossless, to get it back into the same format. Make a master file by appending one or the other based on a coin toss as numbered tracks—too many to remember the order, say, 30. Hide the list. Wait a week. Keep a pad next to the listening chair, and listen to the tracks, writing down impressions numbered by track. Play the tracks in order, or not.

Or, is there ABX software that allows album-length samples? The problem there is that it’s not enough to be able to tell the difference (and we would want that to be difficult in any case)—one must make subjective notes, and ABX software never tells you which was which after the fact.

Rick “not sure he’s up for that experiment except during the dead of winter” Denney

 

[dscottj]

I'd like to emphasize that I consider myself a HUMAN objectivist. Show me the stats an I'll believe them. I couldn't reliably pass a quick ABX test when it came to bit rate. Therefore, I was not hearing a difference. It was subjective. I was rooting for the lower bit rate because that was all I had for streaming. Free music! Turn it up!

AND YET... that volume knob went down instead of up, consistently. Shift to high bit rate, and the volume went up, consistently. This is a long term test measured in MONTHS.

You guys are giving me rationales, but no citations.

 

[SIY]

• Which is why I was wondering if anyone has done any serious long format ABX testing. It seems that, forty years after digital music was introduced, nobody has.

The people who claim that all the "magic" can't be captured in a short duration ABX test have over the decades refused to do controlled experiments to support the hypothesis.

ABX would likely not be the correct format to use for long term controlled testing. The classic BAS tests using take home boxes was one example of a more suitable experimental protocol, and of course, gave the expected null results. My own approach using timers and random selection, described in my Linear Audio article on controlled testing, likewise gave no unexpected results, though I cut it short after a few months and haven't bothered to try to do it again.

 

[rdenney]

Citations?

Rick “may be old news to you, but not the OP (or me)” Denney

 

[GGroch]

• Live music has different goals than hi-fi IMO. Our hobby is recreating a performance as accurately as possible. Live music is ensuring everyone can hear what's going on. Apples vs. oranges IMO.

If you define the goal of enjoying music at home as "recreating a performance as accurately as possible" then by definition listening to lower fidelity sources is less enjoyable. How is that enlightening?

In the real world audiophiles and melophiles are distinct groups. In my experience, melophiles who would spend hours in record shops, or discussing jazz, or attending live performances, or performing themselves, are not directly correlated to those owning the most expensive audio systems and listening to only Hi Res sources. By contrast, there is very little discussion of musical performances on ASR compared to a discussion of DAC specs.

If we whittle down our discussion to audiophiles whose enjoyment depends on accuracy of playback, then the OP's question is much less interesting. Cognitive bias would render it meaningless if the audiophile listener knows their source material is not optimal. Of course they will enjoy it less.

SIY in post #13 describes blind testing over time, and I would also like links to his or other similar citations. (update, SIY thanks for the citations But, this does not describe the situation described in the O.P. in which cognitive bias overwhelms any minute differences in source quality that are not audible in short A/B/X tests.

 

[SIY]

Citations?

Rick “may be old news to you, but not the OP (or me)” Denney

My apologies, it wasn't BAS, it was Clark. The results are summarized and discussed here.
Power cords were also tested long term by Kiang. I still have the spreadsheet with the results in case that's disappeared over the years.
My timer tests were described in my article on testing in Linear Audio (linked to on this forum a few dozen times).

 

[dscottj]

My apologies, it wasn't BAS, it was Clark. The results are summarized and discussed here.
Power cords were also tested long term by Kiang. I still have the spreadsheet with the results in case that's disappeared over the years.
My timer tests were described in my article on testing in Linear Audio (linked to on this forum a few dozen times).

I don't feel that long term tests with things like cabling is necessarily relevant. In that case, there are no measurable differences between the cables and cords, just a claimed perception. As expected, when all the variables are eliminated no difference is observed.

Here we have a (IMO) different situation. There most definitely IS a measurable difference between the files being tested. Short ABX testing reveals that many (most?) humans can't reliably tell the difference between high and low bitrate music streams.

However, I found this extremely interesting and (IMO) relevant conclusion from the first article you linked:

"However, using the A/B/X test, the SMWTMS not only proved audibility of the distortion within 45 minutes, but they went on to correctly identify a lower amount."

I think this may indicate a confusion in my vocabulary. When I say "short term" I mean 3-5 minutes or less. When I say "long term" I mean 45 minutes to 1.5 hours, not weeks or months. I can see how my later comments may have confused this distinction and inadvertently tread on some objectivist mines.

So it would seem not only has there been testing done, it has proven that given a moderate amount of time a trained listener can in fact reliably distinguish noisier recordings. Or am I misunderstanding the conclusion?

 

[rdenney]

That research is interesting, thanks.

The conclusions in the thread are built on the assumption that a difference must be detected reliably before a preference can emerge. But this was precisely my question: detecting a difference requires analytical evaluation. Even asking the question to the subject “Are these different” invites that analytical pathway.

Even if the analytical brain has fully concluded that no difference exists, is there another sensory pathway—a non-analytical emotional or artistic pathway—that might detect something the analytical pathway is too focused to detect? It may be a forest and trees thing. When I’m doing ABX tests, I’m looking for artifacts. I’m not even listening to the musical expression—that’s an irrelevant distraction to me in an ABX test. When I’m listening for enjoyment, I’m listening for the encompassing effect and any analytical focus becomes the distraction.

I think this is not a question for engineers, who are often trapped by their analytical processes (I’m an engineer and I teach engineers, so I suffer from and observe the passion for precision over accuracy). I think this is a question for psychologists.

I’m just posing a variation on the OP’s question. He might be fully satisfied with your citations, but what I’ve read so far generalizes about how people perceive from an engineer’s perspective.

Rick “not a psychologist and not defending a position” Denney

 

[GGroch]

.......Even if the analytical brain has fully concluded that no difference exists, is there another sensory pathway—a non-analytical emotional or artistic pathway—that might detect something the analytical pathway is too focused to detect?.....When I’m listening for enjoyment, I’m listening for the encompassing effect and any analytical focus becomes the distraction.

To paraphrase: is there something about blind testing that puts our minds into an analytical mode that reacts differently, perhaps more crudely, than when our minds are listening for enjoyment? Listening for enjoyment is after all, our normal mindset when listening to music. Did I get that right?

I admit to being cognitively biased against that possibility. It is an argument snake oil audiophiles use in rejecting ABX, and it is amazing how this rejection heightens our awareness of all kinds of new immeasurable distortions. A recent example.

I suppose one way to test it would be to send out music selections for people to rate in terms of enjoyment (how much they liked the music, or how enjoyable the listening was, not telling them that the selections were being sent to different people using different bitrates.
More directly tied to the OP's question, if the music was streamed you could also measure if there was a correlation between bitrate and how long each user listened. This could provide insight into if there is a correlation between bitrate and enjoyment. My guess is there is not.

 

[SIY]

To paraphrase: is there something about blind testing that puts our minds into an analytical mode that reacts differently, perhaps more crudely, than when our minds are listening for enjoyment? Listening for enjoyment is after all, our normal mindset when listening to music. Did I get that right?

I admit to being cognitively biased against that possibility. It is an argument snake oil audiophiles use in rejecting ABX, and it is amazing how this rejection heightens our awareness of all kinds of new immeasurable distortions. A recent example.

I suppose one way to test it would be to send out music selections for people to rate in terms of enjoyment (how much they liked the music, or how enjoyable the listening was, not telling them that the selections were being sent to different people using different bitrates.
More directly tied to the OP's question, if the music was streamed you could also measure if there was a correlation between bitrate and how long each user listened. This could provide insight into if there is a correlation between bitrate and enjoyment. My guess is there is not.

I’ve seen this variation of the Dragon in my Garage expressed many times over the years, most notably by Nelson Pass. Problem is that these tests show extremely acute aural sensitivity to things like frequency response, level, localization, compression... so this excuse deteriorates into a special pleading.