I cannot trust the Harman speaker preference score

[tuga]

Y

Most of this research is available at AES e-library, and much of it Floyd's book Sound Reproduction. Some of it is in by blog (www.seanolive.blogspot.com)

We seem to be heading to a loop if I question the methodology and the interpretation of the data then and you refer to said methodology and interpretattion.

 

[tuga]

It is pretty normal for a single speaker(obviously the center does this, but also the surround speakers sometimes) to be the only one playing a vocal line or some sound effect in multi-channel film soundtracks.

Also, multi-channel music is a thing and an increasingly important thing considering the massive amount of effort going towards Atmos music production now. In multi-channel music it is more common for a single speaker to be playing vocals or some instrument, or even for vocals to be panned across multiple speakers(Yello's Point album pans vocals 360 degrees around the listener and one speaker is primary for most of the pan).

So, I do think mono performance has some value standing alone.

I don't question mono for objective observation-driven assessement of loudspeaker performance, only for preference. For starters, a single speaker is unable to create the stereophonic illusion.

 

[Blumlein 88]

I don't question mono for objective observation-driven assessement of loudspeaker performance, only for preference. For starters, a single speaker is unable to create the stereophonic illusion.

And what makes you think a pair of less neutral speakers will create a better stereophonic illusion than a pair of more neutral speakers?

 

[krabapple]

Not to mention that stereophony is mainly affected by speaker position and output levels, and the recording content itself, rather than the speakers themselves.

 

[Sean Olive]

We seem to be heading to a loop if I question the methodology and the interpretation of the data then and you refer to said methodology and interpretattion.

Well I frankly don't understand your point. I found this graph, which I believe you posted, and is basis of your argument.
This is a study done by Floyd Toole in 1983 where he shows the sound quality and spatial ratings of three loudspeakers in mono and stereo.

The graphic tells us the listeners were less discriminating in stereo compared to mono, for both sound quality of spatial quality. Most of the difference in spatial ratings between tests can be attributed to the Quad, a dipole loudspeaker.

In mono, its colorations and higher directivity produced lower fidelity and spatial ratings. In stereo, the colorations were apparently less audible and it had higher sound quality and spatial ratings. As Floyd shows in the paper, the Quad's spatial ratings varied significantly with program material (unlike the others) where independent ratings were given. For choral and pop recordings its spatial ratings were last, and approximate its ratings given in mono. I wonder if its sound quality ratings also fluctuated with program. Unfortunately, it doesn't show this data, but it timbre tracked spatial that would indicate they are not independent.

First, it seems a bit selective to discount mono tests based on 1 test and 1 loudspeaker. I don't think any conclusions or generalizations can be made from one sample. I don't know if separate tests were done for sound quality and spatial ratings so the ratings were independent judgements. This would have been more work but minimize potential halo bias effects ( ratings for different attributes tend to be highly correlated with preference). In other words, did the change in timbre across tests influence the spatial ratings?

Also, the results of this test, did not convince Floyd to abandon mono testing and do tests in stereo. In fact, if anything it convinced him to stick to mono and abandon stereo because mono are more sensitive tests. The spatial differences in the stereo tests were largely isolated to one minimal and largely variable and attributed to the recordings.

His conclusions:

" Conducting the listening tests in monophonic and stereophonic modes revealed some important similarities and some interesting contrasts in the results. In general, assessments of sound quality were very similar in both modes, except for loudspeakers with significant imperfections, in which case the monophonic evaluations result in lower ratings than the stereophonic tests. In other words, in respect of Judging sound quality - the transduction accuracy of the loudspeaker - monophonic tests appear to be more demanding, or stereophonic tests less sensitive. "

AES E-Library » Subjective Measurements of Loudspeakers: A Comparison of Stereo and Mono Listening

Earlier papers have described monophonic comparisons of loudspeakers in controlled listening tests that produce sound quality assessments with considerable consistency and order. Repeating the experiments in stereo, using a positional substitution technique, produced sound quality ratings with...

www.aes.org

 

[Holmz]

And what makes you think a pair of less neutral speakers will create a better stereophonic illusion than a pair of more neutral speakers?

Two negatives make a positive?
(It is sort of like synergy )

 

[Duke]

Thank you very much Sean for taking the time to do an in-depth reply AND provide that chart summarizing the conditions and findings of multiple relevant studies. Very interesting! In general I agree with what you say.

If we ignore frequency response (which does affect spatial perception) which aspect of loudspeaker performance would give different spatial ratings in mono versus stereo? The only one I know is directivity...

Agreed, but let me give an example of a situation where spatial quality in stereo is imo unlikely to have been reliably predicted by spatial quality in mono:

Consider two directional monopole main speakers, such as JBL M2's, set up with the speakers wider than normal and with axes criss-crossing in front of the listening position. The first significant sidewall reflection for the left-hand speaker is the relatively long-path reflection off the right-hand wall, and vice-versa. So the first strong lateral reflections arrive after a considerably longer time delay than in the single-speaker case, and they are de-correlated with respect to the first arrival sound, which again cannot be replicated by a single speaker. Both reflection arrival time and correlation/de-correlation can arguably play a role in spatial quality. (Note that this sort of extreme toed-in configuration does not work well with most speakers. I do not know for a fact that it works well with the JBLs because I've only tried it with speakers having a nominal pattern width of 90 degrees or less, rather than the JBL's 120 degrees.)

... and most of the research on speaker directivity has not come up with definitive guidelines.

Once again, in my opinion (and acknowledging how much credibility such opinions are due):

Looking at spatial quality from a different angle, in the playback room there is arguably a "competition" between the venue spatial cues on the recording (whether they be real or engineered or both), and the "small room signature" cues of the playback room. In this paradigm, the ideal would be to effectively present the recording's spatial cues while minimizing the strength of the playback room's inherent spatial signature, the goal being a plausible "you are there" presentation (rather than a plausible "they are here" presentation). So rather than focusing on pleasing room signature cues, the emphasis is more on how to tip the above-mentioned "competition" towards the venue cues on the recording being perceptually dominant.

Using the example of the toed-in JBLs above, imo the relatively long time delay for the first sidewall reflections reduces the strength of the "small-room signature", while those reflections will be spectrally correct (assuming the walls aren't overdamped, which is all too easy to do) so they will be effective "carriers" for the reverberant tails on the recording, said reverberant tails being an important component of the "venue spatial cues" on the recording. Ime this set of attributes helps to tip the spatial quality towards a "you are there" presentation, and I don't think that could have been predicted from mono listening.

It's difficult to manipulate DI while not manipulating other parameters.

True for conventional direct-radiator monopoles in a stereo system.

I don't think compromising the sound quality of a loudspeaker for better spatial quality is a good tradeoff...

Agreed. The imo highly innovative Acoustic Research "Magic" speaker was an example of a speaker with generally superior spatial quality but arguably reduced sound quality relative to its competition (weak bass response for such a large and expensive speaker).

My (controlled blind but non-peer-reviewed) experience has been that there seems to be arrival time and/or loudness thresholds for reflections beyond which sound quality is degraded, but within which spatial quality can be improved without degradation of sound quality. And it has been my subjective experience that when the reflections are neither too strong nor too weak, arriving neither too early nor too late, decaying neither too fast nor too slow, and of course are spectrally correct, then the perceived spatial quality varies greatly (and enjoyably) from one recording to the next, implying that the venue cues on the recording are perceptually dominating over the "small room cues" of the playback room.

 

[Holmz]

@Sean Olive
If you would be so kind…

How would rate the importance of:
- Olive score
- Compression
- Distortion
- etc

In the overall selection process?
(Personally I use more of an even mix of OS, Distortion, and maybe a touch deweighted on compression.)

I am leaving out looks, WAF-index, etc.

 

[Sean Olive]

I don't question mono for objective observation-driven assessement of loudspeaker performance, only for preference. For starters, a single speaker is unable to create the stereophonic illusion.

Well, please feel free to ignore the research. Others have found it useful. I've done several of these mono vs stereo comparison over the years and found stereo produces little new insight into listener preference, and usually adds more noise. Mono tests are simply more reliable and sensitive methods to assess the performance of a loudspeaker.

Stereo is 1950s, and the research focus now is immersive audio.

 

[ROOSKIE]

And what makes you think a pair of less neutral speakers will create a better stereophonic illusion than a pair of more neutral speakers?

Not to mention that stereophony is mainly affected by speaker position and output levels, and the recording content itself, rather than the speakers themselves.

*I voted "It is a good metric that helps, but that's all"*

Imaging.
After testing many speakers now (all sighted, my GF was blind a few times), this does not align automatically with my experience.

Imaging is not defined exactly the same by all.
I will define the stereo effect as the whole experience of a soundstage and the whole experience of envelopment by the sonic experience. (I am only talking about 2 channel stereo not more, I have no interest in going beyond 2 channel for music at all)
This is driven by the speakers dispersion and my minds ability to let go of the speakers presence. Some speakers disappear and others not so well. This was also something I was able to test with my Gf a bit as she was not informed ahead of time what was being examined.

In any case I am certain (if given a choice) I prefer a less neutral speaker with the dispersion characteristics I prefer over a more neutral speaker with unappealing dispersion. The same goes for dynamics and ability to play loud. (obviously within limits of reasonable frequency neutrality, we are talking about quality speakers on up)
This is significantly more pronounced when sitting side by side with my Gf listening, which I am lucky enough to experience over half the time I spend listening. Some speakers lose almost nothing and others are severely crippled due their dispersion characteristics.
I will give props to The Revel speakers I have heard in this regard as well as the JBL 4309. The Joseph Crowe Audio 1159 is fantastic in this regard, sounding almost exactly the same for 1 or two people.

If I take the KEF R3 with pinpoint style imaging and extremely neutral response for a passive and high H. score vs the 4309 with a fairly poor H. score and a large life sized presentation there is zero question which speaker I preferer. I prefer the 4309 and so does my Gf without any coaching. And bear in mind I only want what I like to "win" because I like it the most. I will sell any speaker on a dime if I find another I prefer. I am a love 'em and leave 'em material possession guy.

 

[Sean Olive]

@Sean Olive
If you would be so kind…

How would rate the importance of:
- Olive score
- Compression
- Distortion
- etc

In the overall selection process?
(Personally I use more of an even mix of OS, Distortion, and maybe a touch deweighted on compression.)

I am leaving out looks, WAF-index, etc.

The "Olive score" is based on frequency response. Frequency response is the single most important predictor of sound quality. Distortion is only important if its well above audible threshold. Compression is essentially distortion.

In most of our listening tests distortion was not a factor with the home speakers we tested in our listening room at the average level ( 80 dB-weighted SPL). Even with the small 5-6 inch bookshelf speakers, distortion was not a factor.

Most loudspeaker researchers (Toole, Klippel) would agree that frequency response is the biggest predictor of sound quality.

However, distortion can be a significant factor in small powered smart speakers. They usually have dynamic electronic limiters that protect them from overload by rolling off the bass as the excursion /power limits are approached. The sound quality is often dependent on the volume setting. I would say, the predictive model is the least useful for these types of loudspeakers.

 

[sarumbear]

I worked at the National Research Council with Floyd Toole from 1985-1993 and then joined him at Harman in 1993 where I've been ever since. The "clever scientist who created a formula" is me

Toole wrote the seminal papers about loudspeaker measurements and listener preferences in 1985-86. Those tests continued at HARMAN in 1993. Predicting subjective data from objective measurements is not so "bizzare." Many examples of it exist in the world of sensory measurements. Klippel did his PhD on the topic (1990s), Moore & Tam (headphones) and many others.

The data from the model is based on loudspeaker tests done at HARMAN -- not NRC tests. So I'm not sure why it's necessary to prove the model works based on NRC tests. However, since the loudspeaker measurements and listening tests are refined versions of what we did at NRC there are very strong correlations and agreement. Speakers that score well at NRC also score well at HARMAN in spite of differences in listening rooms, listeners, equipment and a speaker mover that we have at HARMAN but not at NRC. The only common link is the scientists conducting the experiments and similar methodology. This tells me that the relationship between subjective and objective measurements is quite robust.

Again, you seem to be obsessed with price being a predictor of loudspeaker. Again, I have data showing a $400 speaker being preferred to a $10,000 speaker in a blind listening test. Do they have the same predicted score? No. If they do, then you have to consider whether the SPL output and distortion will satisfy your application.

I have one question: since the Harman tests done almost 30 years ago, audio reproduction technology has improved quite a lot. DSP was not even a thing then. Were there recent test done using modern speakers where the score is verified to be correlating to listener preferences? Will you personally see the following to come out as an expected result?

Sonos Roam: 5.5
JBL M2: 5.1

Thank you for taking time to comment.

 

[Sean Olive]

I understand that loudspeaker colorations are easier to hear in mono. But does spatial quality preference in mono reliably predict spatial quality preference in stereo?

According to a paper written by Wolfgang Klippel and cited by Floyd Toole in his book (third edition, pages 185-186), the “feeling of space” makes a 50% contribution to "naturalness” (realism and accuracy), and a 70%(!) contribution to "pleasantness" (general satisfaction or preference). Here is how Toole sums it up (page 186):

"Sensations of sound quality and spaciousness contribute equally to impressions of "naturalness", and spatial quality dominated the impression of "pleasantness". Therefore whether one is a picky purist or a relaxed recreational listener, the impression of space is a significant factor."

The point of the preceding two paragraphs being, spatial quality matters a lot.

In the study posted by @tuga in post number 53, going from mono to stereo there is more movement in the spatial quality scores than in the sound quality scores, with which speakers rank first and second trading places. Now admittedly this is a very small sample size, but one of the conclusions one might draw from it (in the absence of additional data, which is the situation for most of us) is that preference in mono is a less reliable predictor of spatial quality preference than it is of sound quality preference.

What are your thoughts on the reliability of spatial quality preference in mono as a predictor of spatial quality preference in stereo?

As a general rule, yes. Things like image width, spaciousness seem to track for across mono and stereo for conventional forward-radiating speakers they do, as shown by Floyd's 1983 paper. The only exception shown so far is a dipole that had higher ratings in stereo than mono that depended on the program. For programs the spatial ratings were similar across mono and stereo tests. I discuss that paper in more detail in this thread.

I'm not disputing that spatial quality doesn't matter, and that it affects naturalness, pleasantness, and other attributes. But a lot of this is moot once you move beyond stereo to immersive audio where the spatial impressions are highly influenced by the number and locations of sources and how the sounds are mixed.

 

[abdo123]

@Sean Olive It is so refreshing to hear someone with your credentials say that Stereo is outdated and irrelevant today.

Sometimes Stereo really seems like the hill this industry wants to die on, with all its faults.

 

[ROOSKIE]

Well, please feel free to ignore the research. Others have found it useful. I've done several of these mono vs stereo comparison over the years and found stereo produces little new insight into listener preference, and usually adds more noise. Mono tests are simply more reliable and sensitive methods to assess the performance of a loudspeaker.

Stereo is 1950s, and the research focus now is immersive audio.

As a general rule, yes. Things like image width, spaciousness seem to track for across mono and stereo for conventional forward-radiating speakers they do, as shown by Floyd's 1983 paper. The only exception shown so far is a dipole that had higher ratings in stereo than mono that depended on the program. For programs the spatial ratings were similar across mono and stereo tests. I discuss that paper in more detail in this thread.

I'm not disputing that spatial quality doesn't matter, and that it affects naturalness, pleasantness, and other attributes. But a lot of this is moot once you move beyond stereo to immersive audio where the spatial impressions are highly influenced by the number and locations of sources and how the sounds are mixed.

I love all the work you have published, I loved reading Tooles book as well as many many forum conversations.
I have several Harman speakers and have recommended them to many.
With all due respect I don't think ignoring the research is what anyone is doing and I find it is a bit to easy to suggest that.

Noise goes both ways. It may be noisy to use just one speaker right? If Speaker A rates 8 out of 10, speaker B 5 out of ten but in stereo people almost enjoy them equally (Say A is still 8 but B is now 7.5) then what is noise to some is smoothing to others.

While I know I am in a niche. Stereo is not the 50's to myself and a small lot of us and I think that is where some disconnect is happening. I love the 2 channel experience, and have absolutely zero interest in a home theater in my living room.
I completely understand why testing in mono is easier and in fact likely far more accurate for a multichannel system where total immersion is near inevitable so the focus is not on getting only two speakers to sound real.
Also for a manufacturer isolating all the parts is important even if in the final use case certain flaws will be mitigated.
Pixel peeping makes sense in that atmosphere. (especially with all Klipple users out there now testing products)

At home stereo is fairly easy to manage and testing a finished system of two speakers not hard.
I need to be sure I get the soundstage quality's I prefer, that is very important to be as I mention a few posts prior to this one.
Essentially I view it as sum of all the parts assessment, I mean if stereo covers up a flaw then who cares since the system will always be used in stereo and never any other way ever.
Also I simply do not enjoy the mono experience, so I can either enjoy the listening/testing session or I can pixel peep in mono and not have any fun.
I understand that you have a staggering level of experience. I am certainly not saying that mono is bad idea, I am a hobbyist not an expert and in no position to be anything else.
Just throwing a real world example of what another enthused guy does at home to pick favorite speakers for a 2.x stereo system.

I have to say that based on my reading of Toole's book he emphasized spatial qualities very often and was a pretty big fan of those old Bipolar speakers.
The spatial qualities of the Harman speakers that I have here at home are pretty special to be honest. Even when the "calculated" Harman score is not the highest. (I wonder what the real score would be blind with real listeners)

Anyway I am rambling now but I am willing to submit myself to some intense mono testing this week at home and reevaluate what I stated here. Yikes.

By the way testing was done at only 80db? Seems a bit tame from my end...

 

[Inner Space]

And what makes you think a pair of less neutral speakers will create a better stereophonic illusion than a pair of more neutral speakers?

I'm old enough to have experienced mono as the majority option, and then the transition to majority stereo, and your question could be answered in two particulars - if a pair of less-neutral speakers has better pair matching than a pair of individually more-neutral speakers, its stereo illusion will be better; and if the more-neutral mono speaker achieved its apparently neutral spectral balance with the help of cabinet talk here and there, perhaps fortuitously filling in FR dips, which phenomenon can't be detected by on- or off-axis measurements, then pairing it with a similar speaker will produce a poor stereo illusion. Good mono usually scales to good stereo, but not quite always.

 

[tuga]

And what makes you think a pair of less neutral speakers will create a better stereophonic illusion than a pair of more neutral speakers?

Did I think that?

 

[Blumlein 88]

I'm old enough to have experienced mono as the majority option, and then the transition to majority stereo, and your question could be answered in two particulars - if a pair of less-neutral speakers has better pair matching than a pair of individually more-neutral speakers, its stereo illusion will be better; and if the more-neutral mono speaker achieved its apparently neutral spectral balance with the help of cabinet talk here and there, perhaps fortuitously filling in FR dips, which phenomenon can't be detected by on- or off-axis measurements, then pairing it with a similar speaker will produce a poor stereo illusion. Good mono usually scales to good stereo, but not quite always.

A speaker with off axis peaks and dips scores very poorly by Harman criteria.

 

[Blumlein 88]

Did I think that?

Your comments imply such can happen if testing isn't done in stereo. There is an example with dipoles, but doesn't appear to be the case for monopoles.

 

[Sean Olive]

I have one question: since the Harman tests done almost 30 years ago, audio reproduction technology has improved quite a lot. DSP was not even a thing then. Were there recent test done using modern speakers where the score is verified to be correlating to listener preferences? Will you personally see the following to come out as an expected result?

Sonos Roam: 5.5
JBL M2: 5.1

Thank you for taking time to comment.

I love all the work you have published, I loved reading Tooles book as well as many many forum conversations.
I have several Harman speakers and have recommended them to many.
With all due respect I don't think ignoring the research is what anyone is doing and I find it is a bit to easy to suggest that.

Noise goes both ways. It may be noisy to use just one speaker right? If Speaker A rates 8 out of 10, speaker B 5 out of ten but in stereo people almost enjoy them equally (Say A is still 8 but B is now 7.5) then what is noise to some is smoothing to others.

While I know I am in a niche. Stereo is not the 50's to myself and a small lot of us and I think that is where some disconnect is happening. I love the 2 channel experience, and have absolutely zero interest in a home theater in my living room.
I completely understand why testing in mono is easier and in fact likely far more accurate for a multichannel system where total immersion is near inevitable so the focus is not on getting only two speakers to sound real.
Also for a manufacturer isolating all the parts is important even if in the final use case certain flaws will be mitigated.
Pixel peeping makes sense in that atmosphere. (especially with all Klipple users out there now testing products)

At home stereo is fairly easy to manage and testing a finished system of two speakers not hard.
I need to be sure I get the soundstage quality's I prefer, that is very important to be as I mention a few posts prior to this one.
Essentially I view it as sum of all the parts assessment, I mean if stereo covers up a flaw then who cares since the system will always be used in stereo and never any other way ever.
Also I simply do not enjoy the mono experience, so I can either enjoy the listening/testing session or I can pixel peep in mono and not have any fun.
I understand that you have a staggering level of experience. I am certainly not saying that mono is bad idea, I am a hobbyist not an expert and in no position to be anything else.
Just throwing a real world example of what another enthused guy does at home to pick favorite speakers for a 2.x stereo system.

I have to say that based on my reading of Toole's book he emphasized spatial qualities very often and was a pretty big fan of those old Bipolar speakers.
The spatial qualities of the Harman speakers that I have here at home are pretty special to be honest. Even when the "calculated" Harman score is not the highest. (I wonder what the real score would be blind with real listeners)

Anyway I am rambling now but I am willing to submit myself to some intense mono testing this week at home and reevaluate what I stated here. Yikes.

By the way testing was done at only 80db? Seems a bit tame from my end...

We design and test speakers in mono but always demonstrate them in stereo.. As Floyd writes in his book, the dominant factor in perceived spatial attributes for stereo tests is the recording itself. And the data from the 1983 test supports that. I've heard the Mirage M1 at NRC and in his home. They sounded wonderful. Very spacious in his large living room, but too diffuse , unfocussed sound stage in the NRC listening room without curtains on the rear and wide wall to absorb the strong rear reflection. More proof that the desired directivity depends on the room acoustics and to some extent the recording.

80 dB is the average level that varies up and down depending on the dynamics and crest-factors of the music. That is what most people consider comfortable in a domestic room.