Limitations of blind testing procedures

[oivavoi]

The other end of things, but 2 or 3 years ago Sound on Sound recorded an electronically controlled grand piano in a large studio (possibly Abby Road). They used a couple different pair of mics and each of those when thru AD/mic pre units ranging between $450 and something like $10-12 k. They put the files up and let people vote on them for preference. There was also a discussion on their forums for identifying which was which. No consensus and not many takers originally. Once they had a fair number of votes for preference the results were split pretty evenly. The number one vote getter: the $450 unit. 2nd was a moderately expensive unit, and third was the 2nd least expensive unit. The real take away was no unit got overwhelming support as best. Now this may not mean they sound the same as much as different people had different preferences. However, this along with a few similar tests posted simply for difference finding indicates ADC's at any level from somewhere below $1k is a solved problem. Microphone pre's unless you are looking for character and color are a solved problem. I think the playback end is similar. Pre's, DAC's and such are all a solved problem unless you purposely skew the device for character. Amps come close, but I agree they interact with speakers or at least often can. So they might not be solved for all speakers, but it is getting very close to a solved problem. Transducers are the areas yet to be solved and may be somewhat unsolvable in the near future.

I posted the 8th gen copies and the digital originals using a pair of $400 recording interfaces. That is in my opinion pretty convincing evidence lower price digital does little harm.

Good comment. I would argue that if you couple performance as in the Hypex amps (or performance which comes close) with active crossovers, the amp/speaker interaction is essentially a solved problem by now. Transducers – not so much.

 

[watchnerd]

So if thats the case I would ask you what is the minimum price I need to pay for a dac to acheive state of the art performance?

I think the bit you are missing here is its not about research. Equipment is still built to a price point and compromises are always made. So whilst I have some agreement with your assertion that a lot of electronics is a solved problem, the reality is not all equipment is equal.

A simple example is for the amplifiers you mention. Reduce the transformer size to reduce cost but in doing so reduce drive capability for difficult speakers.

State of the art performance or audibly transparent performance? State of the art is a label and untestable. Audibly transparent is testable. And based on past results, I contend that we know what makes for an audibly transparent amp or DAC.

Of course not all equipment is equal. But listening tests are only worth the hassle if they inform us of things that measurements cannot.

In the case of amplifiers and the ability to drive difficult speakers, you look at slew rate and damping factor, among other things.

 

[watchnerd]

Good comment. I would argue that if you couple performance as in the Hypex amps (or performance which comes close) with active crossovers, the amp/speaker interaction is essentially a solved problem by now. Transducers – not so much.

Amp/speaker I would agree. Active speaker + room, long ways to go.

 

[watchnerd]

I would also assume the dac in the Emotiva Stealth to be the limit of audibility. Another question is whether there's any point in using independent dacs for each channel in actively run systems. The Kii Three do that, as do Linn in their active systems. My hunch would be that it's not necessary. But I struggle to believe that Kii and Bruno Putzeys would do something just to show off. Maybe it does measure demonstrably better?

It could be related to noise and signal path.

 

[watchnerd]

Audible SOTA maybe $500 for the Emotiva Stealth. Or perhaps a similar amount for some prosumer DACs. Possibly between $500 or $1000 for various versions of the TEAC offerings.

The TEAC UD-301 gets you a dual-mono, balanced DAC with asynchronous USB for $350. It's hard to think of what more would be needed for home playback, 2-channel use.

If you don't need to drive long runs of cables or can't use XLR, the OL DAC has noise, crosstalk and J-test specs (don't see those published often, so kudos to them) < -100 dB for $129.

When @amirm finishes measuring the FiiO Taishan we'll get to see how close a stupid cheap $20 DAC can get to those numbers.

 

[oivavoi]

Amp/speaker I would agree. Active speaker + room, long ways to go.

Agree. I think it's also more difficult to speak of something as a problem to be "solved" when it comes to how loudspeakers fill a room with soundwaves. I believe there might be unavoidable trade-offs here. Spaciousness and envelopment may conflict with razor-sharp imaging, for example. It is also the issue of whether the goal is to create the illusion of looking into the studio/recording venue, or to create the illusion that the music is being played live in one's own living room. My experience with omnis, and my limited experience with multichannel, is that these types of playback solutions increase the feeling of being in the same room as the musicians. I don't have any experience with them, but I think dipoles such as the Linkwitz speakers can also be good for this. But when it comes to looking into the recording venue through a virtual window, as if looking onto a stage, small active monitors together with subs in the near-field can be hard to beat, in my experience.

EDIT:
And, to tie it to the topic of the thread: I'm not sure whether it will be very illuminating to study such trade-offs by trying to get statistically significant preference ratings among college students.

 

[fas42]

Agree. I think it's also more difficult to speak of something as a problem to be "solved" when it comes to how loudspeakers fill a room with soundwaves. I believe there might be unavoidable trade-offs here. Spaciousness and envelopment may conflict with razor-sharp imaging, for example. It is also the issue of whether the goal is to create the illusion of looking into the studio/recording venue, or to create the illusion that the music is being played live in one's own living room.uminating to study such trade-offs by trying to get statistically significant preference ratings among college students.

The problem to be solved is to ensure that systems work at a certain level of competence. This is currently hard to achieve because there is poor knowledge of what has to be measured, and even then the measuring may not be straightforward to do. If the required capability is reached then one gets it all: spaciousness, envelopment, razor-sharp imaging, "realistic" timbres, a convincing illusion - the whole shebang.

The goal is to precisely reproduce what's on the recording, at any desired volume level - no audible artifacts added. Stress testing, playing a high energy recording at elevated volumes, makes the limitations of most systems very clear - and the failings of the setup as revealed by this method need to be addressed; a necessary step to reaching competence.

 

[hvbias]

I'm objectively subjective when it comes to blind testing Everything written below as it applies to me, so I don't have to add IMHO or for me every time. I'm also going to ramble for a bit since there is still a bit of time for the next good football game

Blind testing serves as a useful reference point but they are not the be all end all. I am sincerely thankful that we have it in my industry (medicine) since we are dealing with life or death. When it comes to hifi this is part fun, part accuracy, part illusion. Feel free to assign percentages to each one as it applies to you. I don't think I can do that yet.

As someone who got into the "hobby" as a subjectivist I think I will always have that part of me. For instance I greatly enjoy reading what Nelson Pass publishes. He writes in a very comforting manner; friendly, never authoritative or arrogant, plenty of subjective opinions and objective data and makes understanding electronics/schematics for a non EE a cinch. One thing that is a continuing theme through out his papers is harmonic distortion and how it relates to what we hear. However the highest amplitudes of these types of distortion in his designs most often fall well below -80 dB. I fully respect that he writes about the audibility in these instances with how he hears it even if they are not blind. I think if these amps were put up against the class D Putsys modules people wouldn't be able to tell them apart. Assuming neither amp is clipping and you'd need to be extra vigilant of that since the FirstWatt amps are lower power. However if I were to end up with multichannel speakers that could make use of the FirstWatt amps there is a chance I would chose them over the class D amps because of my preconceived biases.

Rambling point number two that makes me less enthusiastic about blind tests. Everyone does not hear the same. My friends and I conducted a blind test using my turntable to make a needle drop of a Blue Note album that was reissued on vinyl and SACD. The mastering engineer is on record as saying there are no mastering differences between the formats besides the stuff that naturally goes into cutting a lacquer. My cartridge is loaded to more or less be flat (given the limitations). We used my needle drop (some mild processing like clean up to make it so it's not dead obvious) and the redbook portion from the SACD and burned the two to a CDR, levels were matched as closely as I could with Izotope. Most people could not hear the difference between the two. It was quite obvious to me which one was the vinyl since on some of the transients you could hear the "dirtiness" of the vinyl transfer. This was extremely subtle but manifested itself as something that did not sound as clean as the redbook; almost like a transient smearing. I can't describe this properly since it wasn't outright distortion. This is not imply that I'm a golden eared god, but that it is one of the limitations of blind tests, some people are more sensitive of certain things than others. I really respect that Harman had a panel of listeners that were put through their rigorous listening training when it comes to frequency response.

 

[fas42]

IMy friends and I conducted a blind test using my turntable to make a needle drop of a Blue Note album that was reissued on vinyl and SACD. The mastering engineer is on record as saying there are no mastering differences between the formats besides the stuff that naturally goes into cutting a lacquer. My cartridge is loaded to more or less be flat (given the limitations). We used my needle drop (some mild processing like clean up to make it so it's not dead obvious) and the redbook portion from the SACD and burned the two to a CDR, levels were matched as closely as I could with Izotope. Most people could not hear the difference between the two. It was quite obvious to me which one was the vinyl since on some of the transients you could hear the "dirtiness" of the vinyl transfer. This was extremely subtle but manifested itself as something that did not sound as clean as the redbook; almost like a transient smearing. I can't describe this properly since it wasn't outright distortion. This is not imply that I'm a golden eared god, but that it is one of the limitations of blind tests, some people are more sensitive of certain things than others. I really respect that Harman had a panel of listeners that were put through their rigorous listening training when it comes to frequency response.

Listening, very specifically, for that "dirtiness" is how to distinguish levels of quality of playback - it is a form of distortion, because it is not what the microphones transferred to the recording media. In fact, this "dirtiness" is what forms a constant background "noise" to most playback, and allows most people to easily distinguish reproduction from the "real thing".

 

[watchnerd]

It was quite obvious to me which one was the vinyl since on some of the transients you could hear the "dirtiness" of the vinyl transfer. This was extremely subtle but manifested itself as something that did not sound as clean as the redbook; almost like a transient smearing. I can't describe this properly since it wasn't outright distortion.

Sounds like crosstalk to me. Vinyl often has much higher crosstalk and making a digital rip doesn't make that go away.

 

[oivavoi]

If the required capability is reached then one gets it all: spaciousness, envelopment, razor-sharp imaging, "realistic" timbres, a convincing illusion - the whole shebang.

I agree with much of what you're saying. But I tend to think that this is wrong. Let me ask you: Have you ever heard a system which had razor sharp imaging, and at the same time had a huge sense of spaciousness and envelopment? If we stick with two channel stereo for now: I know that I've heard systems with razor sharp imaging (I would say that my system has that), and I've also heard systems with a huge sense of spaciousness and envelopment (I don't have that). But I've yet to hear a system which I would give the maximum score on both of these parameters.

Based on my understanding of psychoacoustics, this is because the thing which leads to spaciousness and envelopment is having lots of reflections, and this very thing may be detrimental for the stereo image. I suspect that the only way of solving this is to listen in a very large room in which reflections come quite late, with loudspeakers and acoustic treatments (diffuskon etc) that match the room.

 

[Blumlein 88]

I agree with much of what you're saying. But I tend to think that this is wrong. Let me ask you: Have you ever heard a system which had razor sharp imaging, and at the same time had a huge sense of spaciousness and envelopment? If we stick with two channel stereo for now: I know that I've heard systems with razor sharp imaging (I would say that my system has that), and I've also heard systems with a huge sense of spaciousness and envelopment (I don't have that). But I've yet to hear a system which I would give the maximum score on both of these parameters.

Based on my understanding of psychoacoustics, this is because the thing which leads to spaciousness and envelopment is having lots of reflections, and this very thing may be detrimental for the stereo image. I suspect that the only way of solving this is to listen in a very large room in which reflections come quite late, with loudspeakers and acoustic treatments (diffuskon etc) that match the room.

I would say if your system has lots of envelopment in its character you can't have razor sharp focus. However, if it has razar sharp focus it might also have good envelopment. Just experimenting with microphone techniques in a large space I have concurrently recorded with different mic methods. Over a playback system not overly enveloping by nature, one technique gives a focused sound. Not razor sharp, but pretty sharp. Another sounds fairly enveloping. Yet another sounded like too much hall and not enough music.

You get the same thing with different sound spaces. Some venues are more sharply focused than others. Some more enveloping.

This less than 3 minute video recorded in 15 different spaces illustrates this nicely.

 

[fas42]

I agree with much of what you're saying. But I tend to think that this is wrong. Let me ask you: Have you ever heard a system which had razor sharp imaging, and at the same time had a huge sense of spaciousness and envelopment? If we stick with two channel stereo for now: I know that I've heard systems with razor sharp imaging (I would say that my system has that), and I've also heard systems with a huge sense of spaciousness and envelopment (I don't have that). But I've yet to hear a system which I would give the maximum score on both of these parameters.

Based on my understanding of psychoacoustics, this is because the thing which leads to spaciousness and envelopment is having lots of reflections, and this very thing may be detrimental for the stereo image. I suspect that the only way of solving this is to listen in a very large room in which reflections come quite late, with loudspeakers and acoustic treatments (diffuskon etc) that match the room.

Not many. But if the recording is reasonable, the playback is competent, and the rig can be raised to realistic volume levels then it happens. One of my test CDs is Led Zepplin I, the original mastering - this produces mammoth spaciousness, and if it is able to go loud without problems the sense of envelopment is intense - yet the individual instruments and other sound elements are extremely precisely defined, a midst the "cacophony" - of course, it's a rare rig that can pull this off, which, of course, makes it good testing material.

 

[watchnerd]

yet the individual instruments and other sound elements are extremely precisely defined

Precisely defined?

The original Led Zep I is noted for the marked 'leakage' of Plant's vocals into the other tracks and even pre-echo from tape saturation on "You Shook Me."

 

[March Audio]

State of the art performance or audibly transparent performance? State of the art is a label and untestable. Audibly transparent is testable. And based on past results, I contend that we know what makes for an audibly transparent amp or DAC.

Of course not all equipment is equal. But listening tests are only worth the hassle if they inform us of things that measurements cannot.

In the case of amplifiers and the ability to drive difficult speakers, you look at slew rate and damping factor, among other things.

You are still missing the point here. We might know what makes a transparent amp or da but that doesnt mean in reality that is what is delivered in products for a bunch of reasons.

If you want to compare thosee products then you still want to minimise the biases of sighted listening.

Btw Im not an advocate of "measurements tell us everything"

 

[watchnerd]

You are still missing the point here. We might know what makes a transparent amp or da but that doesnt mean in reality that is what is delivered in products for a bunch of reasons.

Do you think there is something that can't be electrically measured that distinguishes one amp from another?

 

[tomelex]

sorry, cant resist, NO. now back to BE718

 

[fas42]

Precisely defined?

The original Led Zep I is noted for the marked 'leakage' of Plant's vocals into the other tracks and even pre-echo from tape saturation on "You Shook Me."

There may be technical shortcomings, from the POV of a recording engineer, but I was referring to being able to "see" the image of each sound element very specifically, in space. Amusingly, I have an audio friend down the road, and he has been "conditioned" by me to use this album - when he's feeling confident, he gives me a look, and says something along the lines of "Well, we have to pull out the big test now!!" ... .

 

[watchnerd]

There may be technical shortcomings, from the POV of a recording engineer, but I was referring to being able to "see" the image of each sound element very specifically, in space. Amusingly, I have an audio friend down the road, and he has been "conditioned" by me to use this album - when he's feeling confident, he gives me a look, and says something along the lines of "Well, we have to pull out the big test now!!" ... .

I get the purpose of the scale test, but why that album, in particular?

I usually stress test things with the Mercury Living Presence recording of Dvorak's 9th, "New World", 4th movement.

 

[fas42]

I get the purpose of the scale test, but why that album, in particular?

I usually stress test things with the Mercury Living Presence recording of Dvorak's 9th, "New World", 4th movement.

Largely because when a rig can't do it, it fails miserably. As an example, in "Babe I'm Gonna Leave You", whether the extended drumming workout is handled cleanly - the dynamic contrasts, and ebb and flow of the songs should bowl you over, throughout the album.