Master Thread: Are measurements Everything or Nothing?

[ahofer]

I honestly don't think there are many musicians on this site. I say this because a lot of musicians like things that color the sound. Like there are a ton of plugins and hardware that is made just for this very reason. Like you would run a clean signal through it to color the sound. A lot of people also like something like this in their hifi setup. People here seem to think it is wrong to like such things. I think that is very narrow minded myself.

The whole "musician" thing is a red herring. I am a former musician, as is my wife. We know hundreds of pros.

1) There's no evidence they do better in blind tests ranking and telling equipment apart
2) All but one of the musicians I know just don't care about audio equipment
3)anecdotally, I see more musicians on this site than any other audio site I've been on
4) The only alleged advantage they have is a lot of experience with live reference, often from a non-audience perspective. Of course, in many cases, that means they are hard of hearing. And not just the amplified music. If you are seated in front of the brass, you've taken some damage.
5) This is literally the first time I've seen anyone suggest that musicians like to "color" reproduced music.

This line of criticism should probably be retired.

 

[zvukofor]

Musicians just aren't worried about any of these things.

Oh, they do worry about a lot more things than we are here!
And, unfortunately, they're often very uneducated on engineering things...THD? — isn't it a guitar amps brand? )))

 

[ahofer]

Firstly, I have huge respect for ASR and have learned loads from both the reviews and forums over the years. I think that there are only two things that I would say that I generally disagree with. The first is the popular belief that all competently designed amplifiers sound the same. Surely simple logic says that given the huge diversity of said amplifiers, it is highly unlikely that there are not some subtle differences between at least some of them.

The second is that tube distortion or "warmth" can be simply added digitally with DSP. I don't think that it is quite that straightforward...The current plugins/simulations are not an accurate emulation as yet.(Probably will be at some point in the near future though.)

I think the Stereo Review and Clark tests have revealed how alike amps are, and we have no controlled evidence to contradict it (A Futterman OTL and a Pioneer receiver - indistinguishable!). Not perfectly alike, indeed, but literally imperceptibly close. As you saw above, Amir gave amps a "90% measurement" rating, which seems about right.

(Stereo Review, Jan 1987, start around page 78 of the magazine, 80 of the pdf)

But for now, the evidence would seem to suggest that distinctive amplifier sounds, if they exist at all, are so minute that they form a poor basis for choosing one amplifier over another. Certainly there are still differences between amps, but we are unlikely to hear them.

You may be right on reproducing tube warmth, I haven't seen any blind testing. However, I suspect DSP is much further along than purists will ever allow. Certainly in guitar amp modeling, I've seen people hard pressed to tell the difference when they can't hear the amp hum or didn't know beforehand. The problem is more a lack of (more subtle than guitar modeling) devices designed for consumer audio to test. There's a lot of it in studio (software) though.

 

[ahofer]

And that perspective isn't just a little different from the listener's perspective, it's worlds apart! Jim

Yeah, in another thread we were talking about how strings sound when close mic'd. I've sat practically in the middle of my wife's chamber music groups, and the sound of players of all levels is quite harsh close up compared to concert perspective. Their left ear is right on it.

 

[MakeMineVinyl]

Yeah, in another thread we were talking about how strings sound when close mic'd. I've sat practically in the middle of my wife's chamber music groups, and the sound of players of all levels is quite harsh close up compared to concert perspective. Their left ear is right on it.

That's part of the reason recording violins is so difficult. Get close enough to give them 'presence', and the inharmonic sounds produced by playing the instrument makes the overall sound harsh. Get far enough away that these are not overpowering and the sound is dull and lifeless. Listeners have been complaining about harsh string sound since forever, and the recordings I have (or have made) which really nails string sound without harshness and shrillness in in the minority, especially when played on horn speakers which project every little detail, and which non-horn speakers tend to smooth over.

 

[rdenney]

Richard Clark Amplifier Challenge FAQ

Maybe the parameters of the Richard Clark amp challenge can fill in some blanks. He required amps being compared to have the same power without clipping, no EQ or other response differences, 30 db channel seperation over the whole audio band, and no more than 2% THD. You had to get 12 of 12 correct twice to collect the money. Yet over more than 2000 attempts no one even got enough right to meet a p-value of 5% which would require 17 out of 24. These tests are all done with music.

If anything, the challenge shows that differences appear in 1.) the boundary conditions, and 2.) as a result of interaction with surrounding equipment.

Don has shown us that amps can respond differently into very different loads, and we knew that already. That's an example of a difference in a surrounding condition. Another example is when the gain of an amp is low enough that the driving line-stage amp is driven to nonlinearity.

And boundary conditions. I think clipping happens a lot more often with real music than people suppose, particularly with the lower-sensitivity speakers used in home systems, and particularly for those who play the music loudly.

So, the challenge sounds to be a bit like "show me you can tell when the amps are different if I constrain them to the narrow solution space where they are most likely to be the same."

Granted, the challenge was meant to undermine the notion that amps have a signature sound even within those limiting constraints those boundary conditions. Surely some do. As Bruno Putzeys put it (my paraphrase from memory): "Pass and I have different objectives. He designs amps to sound good; I design them to have no sound at all." But most don't, or when they do, most can't detect it if all they use is their ears and not their eyes. All that is theory, though. What isn't is that in the real world, amps are used differently than in those tests. They are played louder, or into different loads, and show the results of pushing against their limits. How they respond to those limits explains much of the difference in amps, it seems to me. I'm reminded of the Carver challenge, which can be explained by the high output impedance of the "reference" amp compared to the low output impedance of the challenger. The suggestion is simple (though speculation reported only third-hand that I've seen): Carver added resistance to the output to increase the output impedance so that it was altered by the speaker in the same way as the reference amp.

There was another amp challenge linked here that I cannot now recall, where they determined that the amps being compared were clipping up to 1% of the time. I suspect that happens more than people realize, and it would take either putting the output on a scope, or recording the output of the speaker and analyzing the waveform for clipping (which is probably the only thing one would get out of making that recording in support of the comparison).

When Amir tests headphone amps, he's testing for sufficient power, it seems to me. It's not like clipping is a switch--even with a steady-state test signal, a bit of clipping has a bit of an effect--maybe not even audible as such. A lot of headphone amps are driven to their limits, it seems to me. I know there are times when I really crank up my Atom, depending on the sensitivity of the headphones. But the SINAD is measured below clipping, and won't catch that.

Rick "responding before reading the whole thread--always a risk" Denney

 

[Slayer]

Alas it doesn't work that way.

A DAC can have 120dB+ SINAD and not be 'audible transparent'. SINAD ONLY says something about 1kHz distortion + noise at a specific level in a 100k load. That's all it says. All measurements are important and that's why there are multiple measurements shown.

A 70 SINAD DAC can be noise (or hum) dominated yet have low distortion and sound audible different. Another 70 SINAD DAC can have low noise and high distortion and sound fine using music.

So... measurements matter... not just 1 but a the whole suite.

Simply put, yet well said.
I have seen hundreds of comments and questions asked, where your answer should be copied and pasted for the reply.

 

[Chr1]

If anything, the challenge shows that differences appear in 1.) the boundary conditions, and 2.) as a result of interaction with surrounding equipment.

Don has shown us that amps can respond differently into very different loads, and we knew that already. That's an example of a difference in a surrounding condition. Another example is when the gain of an amp is low enough that the driving line-stage amp is driven to nonlinearity.

And boundary conditions. I think clipping happens a lot more often with real music than people suppose, particularly with the lower-sensitivity speakers used in home systems, and particularly for those who play the music loudly.

So, the challenge sounds to be a bit like "show me you can tell when the amps are different if I constrain them to the narrow solution space where they are most likely to be the same."

Granted, the challenge was meant to undermine the notion that amps have a signature sound even within those limiting constraints those boundary conditions. Surely some do. As Bruno Putzeys put it (my paraphrase from memory): "Pass and I have different objectives. He designs amps to sound good; I design them to have no sound at all." But most don't, or when they do, most can't detect it if all they use is their ears and not their eyes. All that is theory, though. What isn't is that in the real world, amps are used differently than in those tests. They are played louder, or into different loads, and show the results of pushing against their limits. How they respond to those limits explains much of the difference in amps, it seems to me. I'm reminded of the Carver challenge, which can be explained by the high output impedance of the "reference" amp compared to the low output impedance of the challenger. The suggestion is simple (though speculation reported only third-hand that I've seen): Carver added resistance to the output to increase the output impedance so that it was altered by the speaker in the same way as the reference amp.

There was another amp challenge linked here that I cannot now recall, where they determined that the amps being compared were clipping up to 1% of the time. I suspect that happens more than people realize, and it would take either putting the output on a scope, or recording the output of the speaker and analyzing the waveform for clipping (which is probably the only thing one would get out of making that recording in support of the comparison).

When Amir tests headphone amps, he's testing for sufficient power, it seems to me. It's not like clipping is a switch--even with a steady-state test signal, a bit of clipping has a bit of an effect--maybe not even audible as such. A lot of headphone amps are driven to their limits, it seems to me. I know there are times when I really crank up my Atom, depending on the sensitivity of the headphones. But the SINAD is measured below clipping, and won't catch that.

Rick "responding before reading the whole thread--always a risk" Denney

Interesting. I guess that this might suggest that there are quite often subtle differences in the way amplifiers and speakers/headphones interact, some audible... and most likely measurable somehow.
Though not necessarily easily...

 

[rdenney]

I honestly don't think there are many musicians on this site. I say this because a lot of musicians like things that color the sound. Like there are a ton of plugins and hardware that is made just for this very reason. Like you would run a clean signal through it to color the sound. A lot of people also like something like this in their hifi setup. People here seem to think it is wrong to like such things. I think that is very narrow minded myself.

As a musician, I make the distinction between creation and playback. If I use an effect as part of the creation, why would I want someone else to add their different effects, one that runs a high likelihood of undermining my intentions, when they play my creation back?

This is a philosophical question. What is the purpose of audio equipment in the home? It's to express the artist's intentions clearly, it seems to me. But if I want to color it, why can't I add a separate effects processor? Building it into the amp is like a colored sunglasses--everything becomes that color, whether it's flattering or not. Or adding cinnamon to every dish in equal amounts.

Photographers see a difference between correction, creation, and targeting. Corrections are the changes they make to reverse the faults of the camera/lens/lighting. Creation is what they add to attain their artistic visualization. Targeting is what they subsequently do to express that creation accurately on whatever medium they will use to display it. We expect the designer of source equipment to perform corrections, so that the audio output is as neutral as possible. We expect the musician and the mixing engineer to perform the creation. We expect the mastering engineer to perform the targeting (which was a much bigger thing when distribution media were much more restricted in their capabilities than they are now). We don't want any of these in the playback equipment downstream from the source, it seems to me, and do what we can to eliminate even the effects in the room, so that the recorded room comes through and isn't buried by the playback room.

I grew up in the South, and thus became used to everything being too salty--salt added in the kitchen and more added at the table. Food in other places tasted bland to me, until I replaced that expectation of saltiness with an expectation of receiving the cook's desired flavor. Now, food "back home" tastes salty, except that back home they've learned to back off on the salt as people have experienced food more broadly. That conversion away from colored towards neutral didn't happen overnight.

Again, this is a philosophy difference.

Rick "who wants tubas to sound like tubas" Denney

 

[bravomail]

Thanks. You all are free to discuss this topic, provide answers, argue, whatever, in this thread.

I had some cases where measurements didn't tell the whole story.
SMSL M3 (not measured here, but in some other places) - decent 90-100 dB SINAD. Sound is rough.
Speaka USB stick - was "recommended" based on price, but measurements (and real sound) were bad.
SMSL Idea USB stick - measured well here, sounded weak and anemic, issues with power delivery?
Schiit Magni 2 - didn't measure well, but sounded really good.
Hifime ES9018 - didn't measure well here, but sounded really sweet.
Yamaha AVRs - all other review sites sing praises to them, here measured badly, my experience is mixed, I ended up decommissioning my RX-683. It could not beat a tiny ClassD amp with similar SINAD. Again, power delivery issues or reencoding audio for DSP processing?
I wish we had a followup deep-dives into those cases. Where audio-device sounds bad despite good measurements. You need to come up with some challenging tracks - bass heavy, treble heavy, classical, 3d binaural etc - to test subjectively.

 

[DanielT]

No measurements are absolutely not everything. Most people will then have a Hi-Fi gadget in their living room so. Some examples. To reduce, or eliminate reflections from the floor, Snell 1.Certainly a smart design..but.
Subwoofer drives with motors positioned in /out. A way to reduce distortion. Mechanical inequalities take each other out, from a distortion point of view, but who wants to stare into a couple of sub drives asses in the living room?
Carpet all over the floor?

 

[Chr1]

No measurements are absolutely not everything. Most people will then have a Hi-Fi gadget in their living room so. Some examples. To reduce, or eliminate reflections from the floor, Snell 1.Certainly a smart design but.Subwoofer drives with motors positioned in /out. A way to reduce distortion. Mechanical inequalities take each other out, from a distortion point of view, but who wants to stare into a couple of sub drives asses in the living room?Carpet all over the floor?

Personally I would rather stare into a couple of subwoofers asses than look at that wallpaper. WTF.

 

[DanielT]

Personally I would rather stare into a couple of subwoofers asses than look at that wallpaper. WTF.

But do not you have a wife? I dare, well I take that smiley anyway

 

[dwkdnvr]

In my view, if we're trying to orient new members/visitors around the basic perspective of the site, we have to back up another step to 'before' the measurements.
I think the basic primary gulf between the 'objectivist' leaning crowd and the 'subjectivist' leaning crowd rests in the difference in perspective on the human auditory system. The 'objectivist' leaning cohort accepts the results from psychology that our hearing is actually rather flawed, and that bias (both conscious and unconscious) has a large influence on subjective perception. i.e. they understand that we really can't trust our ears in many cases. Conversely, the subjective camp believes strongly in both the resolution and repeatability of our hearing. They have a perspective that seems to be rooted in the belief of an 'invariant' listener, leading to the belief that the only possible explanation of a difference in subjective experience is that there is an objective difference in the sound field.

So, I think it's a mistake to jump straight to discussions of 'the measurements' in these cases. If the new member doesn't understand or isn't willing to accept the basic premise, then there isn't likely to be any productive discussion.

From there, I do find the question of exactly how well the measurements capture the potential range of audible differences to be interesting. I do still 'believe' that there are aspects of system performance - particularly spatial perception - that probably aren't captured by the single-channel types of measurements done here. But, it's also highly likely that speaker and room completely dominate that discussion, and so it's probably not relevant to electronics. (probably)

 

[rdenney]

I am sure you are right, with my limited technical knowledge I thought SINAD was an average across the frequency band.

In that case, what does the chart below actually rank? And why can't we come up with a rating system, more representative of the overall performance of each tested device? Would a SINAD performed at different frequencies and then take a weighted average of all SINADs work? Is it even possible that a device has a SINAD of 120 at 1kHz but only 80 or 90 on a different frequency?

The point is not that SINAD isn't helpful, it's that it isn't the only thing one might care about. It has long been advised on ASR that any DAC on the left half of this ranking might sound transparent, if the other measures are consistent with SINAD (which, truth to tell, they usually are). If somebody claimed they could tell the difference between my MF V90 (middle of the green band) and my Topping E30 (middle of the blue band), despite the 8 dB difference in SINAD, in any real-world application, I'd demand ABX testing to prove it. The usual advice goes on to suggest that the selections made within the left half hinge on other measurements, needed inputs and outputs, features, user interface, aesthetics, and so on.

(While it's possible to game SINAD such that it performs well at 1 KHz and 2.1 VRMS output and poorly elsewhere, it's unlikely, particularly with respect to noise. But it's far more likely that performing poorly at 1 KHz and 2.1 VRMS output predicts poor performance at, say, 3 KHZ and 1.5 VRMS output, etc.)

What the chart does do is protect the unwary from the items at the right end, which may actually have audible faults in some circumstances, particularly after downstream amplification.

Rick "necessary but not sufficient" Denney

 

[Chr1]

But do not you have a wife?

Not married but I have a girlfriend. That floral wallpaper is brutal and I guess, being an audiophile, I kinda dig subwoofers... In my ideal mancave I could stare at my motorcycles too. (used as room treatment perhaps?)

 

[rdenney]

Understood, but something like this could work in order to give an overall rating:

View attachment 172832

too much work though, perhaps ...

It would be sorta hard to fit all that in a chart that showed ever DAC ever made in the history of man, as the SINAD chart shows.

Rick "who needs a 27" monitor just to read the damn thing" Denney

 

[rdenney]

@restorer-john

I do believe that Harman is cashing in, I mean, they are a business, why wouldn't they.

But here is what I would like to know (honest curiosity), what makes these Mark Levison amps $10k? Aside from better parts and the luxurious chasis. Does it have better engineering? Does those better parts make better sound? Does the Mark Levison engineers honestly tried to give it a "sound signature." Apparently the Hegel engineers may have (if you read their response to the H190 review, they use the word "subjective"). And if these true and legit engineers believes in "sound signature" can we blast them for it? Remind you, they are real legit EE many have PhD. Or are all of these EE just playing the game because, let's face it, it's a business at the end of the day. And there is an audiophoolia ecosystem that keeps the machine running and the cash coming.

Fundamental principle of market-based economics: Price and cost are unrelated. Price is dictated by the market; cost is what it is. If the market will bear a price that exceeds costs (writ large), then the product will produce profit and should be made. If the costs exceed the price the market will bear, then the product isn't feasible and should not be made.

People with wealth often look for ways to display (or, more charitably, express) that wealth. Harman provides a range of products at a range of price points to find markets, not to qualify products. People who buy Mark Levinson amps will have something that creates an ownership experience those people would likely not get from a Buckeye, even if the Buckeye sounds as good in objective terms. That's worth something, or it isn't. If it isn't, the brand won't last at that elevated price point.

Rick "there is nothing wrong with this" Denney

 

[zvukofor]

...The first is the popular belief that all competently designed amplifiers sound the same. Surely simple logic says that given the huge diversity of said amplifiers, it is highly unlikely that there are not some subtle differences between at least some of them.

The second is that tube distortion or "warmth" can be simply added digitally with DSP. I don't think that it is quite that straightforward...The current plugins/simulations are not an accurate emulation as yet.(Probably will be at some point in the near future though.)

Re: first: nothing in universe is the same, yet, two different amps carefully designed to be as transparent as they could be do share the same qualities, namely - transparency. There're a lot of colors until you just wash off all of them to see clear canvas.

Second: any distortions can be added via appropriate DSP. You just need to care about DSP rules and design what you need. If you cannot describe all details of a model - you cannot model...yet you can make "snaphots"! LUTs, volterra Kernels... Another very big topic by itself.
I do adore modern digital processing as being able to bring all sorts of needed harmonic distortions without any problems. A lot of people replaced their old hardware with code, i did too, to some degree...(a can solder, but cannot code!...)

 

[rdenney]

Interesting. I guess that this might suggest that there are quite often subtle differences in the way amplifiers and speakers/headphones interact, some audible... and most likely measurable somehow.
Though not necessarily easily...

Changes in coloration because of high output impedance interacting with low input impedance at the speaker (which mostly only affects tube amps) are most definitely measurable, and it isn't difficult to do so. But one has to measure the amplifier/speaker system, not just the amplifier, to see the effect. See @DonH56 's sticky treatise on the topic.

Headphone amps mostly fail at providing sufficient power unto insensitive headphones, it seems to me, but I haven't looked at tube-output headphone amps and don't know if they also have impedance interactions. Probably not--headphones, even at the low end of the spectrum, have much higher impedance than do speakers.

The amp challenge sought to equalize these colorations out before making comparisons, which to my thinking makes the challenge much less instructive.

Rick "yes, it's measurable" Denney