Master Complaint Thread About Headphone Measurements

[isostasy]

@Chagall I find the concept of "detail retrieval" completely bogus, along with "technicalities" or any other word which suggests that more can be extracted from the recording than already exists within it.

The furthest you can get to any definition rooted in reality is that it's the lack of masking of any sounds intended to be reproduced. This can entirely be described by frequency response and distortion. Note how this definition is finite, i.e. once the headphone can be said to not be masking sounds, it is detailed; you can't get any more than that. The definitions of "detail" I usually see are not limited, but make suggestions that additional "details" can always be "extracted" from a recording beyond simply reproducing it with correct tonality. Coincidentally, expensive headphones always seem to be better at this. Or there's the good one where a "cheap" headphone like the HD600 lacks detail until you spend 3x the price on an amp+dac stack.

I can see how it may have meant something some years ago when headphones were such a crapshoot that you could have one with such bad frequency response that you could genuinely not perceive parts of your music which you then could when upgrading to a more "detailed" headphone. I had this when I first got my HD6XX.

Beyond this I think many reviewers just think boosted treble sounds more "detailed". This is ironic because many headphones with too little bass, which could genuinely preclude you from hearing "details" in some bass instruments, are considered highly "detailed". They never seem to equate "detail" with anything in the low frequencies which offends me deeply as a bass player

I love the crap reviewers make up to convince themselves of the existence of higher levels of detail too. If a cheap headphone sounds perfectly fine but a bit bright, they say it has "exaggerated treble" which gives it the sense of "fake detail". If an expensive headphone is bright then it's just "detailed" and there's no awareness that it's still only the brightness that is contributing to this.

What I find even more stupid is that there are ways to enhance the "detail" of songs, and as a musician I use it all the time: EQ, low/high-pass filters, plugins like vocal reduction/isolation in Audacity... if I want to hear what a specific instrument is playing in a song these can help. But you wouldn't want to listen to music this way all the time. So for sure I believe that a headphone with boosted treble response might make the sounds of some instruments more prominent but a) I doubt they were inaudible before and b) it's not how it was intended to sound like.

My last point which is perhaps most important is that once you hear something it's really hard to unhear it. Once I've heard a song on an accurate set of speakers or headphones, and especially if I know parts of it by heart, I find it very hard not to hear all the "details" over any other system, even just my phone speakers in some cases. Exceptions are if the headphones are so bad it is super obvious they're failing to reproduce some parts of the frequency response but in general my point is once you have that song and all parts of it in your brain it's difficult to detach that from the experience of the soundwaves entering your ear.

Corollary to this is that the most rewarding way to get "details" from a song is to learn parts of it on an instrument and/or learn the music theory behind it. It takes more work than spending $$$ on boxes and transducers but it's a way to actually increase appreciation and enjoyment.

 

[_thelaughingman]

Corollary to this is that the most rewarding way to get "details" from a song is to learn parts of it on an instrument and/or learn the music theory behind it. It takes more work than spending $$$ on boxes and transducers but it's a way to actually increase appreciation and enjoyment.

This concisely summarizes and closes out the argument to be made about "details". I have numerous tracks that I've memorized based on repetitive listening habits, that I can hum and sing the portion of instrument being played before it even plays in the track. The brain has that inherent ability to remember the sound and how it should sound if it's reproduced in accordance to the instrument's output.

 

[Chagall]

Thank you for your response.

@Chagall I find the concept of "detail retrieval" completely bogus, along with "technicalities" or any other word which suggests that more can be extracted from the recording than already exists within it.

Sure, I have a hunch that is why @amirm dismissed it. But if we all know what is meant by the term there will be less confusion.

The furthest you can get to any definition rooted in reality is that it's the lack of masking of any sounds intended to be reproduced. This can entirely be described by frequency response and distortion. Note how this definition is finite, i.e. once the headphone can be said to not be masking sounds, it is detailed; you can't get any more than that. The definitions of "detail" I usually see are not limited, but make suggestions that additional "details" can always be "extracted" from a recording beyond simply reproducing it with correct tonality. Coincidentally, expensive headphones always seem to be better at this. Or there's the good one where a "cheap" headphone like the HD600 lacks detail until you spend 3x the price on an amp+dac stack.

The problem here is that you forgot to mention the cable - that can make all the difference
Joking aside, detail is in the recording, and the headphones' job is to reproduce it. The problem here is that even if a headphone is tonally corrected with EQ it might not be as detailed as another. Because the other headphone has lower distortion and the absence of other masking elements. Assuming tonally correct is the adherence to the Harman target, HD 600 needs a bass boost which also increases distortion, making it tonally correct, but also less detailed - this is my assumption, but it follows the logic of a given definition. We all agree that if HD600 lacks detail, no amount of DACs or amps will infuse it and can be explained by a placebo and proved as imaginary with a blind AB test.

I can see how it may have meant something some years ago when headphones were such a crapshoot that you could have one with such bad frequency response that you could genuinely not perceive parts of your music which you then could when upgrading to a more "detailed" headphone. I had this when I first got my HD6XX.

Do you think it's not relevant today? Me too with HD 600, and then to a lesser extent with HD800s, but it is noticeable in bass and the highs. I really don't think this is a bias on my part, but others can chime in.

Beyond this I think many reviewers just think boosted treble sounds more "detailed". This is ironic because many headphones with too little bass, which could genuinely preclude you from hearing "details" in some bass instruments, are considered highly "detailed". They never seem to equate "detail" with anything in the low frequencies which offends me deeply as a bass player

Yes, agreed! Again a good reason to define it and I feel your pain.

What I find even more stupid is that there are ways to enhance the "detail" of songs, and as a musician I use it all the time: EQ, low/high-pass filters, plugins like vocal reduction/isolation in Audacity... if I want to hear what a specific instrument is playing in a song these can help. But you wouldn't want to listen to music this way all the time. So for sure I believe that a headphone with boosted treble response might make the sounds of some instruments more prominent but a) I doubt they were inaudible before and b) it's not how it was intended to sound like.

Interesting point but that would be visible in the FR as boosted treble so not tonally accurate.

My last point which is perhaps most important is that once you hear something it's really hard to unhear it. Once I've heard a song on an accurate set of speakers or headphones, and especially if I know parts of it by heart, I find it very hard not to hear all the "details" over any other system, even just my phone speakers in some cases. Exceptions are if the headphones are so bad it is super obvious they're failing to reproduce some parts of the frequency response but in general my point is once you have that song and all parts of it in your brain it's difficult to detach that from the experience of the soundwaves entering your ear.

Brain-infused detail or Brain retrieval if you will, I love it Yeah, could this be a musician thing?

Corollary to this is that the most rewarding way to get "details" from a song is to learn parts of it on an instrument and/or learn the music theory behind it. It takes more work than spending $$$ on boxes and transducers but it's a way to actually increase appreciation and enjoyment.

I don't know, Amir is making a AI music thread for 2074...are there even any musicians in 2074

 

[amirm]

The #1 reason why some headphone characteristic is bogus when you can't say what created it, and how to test for it. Every headphone is tested (or should be) at the end of the manufacturing line. If you have no idea of how to test for "detail retrieval," Technicalities," etc., how do you know the headphone just manufactured has it? Or has it in the right amount? Likely the main and only test performed is frequency response (and maybe buzz and rub). If that is it, then there is nothing else.

 

[Chagall]

The #1 reason why some headphone characteristic is bogus when you can't say what created it, and how to test for it. Every headphone is tested (or should be) at the end of the manufacturing line. If you have no idea of how to test for "detail retrieval," Technicalities," etc., how do you know the headphone just manufactured has it? Or has it in the right amount? Likely the main and only test performed is frequency response (and maybe buzz and rub). If that is it, then there is nothing else.

Thank you for chiming in Amir.

In your Stealth review, you describe it as "transparent" and "clean". These qualities were determined with an accurate frequency response, low distortion, and the absence of resonances - exactly what determines the headphones' "detail retrieval" or "resolution". @Robbo99999 said not to get hung up on semantics, so is it fair to say that "transparent" headphones mean the same thing as "detailed" or "resolving" headphones in this context?

 

[amirm]

We could say that but it is not how your typical youtuber/reviewer uses the term. They think it is something special and different not captured in any of the measurements.

 

[IAtaman]

These scores are not for the headphones but the comparison was by means of "virtual" headphones. So these scores are more the scores for the FRs as these were created after the measurements of the original headphones.

More relevant here: these scores are the scores for the full test group. If one takes the scores from trained listeners only the results look quite different.
None of the FRs from your choice had a score above 60 from trained listeners.

Why would what trained listeners prefer would be more relevant? Given almost everyone is an untrained listener in the context of Harman How to Listen training, scores from untrained listeners would be far more relevant to general public, including you and I I assume, if we were forced to pick one category of listeners, no?

The highest score [≈60] from trained listeners is for HP25.

View attachment 338880View attachment 338881

Please note 60 is the non-normalized score i.e. when HP25 got 60, target got sth like 65 maybe (guessing from the graph) which linearly normalized to make target 100, brings the score of HP25 to ca 92.

 

[IAtaman]

Partially. First, in almost every case you listed above, the target did better than those headphones did. Here is comparison of HP7 to target:
View attachment 338875

Second, the study is incomplete with respect to sub-bass performance. But target is correct. Having done AB tests of over 100 headphones, I am very confident in importance of sub-bass compliance which the above headphones lack.

Third, I do listen to headphones as is and opine just like listeners did. If the sound is good as is, I note that subjectively. So this aspect is not lost.

Fourth, I evaluate if compliance with the target improves things. If it does, then that validates the deficiency to the extent that I hear a difference. This is reported independently than the stock performance.

Fifth, you all can judge the deviation from target and apply your own judgement. If it seems 80% as correct, you can assume if you like that the sound is good as is. I know I don't distinguish between 90% compliance and 100%.

Sixth, the listeners were not given the option of rating a fully compliant headphone as I have tested to compare. Had this been the case, it is entirely possible that they would rate those headphones much lower. Let's remember that all of these ratings are created in multi-way comparisons and not in absolute. This is why even the target itself gets different scores in different studies. It is all relative.

Lastly, my job is to get companies to strive for excellence. I am not here to give awards to people for "good enough" if excellence can be had and at no cost to consumers. You all can again, establish a lower standard than me.

Net, net, what I do is fully compliant with the research. A target has been established for excellent tonality in a headphone. I have verified this over and over again across huge number of headphones that it does indeed produce most excellent fidelity. So for me, it is the gold standard and I give little or no award for silver or bronze status. If your headphone is not fully compliant, I expect to use EQ to correct it.

Partially. First, in almost every case you listed above, the target did better than those headphones did. Here is comparison of HP7 to target:

I agree that almost all headphones can be improved with EQ. And for trained and untrained combined scores, what you point out is 100% correct. But for untrained listeners, which is effectively almost everyone, that is not always the case.

"While the overall trend in headphone preference was similar between the two listening groups, there were some notable differences for specific headphones: HP13 in Test Three, HP22 in Test Four, and HP26 in Test Five, which for untrained listeners was preferred equally to the Target."

(If you look at the graphs on Figure 3, it seems like HP13, HP19, HP25 and HP26 are rated equally or better by untrained listeners, but this is what the text says, which is a bit confusing I have to admit)

That is 3 out of 5 tests. I do not want to make a big point out of it, but I think it is important to touch upon the fact that for average Joe, a tuning that is "deficient" compared to the target might still be equally preferable.

Same story for distortion - I would not buy a headphone that have uncontrolled resonances here and there, and distortion that is not close to best in class, but to the extend they are not clearly audible, they are irrelevant to the average listener's experience.

I say this because the point I want to make is that maybe your expectation for excellence is creating a mismatch between your reviews and conclusions and people's experiences sometimes. Add on top of that your claim that your reviews are objective, then it becomes almost inevitable for some people to reach to the conclusion that science is not correct and conclusive yet, that there is something that can not be explained and all that bull crap we hear all the time.

If you chose to set the bar a bit lower than engineering excellence maybe, it would set a wider net to catch more people and make them more interested in the science of headphones in general and for ASR in particular. A $1000 headphone can still be a good headphone even if it does not follow the target in a lot of people's mind. Let's get to those people as well, and tell them how science explains why that might be the case indeed, instead of telling them they are wrong - that is my sentiment.

In any case, thank you for the clear response.

 

[Chagall]

We could say that but it is not how your typical youtuber/reviewer uses the term. They think it is something special and different not captured in any of the measurements.

Thanks, Amir.

I'm sure some YouTube reviewers use the term as you said, but for example, the explanation/analogy that Resolve gives in his reviews holds:

Detail Retrieval​

When considering detail retrieval, I like to borrow the 'image clarity' analogy expressed by a friend of mine. If you're looking through a window at a scene, how clear is the window? Even in a semi-opaque or translucent window, the images are identifiable, so it's not that all the various pieces of the music don't come through on lesser headphones, it's just that all the details about the musical elements are that much more clear and well-defined on a headphone like the XXXXXX. In other words, the more clear that image, the better detail retrieval the headphone has. This also redounds to improved representation of textural nuances and image structure in the music.

 

[Resolve]

Thanks, Amir.

I'm sure some YouTube reviewers use the term as you said, but for example, the explanation/analogy that Resolve gives in his reviews holds:

Detail Retrieval​

When considering detail retrieval, I like to borrow the 'image clarity' analogy expressed by a friend of mine. If you're looking through a window at a scene, how clear is the window? Even in a semi-opaque or translucent window, the images are identifiable, so it's not that all the various pieces of the music don't come through on lesser headphones, it's just that all the details about the musical elements are that much more clear and well-defined on a headphone like the XXXXXX. In other words, the more clear that image, the better detail retrieval the headphone has. This also redounds to improved representation of textural nuances and image structure in the music.

So, my stance on all of this stuff is that any description of 'technicalities' is only ever a description of the subjective experience and does not describe any physical or acoustic property. Despite what you may read on this forum from people who regularly enjoy misrepresenting my views on this stuff, my position is that if it can be heard then it can be measured. I just provide a clearly delineated subjective report as well.

EDIT: and I should clarify that I am not in favor of the term 'technicalities' in general. The larger community uses this term and I largely blame Crin for its popularity - even though I expect it originates from one of Tyll's reviews from way back. I much prefer to simply categorize it as "Subjective" because this prevents people from erroneously searching for metrics that shouldn't typically be applied with headphones.

 

[Chagall]

So, my stance on all of this stuff is that any description of 'technicalities' is only ever a description of the subjective experience and does not describe any physical or acoustic property. Despite what you may read on this forum from people who regularly enjoy misrepresenting my views on this stuff, my position is that if it can be heard then it can be measured. I just provide a clearly delineated subjective report as well.

Thank you @Resolve

When it comes to specifically detail retrieval I had my opinion but wanted to learn more about it and have some sort of agreement.

A lot of people dismissed it, but as it turns out - without knowing all the facts (not knowing the definition), misrepresenting what the term stands for, or simply using different terms for the same thing (we are not all native English speakers). Not the first or the last time - these things unfortunately happen.

I have used an excerpt from your review just to show what you mean when saying detail retrieval - that it isn't some magical property that infuses the recording with detail.

If some of this discussion helps someone, I consider it time well spent.

And again thank you to @Robbo99999 @majingotan @isostasy and @amirm for engaging in the discussion.

 

[amirm]

If you chose to set the bar a bit lower than engineering excellence maybe, it would set a wider net to catch more people and make them more interested in the science of headphones in general and for ASR in particular. A $1000 headphone can still be a good headphone even if it does not follow the target in a lot of people's mind. Let's get to those people as well, and tell them how science explains why that might be the case indeed, instead of telling them they are wrong - that is my sentiment.

I don't know how doing what you suggest will cast a wider net. Surely if I identify headphones that have best tonality, that will do that a lot better than sugarcoating the performance of a lesser headphone. My recent review of $20 IEMs have gotten a ton of people interested in using them precisely because they match the target and have superbly low distortion, almost assuring a great experience. This is the power of the way I have been doing reviews. It is working quite effectively and is in no need of modification as you suggest.

You also seem to be assuming people are completely dumb and are not capable of looking at a frequency response and comparing to target. If they are here, they likely have enough common sense to determine if a non-precise match to the target is good enough for them.

 

[amirm]

I'm sure some YouTube reviewers use the term as you said, but for example, the explanation/analogy that Resolve gives in his reviews holds:

Detail Retrieval​

When considering detail retrieval, I like to borrow the 'image clarity' analogy expressed by a friend of mine. If you're looking through a window at a scene, how clear is the window? Even in a semi-opaque or translucent window, the images are identifiable, so it's not that all the various pieces of the music don't come through on lesser headphones, it's just that all the details about the musical elements are that much more clear and well-defined on a headphone like the XXXXXX. In other words, the more clear that image, the better detail retrieval the headphone has. This also redounds to improved representation of textural nuances and image structure in the music.

This makes no sense to me. A blurry window will have cut off the high frequency spectrum of the signal (i.e. it is a low pass filter). It is called reduced MTF (modulation transfer function) due to spreading function. So if that is the analogy, we have it exactly in measurements of headphones.

But let's say it is true. How is it quantified? Just some vague words that could mean a headphone is 40% good or just as well 80%? How do we know the reviewer knows to even evaluate such a thing? Do they have special training? Content that can provably show such issues/performance?

What if we went and surveyed all the people who test a headphone on this criteria. You think they all agree with each other? Something tells me they don't. And that is the problem with not having objective standards.

One of the main problems with such characteristics is that it gives liberty to a reviewer to speak from both sides of their mouth. Headphone doesn't measure well but oh, "it has great detail retrieval." Or that it measures well but not liked because it is not good at detail retrieval. So we are to throw out what we know is reliable and trust the nebulous?

Mind you, nothing wrong with reviewers coming up with new criteria. I have done that with sub-bass performance. Ask me about it though I both show the frequency response and hand you on a platter a single track that will easily qualify the performance of a headphone/speaker.

 

[amirm]

Despite what you may read on this forum from people who regularly enjoy misrepresenting my views on this stuff, my position is that if it can be heard then it can be measured.

Well, we can listen to what you say here, stating that good detail retrieval must be a function of a higher end headphones:

If you don't know what creates this aspect of headphone performance, why are you so sure that it has to do with cost???

 

[Resolve]

A lot of people dismissed it, but as it turns out - without knowing all the facts (not knowing the definition), misrepresenting what the term stands for, or simply using different terms for the same thing (we are not all native English speakers). Not the first or the last time - these things unfortunately happen.

Well, people using different terms to mean different things is essentially the problem of private language, and this is core to many debates in the audio hobby. You can go to an audio show and have ten people try the same headphone and you can get ten different descriptors. However, when these descriptions are criticized, it's usually done so in bad faith as you've just seen done here, because those very same people will also use their own private language descriptions in their judgments that could be just as easily scrutinized.

Once again, I will reiterate my position on the usefulness of providing both objective and subjective information in evaluations - and also clearly delineating them. We need to provide objective data to indicate how something may perform for a wide range of people, but we also need to give an indication of how something sounds to an actual human. I mean ideally many humans, but there's only so much you can do there.

 

[amirm]

Well, people using different terms to mean different things is essentially the problem of private language, and this is core to many debates in the audio hobby. You can go to an audio show and have ten people try the same headphone and you can get ten different descriptors. However, when these descriptions are criticized, it's usually done so in bad faith as you've just seen done here, because those very same people will also use their own private language descriptions in their judgments that could be just as easily scrutinized.

Bad faith? It can't be that folks are making stuff up that they can't back? Or even quantify?

We are here because we are sick and tired of nonsense subjective reviews of audio gear in general. You have a case to make for something, do it based on science and engineering. If you can't then that is that. It has nothing to do with us. It is a problem with making things up on your own based on intuition that is not grounded in any proper way.

 

[Resolve]

Well, we can listen to what you say here, stating that good detail retrieval must be a function of a higher end headphones:

If you don't know what creates this aspect of headphone performance, why are you so sure that it has to do with cost???

Please be serious. You're posting a clip of me asking the question to Oratory1990 on a live stream from 3 years ago, and this is your 'proof' that your take of my position is correct? I'm not saying I haven't had wrong opinions on this topic in the past, but you can do better than that haha. How about that other time when you attributed to me a review I never wrote that contained all kinds of audiophile language, or when you had to search through multiple pages of reviews to find one that didn't include measurements (even though it did, you just didn't full look at the review) all so you could tell your followers "see! they don't focus on measurements". This is the very definition of bad faith, and why I generally don't engage with you.

Edit: Also, you've literally just done it again, even in this clip I'm not at all sure it has anything to do with cost. Cost is never a sufficient condition for goodness in sound quality.

 

[amirm]

Once again, I will reiterate my position on the usefulness of providing both objective and subjective information in evaluations - and also clearly delineating them. We need to provide objective data to indicate how something may perform for a wide range of people, but we also need to give an indication of how something sounds to an actual human. I mean ideally many humans, but there's only so much you can do there.

Measurements absolutely show what a headphone sounds like to a human. What do you think Dr. Olive and crew were doing for five years? Pontificating?

I am not against a bit of subjectivity but the notion that subjective remarks can completely turn measurement results upside down is non sequitur. Don't look to us to sanction reviews where most of what is in it is subjectivity with measurements taking a back seat. Measurements of frequency response are highly authoritative in how a headphone sounds. Subjective remarks from reviewers have very little weight compared to that.

 

[amirm]

You're posting a clip of me asking the question to Oratory1990 on a live stream from 3 years ago, and this is your 'proof' that your take of my position is correct?

Huh? You said we made up things about you. So I post an interview where you claim detail retrieval is a function of high-end headphones. This is the typical audiophile myth that we do away with in this forum. That cost has anything to do with performance. I dealt with your "position" in the other post showing how the vision analogy works completely against your argument.

 

[Resolve]

Measurements absolutely show what a headphone sounds like to a human. What do you think Dr. Olive and crew were doing for five years? Pontificating?

I am not against a bit of subjectivity but the notion that subjective remarks can completely turn measurement results upside down is non sequitur.

Ah yes, another straw man. At no point did I say that "subjective remarks can completely turn measurement results upside down". I merely advocating giving both, and that it's good to indicate when it is a subjective report and not based on anything objective - I think you'd agree with this, no?