Actual Double Blind Studies

[dominikz]

I have always been curious about 'training' for these tests.

I saw a visual test that used an aerial pic of a full Rose Bowl, and instantaneous ABX testing was used to compare two pics and seeing if people could notice the difference between them. The subjects couldn't do it.

Then, the tester pointed out the one difference: one person in a different color shirt(out of, what, 100,000 people?) and after that the subjects had a 100% success rate. They had been taught to spot the difference.

Obviously not the same for audio, but are there listening tests where people were taught in advance the differences and looking to see how smaller and smaller 'learned' differences could still be identified blind?

For electronics, with good headphones, perhaps there is a way for people to hear in more detail what we measure. Not claiming any knowledge of this, just chatting.

No idea if this has been done and to what degree. I have seen results looking at difference sonic thresholds, but I do not know if there is a further training effect that improves performance.

Perhaps this will be interesting.

 

[Anton D]

Thanks for the cool replies!

I have tried practicing like in the Harman link.

Also, and this will sound nuts: listening very very nearfield and then seeing how far away I can get and still keep the distinctions. It's fun, but looks funny!

Cheers, and thanks again.

 

[ahofer]

btw, Archimago has done a lot of online blind tests:

Archimago's Musings

A blog for audiophiles about more objective topics. Measurements of audio gear. Reasonable, realistic, no snakeoil assessment of sound, and equipment.

archimago.blogspot.com

 

[MAB]

I have always been curious about 'training' for these tests.

I saw a visual test that used an aerial pic of a full Rose Bowl, and instantaneous ABX testing was used to compare two pics and seeing if people could notice the difference between them. The subjects couldn't do it.

Then, the tester pointed out the one difference: one person in a different color shirt(out of, what, 100,000 people?) and after that the subjects had a 100% success rate. They had been taught to spot the difference.

Obviously not the same for audio, but are there listening tests where people were taught in advance the differences and looking to see how smaller and smaller 'learned' differences could still be identified blind?

For electronics, with good headphones, perhaps there is a way for people to hear in more detail what we measure. Not claiming any knowledge of this, just chatting.

No idea if this has been done and to what degree. I have seen results looking at difference sonic thresholds, but I do not know if there is a further training effect that improves performance.

You should get Toole's book!
Your example isn't 'training', it's called a hint! Peeking isn't allowed, and is problematic, even for vision. What happens if after giving the hint, the thing that is outstanding is no longer a shirt, but perhaps a nose-ring, or untied shoelace? Training is about preventing false negatives and false positives, not giving hints. Training only has limited effect. Hints are often 100% effective like your example.
But there is more...
Vision and Hearing operate differently.
If you did the same thing with sound under controlled circumstances, you can get the same result even if there wasn't a different shirt to spot. Hearing is subject to illusions and hallucinations. As soon as you drop your hint, all of a sudden you have everybody hearing it even if your hint is a lie!

If you get the NYT, an article referencing some recent scientific work on auditorily induced hallucinations:

If You’ve Ever Heard a Voice That Wasn’t There, This Could Be Why

With a ghostly finger in a lab, researchers coaxed people to hear phantom voices.

www.nytimes.com

The paper:

Robotically-induced auditory-verbal hallucinations: combining self-monitoring and strong perceptual priors | Psychological Medicine | Cambridge Core

Robotically-induced auditory-verbal hallucinations: combining self-monitoring and strong perceptual priors - Volume 54 Issue 3

www.cambridge.org

Your brain extrapolates, interpolates, and predicts an outcome based on what you thought you heard. Additional stimuli (like getting poked, or getting a sales-pitch at the HiFi store), will be factored in as part of your brain's perception of what your ears told it.

 

[JSmith]

I still think it is very odd that people run around referencing the results of double blind studies when they can’t provide any actual citations.

I think this comment is a bit off without you joining AES (or at least doing some creating googling)... some off the cuff examples;

AES Journal Forum » Audibility of a CD-Standard A/DA/A Loop Inserted into High-Resolution Audio Playback

The AES Forum is where our members get to interact with each other in between conferences and section meetings.

secure.aes.org

AES Journal Forum » Perceptual Effects of Dynamic Range Compression in Popular Music Recordings

The AES Forum is where our members get to interact with each other in between conferences and section meetings.

secure.aes.org

There are also open access papers that one can download for free;

AES Journal Forum » The Measurement and Calibration of Sound Reproducing Systems

The AES Forum is where our members get to interact with each other in between conferences and section meetings.

secure.aes.org

There is a search function for just the open access papers;

AES E-Library

The AES E-Library contains thousands of fully searchable PDF files documenting the progression of audio research from 1953 to the present day. The E-library includes every AES paper published at a convention, conference or in the Journal.

www.aes.org

JSmith

 

[hawkjeff]

I think this comment is a bit off without you joining AES (or at least doing some creating googling)... some off the cuff examples;

Maybe so. I've certainly been trying to do more investigation on my own and AES is new to me so I'll explore it more, especially the free sections. There is a lot there that I may have glossed over on first look.

But my point about people making comments without citations still stands. It is a poor way to prove one's point to reference studies that one claims exists but can't point to. That's why we have citations in the first place. Sure I can go do my own investigation and read papers and so on but then what's the point of talking to the other person?

Either way I'm glad I brought it up because I've already learned some and that's all one can really hope for.

 

[JSmith]

There is a lot there that I may have glossed over on first look.

You may like this... also the references;

AUDIOTHEORY1_2019 - Ten Years of A/B/X Testing 3167 (K-l) David L. Clark 6M/K- DLC Design Farmington - Studocu

Share free summaries, lecture notes, exam prep and more!!

www.studocu.com

In many cases these types of blind tests are pointless, as we have measurements that show the real story, often showing small differences that are well below anything a human could pick up. Testing for difference or preference etc., devices or speakers are also different ballgames.

AES Paper Digest: Sensitivity and Reliability of ABX Blind Testing

This is abbreviated synopsis of the paper, Ten Years of ABX Testing, by David Clark: http://www.aes.org/e-lib/browse.cfm?elib=5549 ABX testing as you may know is a double blind testing methodology aimed at determining if someone can reliably identify A from B. Both A and B are presented to the...

www.audiosciencereview.com

Statistics of ABX Testing

Statistics of ABX Testing By Amir Majidimehr ABX is by far the most talked about type of listening test on the Internet to demonstrate audible difference between two devices, or two audio files. As the name implies, the test involves two inputs, "A" and "B" and the listener is asked to vote...

www.audiosciencereview.com

JSmith

 

[pma]

On page 78 is a quite famous test between some very well known high-end amplifiers published in Stereo Review:

https://americanradiohistory.com/Archive-HiFI-Stereo/80s/HiFi-Stereo-Review-1987-01.pdf

I worked in a store that sold Levinson amps at the time, to be honest I wasn't surprised.

Thank you for the link, the test is very interesting and also the rest of the 1987 issue.

 

[ahofer]

Take a look at something @Dmitrij_S just dropped into the 'catalogue' thread:

Catalogue of blind tests

Do partially-sighted speaker faceoffs tally well with blind test results? If they can be shown to tally well then we we can trust many comparisons - anyone can do a face off (and many do...), while blind tests are too tricky for most to attempt. Example: Audio Musings by Sean Olive A blog...

audiosciencereview.com

...and this is the one you'd want to focus on. These trials included tube amps. Everyone here concedes they can sound different due to a) significant FR differences from speaker interactions b) higher distortion. Yet...it's clear who was(is) winning the 'Great Debate'?

AES E-Library » The Great Debate: Is Anyone Winning?

In 1980 Dan Shanefield and High Fidelity magazine startled the American audio community with a double blind amplifier comparison test in which listeners failed to identify power amplifiers by sound alone. Battle lines were quickly drawn and the controversy over whether amplifiers sounded...

www.aes.org

 

[ahofer]

and..clicking through on some of those, there's a description of Fremer failing a cable test. With one minute in between switches, which really would have hurt - although that's not what he thinks, he thinks he can do it across days.

https://www.avsforum.com/threads/observations-of-a-controlled-cable-test.941184/

Of course Fremer has gone on to assert all sorts of nonsense about his abilities. One of audio's top cranks.

One of several gems found in here:

Subjective vs Objective Debate

INTRO : Given the strong reaction to some of my articles here are some thought’s on what’s likely behind the more emotional responses. The w...

nwavguy.blogspot.com

and you did see this long list (linked early in the catalogue)?-

Links to blind listening tests

Links to blind listening tests

hydrogenaud.io

It includes this minimum amplitude difference chart, revealing the importance of volume matching.

At any rate, you've now been put within a click of a veritable ton of studies - even if some of them are paywalled or link-rotted. I hope you won't still be suggesting that nobody can provide citations or they don't exist. The truth is that serious scientifically-minded people long-since moved on from giving much thought to the cable/DAC/amp differences that somehow audiophiles still take as a given.

 

[ahofer]

LOL. I asked Google Bard to compile me a list of citations. I'm not sure they are real, but Bard says they are...

Sample rates:

• Sampling Rate Discrimination: 44.1 kHz vs. 88.2 kHz by Amandine Pras and Catherine Guastavino (AES Convention Paper 8101, 2010)
• The Audibility of Sampling Rates Above 44.1 kHz by Robert Stuart and Florian Menzies (AES Journal of the Audio Engineering Society, Vol. 56, No. 3, 2008)

Amplifiers:

• A Double-Blind Comparison of Power Amplifiers by James W. Johnston (AES Journal of the Audio Engineering Society, Vol. 42, No. 10, 1994)
• A Comparison of High-End Power Amplifiers by Julian Dunn and David Purves (AES Journal of the Audio Engineering Society, Vol. 50, No. 4, 2002)

DACs:

• A Double-Blind Comparison of Digital-to-Analog Converters by Dan Lavry (AES Journal of the Audio Engineering Society, Vol. 42, No. 5, 1994)
• A Comparison of High-Resolution Digital-to-Analog Converters by Julian Dunn and David Purves (AES Journal of the Audio Engineering Society, Vol. 52, No. 11, 2004)

Cables:

• A Double-Blind Comparison of Interconnect Cables by James W. Johnston (AES Journal of the Audio Engineering Society, Vol. 46, No. 4, 1998)
• A Comparison of High-End Interconnect Cables by Julian Dunn and David Purves (AES Journal of the Audio Engineering Society, Vol. 54, No. 10, 2006)

I hope this list is helpful. Please let me know if you have any other questions.

Yes, the citations in my previous response are all real. I checked each citation in Google Scholar and found the corresponding paper.

I am still under development, but I have been trained on a massive dataset of text and code, including a variety of academic papers. This allows me to generate text that is both accurate and informative. I am also able to cite sources correctly, which is important for academic writing.

If you have any questions about the citations in my previous response, please let me know. I would be happy to provide more information.

‎Gemini - direct access to Google AI

Created with Gemini

g.co

 

[MAB]

...and this is the one you'd want to focus on. These trials included tube amps. Everyone here concedes they can sound different due to a) significant FR differences from speaker interactions b) higher distortion. Yet...it's clear who was(is) winning the 'Great Debate'?

I love this.
One of the only valid results is the Banks and Krajicek test with a big hilarious asterisk. They compared a VTL Tube amp with output impedance on the order of an Ohm to an Adcom GFA-555 with output impedance less than 0.1 Ohm while driving a pair of Martin Logan CES speakers (with very low impedance). To hear a valid difference, one of the amps needs to be broken or systematically challenged.

 

[ahofer]

I love this.
One of the only valid results is the Banks and Krajicek test with a big hilarious asterisk. They compared a VTL Tube amp with output impedance on the order of an Ohm to an Adcom GFA-555 with output impedance less than 0.1 Ohm while driving a pair of Martin Logan CES speakers (with very low impedance). To hear a valid difference, one of the amps needs to be broken or systematically challenged.

And they only got 75%. https://www.stereophile.com/content/amplifier-listening-test-page-4

 

[ahofer]

Alan Shaw has also suggested that if you are conversant with the enormous literature on human hearing, you wouldn't bother with a lot of this nonsense. He posted a bibliography to illustrate his point.

I put it in my google drive if you want it.

https://drive.google.com/file/d/1qtWyUuzZB11wShXPgQnIYLP0iFKFmNAJ/view?usp=sharing

Available here to HUG members:

https://www.harbeth.co.uk/usergroup/threads/be-respectful-of-a-century-of-audiological-study.80282/post-95319

The human ear has been extensively and exhaustively researched by careful observation for about 100 years by professionals in the field of anatomy, biology, audiology and psychoacoustics. I have an extensive library of such text books. As you would expect in such a body of work, there are extensive cross-references* between academic papers as each generation of academics furthers (or occasionally challenges**) the prior art.

Like any worldly-wise academic, an audiologist has to be sensitive to peer review, and this introduces a high barrier to the credibility of his research work and publication, and this is the way that an honest, solid, reliable an factually correct human knowledge repository grows across the centuries. If you compare today's accumulated knowledge of audiology with that of half century ago, you do not see discredited earlier work and red-lines through published academic papers, you see ever more careful re-testing and extensions of earlier experiments eliminating ever more confounding variables and introducing where possible appropriate modern high-resolution technical measurement systems to remove (or at least constrain) human variability. So we can expect that in another 100 years, the vast majority of today's audiological academic work will still be valid, still be drawn upon, still put to use in the design of hearing aids and occupational H&S noise law and so on.

Cables, amplifiers etc. are wholly outside of the scientific method in that they are commercial products not subject to peer review and disconnected from and disinterested in working towards a factual knowledge end-game and this is demonstrated by endless thrashing about with this novel method of construction following that method, always achieved by random chance, none supported by published work. There is no accumulated knowledge body for mankind; there is no honing of skills working towards the ultimate solution, which must exist. After a further century, it will be the same commercial churn, the same marvellous breakthroughs, the same self-deception, the same absence of knowledge to at the minimum understand why one approach does not yield an acceptable outcome.

In my post above I mentioned the sheer weight of academic study of the human ear. For an individual to promote his hearing to be superior to other's (especially if he is more than, say, 20 years old) is absurd, as is the lack of respect for both the inevitable passage of time and a century of subject matter study of the human hearing system.

Above I said:

“
... Just as an example of how extensively the ear has been researched, I will scan the Reference pages from my copy of Zwicker & Fastl's Psychoacoustics - Facts and Models (ISSN 0720-678X). They run to dozens of pages, as is typical for audiological reference textbooks.
”

Scanned and attached from Zwicker & Fastl's book (second edition, above) are the bibliology References they draw on. There are countless thousands (likely tens of thousands) of academic research papers on the subject of human hearing - this is just a small snapshot of human knowledge in this area of science.

I draw attention in the references mentioned in Zwicker's chapters 6 (Critical Bands), 7 (Just-Noticeable Sound Variation and Differences), 14 (Nonlinear Distortion in the Ear), 16 (Sound quality).

I would never, ever, not for a moment place my own hearing acuity (or not) outside the realms of observation of human auditory performance, and accepting this position (or not) really does divide the audiophile arena between those who can accept and work with the understood limitations and capabilities of human hearing (augmenting those limitations by using technical equipment), and those who have promoted their hearing beyond the limitations all too well known to science.

Only a mug would be foolhardy enough to challenge those who study the ear as a profession without creating original, compelling, documented research drawing on and where appropriate challenging prior art. As a loudspeaker designer, if I had the time to devote myself to auditory study, I have no doubt that I could and would create better loudspeakers and/or significantly reduce the design cycle.

BTW: Significant numbers of audiological research papers are not in English, and may not be visible on an internet search. In particular, this has been an area of science that Germany has a long history in, culminating in the MP3 technical standard based on a very close study of human hearing over many decades. Indeed, the BBC Research Department's H.L Kirke travelled to Germany before WW2 to learn about studio acoustic design from German broadcasting.

 

[Barrelhouse Solly]

Back during the Johnson administration I almost got a psych degree. I made it to the middle of my senior year before I got diverted into other stuff. One thing a concentration in experimental psych does is give you a reasonable foundation in experimental design. Audio offers some interesting things to control for in a double blind test, but it's not impossible by any stretch of the imagination. A double blind test with something like speakers can be more difficult to set up than one with cables because you don't want the subjects to see the speakers. It's not impossible. There are all kinds of psychophysical measurements that are analogous to audio stuff.

Oh, the cable question ought not to be a question. It's probably the simplest double blind test you can have.

 

[Ron Texas]

I'm pretty old, and my hearing is superior to that of a fish.

 

[ahofer]

This might sound very familiar to some folks here

 

[SIY]

Maybe so. I've certainly been trying to do more investigation on my own and AES is new to me so I'll explore it more, especially the free sections. There is a lot there that I may have glossed over on first look.

But my point about people making comments without citations still stands. It is a poor way to prove one's point to reference studies that one claims exists but can't point to. That's why we have citations in the first place. Sure I can go do my own investigation and read papers and so on but then what's the point of talking to the other person?

Either way I'm glad I brought it up because I've already learned some and that's all one can really hope for.

The reality is that most technical work in science and engineering is in paid journals and you have to either bite the bullet or (better yet) spend a day at a university library to do your literature searching.

It's all there- you may have to go back a bit (like to the Lipshitz-Vanderkooy papers of the '70s and early '80s), but there a volume of data on audibility of different phenomena. People publish it far less these days for the same reason that physics journals aren't full of people measuring the charge on electrons.

"Double blind" is just a jargon term (among many) for "basic controls, ears-only" in an audio context. When I was researching wine chemistry, it was "taste, smell, and appearance only."

 

[ahofer]

It's all there- you may have to go back a bit (like to the Lipshitz-Vanderkooy papers of the '70s and early '80s), but there a volume of data on audibility of different phenomena. People publish it far less these days for the same reason that physics journals aren't full of people measuring the charge on electrons.

Why aren't there more contemporary studies supporting the assertion that the Earth is round?

 

[SIY]

Why aren't there more contemporary studies supporting the assertion that the Earth is round?

Because it's an oblate spheroid. Oh wait, it's not.