• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Limitations of blind testing procedures

Status
Not open for further replies.

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
We aren't using any different listening tests than what the sighted subjectivists is already doing. @ Polk Audio forums they were going on and on about burned in vs non burned in cabling. I offered to send out two sets of cables, randomly labeled. one set burned in. The offer was 30 days of fully sighted, self administered, 100% control of listening length and how quickly to swap out. So no test administrator? Check. Fully sighted? Check. On the own Equipment? Check. In their own room? Check. Using their own music? Check.

Ok, let´s see if it really isn´t different. We assume you´ve send out the two randomly labeled cable sets and that you got after some time the response and after unblinding you realize that the listener´s anwer was correct.
What are you doing next?

What about the time frame? Is it choosen wrt any (possible) burn-in effect? Maybe 30 Days is already to much time and the previously unused cable is already beginning to burn in?!
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
What about the time frame? Is it choosen wrt any (possible) burn-in effect? Maybe 30 Days is already to much time and the previously unused cable is already beginning to burn in?!

You stated what should be the obvious problem without maybe realizing it:

We ALL have burned in componentry by virtue of use.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
You stated what should be the obvious problem without maybe realizing it:

We ALL have burned in componentry by virtue of use.
Sure, but that isn´t the point at this time nor does it render a sound experiment impossible.

Please think about the questions, there is no trap included, it just might help to evaluate if your assertion (i.e. procedure not being different) is correct.
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
Sure, but that isn´t the point at this time nor does it render a sound experiment impossible.

Please think about the questions, there is no trap included, it just might help to evaluate if your assertion (i.e. procedure not being different) is correct.

Sigh. It's not a sound experiment. It's a listener claim and ergo an listener ability experiment.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
Sigh. It's not a sound experiment. It's a listener claim and ergo an listener ability experiment.

Sorry for the misunderstanding, using the word "sound" can be misleading; "sound experiment" only means that the experiment is or should be competent/meaningful/valid.
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
When someone says they can jump over a red 8 foot fence that costs $10,000 vs a blue 8 foot fence that costs $1000 we aren't talking about the fence here folks.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
Jinjuku, i was talking about your assertion, therefore cited it.
You stated:
"We aren't using any different listening tests than what the sighted subjectivists is already doing.<snip>"

So it is not any fence to discuss but your assertion about a specific test protocol. Nothing more nothing less.
I joined the discussion under the impression that you were interested in proposing a competent (i.e. good/sound) experiment to get more data.
Did i get the wrong idea?
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
Jinjuku, i was talking about your assertion, therefore cited it.
You stated:
"We aren't using any different listening tests than what the sighted subjectivists is already doing.<snip>"

So it is not any fence to discuss but your assertion about a specific test protocol. Nothing more nothing less.
I joined the discussion under the impression that you were interested in proposing a competent (i.e. good/sound) experiment to get more data.
Did i get the wrong idea?

I've already proposed one such way to test. You can discuss what I proposed as much as you would like.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
<snip> You can discuss what I proposed as much as you would like.
Thank you very much; i really thought that to discuss things instead of just heralding something would be the purpose of a discussion forum. But you don´t like to answer my questions, right?

Is it a personal matter with me or aren´t you just not used to accept questioning of your proposals?
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
Thank you very much; i really thought that to discuss things instead of just heralding something would be the purpose of a discussion forum. But you don´t like to answer my questions, right?

Is it a personal matter with me or aren´t you just not used to accept questioning of your proposals?

What questions were those?
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
What questions were those?
Please look at this quote:

"Ok, let´s see if it really isn´t different. We assume you´ve send out the two randomly labeled cable sets and that you got after some time the response and after unblinding you realize that the listener´s answer was correct.
What are you doing next?

What about the time frame? Is it choosen wrt any (possible) burn-in effect? Maybe 30 Days is already to much time and the previously unused cable is already beginning to burn in?!"
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
Please look at this quote:

"Ok, let´s see if it really isn´t different. We assume you´ve send out the two randomly labeled cable sets and that you got after some time the response and after unblinding you realize that the listener´s answer was correct.
What are you doing next?

1. It is what it is
2. A larger group of participants would be better

What about the time frame? Is it choosen wrt any (possible) burn-in effect? Maybe 30 Days is already to much time and the previously unused cable is already beginning to burn in?!"

That failure of logic is on the claimant and not on me to place a salve. You've only pointed out the absurdity of a claim.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
In the spirit of the thread title:

Suppose we hypothesise that a person who knows they are taking part in an experiment loses much of their ability to discern audible differences. How are we going to test whether it is true, or not?

If we cannot establish that it is not the case, the results of all listening tests are moot.

Edit: I should have re-read the original post before posting this - it is precisely what was asked in the first place..!

My answer is yes, I think that people's ability to discern differences are reduced when they know they are in an experiment, but no, I don't know how to establish whether it is true or not.
 
Last edited:

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
<snip>

My answer is yes, I think that people's ability to discern differences are reduced when they know they are in an experiment, but no, I don't know how to establish whether it is true or not.

The scientific "countermeasures" are to use positive controls (i.e. a difference as EUT that has to be detected, to see if participants reach sufficient sensitivity) or to raise the number of participants to average it out and/or to let participants train under the specific test conditions to overcome.

Use of postive controls (and usually negative controls as well) is mandatory.

Another method would be to let the listener not know that they are participating in any controlled test.

I´ve surely written it a lot elsewhere; controlled tests of humans are common since a long time and in a lot of different fields and therefore there exists a huge knowledge base covering nearly all problems thinkable and possible solution. It is always surprizing me that in typical audio tests (testing performance/difference overall) most of this knowledge is neglected.
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
In the spirit of the thread title:

Suppose we hypothesise that a person who knows they are taking part in an experiment loses much of their ability to discern audible differences. How are we going to test whether it is true, or not?

I like cash incentive. I bet you could tell blind 128Kbps MP3 vs 24/192 if a Benjamin or two was on the table.

This is why I've always been a fan of a blind test where something is going to be certainly be different. It helps separate the wheat from the chaff.
 

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
1. It is what it is
2. A larger group of participants would be better

Which is imo a bit contradictionary to your statement that it would be listeners claim and at test of listeners ability. A larger sample will not help in examing a listeners ability, it will only give some information about the group or maybe the underlying population.

That was one of the reasons why i did ask. It scratches the core of the problem; if you want to develop a test that gives useful/correct results and allows to draw further conclusions, you have to start with a clearly defined task/question/hypothesis that will be examined.

Most likely you´ll notice at one point that you can´t do a controlled test in the same way "sighted listening" does.

That failure of logic is on the claimant and not on me to place a salve. You've only pointed out the absurdity of a claim.

We all have our believies and expectations, are therefore biased (at least to a certain degree), but if you only evaluate it from the viewpoint of absurdity then you will probably let your bias get to much impact.
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
Which is imo a bit contradictionary to your statement that it would be listeners claim and at test of listeners ability. A larger sample will not help in examing a listeners ability, it will only give some information about the group or maybe the underlying population.

A larger sample = more stable the results and teasing out of overall trends.

That was one of the reasons why i did ask. It scratches the core of the problem; if you want to develop a test that gives useful/correct results and allows to draw further conclusions, you have to start with a clearly defined task/question/hypothesis that will be examined.

In the cable burn in I certainly did ask a clear question. Again you are welcome to poke holes in the method that is based on a claim.


Most likely you´ll notice at one point that you can´t do a controlled test in the same way "sighted listening" does.

It's not meant to since sighted listening is inherently flawed.


but if you only evaluate it from the viewpoint of absurdity then you will probably let your bias get to much impact.

But I'm not evaluating it from the viewpoint of absurdity. I'm using the claimants own words. AGAIN poke a hole in my cable burn-in evaluation method.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
The scientific "countermeasures" are to use positive controls (i.e. a difference as EUT that has to be detected, to see if participants reach sufficient sensitivity) or to raise the number of participants to average it out and/or to let participants train under the specific test conditions to overcome.

Use of postive controls (and usually negative controls as well) is mandatory.

Another method would be to let the listener not know that they are participating in any controlled test.

I´ve surely written it a lot elsewhere; controlled tests of humans are common since a long time and in a lot of different fields and therefore there exists a huge knowledge base covering nearly all problems thinkable and possible solution. It is always surprizing me that in typical audio tests (testing performance/difference overall) most of this knowledge is neglected.
If I were to paraphrase your comment it would be "There is a long history of using the scientific method to test humans. Therefore it is possible to scientifically test every aspect of being human".

But larger samples, positive and negative controls etc. can't demonstrate that the listening 'abilities' (and this is a slightly strange concept in the context of listening to music for pleasure) of all participants aren't impaired by the awareness of being in a test.
 
Status
Not open for further replies.
Top Bottom