1. WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). Come here to have fun, be ready to be teased and not take online life too seriously. We now measure and review equipment for free! Click here for details.

Limitations of blind testing procedures

Discussion in 'Psychoacoustics: Science of How We Hear' started by oivavoi, Jan 13, 2017.

Thread Status:
Not open for further replies.
  1. Jakob1863

    Jakob1863 Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    410
    Likes Received:
    97
    Location:
    Germany
    Ok, let´s see if it really isn´t different. We assume you´ve send out the two randomly labeled cable sets and that you got after some time the response and after unblinding you realize that the listener´s anwer was correct.
    What are you doing next?

    What about the time frame? Is it choosen wrt any (possible) burn-in effect? Maybe 30 Days is already to much time and the previously unused cable is already beginning to burn in?!
     
  2. Jinjuku

    Jinjuku Addicted to Fun and Learning Patreon Donor

    Joined:
    Feb 28, 2016
    Messages:
    544
    Likes Received:
    168
    You stated what should be the obvious problem without maybe realizing it:

    We ALL have burned in componentry by virtue of use.
     
  3. Jakob1863

    Jakob1863 Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    410
    Likes Received:
    97
    Location:
    Germany
    Sure, but that isn´t the point at this time nor does it render a sound experiment impossible.

    Please think about the questions, there is no trap included, it just might help to evaluate if your assertion (i.e. procedure not being different) is correct.
     
  4. Jinjuku

    Jinjuku Addicted to Fun and Learning Patreon Donor

    Joined:
    Feb 28, 2016
    Messages:
    544
    Likes Received:
    168
    Sigh. It's not a sound experiment. It's a listener claim and ergo an listener ability experiment.
     
  5. Jakob1863

    Jakob1863 Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    410
    Likes Received:
    97
    Location:
    Germany
    Sorry for the misunderstanding, using the word "sound" can be misleading; "sound experiment" only means that the experiment is or should be competent/meaningful/valid.
     
  6. Jinjuku

    Jinjuku Addicted to Fun and Learning Patreon Donor

    Joined:
    Feb 28, 2016
    Messages:
    544
    Likes Received:
    168
    When someone says they can jump over a red 8 foot fence that costs $10,000 vs a blue 8 foot fence that costs $1000 we aren't talking about the fence here folks.
     
  7. Jakob1863

    Jakob1863 Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    410
    Likes Received:
    97
    Location:
    Germany
    Jinjuku, i was talking about your assertion, therefore cited it.
    You stated:
    "We aren't using any different listening tests than what the sighted subjectivists is already doing.<snip>"

    So it is not any fence to discuss but your assertion about a specific test protocol. Nothing more nothing less.
    I joined the discussion under the impression that you were interested in proposing a competent (i.e. good/sound) experiment to get more data.
    Did i get the wrong idea?
     
  8. Jinjuku

    Jinjuku Addicted to Fun and Learning Patreon Donor

    Joined:
    Feb 28, 2016
    Messages:
    544
    Likes Received:
    168
    I've already proposed one such way to test. You can discuss what I proposed as much as you would like.
     
  9. Jakob1863

    Jakob1863 Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    410
    Likes Received:
    97
    Location:
    Germany
    Thank you very much; i really thought that to discuss things instead of just heralding something would be the purpose of a discussion forum. But you don´t like to answer my questions, right?

    Is it a personal matter with me or aren´t you just not used to accept questioning of your proposals?
     
  10. Jinjuku

    Jinjuku Addicted to Fun and Learning Patreon Donor

    Joined:
    Feb 28, 2016
    Messages:
    544
    Likes Received:
    168
    What questions were those?
     
  11. Jakob1863

    Jakob1863 Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    410
    Likes Received:
    97
    Location:
    Germany
    Please look at this quote:

    "Ok, let´s see if it really isn´t different. We assume you´ve send out the two randomly labeled cable sets and that you got after some time the response and after unblinding you realize that the listener´s answer was correct.
    What are you doing next?

    What about the time frame? Is it choosen wrt any (possible) burn-in effect? Maybe 30 Days is already to much time and the previously unused cable is already beginning to burn in?!"
     
  12. Jinjuku

    Jinjuku Addicted to Fun and Learning Patreon Donor

    Joined:
    Feb 28, 2016
    Messages:
    544
    Likes Received:
    168
    1. It is what it is
    2. A larger group of participants would be better

    That failure of logic is on the claimant and not on me to place a salve. You've only pointed out the absurdity of a claim.
     
  13. Cosmik

    Cosmik Major Contributor

    Joined:
    Apr 24, 2016
    Messages:
    1,555
    Likes Received:
    566
    Location:
    UK
    In the spirit of the thread title:

    Suppose we hypothesise that a person who knows they are taking part in an experiment loses much of their ability to discern audible differences. How are we going to test whether it is true, or not?

    If we cannot establish that it is not the case, the results of all listening tests are moot.

    Edit: I should have re-read the original post before posting this - it is precisely what was asked in the first place..!

    My answer is yes, I think that people's ability to discern differences are reduced when they know they are in an experiment, but no, I don't know how to establish whether it is true or not.
     
    Last edited: Apr 13, 2017
  14. Jakob1863

    Jakob1863 Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    410
    Likes Received:
    97
    Location:
    Germany
    The scientific "countermeasures" are to use positive controls (i.e. a difference as EUT that has to be detected, to see if participants reach sufficient sensitivity) or to raise the number of participants to average it out and/or to let participants train under the specific test conditions to overcome.

    Use of postive controls (and usually negative controls as well) is mandatory.

    Another method would be to let the listener not know that they are participating in any controlled test.

    I´ve surely written it a lot elsewhere; controlled tests of humans are common since a long time and in a lot of different fields and therefore there exists a huge knowledge base covering nearly all problems thinkable and possible solution. It is always surprizing me that in typical audio tests (testing performance/difference overall) most of this knowledge is neglected.
     
  15. Jinjuku

    Jinjuku Addicted to Fun and Learning Patreon Donor

    Joined:
    Feb 28, 2016
    Messages:
    544
    Likes Received:
    168
    I like cash incentive. I bet you could tell blind 128Kbps MP3 vs 24/192 if a Benjamin or two was on the table.

    This is why I've always been a fan of a blind test where something is going to be certainly be different. It helps separate the wheat from the chaff.
     
  16. Jakob1863

    Jakob1863 Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    410
    Likes Received:
    97
    Location:
    Germany
    Which is imo a bit contradictionary to your statement that it would be listeners claim and at test of listeners ability. A larger sample will not help in examing a listeners ability, it will only give some information about the group or maybe the underlying population.

    That was one of the reasons why i did ask. It scratches the core of the problem; if you want to develop a test that gives useful/correct results and allows to draw further conclusions, you have to start with a clearly defined task/question/hypothesis that will be examined.

    Most likely you´ll notice at one point that you can´t do a controlled test in the same way "sighted listening" does.

    We all have our believies and expectations, are therefore biased (at least to a certain degree), but if you only evaluate it from the viewpoint of absurdity then you will probably let your bias get to much impact.
     
  17. Jinjuku

    Jinjuku Addicted to Fun and Learning Patreon Donor

    Joined:
    Feb 28, 2016
    Messages:
    544
    Likes Received:
    168
    A larger sample = more stable the results and teasing out of overall trends.

    In the cable burn in I certainly did ask a clear question. Again you are welcome to poke holes in the method that is based on a claim.


    It's not meant to since sighted listening is inherently flawed.


    But I'm not evaluating it from the viewpoint of absurdity. I'm using the claimants own words. AGAIN poke a hole in my cable burn-in evaluation method.
     
  18. Vladimir

    Vladimir Member

    Joined:
    Mar 25, 2017
    Messages:
    21
    Likes Received:
    10
    Location:
    Macedonia
    amirm and Jinjuku like this.
  19. Cosmik

    Cosmik Major Contributor

    Joined:
    Apr 24, 2016
    Messages:
    1,555
    Likes Received:
    566
    Location:
    UK
    If I were to paraphrase your comment it would be "There is a long history of using the scientific method to test humans. Therefore it is possible to scientifically test every aspect of being human".

    But larger samples, positive and negative controls etc. can't demonstrate that the listening 'abilities' (and this is a slightly strange concept in the context of listening to music for pleasure) of all participants aren't impaired by the awareness of being in a test.
     
  20. amirm

    amirm Founder/Admin CFO (Chief Fun Officer) Staff Member

    Joined:
    Feb 13, 2016
    Messages:
    11,019
    Likes Received:
    2,901
    Location:
    Seattle Area
Loading...
Thread Status:
Not open for further replies.

Share This Page