• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Limitations of blind testing procedures

Status
Not open for further replies.

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,279
Likes
1,180
If I were to paraphrase your comment it would be "There is a long history of using the scientific method to test humans. Therefore it is possible to scientifically test every aspect of being human".

Anyone can paraphrase any way they want including incorrectly. In the scope of cable burn in I proposed a testing method that is based on the claims being :

1. Sighted. You sure can
2. On my own equipment. You sure can
3. Using my own source material. You sure can
4. I can swap out and determine interval, # of swaps, duration? You sure can
5. No one breathing over my shoulder (i.e. self administer). You sure can

Exactly the same way the sighted evaluation is done excepting you have some cables that aren't burned in. And if that is enough to throw your hearing off then you are just imagining things.

AGAIN you can either find a weakness in my proposed protocol or you cannot.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
Anyone can paraphrase any way they want including incorrectly. In the scope of cable burn in I proposed a testing method that is based on the claims being :

1. Sighted. You sure can
2. On my own equipment. You sure can
3. Using my own source material. You sure can
4. I can swap out and determine interval, # of swaps, duration? You sure can
5. No one breathing over my shoulder (i.e. self administer). You sure can

Exactly the same way the sighted evaluation is done excepting you have some cables that aren't burned in. And if that is enough to throw your hearing off then you are just imagining things.

AGAIN you can either find a weakness in my proposed protocol or you cannot.
You can demonstrate that some person's claim of being able to hear some phenomenon under certain conditions is not true. However, you have not demonstrated that the phenomenon doesn't exist or that it is not audible under all conditions. In fact, it is not possible to prove it. The only way to overcome the fantasy of the Quantum Bybee phenomenon (or whatever it is) is to appeal to reason and rationality.
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,279
Likes
1,180
You can demonstrate that some person's claim of being able to hear some phenomenon under certain conditions is not true.

I'm perfectly happy with that outcome however.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,654
Likes
240,798
Location
Seattle Area
Anyone can paraphrase any way they want including incorrectly. In the scope of cable burn in I proposed a testing method that is based on the claims being :

1. Sighted. You sure can
2. On my own equipment. You sure can
3. Using my own source material. You sure can
4. I can swap out and determine interval, # of swaps, duration? You sure can
5. No one breathing over my shoulder (i.e. self administer). You sure can

Exactly the same way the sighted evaluation is done excepting you have some cables that aren't burned in. And if that is enough to throw your hearing off then you are just imagining things.

AGAIN you can either find a weakness in my proposed protocol or you cannot.
Just to cut through the back and forth :), your experiment is not designed to get scientific data but rather show the fragility of conclusions audiophiles have reached over such things. It is valuable in that regard but it is obviously not how science is done.
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,305
Location
uk, taunton
Just to cut through the back and forth :), your experiment is not designed to get scientific data but rather show the fragility of conclusions audiophiles have reached over such things. It is valuable in that regard but it is obviously not how science is done.
What if jinjuku wears a white coat and protective eye wear? That makes anything scientific, right?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,654
Likes
240,798
Location
Seattle Area
What if jinjuku wears a white coat and protective eye wear? That makes anything scientific, right?
Yes sir. THis would be a good start:

brandsonsale-store_2271_1186624194


That said, if you want to get the results you want from any heteresexual male, hiring this person might do better:

sexy_scientist.jpg


She even comes with that SPL meter on the left!
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,279
Likes
1,180
Just to cut through the back and forth :), your experiment is not designed to get scientific data but rather show the fragility of conclusions audiophiles have reached over such things. It is valuable in that regard but it is obviously not how science is done.

And again. I'm perfectly ok with that.

I understand that taking a group of people that stipulate that they can jump from a stand still 20 feet high and putting a bar 20 feet high only tells me that that grouping can't jump 20 feet high. I understand there are 6 billion other people that may be able to. What subjectivists keep conflating is that I'm not testing the ability of the bar, I'm testing the claim to jump over it. The materials that the bar is made up of is immaterial.

If that is not how science in that instance is done then I'm okay with that.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,654
Likes
240,798
Location
Seattle Area
And again. I'm perfectly ok with that.

I understand that taking a group of people that stipulate that they can jump from a stand still 20 feet high and putting a bar 20 feet high only tells me that that grouping can't jump 20 feet high. I understand there are 6 billion other people that may be able to. What subjectivists keep conflating is that I'm not testing the ability of the bar, I'm testing the claim to jump over it. The materials that the bar is made up of is immaterial.

If that is not how science in that instance is done then I'm okay with that.
It is actually worse than that. The experiment uses the rules of their world to generate what is guaranteed to be false outcome. Science says that having such long switch over time will eliminate detection of small differences. By giving them the rope to hang themselves, they do exactly that. :) In other words, if hypothetically there were small differences, it would not be heard in that test.

So it is a good social experiment and opportunity for the testers to learn something (assuming they go through with it). But it just isn't a science subject to discuss as was going on earlier in the thread.
 

fas42

Major Contributor
Joined
Mar 21, 2016
Messages
2,818
Likes
191
Location
Australia
Probably repeating what was already said at some point, but the elephant in the room to me is that the brain quickly adapts to inferior sound, and if one constantly flips between inferior and better, the brain "tires" of the game quickly - and starts adjusting to make the inferior match the better - "Can't fool me, I know they're the same piece of music! I can't be bothered tuning into the sound being 'worse' some of the time!" is what one's head says ...

Trying to make human hearing testing robotic, is probably always going to be extremely hit and miss, unless done very, very carefully.
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,279
Likes
1,180
It is actually worse than that. The experiment uses the rules of their world to generate what is guaranteed to be false outcome. Science says that having such long switch over time will eliminate detection of small differences. By giving them the rope to hang themselves, they do exactly that. :) In other words, if hypothetically there were small differences, it would not be heard in that test.

So it is a good social experiment and opportunity for the testers to learn something (assuming they go through with it). But it just isn't a science subject to discuss as was going on earlier in the thread.

I'm discussing the Title of the Thread. I get that it has a limit. But it's not me that is giving defining parameters to the test. I'm honoring the claim made.

In the case of cable burn in it's not my fault they are all idiots that don't see the obviousness of a used system is a burned in system. 100 hours of playing material through a cable is a 100 hours of playing material through a cable no matter how you slice it.

I can't be blamed for the false outcome and I certainly can't be blamed for setting up a test that is doomed to failure based on the sighted method of subjectivist. The claimant has done all of that. Ignorance becomes stupidity willfully left uncorrected.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,747
Likes
37,568
That said, if you want to get the results you want from any heteresexual male, hiring this person might do better:

sexy_scientist.jpg


She even comes with that SPL meter on the left!

I am pretty sure that would bias the results. However, being pretty sure is not at all scientific. I will volunteer to take part in the testing to see if the pretty lady wearing the lab coat influences the result. Now we know sighted testing is no good. I will agree to wearing a blindfold and simply going by unsighted feel for this very important testing. :cool:
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,305
Location
uk, taunton
I am pretty sure that would bias the results. However, being pretty sure is not at all scientific. I will volunteer to take part in the testing to see if the pretty lady wearing the lab coat influences the result. Now we know sighted testing is no good. I will agree to wearing a blindfold and simply going by unsighted feel for this very important testing. :cool:
How Noble of you, such sacrifice will not be forgotten...
 
OP
oivavoi

oivavoi

Major Contributor
Forum Donor
Joined
Jan 12, 2017
Messages
1,721
Likes
1,939
Location
Oslo, Norway
If said young lady would administer an experiment, I would perhaps consider to volunteer myself. Who am I to turn down Science?
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
I don't think we have got past the stumbling block that we can't demonstrate scientifically whether people lose their audio discerning abilities when they know they are taking part in an experiment. Obviously, setting up an experiment to test this is probably impossible. Pointing to the fact that science is used to test humans in all sorts of fields that involve asking "How do you feel?" doesn't alter that fact - their results may be similarly dubious.
 

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,321
Location
Albany Western Australia
Probably repeating what was already said at some point, but the elephant in the room to me is that the brain quickly adapts to inferior sound, and if one constantly flips between inferior and better, the brain "tires" of the game quickly - and starts adjusting to make the inferior match the better - "Can't fool me, I know they're the same piece of music! I can't be bothered tuning into the sound being 'worse' some of the time!" is what one's head says ...

Trying to make human hearing testing robotic, is probably always going to be extremely hit and miss, unless done very, very carefully.


Does it????

Well in that case listen to a crap hifi for a few minutes and you wont be able to tell the difference between it and the worlds best hifi. Save yourself a fortune. No need to buy expensive (good kit - not that it follows that it should be expensive to be good). Just fool your brain into not knowing the difference.

...mmmmm....something's wrong with that conclusion
 

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,321
Location
Albany Western Australia
I don't think we have got past the stumbling block that we can't demonstrate scientifically whether people lose their audio discerning abilities when they know they are taking part in an experiment. Obviously, setting up an experiment to test this is probably impossible. Pointing to the fact that science is used to test humans in all sorts of fields that involve asking "How do you feel?" doesn't alter that fact - their results may be similarly dubious.

I think its a very valid point that people could well have their aural abilities impaired if they are stressed, however all it points to is that any differences that cant be detected by the golden eared subjective stress bunnies are by definition very small.

In any case if you test a range of candidates there are going to be a range of emotional states, not everyone, in fact I would say few are going to get as upset as most subjective audiophiles do at the prospect of their aural capabilities being tested.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,747
Likes
37,568
I don't think we have got past the stumbling block that we can't demonstrate scientifically whether people lose their audio discerning abilities when they know they are taking part in an experiment. Obviously, setting up an experiment to test this is probably impossible. Pointing to the fact that science is used to test humans in all sorts of fields that involve asking "How do you feel?" doesn't alter that fact - their results may be similarly dubious.

I don't think so. Now if you are talking about challenges to someone's audiophile manhood, then stress is involved.

When in college I signed up for some experiments given by grad students mostly in the psych dept. Didn't know what they were about ahead of time, never told of the results afterward. These were sensory tests. Some of vision, some of touch, some of manual dexterity and some of listening or at least language recognition ability.

I would do my best, but not uptight about it. Was getting a few bucks for an hour or half hour. For instance one was listening to words spoken in the midst of noise. The loudness of the words varied noise didn't. I was supposed to write what they were. At some point they had hidden a tube to direct a blast of air into your eyes. Enough to slightly startle you. And yet the words kept coming. In this case the words were things like spider, snake, scorpion, bee and some others I am not remembering after so many years. The test was being filmed.

I had nothing on the line based upon doing well or not doing well on the test. Would never be told how I did. Don't know if it was a test of discernment of dangerous critters. A test of if you heard dangerous words better than others (some of the words I simply couldn't make out), a test of whether the next few words are heard less well or perhaps from adrenaline heard more precisely after an unexpected blast of air to your eyes or what it might have been or if the film was looking at your facial or pupil response.

So do people lose their audio discernment during tests? I might buy they have it slightly diminished via stress. Lose it, no that would be some incredible news. Be able to hear beyond all levels immeasurably small effects often claimed, and yet lose that completely when asked to test it? Again that is really bending over backwards to think that is happening.

The other thing to think about his how audiophile claimed hearing ability fits in with all other information. Our regular tests of hearing ability in most ways matches up very well with physical constraints of sound, with physical structure of the hearing mechanism, and so on and so forth. Things that could be or have been determined without the listening tests you have in mind.

There is plenty of literature investigating the frequency hearing range of mammals. It fits very well with the physical size of the basilar membrane. Where the human BM fits indicates something a bit beyond 15 khz. Regular testing, of the kind that you think might cause a loss of discernment, indicate the same thing. Firing impulses into the ear canal and recording what comes back reflexively from the hearing structure indicate about the same thing. What is known about firing along the auditory nerve with various frequency stimulus indicates about the same thing. Sighted audiophile anecdotal testimony indicates we need far, far more than this range.

If you look at the ear drum, and the nerve activity at quiet sound levels, our hearing threshold in young healthy adults drops down to right above the point at which we would hear brownian noise of the air molecules if it were just a bit more sensitive with our physical structure. Looks readily apparent why that would be the lower limit. The ability to hear timing between ears and level differences fits in with the physical size of our heads and spacing of our hears. And these limits are in agreement with a conventional unsighted testing of those abilities. Yet audiophiles claim to be effected by presence or absence of noise levels far below what can even be transmitted via the air when mixed with complex music signals. And in direct opposition to masking abilities of noise that also when tested are in line with what you would expect of the ears physical functioning.

Now we know lots, but probably a small minority percentage about processing or pattern matching done with the signals sent to our brain from our ears. You can't process out perception at levels and frequencies etc that simply are beyond or below what the hearing mechanisms themselves are capable of working with. The brain's input of information is limited by that. The kind of discernment claimed yet to go missing when testing audiophiles would be so far beyond many of the physically real possibilities they should be scoffed at immediately.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
I don't think so. Now if you are talking about challenges to someone's audiophile manhood, then stress is involved.

When in college I signed up for some experiments given by grad students mostly in the psych dept. Didn't know what they were about ahead of time, never told of the results afterward. These were sensory tests. Some of vision, some of touch, some of manual dexterity and some of listening or at least language recognition ability.

I would do my best, but not uptight about it. Was getting a few bucks for an hour or half hour. For instance one was listening to words spoken in the midst of noise. The loudness of the words varied noise didn't. I was supposed to write what they were. At some point they had hidden a tube to direct a blast of air into your eyes. Enough to slightly startle you. And yet the words kept coming. In this case the words were things like spider, snake, scorpion, bee and some others I am not remembering after so many years. The test was being filmed.

I had nothing on the line based upon doing well or not doing well on the test. Would never be told how I did. Don't know if it was a test of discernment of dangerous critters. A test of if you heard dangerous words better than others (some of the words I simply couldn't make out), a test of whether the next few words are heard less well or perhaps from adrenaline heard more precisely after an unexpected blast of air to your eyes or what it might have been or if the film was looking at your facial or pupil response.

So do people lose their audio discernment during tests? I might buy they have it slightly diminished via stress. Lose it, no that would be some incredible news. Be able to hear beyond all levels immeasurably small effects often claimed, and yet lose that completely when asked to test it? Again that is really bending over backwards to think that is happening.

The other thing to think about his how audiophile claimed hearing ability fits in with all other information. Our regular tests of hearing ability in most ways matches up very well with physical constraints of sound, with physical structure of the hearing mechanism, and so on and so forth. Things that could be or have been determined without the listening tests you have in mind.

There is plenty of literature investigating the frequency hearing range of mammals. It fits very well with the physical size of the basilar membrane. Where the human BM fits indicates something a bit beyond 15 khz. Regular testing, of the kind that you think might cause a loss of discernment, indicate the same thing. Firing impulses into the ear canal and recording what comes back reflexively from the hearing structure indicate about the same thing. What is known about firing along the auditory nerve with various frequency stimulus indicates about the same thing. Sighted audiophile anecdotal testimony indicates we need far, far more than this range.

If you look at the ear drum, and the nerve activity at quiet sound levels, our hearing threshold in young healthy adults drops down to right above the point at which we would hear brownian noise of the air molecules if it were just a bit more sensitive with our physical structure. Looks readily apparent why that would be the lower limit. The ability to hear timing between ears and level differences fits in with the physical size of our heads and spacing of our hears. And these limits are in agreement with a conventional unsighted testing of those abilities. Yet audiophiles claim to be effected by presence or absence of noise levels far below what can even be transmitted via the air when mixed with complex music signals. And in direct opposition to masking abilities of noise that also when tested are in line with what you would expect of the ears physical functioning.

Now we know lots, but probably a small minority percentage about processing or pattern matching done with the signals sent to our brain from our ears. You can't process out perception at levels and frequencies etc that simply are beyond or below what the hearing mechanisms themselves are capable of working with. The brain's input of information is limited by that. The kind of discernment claimed yet to go missing when testing audiophiles would be so far beyond many of the physically real possibilities they should be scoffed at immediately.
I agree with much of that, but I feel myself being gently nudged away from the real issue (again!). I didn't mean that we may lose all of our discerning ability when under stress, just a proportion of it - maybe just the bit at the outer edges that can discern between 16 and 24 bits (as an audiophile would claim).

And only a few aspects of audiophile listening ability are functions of straightforward physiology that can be confirmed with clicks and bleeps. Much of the reconstruction of the 'audio scene' takes place in the brain.

Take stereo imaging. What is it? Is it real? Can it be measured? We think we hear stereo imaging - which is mainly why we bother with two speakers. But what conditions are necessary for 'good' imaging? We could embark on a series of empirical experiments to try to work it out - but if we have trouble defining what the word means, and it only takes place inside our heads, we are already on non-scientific, aesthetic judgement territory. It probably won't end well.

Mr. Putzeys achieves brilliant imaging with his Kii Threes, apparently. Did he perform thousands of listening tests to get there? Or did he just build a system that is closer to the formal definition of stereo than anyone else's? Ultimately, do we really think that a listening test-based approach would come up with something different from this? If so, it wouldn't actually be a stereo hifi system at all, but an effects box.
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,279
Likes
1,180
I don't think we have got past the stumbling block that we can't demonstrate scientifically whether people lose their audio discerning abilities when they know they are taking part in an experiment. Obviously, setting up an experiment to test this is probably impossible. Pointing to the fact that science is used to test humans in all sorts of fields that involve asking "How do you feel?" doesn't alter that fact - their results may be similarly dubious.

I would imagine everyone that claims they can jump 20 feet straight up would lose the ability to do so when they are being filmed attempting it.

And again. I'm perfectly ok with that.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
I would imagine everyone that claims they can jump 20 feet straight up would lose the ability to do so when they are being filmed attempting it.

And again. I'm perfectly ok with that.
You are homing in on a very specific application of listening tests: debunking claims made by probably deluded people. I think the thread is more general than that.
 
Status
Not open for further replies.
Top Bottom