• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Limitations of blind testing procedures

Status
Not open for further replies.

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,317
Location
Albany Western Australia
Do you think there is something that can't be electrically measured that distinguishes one amp from another?

Its more the definitive interpretation of the data, where comprehensive test data is actually available, and its relation to sound.

This is not relevant and continuing down that line still misses the point. Not all products are created equal for many reasons. Inequalities that do affect the final sound.

Thats why we will always be comparing equipment, and if we do so its better to minimise visual and other biases.
 
Last edited:

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
Do you think there is something that can't be electrically measured that distinguishes one amp from another?

I doubt there is, but sometimes interactions with speaker loads or dynamics of that and various power supply make it hard to just look at simple spec sheets. Or even complex spec sheets.

Amir pointed to amp bypass testing done by the Swedish AES a few months back. It is discussed some in these posts.
http://www.avsforum.com/forum/91-au...bate-thread-scott-s-hi-res-audio-test-10.html I might come up with the online article about the test of one of the Bryston amps.

I did such testing myself many years back unaware the Swedish AES did. You take the amp under test, connect it to a dummy load mimicking your speaker. Reduce the output level so that you have unity gain input to output. Then feed it to your reference amp and speaker. A perfect amp is fully transparent as in you cannot hear if you have a direct connection or the amp in the loop. This is how I found that all the desirable traits of tube amps are colorations.

Now some good amps with good specs come pretty close to full transparency. The only one I found totally transparent however is a Spectral. The Swedish AES does such testing. First sighted and then blind. I don't know the full list of amps they have tested, but they had one large Audio Research SS amp that seemed transparent sighted, but failed the blind test. They tested one of the Bryston's which was the first to pass both tests as fully transparent. Though only after some modifications to the output circuitry with insight from why it wasn't transparent in the Swedish testing.

I wish more magazines did such testing in the English publications. I also wouldn't be surprised that there are more amps out there that meet straight wire with gain criteria. But amps and speaker loads are complex enough various little things that are hard to predict do seem to interact audibly.
 
OP
oivavoi

oivavoi

Major Contributor
Forum Donor
Joined
Jan 12, 2017
Messages
1,721
Likes
1,934
Location
Oslo, Norway
I doubt there is, but sometimes interactions with speaker loads or dynamics of that and various power supply make it hard to just look at simple spec sheets. Or even complex spec sheets.

Amir pointed to amp bypass testing done by the Swedish AES a few months back. It is discussed some in these posts.
http://www.avsforum.com/forum/91-au...bate-thread-scott-s-hi-res-audio-test-10.html I might come up with the online article about the test of one of the Bryston amps.

I did such testing myself many years back unaware the Swedish AES did. You take the amp under test, connect it to a dummy load mimicking your speaker. Reduce the output level so that you have unity gain input to output. Then feed it to your reference amp and speaker. A perfect amp is fully transparent as in you cannot hear if you have a direct connection or the amp in the loop. This is how I found that all the desirable traits of tube amps are colorations.

Now some good amps with good specs come pretty close to full transparency. The only one I found totally transparent however is a Spectral. The Swedish AES does such testing. First sighted and then blind. I don't know the full list of amps they have tested, but they had one large Audio Research SS amp that seemed transparent sighted, but failed the blind test. They tested one of the Bryston's which was the first to pass both tests as fully transparent. Though only after some modifications to the output circuitry with insight from why it wasn't transparent in the Swedish testing.

I wish more magazines did such testing in the English publications. I also wouldn't be surprised that there are more amps out there that meet straight wire with gain criteria. But amps and speaker loads are complex enough various little things that are hard to predict do seem to interact audibly.

Btw, this is an excellent example of how different kinds of blind testing procedures can lead to different outcomes.
 

watchnerd

Grand Contributor
Joined
Dec 8, 2016
Messages
12,449
Likes
10,408
Location
Seattle Area, USA
I doubt there is, but sometimes interactions with speaker loads or dynamics of that and various power supply make it hard to just look at simple spec sheets. Or even complex spec sheets.

Amir pointed to amp bypass testing done by the Swedish AES a few months back. It is discussed some in these posts.
http://www.avsforum.com/forum/91-au...bate-thread-scott-s-hi-res-audio-test-10.html I might come up with the online article about the test of one of the Bryston amps.

I did such testing myself many years back unaware the Swedish AES did. You take the amp under test, connect it to a dummy load mimicking your speaker. Reduce the output level so that you have unity gain input to output. Then feed it to your reference amp and speaker. A perfect amp is fully transparent as in you cannot hear if you have a direct connection or the amp in the loop. This is how I found that all the desirable traits of tube amps are colorations.

Now some good amps with good specs come pretty close to full transparency. The only one I found totally transparent however is a Spectral. The Swedish AES does such testing. First sighted and then blind. I don't know the full list of amps they have tested, but they had one large Audio Research SS amp that seemed transparent sighted, but failed the blind test. They tested one of the Bryston's which was the first to pass both tests as fully transparent. Though only after some modifications to the output circuitry with insight from why it wasn't transparent in the Swedish testing.

I wish more magazines did such testing in the English publications. I also wouldn't be surprised that there are more amps out there that meet straight wire with gain criteria. But amps and speaker loads are complex enough various little things that are hard to predict do seem to interact audibly.

I've seen similar testing regimes in German magazines.

The English magazines mostly suck. Hardly any measure at all, and the ones that do almost never do measurement bake offs.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
To me it seems strange that the amplifier thing continues to be debated. But the reason is that there is uncharted territory between testing with sine waves (or any synthetic test signal) and the unquantifiable things that audiophiles claim to hear when listening to real music. THD measurements will never convince the subjectivists, and listening tests are always subject to dispute by, well, everyone.

We could measure an amplifier using any test signal we like - even real music. All we need to do is to sample and digitally record the amplifier's output into a real load, and then, in software, model the amplifier from a variety of building blocks until we get near a (time domain) null against the test signal. I don't mean model the circuit in SPICE, but represent the amp as gain blocks, phase shifts etc. If we only use linear blocks what difference do we see between input and output? Is it distortion caused by a static nonlinear transfer function (crossover distortion or the onset of clipping, for example) or do we see interesting little instabilities such as bursts of oscillation every time a castanet is played?

It may turn out that all amplifiers are very boring and simple to model. If so, that settles that argument. But it may be that for some amplifiers, the only way to get a null is to introduce some very tricky little 'wrinkles' into the output - these will be the things that may not show up with standard test signals and loads but are the sonic differences between amps.

Setting up a meaningful null test in hardware would be a nightmare. Doing it in software would be simple.

Where people don't 'get' this idea is that they like to think that all signals can be thought of as being made up of sine waves, therefore a sweep of sine waves will cover everything. But the interesting things about amps are those aspects that are often speculated about but never proved: thermal effects, instability set off by power supply dips etc. These may be triggered by certain combinations of transients but not steady state sine waves, pulses etc. but we can never know for sure when using standard tests or models.
 

watchnerd

Grand Contributor
Joined
Dec 8, 2016
Messages
12,449
Likes
10,408
Location
Seattle Area, USA
I doubt there is, but sometimes interactions with speaker loads or dynamics of that and various power supply make it hard to just look at simple spec sheets. Or even complex spec sheets.

Amir pointed to amp bypass testing done by the Swedish AES a few months back. It is discussed some in these posts.
http://www.avsforum.com/forum/91-au...bate-thread-scott-s-hi-res-audio-test-10.html I might come up with the online article about the test of one of the Bryston amps.

I did such testing myself many years back unaware the Swedish AES did. You take the amp under test, connect it to a dummy load mimicking your speaker. Reduce the output level so that you have unity gain input to output. Then feed it to your reference amp and speaker. A perfect amp is fully transparent as in you cannot hear if you have a direct connection or the amp in the loop. This is how I found that all the desirable traits of tube amps are colorations.

Now some good amps with good specs come pretty close to full transparency. The only one I found totally transparent however is a Spectral. The Swedish AES does such testing. First sighted and then blind. I don't know the full list of amps they have tested, but they had one large Audio Research SS amp that seemed transparent sighted, but failed the blind test. They tested one of the Bryston's which was the first to pass both tests as fully transparent. Though only after some modifications to the output circuitry with insight from why it wasn't transparent in the Swedish testing.

I wish more magazines did such testing in the English publications. I also wouldn't be surprised that there are more amps out there that meet straight wire with gain criteria. But amps and speaker loads are complex enough various little things that are hard to predict do seem to interact audibly.

I thought this was an interesting quote from one of those Swedish AES experimenters on the AVS thread:

"So my stance is that there certainly exists sonic differences between power amps that drive a reasonably difficult speaker load that is clearly detectable in DBTs, but in reality, they are so small that I do not concider them an issue because there are other issues that impact sonic performance orders of magnitues more."
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
I thought this was an interesting quote from one of those Swedish AES experimenters on the AVS thread:

"So my stance is that there certainly exists sonic differences between power amps that drive a reasonably difficult speaker load that is clearly detectable in DBTs, but in reality, they are so small that I do not concider them an issue because there are other issues that impact sonic performance orders of magnitues more."

Having done this I would agree for the most part. There were still odd possibilities. For years people complained big MOSFET amps had mosfet mist or grain in the upper-midrange.

At one time I had some of these on hand that belonged to a friend.

http://www.museatex.com/str.htm

Low feedback MOSFET power amp design by Ed Meitner.

I fed it into the dummy load, and fed the result into a VTL triode amp. The VTL had a smooth, big, wide, spacious, dynamic 3D sound. The Museatex amp seemed fully transparent except for this grain thing. You had this big, colorful, spacious 3D sound with grain in the air. More than mist. Listening to jazz combos it was like an airy space in a bar from the 1970's with smoky heavy air from people smoking cigars there. Or maybe more like a heavy smoldering fire in the kitchen of a jazz bar. The air had texture considerable texture. Yet the MOSFET was totally colorless except for that one thing. Fed into the Spectral DMA50 and it sounded transparent except for this grain. A texture that was stuck in the music rather than letting it ring free and sound natural.
 

watchnerd

Grand Contributor
Joined
Dec 8, 2016
Messages
12,449
Likes
10,408
Location
Seattle Area, USA
I fed it into the dummy load, and fed the result into a VTL triode amp. The VTL had a smooth, big, wide, spacious, dynamic 3D sound. The Museatex amp seemed fully transparent except for this grain thing. You had this big, colorful, spacious 3D sound with grain in the air. More than mist. Listening to jazz combos it was like an airy space in a bar from the 1970's with smoky heavy air from people smoking cigars there. Or maybe more like a heavy smoldering fire in the kitchen of a jazz bar. The air had texture considerable texture. Yet the MOSFET was totally colorless except for that one thing. Fed into the Spectral DMA50 and it sounded transparent except for this grain. A texture that was stuck in the music rather than letting it ring free and sound natural.

By air do you mean the top octave? By mist do you mean distortion?

Sorry, I'm not trying to be a butthead, just trying to relate your description to something happening electrically.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
By air do you mean the top octave? By mist do you mean distortion?

Sorry, I'm not trying to be a butthead, just trying to relate your description to something happening electrically.

Didn't have the ability to test much in those days. So it is your regular subjective description. Air and space is more than top octave I would say. Mist was not distortion. At least not obviously such, and it did not occur just during high levels. Many complained that all MOSFET amps had this character. That I didn't find to be true. CJ Mosfet, and BEL Mosfet amps were fine. I even have a little Sherwood mosfet receiver that is fine. Some say the mist is a myth. This one amp in the bypass test however showed the mist. They were supposedly a common trait of amps using Hitachi mosfets.

Imagine a clean clear soundstage with some depth, vs one that had some sort of mist or smoke or a little something that made the edges and background just not quite so clear or clean. It didn't come at high levels and disappear at low levels. There was always the impression of something not quite clean or clear about the sound picture. Yet some amps with it had exemplary SNR. Some said it was mostly in evidence with capacitive loads and as I was using Acoustats or Quads during this time maybe there is something to that. Has puzzled me too.
 

watchnerd

Grand Contributor
Joined
Dec 8, 2016
Messages
12,449
Likes
10,408
Location
Seattle Area, USA
Didn't have the ability to test much in those days. So it is your regular subjective description. Air and space is more than top octave I would say. Mist was not distortion. At least not obviously such, and it did not occur just during high levels. Many complained that all MOSFET amps had this character. That I didn't find to be true. CJ Mosfet, and BEL Mosfet amps were fine. I even have a little Sherwood mosfet receiver that is fine. Some say the mist is a myth. This one amp in the bypass test however showed the mist. They were supposedly a common trait of amps using Hitachi mosfets.

Imagine a clean clear soundstage with some depth, vs one that had some sort of mist or smoke or a little something that made the edges and background just not quite so clear or clean. It didn't come at high levels and disappear at low levels. There was always the impression of something not quite clean or clear about the sound picture. Yet some amps with it had exemplary SNR. Some said it was mostly in evidence with capacitive loads and as I was using Acoustats or Quads during this time maybe there is something to that. Has puzzled me too.

Interestingly, Soundstage seems to think MOSFET Mist is not 100% real, or at least not an inherent 'sound' of MOSFETs. I have no dog in this fight, but it is interesting:

"As long as MOSFET has been brought up, let me debunk something. I don’t know where it originated, some tube amp dealer, some rabid tube audiophiles, not important. What is important is that certain non-technical high-end reviewers have used the term "MOSFET mist" in their writing, and now people think there is such a thing. It’s a bunch of baloney, nothing but made-up cute words drawn from mistaken conclusions. MOSFET mist is no more useful or descriptive of anything real than Triode Tarnish or Pentode Pallor. I don’t doubt that some people may have listened to X number of MOSFET amps and found them all to be sonically lacking due to hazy sound. That’s fine. But what they heard was X number of not particularly good-sounding MOSFET amplifiers, and there are plenty of those, including some made by well-known manufacturers. They were not hearing something -- specifically a trait of MOSFETs.

How do I know this? Because I have heard amps employing MOSFETs which don’t contain an iota of "mist" in their sound. In fact, these amps are so "mist-free" that your average tube amp (from the middle of the tube-amp bell curve) sounds cloudy and hazy in comparison. I have no problem describing certain (numerous, even) MOSFET amps as sounding hazy or cloudy. But this is not a characteristic of the MOSFET -- you’re just hearing an amp that doesn’t sound particularly good. So promise to help me banish the phrase "MOSFET mist" from the high-end lexicon and I promise I won’t popularize Triode Tarnish or Pentode Pallor. We don’t need a bunch of meaningless obfuscating descriptive terms in wide use. It’s hard enough to describe the way things sound in words without confusing things further."


That's from 1998
 

Sal1950

Grand Contributor
The Chicago Crusher
Forum Donor
Joined
Mar 1, 2016
Messages
14,068
Likes
16,598
Location
Central Fl
Many complained that all MOSFET amps had this character. That I didn't find to be true. CJ Mosfet, and BEL Mosfet amps were fine.
Nelson Pass fell in love with Mosfet's at one point. He had originally designed the Adcom 5X5 series of amps using bipolars cicra late 1980s, but when the guys at Adcom came back asking for a fresh design, he did the 5x00 series using Mosfets. I've got both in house and listened closely to them deciding which I would use for my front L & R channels but couldn't hear any difference. On the other hand a couple pro's I've talked to that do cap updates or upgrades to these amps both claim they prefer the sound of the earlier 5x5 bipolars?
Whatever--------------------
 

watchnerd

Grand Contributor
Joined
Dec 8, 2016
Messages
12,449
Likes
10,408
Location
Seattle Area, USA
Nelson Pass fell in love with Mosfet's at one point. He had originally designed the Adcom 5X5 series of amps using bipolars cicra late 1980s, but when the guys at Adcom came back asking for a fresh design, he did the 5x00 series using Mosfets. I've got both in house and listened closely to them deciding which I would use for my front L & R channels but couldn't hear any difference. On the other hand a couple pro's I've talked to that do cap updates or upgrades to these amps both claim they prefer the sound of the earlier 5x5 bipolars?
Whatever--------------------

Whatever happened to Adcom?

GFA-535 was my first power amp, couple to a GTP-400 pre/tuner. Had it through college and a bit beyond until I had a brief detour into AVRs.
 

Sal1950

Grand Contributor
The Chicago Crusher
Forum Donor
Joined
Mar 1, 2016
Messages
14,068
Likes
16,598
Location
Central Fl
Whatever happened to Adcom?
??? Their Pass designed amps plus a preamp or two he did for them where their best of times IMO.
They moved manufacturing from Japan to China in the late 90s and things were going down hill there after.
I had 2 different design, later 5.1 pre-pros from them that were both crap. The first one developed issues after about 6 months and made 2 trips in for service which kept failing, on the 3 time my dealer just took it back and gave me a newer more expensive model in exchange. Guess what, it failed in the same time frame and was serviced twice with the same outcome. It was still at the head of my 5.1 HT system (working intermittently) when I made my retirement move. I took great pleasure in throwing it right out the front window into a garbage can I had rolled there out of the ally.
I'm sure I was not the only one having these experiences with the Adcom Chinese gear. :(
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
Interestingly, Soundstage seems to think MOSFET Mist is not 100% real, or at least not an inherent 'sound' of MOSFETs. I have no dog in this fight, but it is interesting:

"As long as MOSFET has been brought up, let me debunk something. I don’t know where it originated, some tube amp dealer, some rabid tube audiophiles, not important. What is important is that certain non-technical high-end reviewers have used the term "MOSFET mist" in their writing, and now people think there is such a thing. It’s a bunch of baloney, nothing but made-up cute words drawn from mistaken conclusions. MOSFET mist is no more useful or descriptive of anything real than Triode Tarnish or Pentode Pallor. I don’t doubt that some people may have listened to X number of MOSFET amps and found them all to be sonically lacking due to hazy sound. That’s fine. But what they heard was X number of not particularly good-sounding MOSFET amplifiers, and there are plenty of those, including some made by well-known manufacturers. They were not hearing something -- specifically a trait of MOSFETs.

How do I know this? Because I have heard amps employing MOSFETs which don’t contain an iota of "mist" in their sound. In fact, these amps are so "mist-free" that your average tube amp (from the middle of the tube-amp bell curve) sounds cloudy and hazy in comparison. I have no problem describing certain (numerous, even) MOSFET amps as sounding hazy or cloudy. But this is not a characteristic of the MOSFET -- you’re just hearing an amp that doesn’t sound particularly good. So promise to help me banish the phrase "MOSFET mist" from the high-end lexicon and I promise I won’t popularize Triode Tarnish or Pentode Pallor. We don’t need a bunch of meaningless obfuscating descriptive terms in wide use. It’s hard enough to describe the way things sound in words without confusing things further."


That's from 1998

Yes I get it. I used a big CJ total MOSFET to power my Soundlabs for a few years. No mist.

On the other hand the amp I referenced in comparison to tubes and other SS was misty. As far as that goes the Spectral used MOSFETs.

Bypass testing of amps is very interesting. I suggest more people do it.

The only other amp with something similar were hybrid Counterpoint amps. 6dj8 input and driver stages with MOSFET output. They sounded hazy. Just slightly out of focus.
 

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
There are unfortunate aspects to this sort of 'science' in general:
  • often the person who instigated the experiment (who maybe wants to 'prove' something) also carries out the experiment
  • the same person decides whether the results are 'any good' and can bin any experiments that don't 'give the right answer'
  • no matter how perfect the methodology, the same person writes the headline, the summary and conclusion at the end, and in doing so attempts to 'interpret' the results. These are the only bits that most people can be bothered to read. In one fell swoop, the whole experiment boils down to a variant of the experimenter's own assumptions, biases and motivations.
So that is what I get from listening test-based experiments: tiny fractions of the overall 'problem domain' barely tested sufficiently to scrape someone's definition of "statistical significance" (almost certainly falling short of the strict conditions required for statistics to be valid) and then 'laundered' by the person who did the experiment under a single headline "Phase doesn't matter!" which everyone in the industry then refers to forever more.

Apart from that, they're a truly wonderful thing.

Not true. I offered a self proctored test of burned vs non-burned in cables to the howl at the moon members at Polk audio. I would burn in two cables and leave two virgin. Randomly labeled and ship them on out (There are more than a few companies that ship pre-burned cables).

The participant would self administer everything and it addressed their biggest concern: have the 'un-natrulness' of a being blinded. They could do a fully sighted test, at interval and length of time that they considered best for them. They even had 30 days.

I believe Amir even commented on this protocol @ AVS.

I published the setup in it's entirety to invite hole poking. None could and I was subsequently banned :)
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
Not true. I offered a self proctored test of burned vs non-burned in cables to the howl at the moon members at Polk audio. I would burn in two cables and leave two virgin. Randomly labeled and ship them on out (There are more than a few companies that ship pre-burned cables).

The participant would self administer everything and it addressed their biggest concern: have the 'un-natrulness' of a being blinded. They could do a fully sighted test, at interval and length of time that they considered best for them. They even had 30 days.

I believe Amir even commented on this protocol @ AVS.

I published the setup in it's entirety to invite hole poking. None could and I was subsequently banned :)
Well you could pick this hole in such a test: the cables that have been burned in have been handled by you (were you wearing cotton gloves?) and their connectors are no longer virgin. The participant could pick up on such a clue, even if just subconsciously. Ideally you (or preferably someone else) would have treated both cables identically except for the burning in bit. Other holes include questioning whether your "random" really was random. This was not a double blind experiment.

I'm not sure whether your motivation was neutral, or to prove that burning in works, or doesn't, but if you label people as "howl at the moon", it suggests some lack of neutrality, and maybe some animosity between experimenter and subjects! One of the drawbacks of any testing involving humans is called Demand Characteristics.
A possible reason for demand characteristics is the participant's expectation that he or she will somehow be evaluated and thus figures out a way to 'beat' the experiment to attain good scores in the alleged evaluation. Demand characteristics cannot be eliminated from experiments, but demand characteristics can be studied to see their effect on the experiment. Examples of some common demand characteristics:
  • Rumors of the study – any information, true or false, circulated about the experiment outside of the experiment itself.
  • Setting of the laboratory – the location where the experiment is being performed, if it is significant.
  • Explicit or Implicit communication – any communication between the participant and experimenter whether it be verbal or non-verbal that may influence their perception of the experiment.
Weber and Cook have described some demand characteristics as involving the participant taking on a role in the experiment. These roles include:
  • The good-participant role in which the participant attempts to discern the experimenter's hypotheses and to confirm them.[3] The participant does not want to "ruin" the experiment.
  • The negative-participant role (also known as the screw-you effect[4]) in which the participant attempts to discern the experimenter's hypotheses, but only in order to destroy the credibility of the study.
  • The faithful-participant role in which the participant follows the instructions given by the experimenter to the letter.
  • The apprehensive-participant role in which the participant is so concerned about how the experimenter might evaluate the responses that the participant behaves in a socially desirable way.[5]
In your experiment, maybe there would have been some "negative-participant role" stuff going on..? If participants know what the aim of the study is, the experiment is already in trouble.

If you read about it, it is clear that audiophiles' versions of listening tests are positively amateurish. Maybe we are deluded into thinking that our experiments are about something physical, not merely psychological, but I think that they are 99% psychology!

But I am being disingenuous in even giving the experiment any credence at all: in engineering terms we know what a cable is supposed to do. If it doesn't do that, then we are in trouble! If the results of burning in cannot be measured objectively using the finest equipment known to man, then there is no point asking ordinary people whether they can discern a change through the use of their own primitive equipment in their living rooms.
 
Last edited:
OP
oivavoi

oivavoi

Major Contributor
Forum Donor
Joined
Jan 12, 2017
Messages
1,721
Likes
1,934
Location
Oslo, Norway
Just to add to what Cosmik was saying (which I agree with): Assuming you were to publish this test in a scientific journal, and you wanted to show that cables could make a difference. What you could do then, is to administer the test several times, getting no results, and lo and behold! - suddenly after one of the test you're finally able to crunch your numbers in order to arrive at statistical significance which shows a small cable-effect. Here's for example a guy from psychology who was able to show statistically that people can look into the future, published in a top journal (not kidding): http://psycnet.apa.org/journals/psp/100/3/407/

Of course, that experiment has never been replicated. It was a combination of bad experimental design and stupid use of statistics. But that's the problem with those kind of studies which rely on statistical significance in a simplistic manner.
(btw, the publication of that article led to some serious soul searching among psychologists about the statistical methods they accept as valid)

I don't mean that experiments are worthless. But it's hard to do right, and if the effects one finds are very small, they are probably not that important. If one finds big and substantial effects in an experiment, and if they can be replicated by other researchers who use other approaches, then we're talking.
 
Last edited:

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
Well you could pick this hole in such a test: the cables that have been burned in have been handled by you (were you wearing cotton gloves?) and their connectors are no longer virgin. The participant could pick up on such a clue, even if just subconsciously. Ideally you (or preferably someone else) would have treated both cables identically except for the burning in bit. Other holes include questioning whether your "random" really was random. This was not a double blind experiment.

To get into further detail:

All cables would have to be handled just for the

a. Unpacking b. Random labeling c. repackaging.

Your above comment goes to my point about publicly posting the setup and testing protocol: It invites feedback and it's a way to improve the apparatus. And you are right about the cable treatment equality excepting the burn in process up to and including plugging the virgin cables into and RCA to simulate wear.

To answer one question: I offered to video document the entire process including a web cam on the cables under a 96 hour burn in process live. So if anyone wanted to actually watch cables burn in they could.

Lastly I'm not sure what you mean by *"random" really was random"*.

The cables would be color labeled and a password protected zip file sent out ahead of shipping the cables with the answer key. After the participant turned in their results publicly I would then send the password to open the zip .

I'm not sure whether your motivation was neutral, or to prove that burning in works, or doesn't, but if you label people as "howl at the moon", it suggests some lack of neutrality, and maybe some animosity between experimenter and subjects! One of the drawbacks of any testing involving humans is called Demand Characteristics.

In your experiment, maybe there would have been some "negative-participant role" stuff going on..? If participants know what the aim of the study is, the experiment is already in trouble.

In this particular instance I was actually testing claims: "Burned in cables sound better". So my goal was to find out if they indeed did to the people making such claims.

I was interested if: 1. No difference 2. A difference but preference didn't track to the burned in pair 3. A difference and the preference tracked to the burned in pair. To add incentive I offered $100 to a successful outcome. They had a random chance of 12.5% in getting it right by mere guess.
 
Last edited:

Jinjuku

Major Contributor
Forum Donor
Joined
Feb 28, 2016
Messages
1,278
Likes
1,180
Just to add to what Cosmik was saying (which I agree with): Assuming you were to publish this test in a scientific journal, and you wanted to show that cables could make a difference. What you could do then, is to administer the test several times, getting no results, and lo and behold! - suddenly after one of the test you're finally able to crunch your numbers in order to arrive at statistical significance which shows a small cable-effect. Here's for example a guy from psychology who was able to show statistically that people can look into the future, published in a top journal (not kidding): http://psycnet.apa.org/journals/psp/100/3/407/

Of course, that experiment has never been replicated. It was a combination of bad experimental design and stupid use of statistics. But that's the problem with those kind of studies which rely on statistical significance in a simplistic manner.
(btw, the publication of that article led to some serious soul searching among psychologists about the statistical methods they accept as valid)

I don't mean that experiments are worthless. But it's hard to do right, and if the effects one finds are very small, they are probably not that important. If one finds big and substantial effects in an experiment, and if they can be replicated by other researchers who use other approaches, then we're talking.

That's why the protocol is open and public.
 
Status
Not open for further replies.
Top Bottom