I can't say with certainty if
@MattHooper 's test was done with a level of rigor and precision that would pass peer-reviewed scientific muster. But Matt was open about the fact that this was a best effort and he himself is not claiming that level of scientific validity.
With that out of the way, I don't know with certainty if Matt's results are meaningful, but I am inclined to believe they are. They are consistent with what many folks have reported over many years about the sonic signature of some tube gear - and crucially, those reports seem to have a degree of consistency regardless of whether people prefer the tube sound or the non-tube sound. There are also a number of potential factors that could sensibly contribute to what he heard: the distortion of the CJ amp, the high-frequency response, and I suppose also things like output-input impedance of the CJ-Benchmark connection and the simple fact that he was actually listening only for the additive effect of the CJ being put into the chain, rather than comparing just the CJ with just the Benchmark.
So it's only one data point, and it's not conclusive, but I find it a heck of a lot more probable, and therefore more useful. than 99% of the other listening impressions/tests that folks post on audiophile forums.
Thanks Matt!
I appreciate the nuanced post tmtomh!
I think you've captured my own view of the results.
We can't all have the Absolute Answers to everything, even in the realm of audio, so our individual views are going to be influenced by what we read and also very much by personal experience. If for instance you have personal experience blind testing AC cables and finding no sonic difference (as I have) that (along with technical arguments) is going to influence your attitude towards the prospect they likely sound different.
Similarly, if one has done blind tests on their own gear (e.g. tube amps) and found no detectable difference, that's personal experience is going to strongly influence their view of these things.
It's one reason why even engineers can disagree - whether it's the "best" way to design a motorcycle, boat, turntable or speaker or whatever, there will be technical agreements and disagreements, but the scope of one's personal experience even as an engineer - little epiphanies along the way when you've tried certain designs - will lead you down one path, another person down another. (This is what I've observed in seeing many discussions among technically knowledgeable folks).
Since most of us cannot strictly test everything we do, pragmatism means we are often making choices based on non-scientific-level inferences. That's different from 'anti-scientific' which I take to be coming to overconfident beliefs that actually contradict our best science. In other words: "tube preamp sounds a little different? Eh...could be." "Expensive HDMI cable upgrades the sound/picture on your TV vs a competently designed cheap HDMI cable?" Sorry...that's not how HDMI actually works. Even if I felt I saw a difference I'd want more rigorous confirmation. And in fact, I did that for a range of cheap to expensive video cables years ago - blind tested, also with other participants - no difference.
On the other hand, in the 90's I was told that the differences I seemed to hear between some DACs/CDPs I owned shouldn't be there. So I blind tested them, using the method suggested by the skeptics, and easily identified between them. What to do with such "data?"
Well, the skeptics, and I understood their skepticism (the reason I did the blind test was due to my own skepticism!), could always say "
Well, even if we can't find a problem with your method, we weren't there and you could have screwed up somehow." Fair enough. I was there, was familiar with how we did the test, and could not find any likely flaw at all. So I tentatively accept the results and move on, people reading the results could make their own assessment.