So I'm not a scientist just a poor dope who is trying to understand why I hear differences between, say, amplifiers.
Well you hear it because your brain think it is what it hear. Why? Maybe what you hear is a real difference. Or you hear a difference because you expect to hear a difference. Who am I to know? We hear with our brains, and the brain never register only an isolated sensory impression, as in a measuring device. In the brain, hearing impressions are constantly mixed with other impressions and with our stored experiences and prejudices. On the personal level, it is very difficult to sort out what we hear from other impressions. Some differences are of course obvious. But in HiFi we mostly talk about small subtle differences.
I believe that you confuse personal experiences and impressions with what constitutes a scientific method. It may be difficult and not practical for the individual consumer to carry out scientifically based evaluations, but one could wish that HiFi magazines and reviewers would avoid the purely subjective and rely on scientific methods.
What I find strange in your argument is that you claim that audiophiles hear differences when listening is not performed under controlled forms, because the controlled forms affect what we hear. You then conclude that it is the scientifically controlled situation that leads to the wrong conclusion, not that it is in the uncontrolled situation that mistakes or bias arise. You therefore turn the burden of proof, and dismiss the objective with the subjective. That's a problematic argument. How do you know it's not exactly the opposite?
The problem we deal with, when it comes to scientific methods and determine what is good hi-fi equipment, is to bridge the risk of subjective impressions. Consequently, we perform controlled listening tests, or we measure the properties of the devices. If you have objections to this, then you must specify on a principal level what it is that cannot be measured, and why it on a principal level would be safer with subjective impressions than with controlled listening tests. After all, audio is a physical and not a magical reality, and there are no oddities involved in this. However, there are many parameters to take into account, and the discussion of the balance between these parameters is of course relevant, but it must then be based on real and objectively established parameters and not imagined subjective opinions.
I myself have participated in and carried out controlled listening tests, and I am often amazed how the listeners can deceive themselves when they think they are listening to something other than what they are actually listening to. There are lots of tests that show the same thing. For example, many people prefer the sound of vinyl, when they think it is vinyl they are listening to, but when the sound is in fact digital. Or. Many people hear a sound improvement when they think they are listening to a product that pretends to provide cleaner electricity, such as an exclusive power cable, even though it is in fact a simple standard cable they are listening to.
It is perfectly okay to have subjective perceptions and personal favors. No one can demand of anyone else that they report their experience in scientific terms. It's no problem. The problem arises when the industry, not infrequently assisted by HiFi magazines and reviewers, manipulates consumers. I have been to hifi fairs or other so-called test, where representatives of the products clearly manipulated the audience. This may be conscious, or unconscious, but reflects the total absence of a critically grounded scientific approach. We know, for example, that network switches cannot affect the sound in a DAC. Still, I have participated in demonstrations where the audience still hears differences. Why? Because they expect differences. Psychological cognitive bias in other words. And often, even those who are aware of these risks are not safe from this bias. I myself have on several occasions found myself hearing differences where such have not existed. What the heck! I think my HiFi sounds better in the evening. And if I turn off the light, it sounds even better.