Limitations of blind testing procedures

[oivavoi]

Well by being rational we can figure out how far it makes sense to take it. Now people who rationalize that is another thing altogether. So if we put together metrics for a transparent DAC, and we know what is sufficient, then we can see what meets those criteria. Anything more is not for performance. That doesn't mean there aren't other good reasons. That is why I am interested in Amir's reviews of budget DACs. He has crossed a couple of really cheap ones off the list for serious audiophiles. I have presented one that fails a bit. I am of course hoping he finds some that are sufficient for little money. I know of some for $350 to $500 that do the trick. Spending more is not for performance.

Keep getting that message out, let people hear it and in time maybe most of those other products aren't the status symbols they seem currently. People can spend money as they wish, but I feel high end audio has become a case where a significant portion of the money in that market goes for things of no consequence. One result is harm to working products that move things forward.

So for instance I right now have one DAC that cost me $350 I would not be ashamed of dropping into any system. There are some DACs for around $1 k and maybe $2k that I might consider purchasing for other reasons. Beyond that I don't know. We need to get the point across that $2k doesn't get you 90% of the way there, it gets you all the way there.

I do remember Wilson demoing their second most expensive speaker at a big high fi show in 2004 to much accolades. They revealed the last day the actual source for it all was a then current iPod. Lots of motivations and effects one could examine there. I think the main worthy one is with modern digital gear the place to spend the money is the place that can really make the difference and that is speakers.

Good post.

If I was to make a list of recommendations over the most rational way to achieve good sound as I see it now, it would go something like this:

1) Big listening room with high ceilings (high celings most important actually), if can afford
2) Good speakers - best practice IMO implies low distortion, even dispersion with frequency, good dynamic abilities and phase coherent active crossovers (beyond that, there is no "correct way" I believe - electrostats, omnis, dynamic dipoles, monkey coffins, horns... they all have their merits)
3) Good placement of speakers and listening position
4) Room eq/correction in the bass region, above that optional, according to taste
5) Sufficient amplifier headroom, depending on speaker/driver sensitivity

Beyond that, other things pale in importance. Still, I'm currently agnostic as to what constitutes "good enough", and what things may represent actual and objective improvements. How good a dac can still improve the experience? What about amplifiers? Etc. I tend to go for no nonsense budget solutions in the electronics department, and there's no doubt that the traditional box oriented audiophilia has gotten everything backwards. But I don't feel certain that we know all there is to know about how electronics etc work, and how it may affect the listening experience.

(edited: added a couple of points to the list)

 

[Cosmik]

In this post, a blind listening test is suggested for amplifiers. A question we might ask is: how do we know they are actually amplifiers? They may be intended to be amplifiers, but who can actually say for certain? If a difference is heard between them, can we say that at least one of them is not, actually, an amplifier?

Is a home theatre amp with an 'Aural Spatializer' feature strictly speaking an amplifier? What if we press the button during the listening test and find that the listeners actually prefer its version of "amplification"? Philosophically it is no different from any "amplifier" with a known deviation from linearity, such as a valve amp with output transformer etc. that is 'intended' to provide (even if we pretend not to realise) harmonic distortion and a degree of artificial frequency response 'warmth'. Should this device strictly be called an amplifier?

How far can we take the known deviation from linearity? An "amplifier" might be able to intelligently enhance vocals and correct out-of-tune singing. It might even be able to substitute a different tune altogether: if the listeners were being subjected to some atonal process music and the "amplifier" automatically substituted the tune of My Way, I am pretty sure most would prefer it! It would be a really good "amplifier", and science would have demonstrated it through the amazing power of the blind listening test

 

[hvbias]

Good post.

If I was to make a list of recommendations over the most rational way to achieve good sound as I see it now, it would go something like this:

1) Big listening room with high ceilings (high celings most important actually), if can afford
2) Good speakers - best practice IMO implies low distortion, even dispersion with frequency, good dynamic abilities and phase coherent active crossovers (beyond that, there is no "correct way" I believe - electrostats, omnis, dynamic dipoles, monkey coffins, horns... they all have their merits)
3) Good placement of speakers and listening position
4) Room eq/correction in the bass region, above that optional, according to taste
5) Sufficient amplifier headroom, depending on speaker/driver sensitivity

Beyond that, other things pale in importance. Still, I'm currently agnostic as to what constitutes "good enough", and what things may represent actual and objective improvements. How good a dac can still improve the experience? What about amplifiers? Etc. I tend to go for no nonsense budget solutions in the electronics department, and there's no doubt that the traditional box oriented audiophilia has gotten everything backwards. But I don't feel certain that we know all there is to know about how electronics etc work, and how it may affect the listening experience.

(edited: added a couple of points to the list)

Hence my signature

I've never seen objectively correct measurements from any planar speaker. And I say this as someone that will always have my rebuilt ESL57 in some room or another.

 

[Blumlein 88]

In this post, a blind listening test is suggested for amplifiers. A question we might ask is: how do we know they are actually amplifiers? They may be intended to be amplifiers, but who can actually say for certain? If a difference is heard between them, can we say that at least one of them is not, actually, an amplifier?

Is a home theatre amp with an 'Aural Spatializer' feature strictly speaking an amplifier? What if we press the button during the listening test and find that the listeners actually prefer its version of "amplification"? Philosophically it is no different from any "amplifier" with a known deviation from linearity, such as a valve amp with output transformer etc. that is 'intended' to provide (even if we pretend not to realise) harmonic distortion and a degree of artificial frequency response 'warmth'. Should this device strictly be called an amplifier?

How far can we take the known deviation from linearity? An "amplifier" might be able to intelligently enhance vocals and correct out-of-tune singing. It might even be able to substitute a different tune altogether: if the listeners were being subjected to some atonal process music and the "amplifier" automatically substituted the tune of My Way, I am pretty sure most would prefer it! It would be a really good "amplifier", and science would have demonstrated it through the amazing power of the blind listening test

Your working hard to confuse yourself I think. If we hook up amps and listen for preference then possibly anything goes or perhaps there is a consensus. Typically testing would be to hear a difference. Is A audibly different than B or not. If we find a yes, then we can go on and try to test for preference as in is A preferred to B or the reverse. This is another place casual audiophiles confuse themselves. They start off on jump street with a preference in their mind and look for confirmation. Yet they skipped the step to see if any difference is actually there or not. This is right next door to where a real difference is heard, preferred and audiophiles assume the preferred is of higher fidelity. Maybe it is and maybe it isn't. That is how you end up saying measurements are wrong or incomplete or aren't telling us the whole story. You do that by getting the story wrong all the way around.

 

[Blumlein 88]

Good post.

If I was to make a list of recommendations over the most rational way to achieve good sound as I see it now, it would go something like this:

1) Big listening room with high ceilings (high celings most important actually), if can afford
2) Good speakers - best practice IMO implies low distortion, even dispersion with frequency, good dynamic abilities and phase coherent active crossovers (beyond that, there is no "correct way" I believe - electrostats, omnis, dynamic dipoles, monkey coffins, horns... they all have their merits)
3) Good placement of speakers and listening position
4) Room eq/correction in the bass region, above that optional, according to taste
5) Sufficient amplifier headroom, depending on speaker/driver sensitivity

Beyond that, other things pale in importance. Still, I'm currently agnostic as to what constitutes "good enough", and what things may represent actual and objective improvements. How good a dac can still improve the experience? What about amplifiers? Etc. I tend to go for no nonsense budget solutions in the electronics department, and there's no doubt that the traditional box oriented audiophilia has gotten everything backwards. But I don't feel certain that we know all there is to know about how electronics etc work, and how it may affect the listening experience.

(edited: added a couple of points to the list)

I'll give you my opinion, some of which has facts behind it. Some of which has suggestive if not conclusive facts behind it.

Everything from the input of the microphone preamplifier to the input of the power amp for speakers when built to decent modern performance standards and usually without costing a bundle is capable of full audible transparency. Audiophiles are continually beating themselves up and making some people preying on them rich by worrying about this. There are some who can never ever ever be convinced any type of gear has gotten so good improving it is of no consequence. So even when such things are made, become widely available and we have good evidence of it they ignore it or refuse it. At a bare minimum all the gear has gotten so good any differences that may remain are so swamped, covered up and overpowered by an order or two of magnitude by the sins of transducers any improvement is going to be so small as to be truly, truly trivial.

In terms of DACs I don't know it has happened we have full transparency at $200, I have a couple recording interfaces that seem to do the trick for $350. There is gear at $1-2k that surely is going to be enough. You find nothing other than sighted, anecdotal, subjective, biased opinions to the contrary.

I don't think it even necessary to go this far, but a Benchmark DAC 2 for about $2k has some extraordinary measured results of near SOTA accuracy. It is truly all, but completely impervious to jitter in any source. Distortion is vanishingly low as in things like AP gear used to measure it may mostly be measuring themselves.

So the question you might still have to answer is firstly: what demonstration, what testing, what comparisons, what information, just what would convince you that a given DAC is good enough improvement is not something left possible in terms of fidelity?

For me being able to play back a good quality recording, record it with an ADC, and then blind test the original vs the recording and not tell a difference over the best gear I have was convincing. Is there possibly the chance if my playback gear improved I might then hear a difference? Maybe, maybe not. So I did the same thing 8 times. When it was not readily apparent whether one is listening to an 8th generation copy vs an original one has to think just playing back a file is not anywhere close to being audible. Maybe there is someone somewhere with better hearing that could, but as I can't hear with their ears I don't have to worry about it.

 

[RayDunzl]

I've never seen objectively correct measurements from any planar speaker.

I don't understand.

 

[Cosmik]

Your working hard to confuse yourself I think.

Almost certainly!

If we hook up amps and listen for preference then possibly anything goes or perhaps there is a consensus. Typically testing would be to hear a difference. Is A audibly different than B or not. If we find a yes, then we can go on and try to test for preference as in is A preferred to B or the reverse. This is another place casual audiophiles confuse themselves. They start off on jump street with a preference in their mind and look for confirmation. Yet they skipped the step to see if any difference is actually there or not. This is right next door to where a real difference is heard, preferred and audiophiles assume the preferred is of higher fidelity. Maybe it is and maybe it isn't. That is how you end up saying measurements are wrong or incomplete or aren't telling us the whole story. You do that by getting the story wrong all the way around.

I think that if we take a step back, we can see that it is all pointless. If two devices or processes sound identical, so what? Does that tell you which one to use? Or can we say they are interchangeable? Well, if measurements showed that one DAC had 0.01% distortion and the other 0.00001%, yet (as is highly likely) the listeners hadn't heard the difference in this test, you would be misguided to decide that the 0.01% distortion device was as good as the other. Maybe one of the devices or processes is cheaper than the other, like a $2k amp versus $20k, so a 'sounds identical' verdict appears to be a great result - saving $18k! But did you check that a $20 amp off eBay didn't also sound identical? If so, would you buy it? Even if you could overcome the psychological hurdle, I think you would first consult the measurements and other specifications- so the listening test becomes moot.

And if there's an audible difference, it doesn't tell you which of two choices is better - so what use is the information? Maybe there is an audible difference between the headphone output of an iPad and a $30,000 DAC, but it doesn't tell you that the iPad isn't actually better than the 'boutique' (designed by a 'maverick' from discontinued 'classic' chips and valves) DAC. Measurements would probably confirm which is the closer to being transparent - so again the listening test is moot. If there's an audible difference, then at least one of the devices cannot actually be a DAC; it isn't 'definable'. If we go down the preference route, we are in 'effects box' territory where listener preference may be for some arbitrary effect, not transparency. Unless you include all possibilities, your listening test cannot show that the listeners wouldn't prefer an 'Aural Spatializer' effect or a guitar fuzzbox, so again the listening test results are pointless.

 

[Thomas savage]

Almost certainly!


I think that if we take a step back, we can see that it is all pointless. If two devices or processes sound identical, so what? Does that tell you which one to use? Or can we say they are interchangeable? Well, if measurements showed that one DAC had 0.01% distortion and the other 0.00001%, yet (as is highly likely) the listeners hadn't heard the difference, you would still be stupid to decide that the 0.01% distortion device was as good as the other. Maybe one of the devices or processes is cheaper than the other, like a $2k amp versus $20k, so a 'sounds identical' verdict appears to be a great result - saving $18k! But did you check that a $20 amp off eBay didn't also sound identical? If so, would you buy it? Even if you could overcome the psychological hurdle, I think you would first consult the measurements and other specifications- so the listening test becomes moot.

And if there's an audible difference, it doesn't tell you which of the two choices is better - so what use is the information? Maybe there is an audible difference between the headphone output of an iPad and a $30,000 DAC, but it doesn't tell you that the iPad isn't actually better than the 'boutique' (designed by a 'maverick' from discontinued 'classic' chips and valves) DAC. Measurements would probably confirm which is the closer to being transparent - so again the listening test is moot. If there's an audible difference, then at least one of the devices cannot actually be a DAC; it isn't 'definable'. Straight away we are in 'effects box' territory where listener preference may be for some arbitrary effect, not transparency. Unless you include them, your listening test cannot show that the listeners wouldn't prefer an 'Aural Spatializer' effect or a guitar fuzzbox, so again the listening test results are pointless.

Best just do what I do.., buy the one that looks the best to you. If it measures well and looks good it's a GOAL.....

 

[oivavoi]

If there's an audible difference, then at least one of the devices cannot actually be a DAC; it isn't 'definable'.

For the record, I know people who've ABXed dacs. According to them, this is mostly the case with "audiophile" dacs, which often impart a signature of their own. They find it much harder to ABX pro dacs. Bob Katz claims that he wasn't able to ABX dacs (the most transparent ones) with the Revel speakers he used for mastering before, but that he's been able to do it with very revealing headphones, and with his new Dynaudio speakers (they're only 100000 USD a pair or something).

Here's a Gearslutz thread where they compare AD/DA-loops using Audio Diffmaker btw. We see that no AD or DA converters seem to be fully transparent from an objective point of view:
https://www.gearslutz.com/board/12451994-post1433.html

This doesn't necessarily mean that it would be possible to tell them apart with ordinary music listening, off course.

 

[oivavoi]

I'll give you my opinion, some of which has facts behind it. Some of which has suggestive if not conclusive facts behind it.

Everything from the input of the microphone preamplifier to the input of the power amp for speakers when built to decent modern performance standards and usually without costing a bundle is capable of full audible transparency. Audiophiles are continually beating themselves up and making some people preying on them rich by worrying about this. There are some who can never ever ever be convinced any type of gear has gotten so good improving it is of no consequence. So even when such things are made, become widely available and we have good evidence of it they ignore it or refuse it. At a bare minimum all the gear has gotten so good any differences that may remain are so swamped, covered up and overpowered by an order or two of magnitude by the sins of transducers any improvement is going to be so small as to be truly, truly trivial.

In terms of DACs I don't know it has happened we have full transparency at $200, I have a couple recording interfaces that seem to do the trick for $350. There is gear at $1-2k that surely is going to be enough. You find nothing other than sighted, anecdotal, subjective, biased opinions to the contrary.

I don't think it even necessary to go this far, but a Benchmark DAC 2 for about $2k has some extraordinary measured results of near SOTA accuracy. It is truly all, but completely impervious to jitter in any source. Distortion is vanishingly low as in things like AP gear used to measure it may mostly be measuring themselves.

So the question you might still have to answer is firstly: what demonstration, what testing, what comparisons, what information, just what would convince you that a given DAC is good enough improvement is not something left possible in terms of fidelity?

For me being able to play back a good quality recording, record it with an ADC, and then blind test the original vs the recording and not tell a difference over the best gear I have was convincing. Is there possibly the chance if my playback gear improved I might then hear a difference? Maybe, maybe not. So I did the same thing 8 times. When it was not readily apparent whether one is listening to an 8th generation copy vs an original one has to think just playing back a file is not anywhere close to being audible. Maybe there is someone somewhere with better hearing that could, but as I can't hear with their ears I don't have to worry about it.

Good point. In practical terms, we agree. As for myself, I'm currently debating with myself whether the internal dac in my sonos connect is good enough, or whether I should invest in an external dac. This will feed into a speaker/amp combo that's costing more than I ever thought I would invest in something like hifi. I'm not convinced an external dac would represent an audible improvement for me. Exactly for reasons such as those you propose.

But still... I tend towards the view that I'll buy a better dac. Just for reasons of "high fidelity": Knowing that my system is providing higher fidelity, even though my ears might not perceive it. Where to put the limit? I don't know. I think Emotiva will come out with a DC-2 dac this year. That's probably as far as I'll be willing to stretch it.

 

[FrantzM]

After many years as a classic by that I mean subjective audiophile, I have come (very late) to term with the fact that our senses are fallible. Our ears are not as good as we (audiophiles) think and the differences between gears (electronics) are not that great. I also know that Blind testing is not a panacea nor absolute in its results. It also remains a very difficult tool to use ,let alone master, more so than many realize. It is however the best we have to provide ourselves a healthy dose of perspective, of reality.
This said andt in spite of training myself to hear differences between mp3 and lossless and being able on test tones and music I know, to tell some differences on mp3 256 k , I have found myself too often failing to discern 320 kbps mp3 from lossless redbook. It has been almost a revelation, as I focus, now more on enjoying music rather than eternizing on its provenance... I have discovered so much more music through Spotify that I enjoy since the discovery of how my vaunted hearing was in fact pedestrian ... Still it remains to my ears, after a modicum of level matching that the Berkeley DAC is superior to the many DACs i have had to the extent I have sold all of these ( Mytek , Luxman, etc) to keep 2 DACS the Ultra portable iFI microDAC (amp, battery, preamp and DAC) and the Berkeley. To my ears still and in the absence of measurements the Berkeley is superior also to the iFi ... Price bias? Perhaps.. Yet I have heard the BADA compared to some extremely pricey DACs and wasn't impressed ... Were those evaluations blind? No.

All this bring this saying from Winston Churchill. It is about Democracy but can be applied to Blind Testing, just replace "Government" by "Testing" and "democracy" by "Blind Testing":

Many forms of Government have been tried, and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed it has been said that democracy is the worst form of Government except for all those other forms that have been tried from time to time.…

 

[Thomas savage]

After many years as a classic by that I mean subjective audiophile, I have come (very late) to term with the fact that our senses are fallible. Our ears are not as good as we (audiophiles) think and the differences between gears (electronics) are not that great. I also know that Blind testing is not a panacea nor absolute in its results. It also remains a very difficult tool to use ,let alone master, more so than many realize. It is however the best we have to provide ourselves a healthy dose of perspective, of reality.
This said andt in spite of training myself to hear differences between mp3 and lossless and being able on test tones and music I know, to tell some differences on mp3 256 k , I have found myself too often failing to discern 320 kbps mp3 from lossless redbook. It has been almost a revelation, as I focus, now more on enjoying music rather than eternizing on its provenance... I have discovered so much more music through Spotify that I enjoy since the discovery of how my vaunted hearing was in fact pedestrian ... Still it remains to my ears, after a modicum of level matching that the Berkeley DAC is superior to the many DACs i have had to the extent I have sold all of these ( Mytek , Luxman, etc) to keep 2 DACS the Ultra portable iFI microDAC (amp, battery, preamp and DAC) and the Berkeley. To my ears still and in the absence of measurements the Berkeley is superior also to the iFi ... Price bias? Perhaps.. Yet I have heard the BADA compared to some extremely pricey DACs and wasn't impressed ... Were those evaluations blind? No.

All this bring this saying from Winston Churchill. It is about Democracy but can be applied to Blind Testing, just replace "Government" by "Testing" and "democracy" by "Blind Testing":

I'm with you on all of that.

 

[Sal1950]

Best just do what I do.., buy the one that looks the best to you. If it measures well and looks good it's a GOAL.....

And it's CHEAP!

 

[Thomas savage]

And it's CHEAP!

Unless it's a TV.....

 

[Sal1950]

Good point. In practical terms, we agree. As for myself, I'm currently debating with myself whether the internal dac in my sonos connect is good enough, or whether I should invest in an external dac. This will feed into a speaker/amp combo that's costing more than I ever thought I would invest in something like hifi. I'm not convinced an external dac would represent an audible improvement for me. Exactly for reasons such as those you propose.

But still... I tend towards the view that I'll buy a better dac. Just for reasons of "high fidelity": Knowing that my system is providing higher fidelity, even though my ears might not perceive it. Where to put the limit? I don't know. I think Emotiva will come out with a DC-2 dac this year. That's probably as far as I'll be willing to stretch it.

I believe most of us here will struggle with the Monster from the ID.
I had a number of ways to perform the dac from my files and had listened to the options. They all sounded very good to me so I wrestled with the idea of spending the $ for a quality stand alone DAC. In the end it came down to a "piece of mind" decision for me to make the purchase.
Only then after doing a bunch of homework did I end up spending $450 for my Emo DC-1.
I'm completely happy but still sometimes torture myself with doubts, wondering if a $2k Benchmark might make a difference. Damn audiophilia nervosa can never be completely cured.

 

[Jinjuku]

But how far do you take it? Common sense says that the headphone output of an iPad or a $80 USB DAC (Creative Soundblaster, say) sounds identical to the $20k DAC. How many audiophiles would simply use the iPad output? Not many, I'd wager. What they would do would be to rationalise that if they spend $2k on a DAC, it will get them "90% of the way there", and then they'll congratulate themselves on saving the $18k, rather than weeping over the $2k they just effectively burned. Me, I'd use the iPad (if I wasn't using a $100 multichannel DAC for my active system).

Common sense says that a HP amp can only be designed to a point of diminishing returns. I've had the HP output of my Emotiva DC-1 compared to a $5000 Ayre and there simply wasn't any difference. It was a pair of Sennheisers that weren't the easiest to drive either.

Bottom line is $50 in parts gets you a world class HP amp.

 

[Blumlein 88]

For the record, I know people who've ABXed dacs. According to them, this is mostly the case with "audiophile" dacs, which often impart a signature of their own. They find it much harder to ABX pro dacs. Bob Katz claims that he wasn't able to ABX dacs (the most transparent ones) with the Revel speakers he used for mastering before, but that he's been able to do it with very revealing headphones, and with his new Dynaudio speakers (they're only 100000 USD a pair or something).

Here's a Gearslutz thread where they compare AD/DA-loops using Audio Diffmaker btw. We see that no AD or DA converters seem to be fully transparent from an objective point of view:
https://www.gearslutz.com/board/12451994-post1433.html

This doesn't necessarily mean that it would be possible to tell them apart with ordinary music listening, off course.

Null testing can be wonderful or confusing. Basically if you can get a deep null, say 70 db or better you can be sure the difference is not audible. However, lesser nulls all depend on what corrupted the null.

Also they are using Diffmaker, which sometimes is wonderful and sometimes gets very confused. It mostly gets confused with speed differences and timing differences. It purports to be able to fix those and sometimes it can.

Now as an example, I just created a file with tones at 25 hz, 200 hz, 4khz, 10 khz and 16 khz. All at -20db and then mixed to come out around -16 db combined.

Supposedly the default setting for Diffmaker only compares 100 hz to 12 khz. I used a filter at 50 hz that shelved down to 1 db which essentially would only drop the 25 hz tone by a decibel. The Diffmaker result of that vs the original (all this done digitally by the way no analog involved yet) was a null of 21 db. Curious don't you think? I then instead used a file with a 1/10th db shelf down at 30 hz. Diffmaker said -43 db. At the other end of the scale I did a similar thing. 15 khz shelf of 1/10th db and Diffmaker said null depth of -43 db. Such results would have been a touch lower if I had used a truly wideband music signal. So all those results in the 40's you see, they aren't necessarily very poor results and quite probably inaudible. Quite a few recording interfaces will droop the low end just a bit by a half db at 25 hz and roll off below that to prevent issues while recording when people bump mics or you have HVAC rumble in a space.

Next I digitally slowed down the file by 200 ppm per million. Default setting in Diffmaker doesn't include sample rate drift. It says 47 db null in that case. One of those times it gets confused, when I engaged rate correction which I have seen work it was worse in this case reporting a null depth of -32 db.

I have seen Diffmaker work well with a meter of loopback cable, get confused with 3 meters and work well again with 10 meters. It seems some timing issues it can correct for and others fall in a range it doesn't handle well. Remember just sending a signal on a cable takes roughly 1 nanosecond per foot, and that is enough to cause havoc with null results if not corrected for in the comparison.

You'll notice a section where they tested with external clocks and some devices achieved very good results. All of those were better than average anyway, but many jump to the conclusion that when a Forsell/Focusrite 245 combo jumps to 80 db nulls with an external clock the clock was really beneficial. What probably happened was everything clocking together removed any drift and if done correctly can make the DAC sample with the appropriate delay for when the ADC samples. This allowed a better result in Diffmaker though audibility of anything may have been completely uneffected. I have used a source DAC that also puts out a clock, made sure to use the same length and type cable to feed the clock to the ADC as the analog cable. So the time traveling the clocking cable is very close to the time the analog signal took travelling between devices, and you get much less of a delay between DAC output and ADC sampling . Diffmaker results jumped 15 db when I did that. You can't do that with a loopback even though the same clock is being used because it doesn't compensate for time travelling the loopback cable itself.

So like so many things a single number is not always as illuminating as it seems without all the details. Those lesser results you have to dig into why before understanding what the number is telling you.

 

[oivavoi]

Null testing can be wonderful or confusing. Basically if you can get a deep null, say 70 db or better you can be sure the difference is not audible. However, lesser nulls all depend on what corrupted the null.

Also they are using Diffmaker, which sometimes is wonderful and sometimes gets very confused. It mostly gets confused with speed differences and timing differences. It purports to be able to fix those and sometimes it can.

Now as an example, I just created a file with tones at 25 hz, 200 hz, 4khz, 10 khz and 16 khz. All at -20db and then mixed to come out around -16 db combined.

Supposedly the default setting for Diffmaker only compares 100 hz to 12 khz. I used a filter at 50 hz that shelved down to 1 db which essentially would only drop the 25 hz tone by a decibel. The Diffmaker result of that vs the original (all this done digitally by the way no analog involved yet) was a null of 21 db. Curious don't you think? I then instead used a file with a 1/10th db shelf down at 30 hz. Diffmaker said -43 db. At the other end of the scale I did a similar thing. 15 khz shelf of 1/10th db and Diffmaker said null depth of -43 db. Such results would have been a touch lower if I had used a truly wideband music signal. So all those results in the 40's you see, they aren't necessarily very poor results and quite probably inaudible. Quite a few recording interfaces will droop the low end just a bit by a half db at 25 hz and roll off below that to prevent issues while recording when people bump mics or you have HVAC rumble in a space.

Next I digitally slowed down the file by 200 ppm per million. Default setting in Diffmaker doesn't include sample rate drift. It says 47 db null in that case. One of those times it gets confused, when I engaged rate correction which I have seen work it was worse in this case reporting a null depth of -32 db.

I have seen Diffmaker work well with a meter of loopback cable, get confused with 3 meters and work well again with 10 meters. It seems some timing issues it can correct for and others fall in a range it doesn't handle well. Remember just sending a signal on a cable takes roughly 1 nanosecond per foot, and that is enough to cause havoc with null results if not corrected for in the comparison.

You'll notice a section where they tested with external clocks and some devices achieved very good results. All of those were better than average anyway, but many jump to the conclusion that when a Forsell/Focusrite 245 combo jumps to 80 db nulls with an external clock the clock was really beneficial. What probably happened was everything clocking together removed any drift and if done correctly can make the DAC sample with the appropriate delay for when the ADC samples. This allowed a better result in Diffmaker though audibility of anything may have been completely uneffected. I have used a source DAC that also puts out a clock, made sure to use the same length and type cable to feed the clock to the ADC as the analog cable. So the time traveling the clocking cable is very close to the time the analog signal took travelling between devices, and you get much less of a delay between DAC output and ADC sampling . Diffmaker results jumped 15 db when I did that. You can't do that with a loopback even though the same clock is being used because it doesn't compensate for time travelling the loopback cable itself.

So like so many things a single number is not always as illuminating as it seems without all the details. Those lesser results you have to dig into why before understanding what the number is telling you.

Thanks. Super interesting. Kudos for testing Diffmaker so thoroughly! I thought Diffmaker was a foolproof way of finding differences, but seems like that assumption was wrong...

 

[Cosmik]

For the record, I know people who've ABXed dacs. According to them, this is mostly the case with "audiophile" dacs, which often impart a signature of their own. They find it much harder to ABX pro dacs.

Yes, for there to be an audible difference, at least one of the 'DACs' has to be quite a long way off from being a DAC, and instead must be some sort of effects box - the deviation would clearly show up in measurements so you wouldn't need to do a listening test for difference.

But each to their own: some people may genuinely like the effects box. But my claim is that nothing useful can be gained from a scientific listening test for preference - it is a completely open-ended aesthetic judgement kind of thing, like asking people what their favourite colour is.

 

[Blumlein 88]

Thanks. Super interesting. Kudos for testing Diffmaker so thoroughly! I thought Diffmaker was a foolproof way of finding differences, but seems like that assumption was wrong...

Diffmaker is flaky. I have sometimes compared things with it I thought were impossible and it lined everything up and did a seemingly magical job. Other times it is thrown off by seemingly trivial issues.

And btw, Diffmaker does match level super precisely, and that sort of thing. In those cases you get nulls of 200 db with digital only files. Often 100 db with exact analog files only corrupted by thermal and flicker noise. For instance nearly any device if you record its output and test it again in 5 minutes usually you get deep nulls down near noise limits. So it can be very useful, but I don't trust it without some other measures to make sure it didn't flake out.