Can you really hear the lossy codec? - a public blind test (RESULT)

[JohsonChou]

On a more serious side, this shows how difficult it really is to discern lossy from lossless. I thought my threshold was mp3/320. It seems to be lower. . My ears’ Golden platina is flaking.

256kbps AAC should be some what significantly better than 320kbps MP3, MP3 is a very old codec without recent improvements and updates, it is not really suitable for today’s equipment.

 

[Soniclife]

Just the regular aac codec in ffmpeg, highest quality, 256kbps VBR

Thanks, it's probably a lot better than when I played with it last, if I had a use for lossy these days I might well use it.

 

[Soniclife]

256kbps AAC should be some what significantly better than 320kbps MP3

Seconded.
Do you know how ogg compares to aac?

 

[JohsonChou]

Seconded.
Do you know how ogg compares to aac?

Ogg is actually a container (or you can say it is just a file format), so it is not a codec , you can even put FLAC encoded audio in a ogg file. Vorbis is the most used lossy audio codec with Ogg, so I will just assume you are talking about Vorbis...
Somebody did do a blind test in 2005
https://web.archive.org/web/2014060...drogenaud.io/forums/index.php?showtopic=36465

Looks like Vorbis won over AAC. But that was 2005 and just 1 person. 13 years of developer sweat and tears can change things a bit. I personally believes AAC is better than Vorbis today.

However, a upcoming audio codec Opus (which is also usually put into Ogg) shows great promise, it is supposed to be better than existing audio codec at any bitrate. https://en.wikipedia.org/wiki/Opus_(audio_format)

 

[andreasmaaan]

Ogg is actually a container (or you can say it is just a file format), so it is not a codec , you can even put FLAC encoded audio in a ogg file. Vorbis is the most used lossy audio codec with Ogg, so I will just assume you are talking about Vorbis...
Somebody did do a blind test in 2005
https://web.archive.org/web/2014060...drogenaud.io/forums/index.php?showtopic=36465

Looks like Vorbis won over AAC. But that was 2005 and just 1 person. 13 years of developer sweat and tears can change things a bit. I personally believes AAC is better than Vorbis today.

However, a upcoming audio codec Opus (which is also usually put into Ogg) shows great promise, it is supposed to be better than existing audio codec at any bitrate. https://en.wikipedia.org/wiki/Opus_(audio_format)

Interesting that new codecs are still being developed. I would have thought that there is just no market for it anymore, as anyone who cares enough about fidelity now has reasonably easy access to lossless downloads/streaming, and anyone who doesn't is probably content with whatever compression codec their software or streaming service happens to support.

 

[JohsonChou]

Interesting that new codecs are still being developed. I would have thought that there is just no market for it anymore, as anyone who cares enough about fidelity now has reasonably easy access to lossless downloads/streaming, and anyone who doesn't is probably content with whatever compression codec their software or streaming service happens to support.

New codec like HE-AAC, Opus are usually attractive because of their low bitrate performance. They are mostly develop to either be a cheaper streaming codec (believe me, stream at 80kbps HE-AAC saves a lot of money comparing to 128kbps AAC) or to replace AMR, which charges a patent fees.

In short, they are not developed for us who wants high quality audio, nobody cares about us. T_T

 

[bunkbail]

B sounds better to my ears. Hmmmm....

 

[JohsonChou]

Poll is closed, result is posted in original post.

 

[cjfrbw]

Hmm, I "guessed" it, won't give myself much credit beyond that. It was based on the fuller reverberation of the bass and the apparent speed and decay of the transient percussions. I listened over Sony and Stax headphones.

 

[ehowarth]

It looks like the poll closed a half hour too soon for me...

Regardless, A was superior. Listen for decays and ambience.

B was acceptable. A sounded better and fatigued less. But I was wondering during the test if my initial impression of A's superiority was clouding all my subsequent observations. Every time I switched to A I preferred it even when I could not distinguish why.

Would different recordings be even less obvious? Would better equipment have been more revealing?

Test equipment was an iPhone, Apple lightning headphone adapter, and Shure 846 in-ear monitors.

 

[bunkbail]

Answer is: A is lossless

Welp.

If you voted for B, don't feel bad, I might just saved you from buying that $1000 DAC.

Jokes on you, now I'm getting one.

 

[SIY]

6 people voted for A, and 6 people voted for B. So basically people are just blindly guessing...
Conclusion is that we generally cannot tell the difference between AAC 256kbps and Lossless.

You cannot draw that conclusion from that experiment. For example, how would you exclude everyone hearing a difference but preferences being split? Or some people hearing a difference and some not? Or... well, you get the idea.

If the question you want to answer is "can anyone tell the difference between AAC 256kbps and lossless," the experiment has to be structured quite differently. Ditto the question, "if anyone can tell the difference, what percentage of the population can?" Or the question, "If some of the population can tell the difference, what is the distribution of preference?" All are valid questions, all would have experiments and data analysis structured differently.

 

[JohsonChou]

You cannot draw that conclusion from that experiment. For example, how would you exclude everyone hearing a difference but preferences being split? Or some people hearing a difference and some not? Or... well, you get the idea.

If the question you want to answer is "can anyone tell the difference between AAC 256kbps and lossless," the experiment has to be structured quite differently. Ditto the question, "if anyone can tell the difference, what percentage of the population can?" Or the question, "If some of the population can tell the difference, what is the distribution of preference?" All are valid questions, all would have experiments and data analysis structured differently.

Well, I cannot draw any serious conclusion from my experiment, it has a sample of 16 and a sound track downloaded from internet archive. I am sure my experiment won't pass any peer review. It is just a experiment for fun.

 

[RayDunzl]

I didn't vote. My ears aren't qualified for jury duty.

Listened on the PC on headphones, looked at the files in Audacity.

Visually, no difference noted, though, eventually, some can be found. The files are aligned, they remain so till the end, and little details appear the same in both.

File A is neatly engineered so there are no peaks past a certain point where many of them reside. File B isn't quite as obsessive about its limits.

File B is a little higher in level than File A. The highest peak is about .4 dB different. Some difference persists throughout.

Invert File B and combine with File A to create a new track, and a nice rythmic swishy sound emerges.

 

[SIY]

Well, I cannot draw any serious conclusion from my experiment, it has a sample of 16 and a sound track downloaded from internet archive. I am sure my experiment won't pass any peer review. It is just a experiment for fun.

Experiments for fun are great. And you can indeed conclude that, "We did not see any significant preference between AAC 256 and lossless." You can't conclude, "We did not see any significant ability to distinguish between AAC 256 and lossless."

What you might also want to do for fun is, once you confirm that the levels are well-matched, have participants run an ABX between the two files to see if anyone can indeed distinguish them.

 

[Krunok]

Regardless, A was superior. Listen for decays and ambience.

A was superior because you knew it was lossless.

 

[bakker_be]

Best way I know to do these kinds of comparisons is the ABX-component fro Foobar2000 (https://www.foobar2000.org/components/view/foo_abx). Of course this also limits the audience to Windows users.
I just found this cross-platform tool (https://lacinato.com/cm/software/othersoft/abx) but haven't tried it yet ...

 

[amirm]

I just found this cross-platform tool (https://lacinato.com/cm/software/othersoft/abx) but haven't tried it yet ...

I have run it in the past and it works well.

 

[Krunok]

I didn't vote. My ears aren't qualified for jury duty.

Or you could vote "no difference", which also would actually be a true and honest opinion from your side.

 

[ehowarth]

A was superior because you knew it was lossless.

Yes. I knew it was lossless after I tried to vote. My main concern was that I immediately preferred it to B and wondered if every time I switched back I still preferred A only because I had decided it was better initially. Maybe ABX software would have helped.

I really appreciate all the people on this site sharing such practical information on our not so practical hobby.