Home / Opinions / Sounds Good To Me / The Results

The Results

Oh dear. We won't name names, but of the seven people tested, including five members of the TrustedReviews team, only one person could accurately pinpoint which tracks were MP3 and which tracks were FLACs in every case. Of the rest, three of our subjects managed to pick the lossless track on two occasions, while the other three only managed a single correct choice.

Interestingly, the results didn't come out as we anticipated. We knew that discerning 320kbps MP3 files from lossless FLAC files was going to be hard, but we expected that, with decent listening equipment and a couple of hearings, most of the test subjects would be able to tell a 192kbps MP3 from the FLAC original. Shockingly, this wasn't so. In the tests where we played 320kbps files against FLACs the number of people who chose correctly and incorrectly were equal. In the tests where we played 192kbps files against FLACs, more of our subjects actually went for the lower-quality file.

Does the style of music make any difference? Certainly Massive Attack's dark, heavily layered sound seemed to wrong foot our panel more than the R&B, rock and classical tracks used. One subject even noted that the 192kbps version sounded clearer, but also flatter, as if the sacrifice of some background tone made what remained seem that little bit more defined. With certain styles, it's also easier to hear certain cues that tell you that the file has been compressed. With Radiohead, for example, those who got it right noted that the percussion - specifically - the cymbals, sounded smeared and less well defined in the 192kbps track as opposed to the FLAC version. This difference isn't so pronounced when you go up to 320Kbps.

Ada Mari Frost

June 14, 2013, 6:07 am

High quality FLAC sounds warmer than MP3. Most people equate the warm sound to not being crisp, that's how you get a lot of people thinking the MP3s are better.


January 4, 2014, 1:41 pm

Those are the worst graphs ever. The axis are so poorly labled that I can't understand what they are supposed to show.


February 16, 2014, 2:40 pm

This article is a few years old now and out if date, but even taking that into consideration......

Unless I missed it, no mention was made of the MP3 decoder being used. They're not all created equally in respect to accurate decoding. WMP had a poor decoder prior to version 7. The Winamp MP3 decoder prior to version 2.666 was known to be substandard.

Constant bitrate is never going to be the best method for audio encoding any more than it'd be the best method for encoding video.

Bitrate is not the only variable which effects quality. The LAME MP3 encoder has settings for CBR encoding which use different algorithms and result in different encoding speeds at exactly the same bitrate. The encoding time for 5 minutes of stereo audio using my (aging) PC is around 7 seconds for the LAME default of q3, and 38 seconds for q0 (CBR 192k, LAME 3.99). The article makes no mention of the quality setting used.

From version 3.99, LAME's CBR encoding uses the PSY model from the VBR code. Even today, at the same average (lower) bitrate, CBR encoding will never match the quality of VBR encoding, bitrate. Back in 2009 it was possible it wouldn't even at very high bitrates.

The golden eared folks over at hydrogenaudio have a page dedicated to LAME which states any of the LAME VBR presets from V3 to V0 should normally be "transparent". ie it's not possible to distinguish the MP3 from the original. The average bitrate for V3 is around 175 kbps. For V0 it's 245 kbps.

The article refers to lossless audio and bitdepths/sample rates higher than those used for CD sounding better. A claim which seems to have been debunked.

Hannes Minkema

June 8, 2014, 2:00 pm

"Only one person could accurately pinpoint which tracks were MP3 and which tracks were FLACs in every case."

I appreciate the effort, and I believe the general conclusion is about right. Yet it is unwarranted to state that only one person *can* (or *could*, which is the past tense of *can*) accurately distinguish MP3 from FLAC. The statement should have been that only one person *did* this. But this, by itself, says *nothing* about his general capacity to do so.

This is not nitpicking. I am sure that many readers misinterpret this statement, and believe that this one guy actually has better ears than all the rest. Heck, you might even believe this yourself. But there is no proof of that.

Take one hundred people. Ask them to draw the Queen of Hearts from a stack of playing cards. Two, maybe three of them will do so at the first draw. How is that possible? Should we conclude that they are psychic? Of course not. They are just lucky.

Take one hundred people. Ask them to tell which of two playing cards is red, and which is black. Fifty of them will be totally right at the first guess. Ask these fifty people to do it again. Twenty-five will succeed. Ask them again: twelve of them will be so lucky as to perform the trick thrice. Six will do the trick four times in a row, and three are so 'psychic' that they manage to guess right five times in a row. Incredible! Not.

They have no special gifts, of course. They are just assisted by blind chance. The next time they are put to the same test, they most probably won't be so lucky. So far for their 'general capacity'.

During the MP3-to-FLAC trials, the one guy who was able to guess right four times out of four trials was assisted by blind chance either. You can't erase blind chance. That's why scientific inductive statistics take 'blind chance' into account. And that's why the statement 'one guy *could* do it' is misleading. He couldn't. He just did. The combination of his ability AND blind chance did the trick. But that's not the same.

The question was not if one person *could* distinguish MP3 and FLAC, but whether a more general null hypothesis was rejected or not. That null hypothesis would be that out of seven people trying four times to distinguish MP3 from FLAC, a greater number of guesses turned out to be correct than can be accounted for by blind chance, with an uncertainty of less than 5%.

Anyone with a scientific training could have told you this. It is Statistics 101.

comments powered by Disqus