FLAC or MP3? How Can I Determine Which Is Best to Use on My Digital-Audio Players?
Our members requested that this thread be made a sticky. Did you ever wonder why some of our members recommend against using lossless files on digital-audio players? If yes, this is the post for you. The information in this post was spread across several dozen threads.
This thread is just to let you know what is the best way to determine which is the best audio format for you—nothing more, nothing less. You can accomplish this by using your ears and your equipment to listen for differences once variables that can fool you have been removed. This is done with ABX testing. Let us make one thing clear.
No one is trying to tell you what files to use under any circumstances. You should use the files that suit your purposes best. This sticky was created so that an easily referenced set of directions is available. You can now decide what is best by using ABX testing to make an apples-to-apples comparison of various audio formats.
One of the early titles for this thread was FLACs and MP3s. Unfortunately, some read that as FLACs versus MP3s and felt that there was an attempt being made to demonstrate that lossless and lossy files were equal under all circumstances. No. Sorry for the long introduction, but considering the way that this thread has been derailed in the past, we felt that it was best to try to clarify the purpose of this thread.
The difference in sound quality between FLACs and MP3s appears to be a popular topic. This post, therefore, will attempt to answer some of the questions about the sound quality of FLACs compared to the sound quality of MP3s. We have taken part in many discussions on this subject in these forums. Many members have raised objections to what we are going to present.
So, for this reason, we are presenting this post as a conversation between ourselves and a notional member. This notional member will be a composite of some of the members that have raised objections to what we are about to present. What the notional member will have to say will be in italics.
I can hear a big difference between FLACs and MP3s. So can many other individuals. So why all these so-called discussions?
This is actually an excellent question, because the answer is rather complicated. First, you must be aware of the placebo effect. Because of the placebo effect, it is not enough to just listen to a FLAC and then just listen to an MP3. What is needed is an ABX test.
But I have done blind listening before, either by myself or with help from my friends.
That is good, but an ABX test is better because it is one of the most rigorous ways to evaluate the sound quality of lossy files. It is a true double-blind test, and the best way to listen for extremely subtle compression artefacts.
But no one listens to music this way. Sorry, but I really do not see the need to do any ABX tests at all. I have better things to do with my time.
Please read item 4 in this PDF. Because of the placebo effect, many myths are associated with audio. Some unscrupulous individuals have seized this as an opportunity to make money from these individuals who are experiencing the placebo effect.
ABX tests can, therefore, be very useful in leveling the playing field. Yes, ABX tests take time and effort. But it really is the better way to evaluate sound quality. ABX tests have been used to debunk audio myths. Please see, for example, the following.
• Fancy wires sound better than ordinary wires debunked.
• The so-called high-resolution formats sound better than CDs debunked here and here.
• Digital-audio recordings sound bad debunked.
• Expensive amplifiers sound better than ordinary amplifiers debunked here and here.
• More audio myths debunked.
Well, OK. So how does one use an ABX test to evaluate the sound-quality difference between FLACs and MP3s?
Please download foobar2000 and its ABX comparator. They are free downloads. To ABX, please do the following. But first, a couple of recommendations before you start.
• We recommend that you use headphones, Exact Audio Copy or something similar, the newest stable version of LAME—e.g., 3.99.5—and quality settings –V0 to –V9. If you use an MP3 encoder other than LAME, use the newest stable version of that encoder. You should also use a wide range of quality or bitrate settings.
• You need an uncompressed WAV and a perceptually encoded—lossy—copy of the uncompressed WAV. A FLAC or other types of lossless copies of the uncompressed WAV is also OK instead of using an uncompressed WAV. The audio information in any lossless format is the same as in the uncompressed WAV, so using any lossless format instead of an uncompressed WAV allows the same comparison.
1. Open foobar2000, then the File menu. From there, click Add Files. Navigate to where you have the files you want to compare stored. Click on the lossy or lossless file that you want to compare. Repeat to add the other file that you want to compare. Some may find it easier to just drag the files into an open playlist pane.
2. In the open playlist pane, select both of the files that you want to compare and then right click. From the context menu that opens, select Utils. Select ABX Two Tracks. A new window will open.
3. In that window, select the options that you want to apply to your test. You can choose to use ReplayGain if the tracks have that information in their tags. You can also choose to use digital-signal processing (DSP). Once you have made your choices, click OK.
4. In the new window that opens, make the final choices that you want to apply to your testing. You can choose to keep the playback position during track changes or not to keep the playback position during track changes. We recommend Keep playback position when changing track. You can also choose to see the results as you test or to hide them until you are finished with the test.
saratoga recommends that the Hide results box be checked, because “seeing how good or bad you're doing can really mess with your head and throw off your concentration.” “It'll work much better if you don't see the numbers since you won't bias yourself.”
5. Play A and B to get a good idea of how they sound. Unless there is a big difference between the two files, it is not unusual to not be able to hear a difference at this point. Select Play X or Play Y.
6. Compare X or Y to A and B. You can choose to listen to the entire track or just a part of it. You can choose to listen from the beginning or any starting point. These choices can be changed at any time during any part of a single trial in the course of the test.
They can then be changed again in the next or any other following trial in the course of the test. Once you believe that you can hear a difference, choose one of X is A, Y is B or Y is A, X is B. This choice is not locked in until you select Next Trial.
7. Once you have completed the number of trials you have decided on before the start of the ABX test—it must be at least 5 trials—select Exit. A window will come up to allow you to save the results. The saved results will include the files used and the percentage of correct identifications among other information.
8. If Probability that you were guessing reaches 5% or lower, you can hear a difference. If Probability that you were guessing does not reach 5% or lower, you cannot hear a difference.
Why is the minimum number of trials 5?
First, you must know a little about the p-value. For you to legitimately claim to be able to hear a difference, the p-value must be 0.05 or lower. Values like 0.07 or 0.06 are not good enough. Next, please take a look at this ABX binomial probability table. So from this table, we can see that 5 is the minimum number of trials.
Is it also possible to ABX using loudspeakers?
Yes. But, as mentioned earlier, because the compression artefacts can sometimes be extremely subtle, many individuals prefer to use headphones. But if you prefer to use loudspeakers, there is no reason not to. You can also try ABXing with loudspeakers and then with headphones or first with headphones and then later with loudspeakers.
Do I need extremely expensive and high-quality sound cards, headphones, or loudspeakers?
No. You can use any sound cards, headphones, or loudspeakers.
Why use Exact Audio Copy?
Why use LAME?
Why use the newest stable version of LAME?
LAME is always undergoing improvements.
Why use the quality settings –V0 to –V9?
So why is FLAC such a popular format?
FLAC is great for external hard drives and for archiving. And it is also great if you must have a perfect copy of the music on your digital-audio player. But if you are only after great sound quality, MP3 is often a better choice because of the much smaller size. If you have followed all of the recommendations given above and have actually done some ABX tests, I think you too will realize that properly encoded MP3s sound very similar or the same as FLACs.
What do you mean by very similar?
There are rare cases of some music tripping up lossy encoders no matter how good the lossy encoder.
So should I not then be using FLACs on my digital-audio players instead of a lossy format like MP3?
How many songs did you ABX at –V0? Out of all of these ABX tests at –V0, how many times has the p-value reached 0.05 or lower? Can you honestly claim that, from the music you typically listen to, you always or often hear a difference in ABX tests done at –V0?
What about AAC or Vorbis? Should I also ABX these types of files? What about other lossy formats?
Sure. If you want to, why not? We at anythingbutipod always encourage members to expand their knowledge.
If I still believe that MP3s sound bad or that FLACs sound much better than lossy formats such as MP3, is anyone here going to give me a hard time?
We might if, after reading this far, you still refuse to do ABX tests. If you are going to start making claims, you better have some evidence to support your claims such as the results of ABX tests that you have done.
Why are you guys so insistent on ABX tests?
We take pride in trying to give good advice.
This post was written by skip252 and Enigmatic and edited by Enigmatic. This post may undergo further edits. Thank you, WalkGood, dfkt, saratoga, and The DarkSide.
Last edited by Enigmatic; 05-17-2012 at 07:16 AM.
abi >> | Forum Rules | Glossary | Why Rockbox | FLAC or MP3? | irc
There was extensive discussion that followed the original post. We want to continue the discussion but it was detracting from the original purpose. Those posts have been moved here. FLAC or MP3? How Can I Determine Which Is Best for My DAP? Discussion Thread
This way there won't be as much need to move or remove non-ABX chat. I think a thread focused on ABX testing and questions about audible differences is a great idea. We can talk about it in the new thread.
A Glossary For Newbies Why Rockbox? FLAC or MP3? How To Ask Questions The Smart Way
Last edited by WalkGood; 05-21-2012 at 06:03 PM.
|flac vs mp3|
|Thread Tools||Search this Thread|
All times are GMT -5. The time now is 03:53 PM.