Sansa Clip+ vs iPod Touch Detailed Measurements & Review

[Xinz]

Very analytic review. Actually its almost not a review anymore, there is so much hard facts in your articles. If anything, this is truly a unbias review

[CrowdAds]

[Inks]

Very nice review, it confirms the performance of clip+ dfkt measured but it also opens the eyes for those ipod haters in regards to it's sound.

[Enigmatic]

Originally Posted by meetooman

My concern is whether compromises made when using lossy encoders at different bitrates can be measurable in a useful way using key parameters (channel separation, frequency response, distortion).

No, because you have to understand the reason for using lossy encoders. The theory predicts that lossy encoders can throw away information but still preserve transparency. The way to test for transparency is to ABX—not channel separation, frequency response, total harmonic distortion, intermodulation distortion, etc.

I know many people only trust listening, but I would like to discover is which way (linear or not) different bitrates are giving better or lower results.

Do you know why many people only trust listening? If you are sincerely interested, then why not ABX? Did you ABX?

http://wiki.hydrogenaudio.org/index....uiet_listening.

Please ABX an MP3 at –V0 against its uncompressed WAV and report back to us.

http://www.foobar2000.org/.
http://www.foobar2000.org/components/view/foo_abx.

Then, if it would be possible to draw eventually a quality/file size graph showing us the "sweet spot".

The sweet spots for high fidelity are –V3 to –V0. Please ABX some files to make sure that hydrogenaudio is not trying to pull a fast one on you.

Will that spot change with each different music?

Why not ABX using files of different music and report back to us.

[NwAvGuy]

Thanks to all for the kind words. As some may have read about my thoughts on iTunes, I'm not an Apple fanboi. But I have to admit the Touch 3G performed surprisingly well in most of the test--it's just too bad about the output impedance.

I'd like to measure some of the current iPods. If they could lower the output impedance, with otherwise similar performance to the Touch 3G, Apple would have a world class player in terms of audiophile performance. They're supposedly going to 24 bit DACs so perhaps they'll improve the analog section then as well?

But, manufactures often go backwards with new products despite their marketing claims--especially in areas that are not immediately obvious or easily verified. Apple's marketing team played up the new "much more durable" screen glass on the iPhone 4, when in reality, it's proven to be far more fragile and there's already a class action lawsuit over it.

Apple largely follows a "form over function" design philosophy. And in wanting to make the iPhone 4 slimmer and more sexy, they made the glass far more vulnerable if the phone is dropped. So I wouldn't be surprised if we see "24 Bit DAC" proudly displayed on the Apple website but the 24 bit iPod's actually measure worse.

I haven't tested both, but the now discontinued Sansa Fuze is supposedly a far better player than the Fuze+ that replaced it. Sometimes it's more about marketing and making a product *seem* better than actual performance.

As to battery life, I think the Clip+ does great for its size. In any portable player the battery is usually the biggest and heaviest component. And given the iPod is about 5 times bigger and heavier, the Clip does pretty well in comparison. Also, I haven't verified it, but several people have claimed Rockbox firmware increases the battery life. I can verify Rockbox offers more and better battery information.

[saratoga]

Originally Posted by meetooman

My concern is whether compromises made when using lossy encoders at different bitrates can be measurable in a useful way using key parameters (channel separation, frequency response, distortion).

They cannot. Lossy encoders work by introducing noise at frequencies you can't hear due to masking. Attempting to simply measure the noise will give you ridiculous numbers since it will not take into account masking.

To assess encoder quality, you need to do ABX testing. That said, even at 128k on HA's listening tests people rate songs as comparable to lossless, so if you're using 160-200kbps you're well into the range where theres no noticeable difference.

Originally Posted by meetooman

I know many people only trust listening, but I would like to discover is which way (linear or not) different bitrates are giving better or lower results. Then, if it would be possible to draw eventually a quality/file size graph showing us the "sweet spot". Will that spot change with each different music?

Theres no way to do this.

Originally Posted by meetooman

AudioDiffMaker seems to be a very interesting tool for ABX test fans. Do you think we could compute an "deviation" value out of a "difference" system?

Doesn't that tool just subtract two waveforms? Thats not ABX testing, and its not really useful for lossy audio.

[NwAvGuy]

I second Saratoga's comments. There's quite a bit on the web regarding blind listening tests of various CODECs and their settings. That's the best resource if you want to know more about what the best compromise might be for your need/ears/storage requirements/etc.

And, Audio DiffMaker is just a way to (in a clever way) subtract two digital audio files from each other. I've met Bill Waslo, the creator, and he's a really good guy. While you can't use it to ultimately evaluate CODECs in an objective way, DiffMaker does provide an interesting perspective on what gets removed. In that regard, it can be another subjective tool to help with the evaluation.

Originally Posted by esanthosh

^^ I am not worried about the 'last 2-5%' that people seem to spend loads of money on. But, many of the so called high-end (as in costly) products highlight better jitter correction as one of the features. How much does jitter affect the final output - as in comprehensible to an average listener like me?

My new blog post on jitter might help somewhat and there are some good links there to more info: http://nwavguy.blogspot.com/2011/02/...it-matter.html

[selectortone]

Using AudioDiffMaker to see what mp3s 'take out' is interesting only from a scientific perspective. It doesn't account for the psychoacoustics involved in the human brain decoding and adding-back-in what we ultimately perceive we are hearing.

[Adub]

Great review, a definite interesting read!

Also thumbs up on the blog, and I look forward to your future posts.

[meetooman]

Originally Posted by skip252

Instead of wondering you could do a little research and let us know what you find. You could start with some of the videos on Youtube and an interesting thread over at Hydrogenaudio on comparing the difference between the original lossless file and a lossy encode.

Should I look for answers from our member elsewhere? I already know this substraction is no fit to my question, so I politely asked our fellow member if he has some use for the procedure.
From what I see (your links are good examples), people always only care how badly lossy should compare to lossless/original, but not about a way to measure what you get at one encoding level regarding quality and file size. I am not interested by the lossless/lossy/ABX wars people seem to like so much. I saw how Enigmatic can jump easily on this. It is frightening...
My question is if there are ways to show what makes a 192kbit/sec mp3 better than a 128kbit/sec one. How ABX will tell me how much stereo is shrinking when I go from Vorbis Q5 to Q3?

Originally Posted by saratoga

They cannot. Lossy encoders work by introducing noise at frequencies you can't hear due to masking. Attempting to simply measure the noise will give you ridiculous numbers since it will not take into account masking.

Because a mp3 stream is not supposed to be original signal carbon-copied, but the "best of it" musically, comparisons can be unfair. That's why I wonder if instead of focusing on wave to mp3 problems and artifacts, one could look for what the encoding actually delivers.

Originally Posted by saratoga

Doesn't that tool just subtract two waveforms? Thats not ABX testing, and its not really useful for lossy audio.

It looks more like a tool to monitor "sound leaks". I said in my previous post that substraction software could please ABX users, because it gives something to listen to, not numbers. But it is very difficult to use humor here.

[dfkt]

Originally Posted by meetooman

But it is very difficult to use humor here.

I fail to see 'humor' in any of your posts - looks more passive-aggressive to me, in general. Might be better if you try to contribute to threads in other ways, since it doesn't exactly lighten the mood when you specifically have to indicate an attempt at humor where there is none.

Originally Posted by meetooman

My question is if there are ways to show what makes a 192kbit/sec mp3 better than a 128kbit/sec one. How ABX will tell me how much stereo is shrinking when I go from Vorbis Q5 to Q3?

That seems like a rhetorical question to me, or maybe I'm not understanding it correctly. ABX will tell you what you hear. With some material you might not notice any difference between 128/192 or Q5/Q3, with other material you might hear a difference.

[Enigmatic]

Originally Posted by meetooman

From what I see (your links are good examples), people always only care how badly lossy should compare to lossless/original, but not about a way to measure what you get at one encoding level regarding quality and file size.

We have explained it to you. I even gave you links—e.g.:

http://en.wikipedia.org/wiki/Transpa...compression%29:

The ABX method is normally used, with a null hypothesis that the samples tested are the same and with an alternative hypothesis that the samples are in fact different.

I am not interested by the lossless/lossy/ABX wars people seem to like so much.

Why? It answers your question.

My question is if there are ways to show what makes a 192kbit/sec mp3 better than a 128kbit/sec one.

Answered by me, by that transparency link, by NwAvGuy, by that hydrogenaudio link, by saratoga, and by dfkt—ABX. The explanation is also in my signature.

How ABX will tell me how much stereo is shrinking when I go from Vorbis Q5 to Q3?

Then why not ABX and report back to us?

Because a mp3 stream is not supposed to be original signal carbon-copied, but the "best of it" musically, comparisons can be unfair.

Why?

That's why I wonder if instead of focusing on wave to mp3 problems and artifacts, one could look for what the encoding actually delivers.

It has been explained to you, over and over again, if you want to look at what the encoding actually delivers, you should ABX.

It looks more like a tool to monitor "sound leaks".

I see it as very useful for testing whether things like green ink, stones, cable risers, air bladders, special racks, special knobs, plugging in clocks, etc., make a difference to the music or test signal.

I said in my previous post that substraction software could please ABX users, because it gives something to listen to, not numbers.

With ABX, there is something to listen to.

[skip252]

@meetoman: You posted a question. This is an open forum. I supplied a link that would give you more information that what you seem to have. After your response you can be damn sure that won't happen again.

[saratoga]

Originally Posted by meetooman

My question is if there are ways to show what makes a 192kbit/sec mp3 better than a 128kbit/sec one. How ABX will tell me how much stereo is shrinking when I go from Vorbis Q5 to Q3?

It sounds to me like you're not really understanding what an ABX test does. Its a way of determining if two files sound different. It doesn't quantify how different two files (or devices,etc) sound, only if they sound different at all.

The goal of ABX testing is to determine if a lossy file is encoded such that it is transparent. Generally people don't care too much about how far something is from transparent. If its not transparent, then something went wrong, and they reencode or whatever.

If you want to try and compare files you know will not be transparent (e.g. very low bitrate files), you can try tests like an abc/hr. In this test its assume that the files are of such poor quality that none are transparent, so the test merely asks you to rate how annoying each file sounds compared to a reference. Generally these only work at low bitrates though, since by the time you get up to 128k theres no longer differences for you to rate.

Originally Posted by meetooman

Because a mp3 stream is not supposed to be original signal carbon-copied, but the "best of it" musically, comparisons can be unfair. That's why I wonder if instead of focusing on wave to mp3 problems and artifacts, one could look for what the encoding actually delivers.

Theres no way to do this.

Originally Posted by meetooman

It looks more like a tool to monitor "sound leaks". I said in my previous post that substraction software could please ABX users, because it gives something to listen to, not numbers.

Its unclear to me what you're trying to say here, but that program is not useful for ABX, and theres no use in listening to anything it generates. Its really not useful at all in this context.

Originally Posted by meetooman

But it is very difficult to use humor here.

This is your "humor":

Originally Posted by meetooman

AudioDiffMaker seems to be a very interesting tool for ABX test fans. Do you think we could compute an "deviation" value out of a "difference" system?

Theres basically no way you meant that as a joke, so please don't bullshit me like this when I'm trying to answer your questions.

[NwAvGuy]

Originally Posted by saratoga

If you want to try and compare files you know will not be transparent (e.g. very low bitrate files), you can try tests like an abc/hr. In this test its assume that the files are of such poor quality that none are transparent, so the test merely asks you to rate how annoying each file sounds compared to a reference. Generally these only work at low bitrates though, since by the time you get up to 128k theres no longer differences for you to rate.

I second your comments Saratoga and it's worth pointing out that sometimes in blind tests of CODECs you don't get the obvious result. A professor of music at Stanford has used a fairly high quality system to play tracks encoded, or not, in various ways for groups of his students. To his surprise, many students preferred the tracks with obvious MP3 compression artifacts over the uncompressed versions. These were not rigorous ABX tests, but still an interesting survey of subjective preferences.

And, it's important to note, this isn't some random sample of all Stanford students, these are music students who presumably mostly have a stronger than normal interest in music and may well play musical instruments.

Their preference for the "inferior" tracks has been explained by many as simply being what they're most familiar with. Most of them have been listening to mostly MP3 for many years. And a lot of those MP3s were downloaded from the web in whatever format they happened to be in--often 128K. So, to many of the students, that's what music sounds like and hence that's their preference.

Here's a link to one of many references: http://radar.oreilly.com/2009/03/the...-of-music.html

[saratoga]

Originally Posted by NwAvGuy

I second your comments Saratoga and it's worth pointing out that sometimes in blind tests of CODECs you don't get the obvious result. A professor of music at Stanford has used a fairly high quality system to play tracks encoded, or not, in various ways for groups of his students. To his surprise, many students preferred the tracks with obvious MP3 compression artifacts over the uncompressed versions. These were not rigorous ABX tests, but still an interesting survey of subjective preferences.

So he did a test without a reference? Why is that interesting? A test without a known reference isn't expected to give you any particular result. Thats why ABX and ABC/HR are generally used for these things.

Originally Posted by NwAvGuy

Their preference for the "inferior" tracks has been explained by many as simply being what they're most familiar with. Most of them have been listening to mostly MP3 for many years. And a lot of those MP3s were downloaded from the web in whatever format they happened to be in--often 128K. So, to many of the students, that's what music sounds like and hence that's their preference.

Here's a link to one of many references: http://radar.oreilly.com/2009/03/the...-of-music.html

I'd explain it as "ask a stupid question, get a stupid answer"

[dfkt]

That must've been some pretty shitty MP3s they tested with, if the perceived difference was that big for a whole group of people, no matter if better or worse. Especially played over the air (sans headphones), a standard 128k encode usually isn't that easy to identify. Well, some people still use Xing...

Either way, so far this thread has deviated from its original topic (Clip/iPod hardware test) to the perception of *proper* lossy encodes compared to the uncompressed/lossless original. Throwing some Victrola-quality encodings at a bunch of students is a whole other story than LAME -v0 or aoTuV -q6.

[NwAvGuy]

There's been considerable debate over the actual facts of the test with the students. Somewhere I watched a video interview of the professor and it seemed some of what was written about was in error. But yeah, enough about CODECs.

Closer to being on-topic, I got my Fuze yesterday and have indeed confirmed it measures virtually identically to the Clip+ (as I expected it would). My comments on the Fuze are on my blog. It might be a better option for someone considering a Clip+ depending on their needs. And, with it being discontinued and generally favored heavily over the new Fuze+, stores will no doubt run out of them.

[dfkt]

This is the first time I heard the Fuze is 'favored heavily' over the Clip+. Sounds more like personal preference than general consensus to me.

Sure, it has a bigger color screen and a slightly better battery life, but the imprecise wheel and proprietary port are annoying. Build quality feels somewhat cheaper than the Clip+ (also, no comparison to the e200v2/3/4 with the metal back), and it has no... clip.

EDIT: I think you meant Fuze+ instead of Clip+ - then it all makes sense. The Fuze+ also doesn't use the same AMS SoC as the other current Sansas.

[NwAvGuy]

Originally Posted by dfkt

This is the first time I heard the Fuze is 'favored heavily' over the Clip+. Sounds more like personal preference than general consensus to me.

Sure, it has a bigger color screen and a slightly better battery life, but the imprecise wheel and proprietary port are annoying. Build quality feels somewhat cheaper than the Clip+ (also, no comparison to the e200v2/3/4 with the metal back), and it has no... clip.

EDIT: I think you meant Fuze+ instead of Clip+ - then it all makes sense. The Fuze+ also doesn't use the same AMS SoC as the other current Sansas.

Doh! Thanks for catching that. I fixed it.

I personally think the Fuze has better build quality than the Clip+, but I agree on the other points. The wheel is a mixed feature (although it's better than the really cheesy one on the e200). I like using it to navigate through the long menus and music library. But it's also rather easy to change the volume when you don't want to. And you're right, it can be imprecise.

[meetooman]

Originally Posted by saratoga

It sounds to me like you're not really understanding what an ABX test does.
...
Generally people don't care too much about how far something is from transparent. If its not transparent, then something went wrong, and they reencode or whatever.

ABX is very easy to understand. What's scary is that apparently that I have no right to be interested in something else than transparency. I will see what ABC/HR says, thank you for giving me this lead.

AudioDiffMaker seems to be a very interesting tool for ABX test fans. Do you think we could compute an "deviation" value out of a "difference" system?

I am also surprised how limited the scope of humor is here. I am talking about humor as the ability to apprehend poetic absurdity in life. Humor that is not in my words but in the world around me.
Seeing the "audio diff" on the youtube video, I have found that the so-called mp3 distortion is quite pleasing to hear (that's funny). It sounds like a "forgotten echo" (that's poetic). Now if you imagine you can create 2 different streams from wave-mp3/128k and wave-mp3/192k. Comparing them via ABX may be insane (absurdity). But am I that sure it would be completely meaningless without trying? Having humor is a way to be open minded, to welcome the strangest ideas with a smile and say why not? The same about a number to get, what stat to calculate from 10 seconds of music diff? It is funny is to have a software and not knowing what to do with it, it is funny to have more questions than an answer.

NwAvGuy kindly introduces us to AudioDiffMaker. I don't want to say it's bad just because I don't know what to do with it. So I ask him what he think he can do. The idea that a single value could sum up what happens inside a musical extract can also be a silly/funny thing (humor needed again). But this way I give him the opportunity to expand on the subject. I can't see what's wrong here. You all can say this software is useless, but I will say how it leaves me helpless without closing doors.
Humor is also a way not to take yourself too seriously, so you can leave openness in conversation because you don't pretend to have the truth. You don't use emoticons for that.