Handbrake H264 encoding does work for J3
I've read many posts saying that Handbrake created h264 videos won't work on the J3. However, I just created some using the 'ipod legacy' profile (Handbrake v0.9.5), h264 in an mp4 container with AAC audio, and it works just fine (J3 w/ firmware 2.26). I tried both ripping right from DVD (via DVD Decrypter) and transcoding existing .avi files, and no problems with either.
The only changes I made to the 'ipod legacy' profile were to drop resolution to 480x272, drop bitrate to 768 (though I'm going to try 640 or maybe 512 next, as 768 is a bit too large for such low resolution), save as mp4 (not m4v), unclick 'ipod 5g support', and only used one audio track downmixed to stereo AAC 160kbps. Super easy.
I hope this helps