Actually, the batteries in the A840 and A850 series are really good, when you consider their size. You have to remember that those players are only 7 mm thick, and the battery has to be thinner than that. You know of any other battery that can be that small and still give more then a few hours of power?
Sony's real fault with the A840 and A850 Walkmans was the desire to make them super thin. They could have added a few more millimeters and got a bigger battery in there, and that would have increased the battery life a lot. Look at the S540 series. They're almost the same size, but almost twice as thick. The battery life is about 45 to 50 hours. Same with the E-series.
Besides, most people don't know this, but Sony owns multiple chemical companies and some of them are doing battery research. If Sony, with its various resources and divisions that are only working on battery technology, can't make a better battery, it's probably because they've reached the limits of what they can do with current science.