The H320 and H340 cases are a bit different . The H320 disks are slimmer so the case is also slimmer (about 3mm I think), so while you can't fit the 40GB hard disk into a H320 you ought not to have any problems going the other way. I don't know if there are any differences aside from the disk dimensions but the best place to look will be the Rockbox wiki because it has an awful lot of information on these devices, including data sheets and images of the components disassembled: http://www.rockbox.org/wiki/IriverPort
From what I can tell you shouldn't have much trouble at all but you need to do your own research and make up your own mind.
As an alternative to doing a transplant it may be worth considering a H320 with an SSD. The SSDs are exactly the same dimensions as the 20GB hard disks so you can put a high capacity SSD in a H320 and it fits perfectly. You could use a 128GB Kingspec in for example (I have one in my H140). The caveat is that the iRiver firmware only works with hard disks so you would need to run Rockbox only.