It seems this is possible if I write an audio filter for VLC.
This guy was/is looking for the same thing:
http://mailman.videolan.org/pipermail/v ... 34094.html
Someone who apparently tried to write an audio filter without success:
viewtopic.php?f=2&t=50412
The problem with an audio filter is that VLC still will do the actual playback. Ideally I would like to just call some function decode_frame(byte[] data) or have a callback that gave me the decoded frames as soon as they were available and not have VLC do the playback. I could then do the playback myself in JavaSound.