I had similar problems with my G5 1.8 SP when I was using the interal speaker, and also when I used the G5's stereo minijack to connect it to my stereo power amp (Adcom GFA-555). Most files play fine, but some default to the optical digital output, and I had a hard time changing the output to analog. I discovered the same solution posted here on my own.
However, recently I added an M-Audio FireWire410, with the intention of adding a MIDI keyboard controller later. So now the G5 is outputting via FireWire to the 410, which is connected to my power amp via two 1/4"-to-RCA cables. Now I get no sound whatsoever. Other apps, such as iTunes and QuickTime Player, play sound just fine.
Is there anything I can do about it, or is VLC currently completely incapable of digital audio output?