Hi there,
exactly the same problem occurs for me. I cannot choose the 5.1 Device when using PulseAudio (no matter if another application is using sound or not). With ALSA as the output module it will run fine. Totem can handle 5.1 Output and PulseAudio. Is this a pulseaudio problem or a vlc problem?