I am experiencing a similar problem with 5.1 and 7.1 surround sound using VLC 1.1.12 on Mac OSX 10.7.2 running on a Mac Mini (Mid 2011, M816LL/A) hooked up to a Pioneer SC-1525-K receiver via HDMI.
I have gone through the Audio MIDI Setup tool in OSX and configured all the speakers so the channels come out of the correct speakers using the utility. Before I did this, I clicked the "Apply" button a few times with incorrect channel assignments before finally clicking on the "Apply" button with the correct channel assignments and verifying each speaker.
I then started up a fresh instance of VLC and loaded some files from
http://www.demo-world.eu/trailers/high- ... ailers.php to test the setup. In particular the "Dolby TrueHD Channel Check (Lossless)" and the "DTS HD-Master Audio Sound Check 7.1 (Lossless)" files do not play correctly. VLC detects the files as 3F2R/LFE. That may be expected if VLC does not support 7.1 audio although I thought it was supposed to support Dolby TrueHD. Also, XBMC detects the DTS-HD MA file as 5.1 so most likely VLC is only detecting the core 5.1 stream like XBMC since there is no open source DTS-HD MA decoder out there yet.
When both of these files are played back, VLC plays the sounds out of the wrong speakers (mostly). For example, the Front Left channel comes out of the Left Rear Surround speaker, the Center channel comes out of the Left Side Surround speaker, the Front Right channel comes out of the Right Rear Surround speaker, and the Right Side Surround channel does not play back at all. XBMC is able to play the channels out of the correct speakers but it does not handle the Left/Right Rear Surround channels correctly (both the Side and Rear Surround channels come out of the appropriate Side Surround speakers only). I was testing VLC to see if it handled things any better when I noticed this issue.
I also played the THX -> Terminator 2 video (Dolby Digital 5.1) from the Demo-World site as well and it came out of the rear surround speakers instead of the front speakers. It is hard to say if the channel mapping is the same as the other files but it didn't sound right compared to playback using other apps that have the correct mapping.
Is this a VLC bug or can anyone suggest how to fix this without changing the channel assignment using the Audio MIDI Setup tool so that the sounds don't play out of the correct speakers but fix the mixed up mapping that VLC is using? I don't think it makes sense to change the mapping in Audio MIDI Setup just to fix VLC's incorrect mapping. It might mess up other apps that use the info as well that map the channels correctly.
BTW, I also tried running VLC 1.1.12 using Bootcamp in Windows 7 Home Premium. I configured the speakers in Win7 appropriately in the Control Panel sound settings and the same video files above played better than on OSX. Both the DTS-HD MA and the Dolby TrueHD 7.1 file had the side surround channels come out of the rear surround speakers instead of the side surround speakers. No speakers output sound while the rear surround channels were active. In order to get this far in VLC on Win7, I had to switch the output method to DirectX using AMD HDMI Output. Also, after playback of the video file started, I had to select the 5.1 audio device in the menu or else I only had the front speakers output sound. It looks like the OSX version of VLC 1.1.12 may be a little bit behind the Win7 version.