Actually, the problem is with the Mac. I've hit this problem before, and resolved it, but couldn't remember exactly how. But when the absolute latest VLC 2.1.4 sent audio to the wrong channels on my new MacPro, I went digging again. First I found this and some other threads on videolan.org, but I don't think the problem has been diagnosed correctly. Fortunately, I also found the solution. I don't know why Macs ship this way or why some software still seems to work, but the problem is in the Mac's audio setup, and once you know how, it is quite easy to fix. See the instructions in the top post here:
http://forums.macrumors.com/showthread.php?t=1314744. Particularly the "Instructions" section. In fact, in case that ever goes away, here are the most salient bits excerpted:
INSTRUCTIONS
The key to getting dialog to play (indeed, all channels of the source file to play via the correct speakers in your surround system) correctly, i.e., as it was meant to be heard, is to configure the speaker map on your Mac properly. Follow these instructions, as tested on Mac OS X 10.7.2 (earlier versions may be slightly different):
1. You need to enable AC3/DTS passthrough. Go to Applications>Utilities>Terminal. When the application opens, copy and paste the following, without the quotes, then press enter: “defaults write com.cod3r.a52codec attemptPassthrough 1” and “defaults write org.perian.Perian attemptDTSPassthrough 1”. q.v. FOOTNOTE 1
2. Connect the HDMI cable from your AVR to your Mac. Make sure you select the proper mode on your AVR so your computer is recognized. Make sure your audio output device in OS X is changed to, e.g., "DENON-AVAMP" (other AVRs will show something else) from the default "Internal Speakers", otherwise, your audio will play through your internal computer speakers. You can quickly switch the audio output device by clicking on the sound icon in the top menu bar, whilst holding option on the keyboard. If you don't see the sound/volume icon in your menu bar, go to the apple icon in the top left, click System Preferences>Sound. Then click the box for "Show volume in menu bar". This will make your life easier.
3. Go to Applications>Utilities>Audio MIDI Setup
4. On the left side of the "Audio Devices" window that appears, click "HDMI".
5. Change Format to 48000.0 Hz and in the dropdown to the right, select “8ch-24bit Integer”. q.v. FOOTNOTE 2
6. Now click “Configure Speakers”. In the top right, select the surround system you have. I have 5 speakers (no subwoofer), but I selected “5.1 Surround”. Click each speaker name to make sure sound comes out of the speaker name you just clicked. Adjust as necessary.
Perhaps because I had set all this up on an older Mac from which I imported my settings (using Migration Assistant), all but the last step (6) were already in place on my system. (Note, you can see the state of the defaults you are told to change in step 1 by typing "defaults read com.cod3r.a52codec attemptPassthrough” and “defaults read org.perian.Perian attemptDTSPassthrough” if you want to check them before you change them.) Once I selected "Multichannel" and "5.1 Surround" in step 6, all was well, and VLC works as intended.
Keep up the great work, VLC contributors, and don't waste time on this non-bug.