I'm having the very same issue here.
When I set the audio device in VLC to "A/52 over S/PDIF" all I get out of my speakers is some stuttering sound. If I play the very same mkv file in Windows Media Player I get proper 5.1 DD sound (using AC3filter S/PDIF passthrough).
The stuttering sounds like the receiver decodes half a second of audio, than repeats that piece for three seconds than decodes the next half second. So you can identify the stuttering as being the sound that is supposed to play.
When I switch the output device back to Stereo the receiver will switch back to PLII Music. So the receiver is getting the format switch from DolbyDigital to PCM Stereo and vice versa. It's just not decoding DD5.1 properly.
I tried several audio output modules but ended up reverting it back to default since none of them seems to have a healing effect. Though I can say for sure that when audio output is set to Win32 waveOut instead of default, the stuttering pieces are about twice as long as before. So instead of half a second I maybe get one second of decoded audio that than repeats itself for another three seconds.
The Soundcard is an onboard Realtek Card on a ASUS P5WD2 Premium, which also supports DolbyDigital Live. So of course if I switch the Digital Out default Format from 48kHz Stereo PCM to Dolby Digital Live 5.1 in the Windows Control Panel and set the audio device in VLC to 5.1 instead of A/52 over S/PDIF I will get 5.1 sourround sound out of my speakers. But this is a less than optimal solution as VLC is decoding the AC3 stream to 5.1, hands it over to the system which than does DDL5.1 encoding to the 5.1 channels and sends it over to the receiver who again decodes the channels. That's a bit like mp3->wav->mp3->wav de/encoding, and we all know nothing good can come out of this quality wise
Since DolbyDigital and DTS passthrough works in a variety of apps (AC3filter - thus WMP, PowerDVD, WInDVD) on my system I'm blaming VLC right now for the broken Sound.
The System is running VLC 0.8.6c on Vista Ultimate x64 on the MB mentioned above. The receiver is a Onkyo TX-SR505E supporting Dolby Digital, DTS and PCM of course at sample rates of 44.1, 48 and 96 kHz (these are also the Options that I have activated in the Supported Formats tab of the Digital Output Properties in Windows Control Panel). My default output format is 48kHz PCM Stereo.