I'm having the very same issue here. When I set the audio device in VLC to "A/52 over S/PDIF" all I get out of my speakers is some stuttering sound. If I play the very same mkv file in Windows Media Player I get proper 5.1 DD sound (using AC3filter S/PDIF passthrough). The stuttering sound...