I also had the same problems as all the writers before me with VLC-Player and DTS over S/PDIF.
No problems with AC3- and DTS-playback over S/PDIF within movies.
No problems with AC3-playback over S/PDIF with solo AC3-audiotracks.
Problems with DTS-Playback over S/PDIF with solo DTS audio streams, the "A/52 over S/PDIF" option disappeared!
Sadly there is no real solution for this: It's not a problem with the VLC-Player, it's a problem with the sound card and/or receiver! They cannot send/decode 44khz DTS streams, only 48khz is possible!
Just try to play DTS-WAV's which are encoded with 48khz, no problem with VLC-Player. On the following page there are different audio samples in AC3/DTS format encoded with 44khz and 48khz:
http://www.kellyindustries.com/sounds.html
For those systems only solution for playing 44khz DTS streams over S/PDIF is to decode and resample it to 48khz DTS.