I had not tried but now I have. I have tried all output modes and no diffrence is noticed.
A friend also told me about the GOM player for windows. I have tried that and it plays the MKV files in 5.1 via s/pdif without any issue.
I would still however like to solve this VLC issue.