Recently FFMPEG have updated their h264 codec to allow 10bit depth h264 to be played back when using the x264 10 bit encoder.
MPC-HC plays the 10bit depth MKV ok but with VLC v1.1.11 all I see is a green screen.
----
There might be a problem with the 10bit depth mode as the colours didn't look correct - red looks a bit pink. I'm not sure if the problem is with x264 10bit encoder or with the FFMPEG 10bit decoder. If you do build 10bit support into VLC just do a couple of test encodes (with x264 10bit depth encoder) and check the colour when playing back with your future 10bit h264 enabled VLC.