Same Issue here. Grey pixeled scenes by playing .avi files, but MKV have no problem.
Latest VLC 2.2.5.1 64bit on Mac Pro 5.1 with GTX680.
Fresh install of latest Sierra 10.12.5, standard settings, nothing helps.
EDIT: Setting Cache to 6000 ms seems to fix it by now.