This problem has reappeared for me. I'm doing something non-standard which might explain it. My app dynamically opens libvlc.dylib from whatever release of VLC is installed on the Mac. The 3.x releases seemingly do not respect "--vout=caopengllayer". Everything is fine with 2.2.8.