I just got an M3 Max MacBookPro 16" I run it in Lid Closed with it connected to my 3 Dell 27" displays (2560x1440). Center display is connected to the MBP HDMI port. Left and Right displays are connected via Display Port cables via an OWC Thunderbolt Dual DisplayPort Adapter. I have also tried connecting them individually to the machine as well as making one of them HDMI as well. I am running VLC 3.0.21 and it shows as Apple Silicon under its kind. My machine is a 16-core with 40-core gpu. As powerful as it gets for a MBP right now.
When I open a 2160p mkv (downloaded from YouTube) or other 2160p .mp4 fileson one of the side displays, the playback, at the default -filled screen- is horribly stuttering. Maybe 1 in 300 frames plays back. If I slide the playback window onto my center display, it plays better but still stutters a fair amount. I have checked that the window is not spilling over onto the adjacent screen. The only way to have smooth playback on either side displays or the center is to use the VLC command to half size the video. Then it plays perfectly on whichever screen I have it on.
My previous machine was a 2013 MacPro (trashcan) with the D500 video card driving these same displays, with zero issue. I generally playback movies on a side display while I work off my main display.
The files in question are on the 1TB internal SSD from Apple.
Seems like something is unhappy between Apple Silicon and the scaling code.