Faster play speed causes stutter/full video playback stop

[dabp]

Alright, so a bit of a weird one:

5950X, 32GB DDR4, 3080ti, 2TB 980 Pro NVME.
Windows 10 Pro 19045.4291, VLC 3.0.20
I have weeks worth of video fotage I need to scrub through, so I loaded all the video files onto my PC, seperated days into folders and loaded all files from a day into VLC playlist and started watching x64 speed. Worked great, video playback was smooth until I hit 4:40 in the morning and the camera switches from black/white night vision to daytime color. As soon as it does that, video stops. I can go back and it'll start playback again, still playing x64 and it runs smoothly. If I try to skip forward past that "daylight" point, it simply won't play x64 speed. If I set it down to 1-2-4-8x it will, if I go to x16 or above video stutters, lags, bounces and is unwatchable.
I tried multiple days, the same is true for all tested. If I go into regular daytime videos, it's the same, going to x16 or above is just unwatchable.

I tried disabling hardware acceleration, I tried the direct3d and directx options under hardware accelerated decoding as well, tried enabling fast seek, all to no avail.

VLC never crashes, nor is there any indication that the hardware is lagging as it's using very little ressources on this.

I tried dragging the file where it switches between nightime and daytime into my wondershare filmora video editor and changing speed to 50x and that works just fine for both night and daytime, no stutter or failures in playback

[Rémi Denis-Courmont]

You may need to disable hardware decoding. No matter how great your GPU, the video DSP is not going to scale infinitely.

[dabp]

You may need to disable hardware decoding. No matter how great your GPU, the video DSP is not going to scale infinitely.

Hey, yeah, already tried disabling hardware acceleration. Do you mean something else, or?