Return from pause lags

[twbot]

VLC 3.0.11

If I pause a video, even for a moment, it can take several seconds (varies from 2 to 5 seconds) for the video to start playing again when I hit play - tried this on two recent macs - a 2018 3.6 GHz i3 w/Intel Graphics 630 MacMini and a 2020 3.8GHz 8-core i7 iMac with Radeon Pro 5500 XT. Both have SSDs and 32gb of ram, both are running OSX Catalina.

Doesn't happen in Elmedia Pro or Quicktime Player.