I can confirm the issue. Mojave 10.4.4 and VLC 3.0.6.
VLC autoresizes to fit the screen (ignoring aspect ratio) whenever I press "Playback", "Subtitles" or "Help" in the menu (everything is fine with "File", "Edit", "Audio", etc.) or, for instance, open "About VLC Media Player" window.
===
does --no-macosx-video-autoresize or unchecking it in settings work for you ?
Tried restarting, resetting the settings, reinstalling the app, installing older (2.*) version.
Tried launching VLC with
--no-macosx-video-autoresize. Did not help.
I didn't find the checkbox in the settings though. Can you please point me in the right direction where to find it?