I think the problem here is, in order to help you, people need to know how to replicate the problem. Those two "one instance" options have always worked for me.
Assuming after you close the VLC and reopen it, that the one instance options still remain checked, what I would look into is whether there is something messing with VLC.
One tool I use is the free Process Explorer from Microsoft.
https://docs.microsoft.com/en-us/sysint ... s-explorer
It is a more sophisticated task manager. What I would do is start VLC, have it play a video and then start process-explorer. From the View menu, I would check "show lower pane" and for the lower pane view, then select dlls.
And then I would highlite VLC and review in the lower pane what dll files VLC is loading. In my case, since it is running on my Nvidia card, I see a Nvidia file, a bunch of VLC files, and Microsoft file, and nothing else. One file I have that doesn't belong there is Microsoft's EMET.dll. However, this file plays nice on VLC and provides some security in the event VLC tries to play malicious input. But if I was having problems, that would be the first file to go.
In particular I'm wondering if you have some kind of anti-virus that has a hook into VLC?