Reset the preferences, as this is the default or in preferences, Video, Output modules, DirectX and in Interface, Main interfaces, wxWidgets set to non embedded mode.
You can check this with a modified shortcut that has no options after VLC.EXE
As far as doing this from within another application, it really shouldn't be necessary as it is very unusual to want to shut this off. The software counterpart is slower and really only has use for testing as most video cards do support this. At lease any thing reasonably modern (last 7 or so years). Please note: For this to work with all cards, in many cases the video overlay must also be shut off (video acceleration). So the condition now becomes more complex.