Thanks for the answer!
Now I understand why DLLs from different builds can't be mixed. By the way, in particular, it can be because cygwin appends .exe extension to all the DLLs. Though MAKEFILEs take care of it by removing it, it does not solve the problem with this odd extensions. In particular, if I build VCL with cygwin to use LIBVLC.DLL, the vlc.exe tries to load libvlc.dll.exe. I had to rename libvlc.dll back to libvlc.dll.exe to make it working.
Regarding "Show interface with mouse", this is a bug in VLC. First I thought the different builds work different, then I realized that it is a crop-related bug. My build worked a different way due to my modifications to the "crop" module (see
here for details). Then I've found out that y offset of the "crop-geometry" option affects "mouse-y" internal variable value: it becomes non-zero when the mouse cursor at the top of the screen. Now I see that that the "crop" option (even in the current version, e.g. if I select 16:10) affects "mouse-y" the same way, and also prevents "Show interface with mouse" to work as it should.