Linkage of Plugins in MinGW-LLVM <->MSVC C API/ABI

[bjornss]

Hi,

I have compiled VLC for windows - along the recommended route, using the MINGW-LLVM compiler framework, with matching precompiled contribs for LLVM compiler.
I have made an extra in-tree module in VLC. I am trying to get the module to invoke functionality from a MSVC-based PE+ (EXECUTABLE/DLL)

I previously linked this module DIRECTLY to a MSVC C++ DLL, and found, that I can access pointers to public class instances and datamembers of the MSVC C++ code from INSIDE the MINGW-based VLC module.
I could however NOT call member functions declared/compiled inside MSVC C++ from INSIDE the VLC MINGW-LLVM C++ module due to ABI incompatibilities (name mangling) between the MINGW and MSVC compiler architectures.

Guided by other's experiences of successful mixing of MINGW and MSVC in C++ applications( e.g. https://www.dynamsoft.com/codepool/ming ... c-dll.html, https://nullprogram.com/blog/2021/05/31/), I decided to pursue an approach where all communication between the MINGW-LLVM exclusively C-based VLC module and MSVC C++ code, was routed through a linklayer. This linklayer DLL is compiled/linked in MSVC with appropriate _declspec(dllexports) for the functions to be called from MinGW C

The functions used / called by the VLC module inside the linklayer are therefore entirely C API/ABI (defined inside extern "C" blocks) but the entire layer is not in C, as those C API functions invokes MSVC C++ functions/methods from another source.
The linklayer library is ONLY IMPORTING some functions in (MSVC) KERNEL32.DLL, and NOTHING else.

After the addition of the linklayer I am having a problem listing/loading the module. In VLC I am getting "The module was not found. (error 126)"
Removing the CALLS to the C API functions from inside the module, I can immediately again load/use the same module.

1. What if anything, can I do to increase diagnostic output in libvlc for this issue ("--verbose 2") does not seem to cut it for my problem,( Should I try --log-verbose 3 and scan file)?

In building the Win32 C API linklayer in MSVC and linking the produced DLL with the VLC module, this all seemingly succeeded (There was no warning on dynamic linking, I see it IS IMPORTED by the MINGW compiled / linked DLL module) but the linkage is dynamic. I tried to put the MSVC-based DLL of the linklayer in both the same dir as the module, and in the top dir of main (libvlc.dll) DLLS, but the error still persists, the module refused to load.
This is even the case after adding the -rpath variable to the LDFLAGS, as far as I can discern.

2. Based on earlier cited URLs, it is claimed that there should be no difficulty for the MINGW-LLVM linker to recognize and use a MSVC DLL (as long as only C ABI is used in functions). Is this the case for VLC, or should I provide a MINGW stub (*.a) library produced by gendef / MINGW-LLVM-libtool and link the VLC module to this stub?

3. Could there be a possible issue in using the MSVC/Win native KERNEL32.DLL and libkernel32.a bundled with MINGW-LLVM? (Based on my earlier, partially successful trials with direct linking, I think not)

3. Based on the answer to 2:
a) What libtool config / linker flags should be employed to combine the MSVC-based DLL, with the MINGW-LLVM compiled VLC module?
b) Should a linklayer DLL have its OWN libtool entry / configuration?

4. If everything else fails, do you recommend the LoadLibrary/GetProcAddress scheme for fetching functionality from MSVC DLLs?

PS: I note in ending, that the libplacebo_vk_plugin seems to have similar issues to my module, with the same error code (error 126) - this while having linkage to what looks like a MINGW *.a library stub probably converted from other compiler arch (MSVC?)

A long post, yes, but I hope someone can help clarify these issues, as they may be important also for other developers too.

Sincerely Yours,
Bjorn

[bjornss]

I have found the cause and mitigation to my issue.

These 2 libraries mentioned below (my new plugin module), and the libplacebo_vk_plugin are the ONLY 2 Plugins in my VLC build that invoke Windows - vlc_dlopen function from the vlc\src\win32\plugin.c:53 file.
This again invokes LoadLibraryExtW with the LOAD_LIBRARY_SEARCH_SYSTEM32 flag, so by placing my link-library in the System32 dir, I "fixed" the issue.

I believe however that this FLAG is to restrictive, and should be augmented with LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR, unless the latter would cause conflict with other WIN32 linked modules.
Maybe this is a point for your consideration?

Regards
Bjorn