I use Delphi, so I cannot give you any C++ examples.
You have to use the SetWindowsHookEx API function to declare the hook. As you need a global mouse hook (else you won't get the mouse messages before the vlc active x) you have to use this function and the hook callback function in a dll and not in your exe. It's in this function that you'll treat the mouse messages.
The simple way to do it is to forward every "interesting" mouse message to your application, and to check if you're over the vlc active x client area or not so you can decide if you forward the mouse message to it's destination window or not.
This is a very "basic" explanation of how i did it, but i think you can find a lot of C++ mouse hook examples on the web with detailed explanations, google is your friend