How to intercept keyboard and mouse events on Windows

[gnosygnus]

I am using java1.6 on Windows XP SP3.
Thanks to sherington and the excellent vlcj project, I have a basic video player application. However, I need to intercept keyboard and mouse events to trigger actions (custom menu, log to database, etc.).

The vlcj implementation passes a pointer to a heavyweight Canvas object to vlc. From what I know (and tried), it isn't possible to subscribe to key/mouse events to the Canvas object.

According to the comments, both libvlc_video_set_mouse_input and libvlc_video_set_key_input are only implemented for X11. (I'm not sure how they would be usable in a Win32 context). I've tried setting libvlc_video_set_mouse_input to 0 for curiosity's sake, and it swallows up the mouse cursor when it moves into the frame.

The only other approach is to create a transparent JDialog with the AWTUtilitiesWrapper and have it cover the entire video surface. Unfortunately, this has at least two issues:
- overlays need to be turned off (--no-overlays)
- JDialog still shows up in the Alt-Tab menu cycle

I'm wondering if anyone has come up with better approaches. If possible, I'd like it to be cross-platform (i.e: it needs to intercept key/mouse events on Linux/MacOS).

Also, if any libVLC developers are reading, are there any plans to expose keyboard/mouse input as events for libvlc_event_attach? Is it even possible?

Thanks in advance.

[sherington]

This works on Linux simply by adding listeners, but I I did a quick test of this on Vista and indeed the mouse events don't get sent once the video starts playing.

I even hooked into the AWT system event queue and the events simply never arrive.

I don't think there's anything you can do on the Java side to address this, on Windows at least.

Maybe you can try a JFrame for your application frame so you could set a transparent Canvas glasspane to grab the events. Even _if_ that worked, I think performance would be bad.

Hopefully someone else has a solution for you.

[Rémi Denis-Courmont]

LibVLC is a media library, not a windowing system. Handling input event through it would be just plain wrong.
First, it wouldn't integrate with the event loop of your application's GUI framework.
Second, VLC only knows a limited subset of input events - the rest would be lost.
Third, the current design of VLC internal key handling is quite broken at the moment.

Someone needs to put the missing code in the Windows video output plugins.

[erwan10]

Here is a summary of what I found on Window (Linux is a quite different story),

- Intercepting keyboard events should not be a problem.
By default, keys are dispatched to the top window being active, not the vlc video window which is only a child window. Since you're in charge of developping this top window, you should get key pressed.

- Intercepting mouse events is a lot more difficult
Mouse is delivered to the window on which the cursor lies. So the vlc video window is really the place where to get those events. Things are even made more complicated by the fact that this video window, though being a descendant of your top window, is not created in the same thread, and therefore doesn't share the same event loop.

For vlc skins2 (written in C++ with win32 api), the only way I found was to use hooks (as described here http://msdn.microsoft.com/en-us/library ... 85%29.aspx). Though you don't know the vlc video window, it can easily be retrieved by querying the OS for the list of child windows (the vlc video window is the only child window of the handler you pass to vlc). This is a bit hacky, but it does work.

Now, how that can be implemented in Java is another matter ...... (no experience) ?

Last point, libvlc_video_set_mouse_input should now work on Windows (from vlc1.1 on) to enable/disable mouse event handling at the vlc level. (unfortunately, unlike Linux, disabling it at vlc level doesn't mean it will be propagated to the ancestor windows)

Erwan10

[gnosygnus]

Thanks all for the helpful comments.

@sherington: I've tried a transparent GlassPane, but for some reason I couldn't get it working. I think that this has to do with mixing lightweight and heavyweight components. I discovered the AWTUtilities class here, http://java.sun.com/developer/technical ... index.html but as you guessed the performance is pretty wretched. (eats up 30% of my cpu: AMD 3800 2.0GHZ dual-core)

Btw: I just want to say thanks again for an excellent job with the vlcj project. It was tremendously helpful.

@Rémi Denis-Courmont: Yeah, sorry, I didn't think the request through. I guess I was approaching this from the perspective of an application developer who wants to builds a custom app around vlc. The libvlc library provides me 95% of the functionality -- except for the input event processing. I was naively hoping that last 5% would somehow be in there. Without the input processing, it just means there is a large hole in the middle of the screen that the developer cannot customize/interact with.

If I understand correctly, if this functionality makes it into vlc, it would only be by a Windows specific plugin?

@erwan10: I've tried keyboard listeners, but unfortunately they don't work. (something like videoSurface.addKeyListener(new KeyboardListener.....). I think this has to do with the java.awt.Canvas object, and it being a heavyweight component (I could be mixing this up with other details I've read about Canvas objects).

As for mouse events, I'd like to stay away from c++ win32 code. I have no experience with it and I'm afraid it is just beyond me. Also, being a java app, I'd rather not get into the business of specific win32 code. Still, I'll keep your idea in mind if I ever do need to go down that route.

Also, regarding your point about libvlc_video_set_mouse_input: I definitely do see different behavior when I pass in 1 vs 0 to void libvlc_video_set_mouse_input(libvlc_media_player_t p_mi, int on). When I pass in 1, nothing different happens. When I pass in 0, the mouse pointer disappears as soon as it enters the videoSurface area and it looks like there is no response to any "clicking". I'm guessing that it is as you said: that it is not propagating to any ancestor windows

Finally, I did a little more digging, and discovered that I can hide the java.awt.Window from the Alt-Tab cycle if I just subclass it directly (rather than use JDialog/JFrame/JWindow). This will allow me to give the appearance of custom input processing, and not having to deal with three Windows/Forms that a user could alt-tab into.

I'm going to look at the subclassedWindow + AWTUtilities route as an option. This approach is closest to what I'd want, with major reservations: it requires "no-overlays" and will be very performance-intensive.

I'll try to post a simple example later (probably by next weekend) if anyone else is interested....

[sherington]

Btw: I just want to say thanks again for an excellent job with the vlcj project. It was tremendously helpful.

Glad to hear it, thanks.

@erwan10: I've tried keyboard listeners, but unfortunately they don't work. (something like videoSurface.addKeyListener(new KeyboardListener.....). I think this has to do with the java.awt.Canvas object, and it being a heavyweight component (I could be mixing this up with other details I've read about Canvas objects).

Well, it is platform related. I can mix heavy and lightweight components just fine (just make sure they don't overlap) without any problems and get all the events on Linux.

I tested that the following works for me on Windows for getting key events, not ideal but it might be some use...

Code: Select all

Toolkit.getDefaultToolkit().addAWTEventListener(new AWTEventListener() { @Override public void eventDispatched(AWTEvent event) { if(event instanceof KeyEvent) { KeyEvent keyEvent = (KeyEvent)event; // ...do whatever } } }, AWTEvent.KEY_EVENT_MASK);

[Rémi Denis-Courmont]

From the LibVLC perspective, platform-specific code is unavoidable for video embedding. This stems from the fact that native window handles are required. I reckon it is annoying all the more for portable byte code and/or scripting languages, but you'll have to suck it up.

However, in my opinion, the platform-specific code is best put in the bindings, not in the application using them. So I'd consider it a vlcj bug if it cannot hide this.
I have hardly any clue about Java or Win32, and it might be impossible - but then there is no point in trying to solve it at the application level either.

[sherington]

However, in my opinion, the platform-specific code is best put in the bindings, not in the application using them. So I'd consider it a vlcj bug if it cannot hide this.
I have hardly any clue about Java or Win32, and it might be impossible - but then there is no point in trying to solve it at the application level either.

I think it can be done if the Java can set up a global message hook in Windows. This would require some more native Win32 calls and may require elevated privileges for all I know. I'm not a Win32 programmer so I don't know how to do this.

This page has something that might help the OP...

http://forums.sun.com/thread.jspa?threadID=5407317

[sherington]

OK, this turned out to be a bit interesting to do.

With the newest version of the vlcj bindings, mouse events (move, press, release) via a native message hook works for me when playing video on Vista.

Key events already worked without the hook.

http://code.google.com/p/vlcj/downloads/list

Please see the new "WindowsCanvas" object and use that for your video surface rather than a regular Canvas.

[gnosygnus]

From the LibVLC perspective, platform-specific code is unavoidable for video embedding. This stems from the fact that native window handles are required.

Thanks. That makes complete sense. I was looking at it too much from the pampered portable-language perspective.

With the newest version of the vlcj bindings, mouse events (move, press, release) via a native message hook works for me when playing video on Vista.

I took a look and it works great. I made a few additions on my side. Please feel free to modify and incorporate.

1) The global mouse hook fires even when the vlcWindow is not the activeWindow. EX: launch vlcj; launch eclipse/ide; overlap vlcj with eclipse; activate eclipse; move mouse over overlapped region; -> mouse move messages appear in console
I put the following in WindowsMouseHook to detect if the applicationWindow is active.

Code: Select all

java.awt.Window applicationWindow = null; @Override public LRESULT callback(int nCode, WPARAM wParam, MOUSEHOOKSTRUCT lParam) { if (applicationWindow == null) { Component parent = relativeTo.getParent(); while (parent != null) { if (parent instanceof java.awt.Window) { applicationWindow = (java.awt.Window)parent; break; } parent = parent.getParent(); } } if(nCode >= 0) { // Is the component visible... if(relativeTo.isVisible() && relativeTo.isValid() && applicationWindow != null && applicationWindow.isActive() // isActive should mean that it is the active window in the windowing system

2) I didn't know about addAWTEVentListener. (Full disclosure: C# background). The proc does allow me to capture keyboard events.

I made a wrapper so that one could just do "new WindowsCanvas().addKeyboardListener(....)". I plopped the following in your WindowsCanvas.

Code: Select all

private java.util.ArrayList<WindowsKeyListener> keyListeners = new java.util.ArrayList<WindowsKeyListener>(); @Override public synchronized void addKeyListener(KeyListener l) { WindowsKeyListener wkl = new WindowsKeyListener(this, l); Toolkit.getDefaultToolkit().addAWTEventListener(wkl, AWTEvent.KEY_EVENT_MASK); keyListeners.add(wkl); } @Override public synchronized void removeKeyListener(KeyListener l) { int index = keyListeners.indexOf(l); WindowsKeyListener wkl = (WindowsKeyListener)keyListeners.get(index); Toolkit.getDefaultToolkit().removeAWTEventListener(wkl); } class WindowsKeyListener implements java.awt.event.AWTEventListener { java.awt.Component component; java.awt.event.KeyListener keyListener; public WindowsKeyListener(java.awt.Component component, java.awt.event.KeyListener keyListener) { this.component = component; this.keyListener = keyListener; } @Override public void eventDispatched(AWTEvent event) { KeyEvent keyEvent = (KeyEvent)event; if (keyEvent.getComponent() != component) return; int id = keyEvent.getID(); switch (id) { case KeyEvent.KEY_TYPED: keyListener.keyTyped(keyEvent); break; case KeyEvent.KEY_PRESSED: keyListener.keyPressed(keyEvent); break; case KeyEvent.KEY_RELEASED: keyListener.keyReleased(keyEvent); break; } } }

3) I spent some time trying to capture Mouse Wheel events. I finally got something working, but it requires a lot more changes. You basically need to do the following:
a) add a class called MSLLHOOKSTRUCT

Code: Select all

package uk.co.caprica.vlcj.runtime.windows.internal; import com.sun.jna.Structure; import com.sun.jna.platform.win32.User32.POINT; import com.sun.jna.platform.win32.W32API.HWND; import com.sun.jna.platform.win32.W32API.ULONG_PTR; import com.sun.jna.platform.win32.W32API.DWORD; public class MSLLHOOKSTRUCT extends Structure { public static class ByReference extends MSLLHOOKSTRUCT implements Structure.ByReference {}; public POINT pt; public DWORD mouseData; public DWORD flags; public DWORD time; public ULONG_PTR dwExtraInfo; }

b) in LowLevelMouseProc change the callback signature

Code: Select all

/*LRESULT callback(int nCode, WPARAM wParam, MOUSEHOOKSTRUCT lParam);*/ LRESULT callback(int nCode, WPARAM wParam, MSLLHOOKSTRUCT lParam);

c) in WindowsMouseHook change all your references from MOUSEHOOKSTRUCT to POINT and adjust code accordingly
d) add to switch statement

Code: Select all

case WM_MOUSEWHEEL: // 522 long delta = (lParam.mouseData.longValue() >> 16); // >> 16 HighOrder Word (GET_WHEEL_DELTA_WPARAM). System.out.println(delta == 120 ? "wheel up" : "wheel down"); // TODO: call listener break;

Otherwise, this looks great. I'll still experiment with it a little more, as I had the app lock up a few times -- usually within 15 or so minutes of playback. I'm not sure if it's related directly to the hooks (it could be the video), but I'll post accordingly.

Thanks!

[sherington]

I took a look and it works great. I made a few additions on my side. Please feel free to modify and incorporate.

Nice, thanks. I've incorporated this with a few minor differences.

It's in svn and the next release will have these changes.

Please report any new issues with it.

I had the app lock up a few times -- usually within 15 or so minutes of playback

This is one reason why I didn't bother with this at first, the more native calls that are made the more unstable things seem to become - especially on Windows. Tracing those problems can be very difficult.

[dhirwinjr]

@sherington

I'm trying to get the mouse hook working on Windows XP but I keep getting this exception:

Code: Select all

Exception in thread "Thread-6" java.lang.NoSuchMethodError: com.sun.jna.Pointer.createConstant(I)Lcom/sun/jna/Pointer; at com.sun.jna.platform.win32.W32API.<clinit>(W32API.java:93) at com.sun.jna.platform.win32.W32API$HANDLE.fromNative(W32API.java:38) at com.sun.jna.NativeMappedConverter.fromNative(NativeMappedConverter.java:61) at com.sun.jna.Function.invoke(Function.java:234) at com.sun.jna.Library$Handler.invoke(Library.java:204) at $Proxy3.GetModuleHandle(Unknown Source) at com.delcan.decoder.runtime.windows.WindowsMouseHook$MouseHookThread.run(WindowsMouseHook.java:459)

Since I'm still using VLC 1.0.5 yet I'd like to try to use the mouse hook stuff for testing purposes I copied over all the Windows specific code from the *.runtime.windows package in the VLCj project.

I'm using JNA 3.2.5 and Java 1.5. Any ideas?

Thanks,
Dave

[sherington]

@sherington
Exception in thread "Thread-6" java.lang.NoSuchMethodError: com.sun.jna.Pointer.createConstant(I)Lcom/sun/jna/Pointer;
I'm using JNA 3.2.5.

You have a mismatch on your classpath between the JNA platform.jar and the JNA jna.jar.

[dhirwinjr]

@sherington
Exception in thread "Thread-6" java.lang.NoSuchMethodError: com.sun.jna.Pointer.createConstant(I)Lcom/sun/jna/Pointer;
I'm using JNA 3.2.5.

You have a mismatch on your classpath between the JNA platform.jar and the JNA jna.jar.

Indeed you are correct. I had the correct combination of JNA jar and platform jar but the main project was referencing other projects (within Eclipse) that had older JNA jars on their classpath. Upgraded all to the same version of JNA and now it's working.

Thanks for your help.

Dave

[chirantanaranya]

Hi all,

First, thanks to all you awesome folks for all the great work. I am a newbie so excuse me if I sound naive.

I recently downloaded vlcj which I think is a Java 6 skin around libvlc. vlcj has several i/fs to control the video. I am wondering, should it not be easy to take vlcj source code and replace the event handlers with logic we want executed in response to the mouse events? That way, at least some mouse events can be processed, if not all mouse/keyboard events.

Chirantan

[sherington]

Hi all,

First, thanks to all you awesome folks for all the great work. I am a newbie so excuse me if I sound naive.

I recently downloaded vlcj which I think is a Java 6 skin around libvlc. vlcj has several i/fs to control the video. I am wondering, should it not be easy to take vlcj source code and replace the event handlers with logic we want executed in response to the mouse events? That way, at least some mouse events can be processed, if not all mouse/keyboard events.

Chirantan

Hi,

Two things:

1. vlcj requires Java 5.
2. I'm not sure what you mean because it works exactly like you seem to suggest. The problem is on Windows that the native video component does not (always?) propagate all the events you need into your Java application so you have to hack around with native hooks and generate 'fake' events that do propagate to your Java application.