Opengl video output

[kolor]

Hi there,

When setting libvlc to render in an opengl window, is there a way to interact with opengl window created by libvlc ? I mean, set a custom opengl context for the window, or just get the opengl context created by libvlc, get back framebuffer associated to window, ...

[Jean-Baptiste Kempf]

What are you trying to achieve?

[kolor]

At the end, I would like to get the opengl texture containing the decoded frame.
Since VLC can use an opengl output I think it is already possible but maybe not exposed ?

[FL53]

I think it would be difficult to exploit the OpenGL texture out of its context. If you want the OpenGL texture you need the OpenGL context too (context created by VLC to draw when using OpenGL). This is not something exposed I think.

I suppose that you need an OpenGL texture to display video on a 3D object (or fullscreen quad ?) in your OpenGL application. If so, working from the SDL example (https://wiki.videolan.org/LibVLC_SampleCode_SDL/) seems to be a good idea : you will have to fill your texture each time a new frame has been fully received (after lock/unlock callbacks). You can look at PBO (http://www.songho.ca/opengl/gl_pbo.html) to create your own "double buffer" rendering process to get video data asap in your texture.

[kolor]

Thank you for the links FL53. Indeed I already know the callbacks lock/unlock but it is not the most efficient, I mean there are copies between gpu and cpu. I would like to have a zero copy pipeline, in other words stay on gpu in the whole pipeline.

[FL53]

I had the same concern about reading video in OpenGL texture but without modifying the libVLC sources I think this is not something easy to do. I used as basis the SDL tutorial and resulting performances are pretty good (but I only load a single video and apply it as texture with a simple shader on a mesh).

Nonetheless, I agree with you : avoiding copy (from VRAM to RAM then RAM to VRAM) would be great. Does someone involved in libVLC development have a trick to do that kind of thing ? (I'm quite interested too by the answer )

[Jean-Baptiste Kempf]

Thank you for the links FL53. Indeed I already know the callbacks lock/unlock but it is not the most efficient, I mean there are copies between gpu and cpu. I would like to have a zero copy pipeline, in other words stay on gpu in the whole pipeline.

You need a patch to libVLC to do that.

[neosettler]

Hello Jean-Baptiste,

Code: Select all

You need a patch to libVLC to do that.

Is this something Libvlc user could expect in the future? With the amount of data required for custom 360 stereoscopic videos, copying the texture is killing performance to a point where vlclib becomes useless. I hope you've been giving this feature a serious though.

[sherington]

VLC/LibVLC 4.x, currently in development, has new callbacks for OpenGL.

Code: Select all

LIBVLC_API bool libvlc_video_set_output_callbacks( libvlc_media_player_t *mp, libvlc_video_engine_t engine, libvlc_video_setup_cb setup_cb, libvlc_video_cleanup_cb cleanup_cb, libvlc_video_update_output_cb update_output_cb, libvlc_video_swap_cb swap_cb, libvlc_video_makeCurrent_cb makeCurrent_cb, libvlc_video_getProcAddress_cb getProcAddress_cb, void* opaque );

I have used this with LWJGL+Java and it seems to work really well.

[neosettler]

Thank you for your suggestion sherington. I got 4.0 and it seems to work well so far. I'll give this function I tried. Is there any example I could chew on?

[unidan]

Hi, check in doc/libvlc/sdl_opengl_player.cpp.

[neosettler]

Thank you for your input unidan.

I've got my hands on the latest 4.0 nightly build and the libvlc_video_set_output_callbacks signature is different from the example you've mentioned. I'm on Windows 10 and the behavior seems to be awkward as the cleanup, procAdress and setup callbacks are called every single frame. I'm not quite sure where to go from here.

Also, it might work quite nicely in a one cpp example but chances are that other people like me have their GL context on an other thread, which make this process difficult to implement. In order to move the GL context to the libvlc_video_set_output_callbacks thread, is it possible to access the libvlc_video_set_output_callbacks thread some how?

[unidan]

Hi,

Thank you for your input unidan.

I've got my hands on the latest 4.0 nightly build and the libvlc_video_set_output_callbacks signature is different from the example you've mentioned. I'm on Windows 10 and the behavior seems to be awkward as the cleanup, procAdress and setup callbacks are called every single frame. I'm not quite sure where to go from here.

4.0 and libvlc_video_set_output_callbacks are still experimental and haven't been released; so it might even change again. It was mostly done as a prototype before integrating the same mechanism for d3d11 and vulkan, then it will probably become an unified API for doing this.

Also, it might work quite nicely in a one cpp example but chances are that other people like me have their GL context on an other thread, which make this process difficult to implement. In order to move the GL context to the libvlc_video_set_output_callbacks thread, is it possible to access the libvlc_video_set_output_callbacks thread some how?

You need to provide a context to libvlc. As libvlc output runs into its own thread, you need to dedicate a full context that you don't use in another thread. So you'll likely have two contexts that share resources.
Keep in mind that you're running video, that probably runs with a different framerate than your own application, so you have to clock gate with for example triple buffering, or sync with the video rendering.

[onelistme]

Hi, check in doc/libvlc/sdl_opengl_player.cpp.

When run this example, it's founded that the CPU using is so high than normal relaesed player. Is there any more information on Why the CPU using growth so high and the video play is not smooth. There is more info at https://forum.videolan.org/viewtopic.php?f=14&t=155928
would you like comment on it.

[onelistme]

Thank you for the links FL53. Indeed I already know the callbacks lock/unlock but it is not the most efficient, I mean there are copies between gpu and cpu. I would like to have a zero copy pipeline, in other words stay on gpu in the whole pipeline.

You need a patch to libVLC to do that.

Hi, is there copies between CPU and GPU in this example doc/libvlc/sdl_opengl_player.cpp? I found CPU consumption is so higher(example 30%) than the normal released player(3%). More info at https://forum.videolan.org/viewtopic.php?f=14&t=155928

[unidan]

Hi,

As mentioned in the other post for the sake of search engine completeness:

This is tracked upstream as https://trac.videolan.org/vlc/ticket/25234. The issue is that hardware decoding interops are not used because platform extensions are not available through the callbacks. We're investigating a way to have them anyway. I'll keep you posted.