Use libvlc_video_set_output_callbacks() to work around high CPU usage

[truth]

We are developing a desktop application for viewing all kinds of media including Office documents, text, audio, video, and images.
The application is written in C++ and uses Qt and Qml. Due to this fact, we cannot use libvlc_media_player_set_xwindow() but rather libvlc_video_set_callbacks(). Unfortunately, this results in high CPU usage due to video buffer management and decoding that cannot be done by the GPU.

To work around this issue, we would like to use libvlc_video_set_output_callbacks() from VLCv4.0 so that we can render video frames directly to an OpenGL texture. This should make use of the system GPU.
Unfortunately, the documentation for this function is limited and we cannot find any examples to help us understand how best to use it.

1. Is this the correct path to follow? That is, will libvlc_video_set_output_callbacks() from VLCv4.0 potentially solve our problem?
2. If correct, where can we find examples of how to use libvlc_video_set_output_callbacks()?
3. If not, is there another possible work around?

At least on other developer has tried the same work around: https://code.videolan.org/videolan/vlc- ... ssues/1465

[mfkl]

1. Is this the correct path to follow? That is, will libvlc_video_set_output_callbacks() from VLCv4.0 potentially solve our problem?

Yes.

2. If correct, where can we find examples of how to use libvlc_video_set_output_callbacks()?

https://code.videolan.org/videolan/vlc/ ... widget.cpp

[unidan]

The application is written in C++ and uses Qt and Qml. Due to this fact, we cannot use libvlc_media_player_set_xwindow() but rather libvlc_video_set_callbacks(). Unfortunately, this results in high CPU usage due to video buffer management and decoding that cannot be done by the GPU.

Note that if you plan to integrate other libraries for some purpose (given the extend of features you showcase), you might as well provide X11 windows to those libraries and redirect them into an OpenGL texture and then achieve composition in your app, in a general way.

Otherwise, libvlc_video_set_output_callbacks is indeed the way to go.

[truth]

Thank you both for the information. We're testing and will post results as soon as we can.

[A13x]

1. Is this the correct path to follow? That is, will libvlc_video_set_output_callbacks() from VLCv4.0 potentially solve our problem?

Yes.

2. If correct, where can we find examples of how to use libvlc_video_set_output_callbacks()?

https://code.videolan.org/videolan/vlc/ ... widget.cpp

Hello.
I tried to build that example.
It has been built and it starts. But then it constantly crashes in function:

Code: Select all

static inline bool vlc_gl_StrHasToken(const char *apis, const char *api) { size_t apilen = strlen(api); while (apis) { while (*apis == ' ') apis++; if (!strncmp(apis, api, apilen) && memchr(" ", apis[apilen], 2)) return true; apis = strchr(apis, ' '); } return false; }

line "while (*apis == ' ')"

after invoking

Code: Select all

auto addr = that->mContext->getProcAddress(current);

with "current" value = "eglQueryString"

Sorry. I wasn't able to make debugging session.

Debug information:

Code: Select all

Signal name : SIGSEGV Signal meaning : Segmentation fault

Crash here:

Code: Select all

1 vlc_gl_StrHasToken vlc_opengl.h 147 0x7fffdcd8f40a

Full backtrace: https://pastebin.com/081u6uNT

How to fix?
Or how to debug?
I have built libvlc with

Code: Select all

CFLAGS="-g -Og" CXXFLAGS="-g -Og" ./configure --enable-debug --disable-optimizations --prefix=/srv/build_libs/vlc4

But breakpoints inside this function don't work while I'm debugging the application.

[Rémi Denis-Courmont]

Means you've returned garbage from GetProcAddress

[A13x]

Means you've returned garbage from GetProcAddress

No.

I was debugging a bit.

There (vlc/modules/video_output/opengl/interop_vaapi.c):

Code: Select all

priv->egl.queryString = vlc_gl_GetProcAddress(interop->gl, "eglQueryString"); if (priv->egl.queryString == NULL) goto error; /* EGL_EXT_image_dma_buf_import implies EGL_KHR_image_base */ const char *eglexts = priv->egl.queryString(priv->egl.display, EGL_EXTENSIONS); if (eglexts == NULL || !vlc_gl_StrHasToken(eglexts, "EGL_EXT_image_dma_buf_import")) goto error;

GetProcAddress returns valid pointer to eglQueryString function.
Function queryString works.

But function queryString returns invalid pointer with address: 0xfffffffffffffff8

So, dereference of 'eglexts' in vlc_gl_StrHasToken fails.

[Rémi Denis-Courmont]

eglQueryString() is not allowed to return a garbage value. You'te returning a pointer that is contextually garbage.

I don't know if it's your code, your GL drivers or something between them that causes this, but that does not change the fact that the value returned by the GPA callback is garbage

[A13x]

I agree. Garbage value is not allowed.

This is not my code. It is from Vlc's git, please check a link: https://code.videolan.org/videolan/vlc/ ... t.cpp#L158

"eglQueryString" pointer is provided by getProcAddress method of the instance of QOpenGLContext class.

I just tried to build and run the example.

[Rémi Denis-Courmont]

The comment there does say that it is not portable and won't work on some system...

[A13x]

Thank you.
I got it.
Now I need to figure out how to make it working...

[unidan]

On which EGL implementation are you trying to do that? It should work correctly on Linux MESA and ANGLE/Native Windows implementations.

Qt currently **requires** this behaviour to load its own API, so there shouldn't be an issue with this, and the correct entrypoint seems to be returned.

However, with MESA, the return value of EGL/OpenGL/GLES functions are dependant on the context being currently bound. If there is no context bound, you might have incorrect return values. You should enable MESA debug environment variables to ensure that your state is correct: https://docs.mesa3d.org/envvars.html.

[A13x]

On which EGL implementation are you trying to do that? It should work correctly on Linux MESA and ANGLE/Native Windows implementations.

Qt currently **requires** this behaviour to load its own API, so there shouldn't be an issue with this, and the correct entrypoint seems to be returned.

However, with MESA, the return value of EGL/OpenGL/GLES functions are dependant on the context being currently bound. If there is no context bound, you might have incorrect return values. You should enable MESA debug environment variables to ensure that your state is correct: https://docs.mesa3d.org/envvars.html.

I use Linux OpenGL.

Seems the issue caused by calling QOpenGLContext::makeCurrent from different (rendering) thread.
And provided in the example option QCoreApplication::setAttribute(Qt::AA_DontCheckOpenGLContextThreadAffinity) disables checking for valid thread as Qt documentation describes: https://doc.qt.io/qt-5/qopenglcontext.html#makeCurrent

So, as I understand, I need to implement OpenGL context switching thread-safe.
But I don't have much experience in OpenGL and threads interaction.

[unidan]

So, as I understand, I need to implement OpenGL context switching thread-safe.
But I don't have much experience in OpenGL and threads interaction.

You need two context:

- One will be used by VLC, and indeed need QCoreApplication::setAttribute(Qt::AA_DontCheckOpenGLContextThreadAffinity) for now since we don't guarantee the thread it's being run into.
- One from your application to do the rendering on your side.

You need to configure context sharing so that your second context can access the same texture as the first you gave to VLC, and vice versa.

Then you configure the texture and framebuffer, so they are rendered with the second context, and read afterwards with the first context, while ensuring you don't use a texture in both context at the same time (to avoid racing read/write).

Regards

[A13x]

Thank you.

I integrated VLC4 using discussed callbacks into my application. It works almost as expected.

But now I have a different issue: sometimes the application crashes when I try to stop playback with libvlc_media_player_stop_async which is called from Qt's slot on button clicked.
There is something with threading.

Callstack looks:

Code: Select all

1 ?? 0x7fff7edde18a 2 ?? 0x7fff7eda3ffc 3 ?? 0x7fff7edb7fcb 4 ?? 0x7fff7edb926d 5 ?? 0x7fff7edc0d68 6 ?? 0x7fff7f063eab 7 start_thread pthread_create.c 477 0x7ffff6328609 8 clone clone.S 95 0x7ffff676b163

And I'm not able to catch the point of exception in the debugger.

What do you think, and what should I research?

[Rémi Denis-Courmont]

Stating the obvious here, but install debug symbols and enable debug symbols in your build first?

[AlexeyGromov]

Hello, l launch the same project on MacOS. If I use libvlc_video_set_callbacks() - there is no video. Only sound. If I use  libvlc_media_player_set_xwindow() - everything is fine.
Probably the next logs will be helpful:

Code: Select all

Using libvlc version: "4.0.0-dev Otto Chriek" [0000000158ed7200] videotoolbox decoder: Using Video Toolbox to decode 'h264' [000000011d2634c0] main video output error: video output display creation failed [0000000158ed7200] videotoolbox decoder: Using Video Toolbox to decode 'h264' [0000000158ed7200] main decoder error: buffer deadlock prevented Texture 0x600005db9560 () used with different accesses within the same pass, this is not allowed. Texture 0x600005db9710 () used with different accesses within the same pass, this is not allowed. [000000011d2634c0] main video output error: video output display creation failed [000000011d2634c0] main video output error: video output display creation failed [000000011d2634c0] main video output error: video output display creation failed [000000011d2634c0] main video output error: video output display creation failed [0000000158ed7200] videotoolbox decoder: Using Video Toolbox to decode 'h264'

[Rémi Denis-Courmont]

Isn't that the intended behaviour?

[AlexeyGromov]

The intended behaviour is to have both sound and video