Use libvlc_video_set_output_callbacks() to work around high CPU usage

This forum is about all development around libVLC.
truth
Blank Cone
Blank Cone
Posts: 14
Joined: 06 Apr 2017 09:55

Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby truth » 01 Feb 2022 00:00

We are developing a desktop application for viewing all kinds of media including Office documents, text, audio, video, and images.
The application is written in C++ and uses Qt and Qml. Due to this fact, we cannot use libvlc_media_player_set_xwindow() but rather libvlc_video_set_callbacks(). Unfortunately, this results in high CPU usage due to video buffer management and decoding that cannot be done by the GPU.

To work around this issue, we would like to use libvlc_video_set_output_callbacks() from VLCv4.0 so that we can render video frames directly to an OpenGL texture. This should make use of the system GPU.
Unfortunately, the documentation for this function is limited and we cannot find any examples to help us understand how best to use it.

1. Is this the correct path to follow? That is, will libvlc_video_set_output_callbacks() from VLCv4.0 potentially solve our problem?
2. If correct, where can we find examples of how to use libvlc_video_set_output_callbacks()?
3. If not, is there another possible work around?

At least on other developer has tried the same work around: https://code.videolan.org/videolan/vlc- ... ssues/1465
Last edited by truth on 01 Feb 2022 18:40, edited 1 time in total.

mfkl
Developer
Developer
Posts: 736
Joined: 13 Jun 2017 10:41

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby mfkl » 01 Feb 2022 06:11

1. Is this the correct path to follow? That is, will libvlc_video_set_output_callbacks() from VLCv4.0 potentially solve our problem?
Yes.
2. If correct, where can we find examples of how to use libvlc_video_set_output_callbacks()?
https://code.videolan.org/videolan/vlc/ ... widget.cpp
https://mfkl.github.io

unidan
Developer
Developer
Posts: 1493
Joined: 25 Mar 2018 01:00

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby unidan » 01 Feb 2022 18:22

The application is written in C++ and uses Qt and Qml. Due to this fact, we cannot use libvlc_media_player_set_xwindow() but rather libvlc_video_set_callbacks(). Unfortunately, this results in high CPU usage due to video buffer management and decoding that cannot be done by the GPU.
Note that if you plan to integrate other libraries for some purpose (given the extend of features you showcase), you might as well provide X11 windows to those libraries and redirect them into an OpenGL texture and then achieve composition in your app, in a general way.

Otherwise, libvlc_video_set_output_callbacks is indeed the way to go.

truth
Blank Cone
Blank Cone
Posts: 14
Joined: 06 Apr 2017 09:55

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby truth » 01 Feb 2022 18:44

Thank you both for the information. We're testing and will post results as soon as we can.

A13x
New Cone
New Cone
Posts: 6
Joined: 16 Mar 2022 10:14

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby A13x » 16 Mar 2022 10:40

1. Is this the correct path to follow? That is, will libvlc_video_set_output_callbacks() from VLCv4.0 potentially solve our problem?
Yes.
2. If correct, where can we find examples of how to use libvlc_video_set_output_callbacks()?
https://code.videolan.org/videolan/vlc/ ... widget.cpp

Hello.
I tried to build that example.
It has been built and it starts. But then it constantly crashes in function:

Code: Select all

static inline bool vlc_gl_StrHasToken(const char *apis, const char *api) { size_t apilen = strlen(api); while (apis) { while (*apis == ' ') apis++; if (!strncmp(apis, api, apilen) && memchr(" ", apis[apilen], 2)) return true; apis = strchr(apis, ' '); } return false; }
line "while (*apis == ' ')"

after invoking

Code: Select all

auto addr = that->mContext->getProcAddress(current);
with "current" value = "eglQueryString"

Sorry. I wasn't able to make debugging session.

Debug information:

Code: Select all

Signal name : SIGSEGV Signal meaning : Segmentation fault
Crash here:

Code: Select all

1 vlc_gl_StrHasToken vlc_opengl.h 147 0x7fffdcd8f40a
Full backtrace: https://pastebin.com/081u6uNT

How to fix?
Or how to debug?
I have built libvlc with

Code: Select all

CFLAGS="-g -Og" CXXFLAGS="-g -Og" ./configure --enable-debug --disable-optimizations --prefix=/srv/build_libs/vlc4
But breakpoints inside this function don't work while I'm debugging the application.

Rémi Denis-Courmont
Developer
Developer
Posts: 15263
Joined: 07 Jun 2004 16:01
VLC version: master
Operating System: Linux
Contact:

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby Rémi Denis-Courmont » 16 Mar 2022 12:10

Means you've returned garbage from GetProcAddress
Rémi Denis-Courmont
https://www.remlab.net/
Private messages soliciting support will be systematically discarded

A13x
New Cone
New Cone
Posts: 6
Joined: 16 Mar 2022 10:14

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby A13x » 16 Mar 2022 14:02

Means you've returned garbage from GetProcAddress
No.

I was debugging a bit.

There (vlc/modules/video_output/opengl/interop_vaapi.c):

Code: Select all

priv->egl.queryString = vlc_gl_GetProcAddress(interop->gl, "eglQueryString"); if (priv->egl.queryString == NULL) goto error; /* EGL_EXT_image_dma_buf_import implies EGL_KHR_image_base */ const char *eglexts = priv->egl.queryString(priv->egl.display, EGL_EXTENSIONS); if (eglexts == NULL || !vlc_gl_StrHasToken(eglexts, "EGL_EXT_image_dma_buf_import")) goto error;
GetProcAddress returns valid pointer to eglQueryString function.
Function queryString works.

But function queryString returns invalid pointer with address: 0xfffffffffffffff8

So, dereference of 'eglexts' in vlc_gl_StrHasToken fails.

Rémi Denis-Courmont
Developer
Developer
Posts: 15263
Joined: 07 Jun 2004 16:01
VLC version: master
Operating System: Linux
Contact:

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby Rémi Denis-Courmont » 16 Mar 2022 14:33

eglQueryString() is not allowed to return a garbage value. You'te returning a pointer that is contextually garbage.

I don't know if it's your code, your GL drivers or something between them that causes this, but that does not change the fact that the value returned by the GPA callback is garbage
Rémi Denis-Courmont
https://www.remlab.net/
Private messages soliciting support will be systematically discarded

A13x
New Cone
New Cone
Posts: 6
Joined: 16 Mar 2022 10:14

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby A13x » 16 Mar 2022 15:02

I agree. Garbage value is not allowed.

This is not my code. It is from Vlc's git, please check a link: https://code.videolan.org/videolan/vlc/ ... t.cpp#L158

"eglQueryString" pointer is provided by getProcAddress method of the instance of QOpenGLContext class.

I just tried to build and run the example.

Rémi Denis-Courmont
Developer
Developer
Posts: 15263
Joined: 07 Jun 2004 16:01
VLC version: master
Operating System: Linux
Contact:

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby Rémi Denis-Courmont » 16 Mar 2022 15:07

The comment there does say that it is not portable and won't work on some system...
Rémi Denis-Courmont
https://www.remlab.net/
Private messages soliciting support will be systematically discarded

A13x
New Cone
New Cone
Posts: 6
Joined: 16 Mar 2022 10:14

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby A13x » 16 Mar 2022 15:15

Thank you.
I got it.
Now I need to figure out how to make it working...

unidan
Developer
Developer
Posts: 1493
Joined: 25 Mar 2018 01:00

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby unidan » 18 Mar 2022 12:51

On which EGL implementation are you trying to do that? It should work correctly on Linux MESA and ANGLE/Native Windows implementations.

Qt currently **requires** this behaviour to load its own API, so there shouldn't be an issue with this, and the correct entrypoint seems to be returned.

However, with MESA, the return value of EGL/OpenGL/GLES functions are dependant on the context being currently bound. If there is no context bound, you might have incorrect return values. You should enable MESA debug environment variables to ensure that your state is correct: https://docs.mesa3d.org/envvars.html.

A13x
New Cone
New Cone
Posts: 6
Joined: 16 Mar 2022 10:14

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby A13x » 21 Mar 2022 12:12

On which EGL implementation are you trying to do that? It should work correctly on Linux MESA and ANGLE/Native Windows implementations.

Qt currently **requires** this behaviour to load its own API, so there shouldn't be an issue with this, and the correct entrypoint seems to be returned.

However, with MESA, the return value of EGL/OpenGL/GLES functions are dependant on the context being currently bound. If there is no context bound, you might have incorrect return values. You should enable MESA debug environment variables to ensure that your state is correct: https://docs.mesa3d.org/envvars.html.

I use Linux OpenGL.

Seems the issue caused by calling QOpenGLContext::makeCurrent from different (rendering) thread.
And provided in the example option QCoreApplication::setAttribute(Qt::AA_DontCheckOpenGLContextThreadAffinity) disables checking for valid thread as Qt documentation describes: https://doc.qt.io/qt-5/qopenglcontext.html#makeCurrent

So, as I understand, I need to implement OpenGL context switching thread-safe.
But I don't have much experience in OpenGL and threads interaction.

unidan
Developer
Developer
Posts: 1493
Joined: 25 Mar 2018 01:00

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby unidan » 24 Mar 2022 15:11

So, as I understand, I need to implement OpenGL context switching thread-safe.
But I don't have much experience in OpenGL and threads interaction.
You need two context:

- One will be used by VLC, and indeed need QCoreApplication::setAttribute(Qt::AA_DontCheckOpenGLContextThreadAffinity) for now since we don't guarantee the thread it's being run into.
- One from your application to do the rendering on your side.

You need to configure context sharing so that your second context can access the same texture as the first you gave to VLC, and vice versa.

Then you configure the texture and framebuffer, so they are rendered with the second context, and read afterwards with the first context, while ensuring you don't use a texture in both context at the same time (to avoid racing read/write).

Regards

A13x
New Cone
New Cone
Posts: 6
Joined: 16 Mar 2022 10:14

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby A13x » 18 Apr 2022 13:53

Thank you.

I integrated VLC4 using discussed callbacks into my application. It works almost as expected.

But now I have a different issue: sometimes the application crashes when I try to stop playback with libvlc_media_player_stop_async which is called from Qt's slot on button clicked.
There is something with threading.

Callstack looks:

Code: Select all

1 ?? 0x7fff7edde18a 2 ?? 0x7fff7eda3ffc 3 ?? 0x7fff7edb7fcb 4 ?? 0x7fff7edb926d 5 ?? 0x7fff7edc0d68 6 ?? 0x7fff7f063eab 7 start_thread pthread_create.c 477 0x7ffff6328609 8 clone clone.S 95 0x7ffff676b163
And I'm not able to catch the point of exception in the debugger.

What do you think, and what should I research?

Rémi Denis-Courmont
Developer
Developer
Posts: 15263
Joined: 07 Jun 2004 16:01
VLC version: master
Operating System: Linux
Contact:

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby Rémi Denis-Courmont » 19 Apr 2022 14:58

Stating the obvious here, but install debug symbols and enable debug symbols in your build first?
Rémi Denis-Courmont
https://www.remlab.net/
Private messages soliciting support will be systematically discarded

AlexeyGromov
New Cone
New Cone
Posts: 6
Joined: 22 Apr 2022 14:46

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby AlexeyGromov » 11 May 2022 11:23

Hello, l launch the same project on MacOS. If I use libvlc_video_set_callbacks() - there is no video. Only sound. If I use  libvlc_media_player_set_xwindow() - everything is fine.
Probably the next logs will be helpful:

Code: Select all

Using libvlc version: "4.0.0-dev Otto Chriek" [0000000158ed7200] videotoolbox decoder: Using Video Toolbox to decode 'h264' [000000011d2634c0] main video output error: video output display creation failed [0000000158ed7200] videotoolbox decoder: Using Video Toolbox to decode 'h264' [0000000158ed7200] main decoder error: buffer deadlock prevented Texture 0x600005db9560 () used with different accesses within the same pass, this is not allowed. Texture 0x600005db9710 () used with different accesses within the same pass, this is not allowed. [000000011d2634c0] main video output error: video output display creation failed [000000011d2634c0] main video output error: video output display creation failed [000000011d2634c0] main video output error: video output display creation failed [000000011d2634c0] main video output error: video output display creation failed [0000000158ed7200] videotoolbox decoder: Using Video Toolbox to decode 'h264'

Rémi Denis-Courmont
Developer
Developer
Posts: 15263
Joined: 07 Jun 2004 16:01
VLC version: master
Operating System: Linux
Contact:

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby Rémi Denis-Courmont » 11 May 2022 14:13

Isn't that the intended behaviour?
Rémi Denis-Courmont
https://www.remlab.net/
Private messages soliciting support will be systematically discarded

AlexeyGromov
New Cone
New Cone
Posts: 6
Joined: 22 Apr 2022 14:46

Re: Use libvlc_video_set_output_callbacks() to work around high CPU usage

Postby AlexeyGromov » 11 May 2022 15:34

The intended behaviour is to have both sound and video :)


Return to “Development around libVLC”

Who is online

Users browsing this forum: No registered users and 9 guests