General understanding of libvlc_video_set_callbacks

This forum is about all development around libVLC.
zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

General understanding of libvlc_video_set_callbacks

Postby zack » 21 Sep 2014 20:53

When I use this callback, do I then get support for GPU decoding?

1)
When not, why?

2)
I use this callback for embedding libvlc into QML. Libvlc simply paints into a QImage("RV32").
I get 10%-15% CPU usage. using the same movie in VLC directly, with disabled Hardware-decoding I get 4%-6% CPU usage.
Is there something I can do to lower the CPU usage?

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 22 Sep 2014 08:30

you could use gpu for image format conversion and resizing.

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 22 Sep 2014 08:33

also, you could look to https://github.com/RSATom/QmlVlc - I'm already solve this task.

zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

Re: General understanding of libvlc_video_set_callbacks

Postby zack » 22 Sep 2014 20:30

I guess, this is your answer to 2)
I looked into your code and you are using raw OpenGL to to show the image and you are using the "I420" decoder.
Your CPU usage in the demo APP was lower, but only when I used 1920x1080 movies.
A guy told me that the bottleneck is not the upload of the frames, it is the decoding, which he guessed must not be GPU accelerated.
To show "I420" decoded images I cannot use QImage I guess?
Can you explain how your approach is working?
What are the drawbacks of your approach?
How about scaling,anti aliasing quality?

But can you please answer also 1)?
Is there GPU decoding on, when not why?

Rémi Denis-Courmont
Developer
Developer
Posts: 15279
Joined: 07 Jun 2004 16:01
VLC version: master
Operating System: Linux
Contact:

Re: General understanding of libvlc_video_set_callbacks

Postby Rémi Denis-Courmont » 22 Sep 2014 20:32

1) No. GPU decoding outputs its result in GPU memory, not in CPU memory.

2) Retian the same video format than LibVLC provides, rather than force RV32. Retain the same resolution than LibVLC provides.
Rémi Denis-Courmont
https://www.remlab.net/
Private messages soliciting support will be systematically discarded

zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

Re: General understanding of libvlc_video_set_callbacks

Postby zack » 22 Sep 2014 21:38

1)
That is not clear from the documentation.
And btw. can I not provide some kind of memory space on the VRAM, e.g. a texture, where VLC can directly render to it?

2)
RSATom implemented "I420" only, no other option.
I at the moment only switched the decoding to "RV32", the resolution stays the same, I scale the image by myself. using QPainter.

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 23 Sep 2014 03:47

I420 is just most popular format

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 23 Sep 2014 03:55

To show "I420" decoded images I cannot use QImage I guess?
You could, but you will need implement I420 to RGB transform. It's consuming, and that's why I'm using (btw QtMultimedia too) shaders for it.
If you want to use QImage - it will be better still use "RV32" - libvlc will do conversation for you.
Can you explain how your approach is working?
I don't know what exactly you didn't understand in this approach.
What are the drawbacks of your approach?
maybe some platform specific implementation will be better...
How about scaling,anti aliasing quality?
you have to use shaders for it, and how good you do it, so good it will be.
But can you please answer also 1)?
Is there GPU decoding on, when not why?
I think Rémi Denis-Courmont has much more knowledge in this area.

zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

Re: General understanding of libvlc_video_set_callbacks

Postby zack » 23 Sep 2014 08:14

@ Rémi Denis-Courmont
Is there GPU decoding on, when using these callbacks, when no, why?
Can I not simply provide some memory space on the VRAM and VLC can directly render to it?

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 23 Sep 2014 08:23

Can I not simply provide some memory space on the VRAM and VLC can directly render to it?
You could do it with VBO (if your hardware supports it) - there is no need of any special support on libvlc side.
I want to try it long time...

zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

Re: General understanding of libvlc_video_set_callbacks

Postby zack » 23 Sep 2014 14:25

Can I not simply provide some memory space on the VRAM and VLC can directly render to it?
You could do it with VBO (if your hardware supports it) - there is no need of any special support on libvlc side.
I want to try it long time...
and you failed why?

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 23 Sep 2014 14:59

because lack of time to try, and missing android device to try on.

zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

Re: General understanding of libvlc_video_set_callbacks

Postby zack » 23 Sep 2014 16:14

How would you do it with VBO, that is used for geomerty data?
Is using PBO not a better approach?

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 23 Sep 2014 18:05

You are right. PBO, not VBO. They are very similar, but not the same.

zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

Re: General understanding of libvlc_video_set_callbacks

Postby zack » 23 Sep 2014 23:14

So RV32 hast 32 bit per pixel and I420 12 bit per pixel, so this must be the reason why your code is faster for videos with higher resolution.
You don't need to transfer so much data every frame.

Can be the drawback of your approach also be, that it is not supported on every platform?

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 24 Sep 2014 02:32

the point is I420 is most popular format, and if use it as default, there is big chance that will be not need to do software (read slow) transform of pixel format. It's more effective to do pixel format transform on GPU rather than on CPU. And thats the reason why it's faster.

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 24 Sep 2014 02:36

Can be the drawback of your approach also be, that it is not supported on every platform?
There are no any need to support I420 pixel format directly by platform, since it come to shader just as raw data, and there transforming to RGB. And again, if you want to play video, there are big chance you will have to deal with I420 pixel format, just because it's very popular.

zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

Re: General understanding of libvlc_video_set_callbacks

Postby zack » 24 Sep 2014 09:48

Can be the drawback of your approach also be, that it is not supported on every platform?
There are no any need to support I420 pixel format directly by platform, since it come to shader just as raw data, and there transforming to RGB. And again, if you want to play video, there are big chance you will have to deal with I420 pixel format, just because it's very popular.
What I wanted to say here is, maybe you use some OpenGL commands, which are not supported on every platform?
Or do you use OpenGL ES2?


So you say it the bottleneck is the conversion from I420 to RV32, and not that more data if moved to the GPU?
The formula to convert the image looks pretty simple I can't imagine that this is that expensive.
I think because of both, less pixels and faster conversion.

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 24 Sep 2014 10:11

Or do you use OpenGL ES2?
At this time - yes, since I developed it on Windows and default Qt build use ANGLE lib, which offer only OpenGL ES2 api.
So you say it the bottleneck is the conversion from I420 to RV32, and not that more data if moved to the GPU?
The formula to convert the image looks pretty simple I can't imagine that this is that expensive.
I think because of both, less pixels and faster conversion.
I'm not talking about bottlenecks. I am just talking about effective units using.
Problem is when you instruct libvlc give you frames in RV32 pixel format, libvlc have to decode it on CPU (in addition to decompression media frames itself), but when you ask I420 - libvlc do only decompression, and pixel format conversion work will do GPU. So it's obvious question, what's better, do same job amount on two units or only on one? Same related to resizing/stretching...
Last edited by RSATom on 24 Sep 2014 10:24, edited 1 time in total.

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 24 Sep 2014 10:19

btw, you don't need to use my video output implementation. QmlVlc can work with QtMultimeda's "VideoOutput" (which use Opengl ES2 shaders too). I can tell you more, I think using QtMultimedia's "VideoOutput" more likely will be more effecitve on mobile devices than my implementation.

https://github.com/RSATom/QmlVlcDemo/bl ... asic_2.qml

P.S. confirmation: https://github.com/RSATom/QmlVlc/issues/14

zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

Re: General understanding of libvlc_video_set_callbacks

Postby zack » 24 Sep 2014 20:28

I wonder how your code works, without any locks:
https://github.com/RSATom/QmlVlc/blob/m ... Output.cpp

The CPU usage is the same with your own display and with the abstract videosurface.

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 25 Sep 2014 09:17

It's a magic! :) It just don't use same memory twice, i.e. for every frame it's own memory block allocated ( https://github.com/RSATom/QmlVlc/blob/m ... ut.cpp#L89 ), after that QVideoFrame acts like smart pointer. So there are no need in synchronization. It could be bottleneck on devices with slow memory allocation ( for example mobile devices ), but for desktop it's pretty Ok.

zack
Blank Cone
Blank Cone
Posts: 13
Joined: 21 Sep 2014 16:53

Re: General understanding of libvlc_video_set_callbacks

Postby zack » 25 Sep 2014 10:21

I don't understand how this will work.

when you are doing this:
m_videoFrame = frame;
The existing frame will be destroyed. This happens in the decoding thread of libvlc.
But what, if m_videoFrame is already used by the GUI thread to render it?
Note: Since video frames can be expensive to copy, QVideoFrame is explicitly shared, so any change made to a video frame will also apply to any copies.

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 25 Sep 2014 10:26

Do you know how smart pointers works?

std::shared_ptr for example?

RSATom
Big Cone-huna
Big Cone-huna
Posts: 513
Joined: 24 Nov 2011 06:55
Operating System: Windows/Linux/OsX
Location: Russia, Tomsk

Re: General understanding of libvlc_video_set_callbacks

Postby RSATom » 25 Sep 2014 10:53



Return to “Development around libVLC”

Who is online

Users browsing this forum: No registered users and 34 guests