General understanding of libvlc_video_set_callbacks

[zack]

When I use this callback, do I then get support for GPU decoding?

1)
When not, why?

2)
I use this callback for embedding libvlc into QML. Libvlc simply paints into a QImage("RV32").
I get 10%-15% CPU usage. using the same movie in VLC directly, with disabled Hardware-decoding I get 4%-6% CPU usage.
Is there something I can do to lower the CPU usage?

[RSATom]

you could use gpu for image format conversion and resizing.

[RSATom]

also, you could look to https://github.com/RSATom/QmlVlc - I'm already solve this task.

[zack]

I guess, this is your answer to 2)
I looked into your code and you are using raw OpenGL to to show the image and you are using the "I420" decoder.
Your CPU usage in the demo APP was lower, but only when I used 1920x1080 movies.
A guy told me that the bottleneck is not the upload of the frames, it is the decoding, which he guessed must not be GPU accelerated.
To show "I420" decoded images I cannot use QImage I guess?
Can you explain how your approach is working?
What are the drawbacks of your approach?
How about scaling,anti aliasing quality?

But can you please answer also 1)?
Is there GPU decoding on, when not why?

[Rémi Denis-Courmont]

1) No. GPU decoding outputs its result in GPU memory, not in CPU memory.

2) Retian the same video format than LibVLC provides, rather than force RV32. Retain the same resolution than LibVLC provides.

[zack]

1)
That is not clear from the documentation.
And btw. can I not provide some kind of memory space on the VRAM, e.g. a texture, where VLC can directly render to it?

2)
RSATom implemented "I420" only, no other option.
I at the moment only switched the decoding to "RV32", the resolution stays the same, I scale the image by myself. using QPainter.

[RSATom]

I420 is just most popular format

[RSATom]

To show "I420" decoded images I cannot use QImage I guess?

You could, but you will need implement I420 to RGB transform. It's consuming, and that's why I'm using (btw QtMultimedia too) shaders for it.
If you want to use QImage - it will be better still use "RV32" - libvlc will do conversation for you.

Can you explain how your approach is working?

I don't know what exactly you didn't understand in this approach.

What are the drawbacks of your approach?

maybe some platform specific implementation will be better...

How about scaling,anti aliasing quality?

you have to use shaders for it, and how good you do it, so good it will be.

But can you please answer also 1)?
Is there GPU decoding on, when not why?

I think Rémi Denis-Courmont has much more knowledge in this area.

[zack]

@ Rémi Denis-Courmont
Is there GPU decoding on, when using these callbacks, when no, why?
Can I not simply provide some memory space on the VRAM and VLC can directly render to it?

[RSATom]

Can I not simply provide some memory space on the VRAM and VLC can directly render to it?

You could do it with VBO (if your hardware supports it) - there is no need of any special support on libvlc side.
I want to try it long time...

[zack]

Can I not simply provide some memory space on the VRAM and VLC can directly render to it?

You could do it with VBO (if your hardware supports it) - there is no need of any special support on libvlc side.
I want to try it long time...

and you failed why?

[RSATom]

because lack of time to try, and missing android device to try on.

[zack]

How would you do it with VBO, that is used for geomerty data?
Is using PBO not a better approach?

[RSATom]

You are right. PBO, not VBO. They are very similar, but not the same.

[zack]

So RV32 hast 32 bit per pixel and I420 12 bit per pixel, so this must be the reason why your code is faster for videos with higher resolution.
You don't need to transfer so much data every frame.

Can be the drawback of your approach also be, that it is not supported on every platform?

[RSATom]

the point is I420 is most popular format, and if use it as default, there is big chance that will be not need to do software (read slow) transform of pixel format. It's more effective to do pixel format transform on GPU rather than on CPU. And thats the reason why it's faster.

[RSATom]

Can be the drawback of your approach also be, that it is not supported on every platform?

There are no any need to support I420 pixel format directly by platform, since it come to shader just as raw data, and there transforming to RGB. And again, if you want to play video, there are big chance you will have to deal with I420 pixel format, just because it's very popular.

[zack]

Can be the drawback of your approach also be, that it is not supported on every platform?

There are no any need to support I420 pixel format directly by platform, since it come to shader just as raw data, and there transforming to RGB. And again, if you want to play video, there are big chance you will have to deal with I420 pixel format, just because it's very popular.

What I wanted to say here is, maybe you use some OpenGL commands, which are not supported on every platform?
Or do you use OpenGL ES2?


So you say it the bottleneck is the conversion from I420 to RV32, and not that more data if moved to the GPU?
The formula to convert the image looks pretty simple I can't imagine that this is that expensive.
I think because of both, less pixels and faster conversion.

[RSATom]

Or do you use OpenGL ES2?

At this time - yes, since I developed it on Windows and default Qt build use ANGLE lib, which offer only OpenGL ES2 api.

So you say it the bottleneck is the conversion from I420 to RV32, and not that more data if moved to the GPU?
The formula to convert the image looks pretty simple I can't imagine that this is that expensive.
I think because of both, less pixels and faster conversion.

I'm not talking about bottlenecks. I am just talking about effective units using.
Problem is when you instruct libvlc give you frames in RV32 pixel format, libvlc have to decode it on CPU (in addition to decompression media frames itself), but when you ask I420 - libvlc do only decompression, and pixel format conversion work will do GPU. So it's obvious question, what's better, do same job amount on two units or only on one? Same related to resizing/stretching...

[RSATom]

btw, you don't need to use my video output implementation. QmlVlc can work with QtMultimeda's "VideoOutput" (which use Opengl ES2 shaders too). I can tell you more, I think using QtMultimedia's "VideoOutput" more likely will be more effecitve on mobile devices than my implementation.

https://github.com/RSATom/QmlVlcDemo/bl ... asic_2.qml

P.S. confirmation: https://github.com/RSATom/QmlVlc/issues/14

[zack]

I wonder how your code works, without any locks:
https://github.com/RSATom/QmlVlc/blob/m ... Output.cpp

The CPU usage is the same with your own display and with the abstract videosurface.

[RSATom]

It's a magic! It just don't use same memory twice, i.e. for every frame it's own memory block allocated ( https://github.com/RSATom/QmlVlc/blob/m ... ut.cpp#L89 ), after that QVideoFrame acts like smart pointer. So there are no need in synchronization. It could be bottleneck on devices with slow memory allocation ( for example mobile devices ), but for desktop it's pretty Ok.

[zack]

I don't understand how this will work.

when you are doing this:
m_videoFrame = frame;
The existing frame will be destroyed. This happens in the decoding thread of libvlc.
But what, if m_videoFrame is already used by the GUI thread to render it?

Note: Since video frames can be expensive to copy, QVideoFrame is explicitly shared, so any change made to a video frame will also apply to any copies.

[RSATom]

Do you know how smart pointers works?

std::shared_ptr for example?

[RSATom]

http://ideone.com/TClLGy