LibVLC RC 2.0, RV24 and WebM

[bunjee]

Greetings VideoLAN community,

My name is Benjamin Arnaud. I'm new on this board and looking forward to share interesting stuff with you.

I've been playing around a little bit with LibVLC RC 2.0 lately. Congratulation on switching to LGPL.

I'm using libvlc_video_set_callbacks and libvlc_video_set_format_callbacks inside a Qt QGraphicsScene.

I'm using the RV24 chroma setting and I'm rendering my frames using native Opengl with glTexSubImage2D.

It's working fine with a mp4 h264 file. I tried with a youtube webM video file, my frames turn black and white with a strange tearing. When switching to RV32, the rendering seems ok.

Is there an issue between RV24 and the WebM format ? Is there a performance difference between RV24 and RV32 on VLC side ?

Thanks guys.

B.A.

[Jean-Baptiste Kempf]

Greetings VideoLAN community,

My name is Benjamin Arnaud. I'm new on this board and looking forward to share interesting stuff with you.

I've been playing around a little bit with LibVLC RC 2.0 lately. Congratulation on switching to LGPL.

Thanks! And Welcome!

I'm using libvlc_video_set_callbacks and libvlc_video_set_format_callbacks inside a Qt QGraphicsScene.

I'm using the RV24 chroma setting and I'm rendering my frames using native Opengl with glTexSubImage2D.

It's working fine with a mp4 h264 file. I tried with a youtube webM video file, my frames turn black and white with a strange tearing. When switching to RV32, the rendering seems ok.

Is there an issue between RV24 and the WebM format ? Is there a performance difference between RV24 and RV32 on VLC side ?

You should look at the logs of VLC.

[bunjee]

Thanks I'll take a look.

"--verbose=2" should do the trick.

[Jean-Baptiste Kempf]

Yes, it should

[bunjee]

Code: Select all

[056b7f08] swscale scale debug: 32x32 chroma: YUVA -> 16x16 chroma: RGBA with scaling using Bicubic (good quality) [056b7f08] main scale debug: using video filter2 module "swscale" [056b7f08] main scale debug: TIMER module_need() : 1.083 ms - Total 1.083 ms / 1 intvls (Avg 1.083 ms) [056b8230] main scale debug: looking for video filter2 module: 18 candidates [056b8230] yuvp scale debug: YUVP to YUVA converter [056b8230] main scale debug: using video filter2 module "yuvp" [056b8230] main scale debug: TIMER module_need() : 1.847 ms - Total 1.847 ms / 1 intvls (Avg 1.847 ms) [05600b00] aout_directx audio output debug: found device: Primary Sound Driver [05600b00] aout_directx audio output debug: found device: Speakers (Realtek High Definition Audio) [056012b0] main video output debug: Deinterlacing available [056012b0] main video output debug: deinterlace 0, mode blend, is_needed 0 [056012b0] main video output debug: Opening vout display wrapper [055bc6a8] main vout display debug: looking for vout display module: 1 candidate [055bc6a8] main vout display debug: VoutDisplayEvent 'fullscreen' 0 [055bc6a8] main vout display debug: VoutDisplayEvent 'resize' 854x476 window [055bc6a8] main vout display debug: using vout display module "vmem" [055bc6a8] main vout display debug: TIMER module_need() : 0.543 ms - Total 0.543 ms / 1 intvls (Avg 0.543 ms) [055bc6a8] main vout display debug: A filter to adapt decoder to display is needed [056dffe8] main filter debug: looking for video filter2 module: 18 candidates [056dffe8] swscale filter debug: 854x476 chroma: I420 -> 854x476 chroma: RV24 with scaling using Bicubic (good quality) [056dffe8] main filter debug: using video filter2 module "swscale" [056dffe8] main filter debug: TIMER module_need() : 0.161 ms - Total 0.161 ms / 1 intvls (Avg 0.161 ms) [055bc6a8] main vout display debug: Filter 'Swscale' (056dffe8) appended to chain [056012b0] main video output debug: original format sz 854x476, of (0,0), vsz 854x476, 4cc I420, sar 1:1, msk r0x0 g0x0 b0x0

I don't see anything too shocking here. Am I missing something ?

[Jean-Baptiste Kempf]

What happens in the logs in the case of h264?

[bunjee]

For mp4 / h264 I get the following:

Code: Select all

[055c3e10] swscale scale debug: 32x32 chroma: YUVA -> 16x16 chroma: RGBA with scaling using Bicubic (good quality) [055c3e10] main scale debug: using video filter2 module "swscale" [055c3e10] main scale debug: TIMER module_need() : 0.852 ms - Total 0.852 ms / 1 intvls (Avg 0.852 ms) [055c4138] main scale debug: looking for video filter2 module: 18 candidates [055c4138] yuvp scale debug: YUVP to YUVA converter [055c4138] main scale debug: using video filter2 module "yuvp" [055c4138] main scale debug: TIMER module_need() : 1.568 ms - Total 1.568 ms / 1 intvls (Avg 1.568 ms) [056253b0] main video output debug: Deinterlacing available [056253b0] main video output debug: deinterlace 0, mode blend, is_needed 0 [056253b0] main video output debug: Opening vout display wrapper [055d1ac0] main vout display debug: looking for vout display module: 1 candidate [055d1ac0] main vout display debug: VoutDisplayEvent 'fullscreen' 0 [055d1ac0] main vout display debug: VoutDisplayEvent 'resize' 854x480 window [055d1ac0] main vout display debug: using vout display module "vmem" [055d1ac0] main vout display debug: TIMER module_need() : 0.472 ms - Total 0.472 ms / 1 intvls (Avg 0.472 ms) [055d1ac0] main vout display debug: A filter to adapt decoder to display is needed [0567c4c8] main filter debug: looking for video filter2 module: 18 candidates [0567c4c8] swscale filter debug: 854x480 chroma: I420 -> 854x480 chroma: RV32 with scaling using Bicubic (good quality) [0567c4c8] main filter debug: using video filter2 module "swscale" [0567c4c8] main filter debug: TIMER module_need() : 0.168 ms - Total 0.168 ms / 1 intvls (Avg 0.168 ms) [055d1ac0] main vout display debug: Filter 'Swscale' (0567c4c8) appended to chain [056253b0] main video output debug: original format sz 854x480, of (0,0), vsz 854x480, 4cc I420, sar 1:1, msk r0x0 g0x0 b0x0

[mirswith]

It's working fine with a mp4 h264 file. I tried with a youtube webM video file, my frames turn black and white with a strange tearing. When switching to RV32, the rendering seems ok.

Is this a case of RGB vs. BGR? OpenGL expects BGR for 24bit rendering and VLC (if I remember correcty) supplies it as RGB. This doesn't explain the tearing however.

[bunjee]

Well I had the BGR vs RGB issue with my opengl implementation.

This is not related since I have the same tearing and black and white result even when using a QImage + rgbswapped() function.

Is RV24 supposed to be faster than RV32 anyway ?

[mirswith]

RV24 is faster from a glTexSubImage2D perspective since it is less data being pushed to the video card, however if the rgb is being swapped in software it might counter those gains.

[Rémi Denis-Courmont]

RGB is going to be slow, no matter what kind of RGB.

Use YUV if you care about speed. There are no other options. And yes, that pretty much eliminates OpenGL, unless shaders are involved.

[bunjee]

Thanks Remi.

So I can expect Opengl and YUV to RGB shaders to be significantly faster than VLC's own RGB conversion ?

Doesn't RV32 / RV24 already use Opengl shaders to perform that conversion on VLC 2.0 ?

B.A.

[Rémi Denis-Courmont]

VLC's color spaces conversions are done on CPU. Relative speed depends on the relative speed and use of the CPU and the GPU.

[bunjee]

I'll give YUV + Opengl a shot,

Judging from what I read here: http://wiki.videolan.org/YUV I should use UYVY for 24bit video, then perform the conversion on the GPU using a UYVY to RGB shader.

UPDATE: after further reading it seems YUV 4:2:0 I420 is the best option.

Also using glTexSubImage2D together with a pixel buffer object might increase texture upload performances: http://www.songho.ca/opengl/gl_pbo.html#unpack

[Jean-Baptiste Kempf]

the VLC OpenGL uses fragment shaders ARB to do such conversion.

[mirswith]

Can you point to an example that uses this or some hints about how to use it?

Thanks!

[Jean-Baptiste Kempf]

Read the opengl output code.

[vseryakov]

I'll give YUV + Opengl a shot,

Judging from what I read here: http://wiki.videolan.org/YUV I should use UYVY for 24bit video, then perform the conversion on the GPU using a UYVY to RGB shader.

UPDATE: after further reading it seems YUV 4:2:0 I420 is the best option.

Also using glTexSubImage2D together with a pixel buffer object might increase texture upload performances: http://www.songho.ca/opengl/gl_pbo.html#unpack

Hello,

Any chance to show your example, for some reason it keeps crashing on Mac OS when i am trying to integrate this with QML scene. No worries if it is proprietary, i will keep digging.

Thanks

[Rémi Denis-Courmont]

For the record, UYVY is 16-bits and U420 is 12-bits. Only I444 is 24-bits, but it is rarely used in video streams.

[bunjee]

vseryakov,

First, take a look at the VLC source code for a proper OpenGL implementation:

vlc/modules/video_output/opengl.c

Then use a QGraphicsView together with a QGLWidget viewport.

Then use QPainter::beginNativePaint() in the paintEvent and render your frame.

You get best performances with VLC YUV decoding + multitexturing and a shader.

Also it's a good idea to run VLC from its own QThread communicating with QEvent(s).

B.A.