VLC and Qt Integration - reder decoded video to a QPixmap

[Mauro]

Hi,
I need to render decoded frames on a QPixmap and draw it on a QWidget (I actually need to render each frame on several QWidgets, that's why i need to render on a custom area).

I've studied the libvlc and Qt documentation and googled around a lot... but I still get nothing!

This is what I've done:

FIRST, I've set callbacks and private data to render decoded video to a custom area in memory

Code: Select all

// these are class properties of type uchar*; the second pointer should be 32-byte aligned _nativeBufferNotAligned = (uchar*)malloc((_nativeWidth * _nativeHeight * 4) + 31); _nativeBuffer = (uchar*)((size_t(_nativeBufferNotAligned)+31) & (~31)); libvlc_video_set_format(_vlcmp, "RV32", _nativeWidth, _nativeHeight, _nativeWidth*4); libvlc_video_set_callbacks(_vlcmp, libvlc_media_player_lock, libvlc_media_player_unlock, libvlc_media_player_display, this);

SECOND, I've written the code for the callbacks

Code: Select all

void* VlcMediaPlayer::libvlc_media_player_lock(void *opaque, void **plane) { VlcMediaPlayer* vlcmp = (VlcMediaPlayer*) opaque; *plane = vlcmp->_nativeBuffer; return NULL; } void VlcMediaPlayer::libvlc_media_player_unlock(void *opaque, void *picture, void *const *plane) { } void VlcMediaPlayer::libvlc_media_player_display(void *opaque, void *picture) { VlcMediaPlayer* vlcmp = (VlcMediaPlayer*) opaque; // I keep a pointer to a QWidget that will render the buffer on video vlcmp->_renderSurface->bufferReady(vlcmp->_nativeBuffer, vlcmp->_nativeWidth, vlcmp->_nativeHeight); }

THIRD, RenderSurface is a QWidget where:

Code: Select all

void RenderSurface::bufferReady(uchar* nativeBuffer, uint nativeWidth, uint nativeHeight) { // _nativePixmap is a class property of type QPixmap* if (_nativePixmap == NULL) _nativePixmap = new QPixmap(nativeWidth, nativeHeight); _nativePixmap->loadFromData(nativeBuffer, nativeWidth*nativeHeight*4); update(); } void RenderSurface::paintEvent(QPaintEvent *event) { QPainter painter(this); painter.drawPixmap(0, 0, _surfaceWidth, _surfaceHeight, *_nativePixmap); }

The application crashes.
Has anybody any suggestion?
Thanks!!

[Rémi Denis-Courmont]

There can be many reasons why it crashes. The buffer is too small. The calling conventions are incompatible. The callbacks don't do what is expected. The callbacks are not thread-safe, etc etc.

So can't really help you.

[Mauro]

Buffer size is: (_nativeWidth*_nativeHeight*4) bytes
that should be fine for: libvlc_video_set_format(_vlcmp, "RV32", _nativeWidth, _nativeHeight, _nativeWidth*4);

"lock" implementation I believe is quite standard (just set: *plane = <buffer pointer>); "unlock" do nothing; "display" gives the buffer to a QWidget to render to a QPixmap.

I don't know, everything seems quite straightforward, but it doesn't work.
Some more help?
Thanks

[Mauro]

Ok, the crash was caused by something unrelated.
Now, I can hear audio but on the QWidget I get a blank screen.
libvlc video format is set to "RV32", I've invoked QPixmap::loadFromData(..) and QPainter::drawPixmap(..)
Maybe I have to tell Qt something more about the format of the decoded frame.
Any help?
Thanks!!

[xylosper]

QPixmap::loadFromData loads image from binary format like png or jpeg not raw data.
It does same thing with QPixmap::load, only difference is it loads data from your memory not a file.

You have to make an QImage instance with proper QImage::Format, for instance, QImage::Format_RGB32 or QImage::Format_ARGB32 for RV32.
Then, if you need QPixmap, you can convert the image to pixmap with QPixmap::fromImage.
Also, you can check if the created image or pixmap is ok by outputing to a file with QImage::save or QPixmap::save .

[Mauro]

Thanks a lot xylosper!

Ok, then each time a new frame needs to be displayed, this is what I do:

Code: Select all

QImage image(QSize(nativeWidth, nativeHeight), QImage::Format_ARGB32); image.loadFromData(nativeBuffer, nativeWidth*nativeHeight*4); _nativePixmap.fromImage(image); update();

Then in the paintEvent:

Code: Select all

QPainter painter(this); painter.drawPixmap(0, 0, _surfaceWidth, _surfaceHeight, _nativePixmap);

However, I still get a blank screen (with a few coloured pixels at the top, proably meaning there's still an image format misunderstanding).
I've tried to save the image (both from QImage and QPixmap) and I've got the same blank image.

What do you think I'm still missing?
Thanks for your help.

[xylosper]

QImage::loadFromData does samething with QPixmap::loadFromData which I've already told.

When you want to create an image from raw data, you have to create the QImage instance with the constructor which takes raw data, like:
QImage image(nativeBuffer, nativeWidth, nativeHeight, QImage::Format_RGB32);

[Mauro]

Yes! Thanks, it works... it's just very inefficient: lots of frame drops if I set vlc video format at 1080x1020 and print the image on a large screen.
Do you have any clue for speeding up the process?
I'll try with a QGLWidget and see if I get better results.

What about integrating SDL surfaces or OpenCV with Qt?
My basic need is to render the same frame on more surfaces as I need to manipulate them differently.

Thanks again!

[Mauro]

This works fine:

Code: Select all

QImage image(nativeBuffer, nativeWidth, nativeHeight, QImage::Format_ARGB32); // QImage::Format_RGB32 works fine as well and it's much faster

where nativeBuffer points to the raw RV32 data from VLC.

However, I'm trying to build a QPixmap directly (without having to instantiate a QImage) from the raw data, therefore I added a BMP header before the raw data, and then tried

Code: Select all

_nativePixmap.loadFromData(nativeBuffer, nativeWidth*nativeHeight*4, "BMP");

but I only get the first frame (correct, but upside-down) and nothing more, even audio stops.

Any idea why?

This is the code to write the BMP Header, the raw data starts "imageDataOffset" bytes from the start of the buffer:

Code: Select all

void writeBMPHeader (unsigned char* buffer, unsigned int width, unsigned int height, unsigned int imageDataOffset) { BITMAPFILEHEADER fileHeader; BITMAPINFOHEADER infoHeader; memset ( &fileHeader, 0, sizeof (BITMAPFILEHEADER ) ); memset ( &infoHeader, 0, sizeof (BITMAPINFOHEADER ) ); fileHeader.bfType = 0x4d42; fileHeader.bfReserved1 = 0; fileHeader.bfReserved2 = 0; fileHeader.bfSize = width*height*4 + imageDataOffset; fileHeader.bfOffBits = imageDataOffset; infoHeader.biSize = sizeof(BITMAPINFOHEADER); infoHeader.biWidth = width; infoHeader.biHeight = height; infoHeader.biPlanes = 1; infoHeader.biBitCount = 32; infoHeader.biCompression = 0; infoHeader.biSizeImage = width*height*4; infoHeader.biXPelsPerMeter = 0x0ec4; infoHeader.biYPelsPerMeter = 0x0ec4; infoHeader.biClrUsed = 0; infoHeader.biClrImportant = 0; memcpy( buffer, &fileHeader, sizeof(BITMAPFILEHEADER) ); memcpy( buffer + sizeof(BITMAPFILEHEADER), &infoHeader, sizeof(BITMAPINFOHEADER) ); }

Thanks for any help