Streaming into resizable memory buffer

[callum]

The [Win64/C++/libVLC 3.0.8] app I am working on streams video URLs into a memory buffer that is subsequently used as an OpenGL texture.

It works well but I would like to add support for resizing the memory buffer as the video is playing back and everything I have tried so far fails - the only solution I have been able to come up with is starting playback from the beginning when the texture (and therefore memory buffer) is resized which isn't very useful.

I think I have to override the "libvlc_video_set_format(...)" call make when I establish a connection to the video stream but efforts to do so result in a crash and I haven't been able to get a useful stack trace to inspect.

The source is here - https://github.com/callumprentice/libvlc-glut/blob/master/src/libvlc-glut.cpp - and I broke out the code into a standalone app in a Git repo https://github.com/callumprentice/libvlc-glut that can trivially be built using CMake.

Are there any experienced libVLC developers who could point me in the direction of the additional functions to investigate?

Thank you.

[Rémi Denis-Courmont]

The only way to achieve that is to write a custom video output plugin.

[callum]

That's helpful to know I need to change direction - thank you.

[dm-helper]

I have more or less exactly the same need, although the output goes through some manipulation before being finally drawn into a resizeable window.

@callum - are you already working on a video output plugin? This seems to be the architecturally correct solution and certainly better than restarting the video with every resize. However, the barrier to entry is fairly high and I'm not sure that the cost/benefit for my project would justify doing this alone. Perhaps some collaboration?

[callum]

@dm-helper Understood - I was a bit surprised this scenario isn't more common given the obvious use cases in video games, virtual 'theaters' etc.

My work was for my company and our product team has deprioritized making it work as I'd like so I can't spend any more work time on it.

For my own edification, I'd like to make it work so I plan to chip away at it when I can - evenings and weekends, family commitments permitting - I just can't commit to anything.

Happy to share here when I make some progress - I'll use the same repo listed in my original message.

[sherington]

Do you really need to worry about resizing the buffer?

When I do stuff like this, I keep the buffer the same size as that reported for the media, and then simply scale the output to fill my window (or texture), preserving aspect ratio. By scaling the output I mean setting some operation on the graphics context to apply a scaling transform to everything that is rendered. Because I do it this way, any overlays I need to draw on top of the video also get automatically scaled correctly. When you render a frame, if the window size has changed calculate a new scaling transform.

I've done with this with Java2D, JavaFX and OpenGL and it works just fine, I don't see any scaling artefacts depending on which pixel interpolation I use.

This might not be the best solution, but it's likely to be far easier than writing a new video output plugin.

[dm-helper]

@callum - thanks, I've followed you on github. In case I come up with something, I'll ping you - but this is performance vs basic features for me right now, so also unfortunately not top priority.

@sherington:
I'm pulling the video into a Qt application and manipulating it with masking visibility of some regions and by drawing additional icons and shapes on top of it before finally showing it in a window. Ideally, I was hoping a custom video output would let me leverage HW scaling if available before these other relatively CPU-intensive activities or alternatively to combine SW scaling and masking at least if not also further drawing. I definitely agree that having a two-step SW rendering with scaling, etc on the way would be very inefficient and this is definitely something I am trying to avoid.

One option is indeed to shift the entire visualization in the application to OpenGL, but this is a longer-term solution as it requires significant rework of many interaction features.

[callum]

@sherington:

Understood and thanks - I know that's often a good solution but in my case, there are many layers of code between where the video is decoded and the buffer used as a texture and a lot of assumptions about how it works. I could always decode video into a fixed buffer size and then scale that to match the buffer that was offered to the code but that seems like it'd difficult. Agree about it being difficult to write a video output plugin though.