LibVLCSharp.WPF png overlay and snapshot

[manity]

Hello,

I have got as far as using LibVLCSharp.WPF to play a video file. So its a c# WPF project
I would like to ask

1. is there a way to overlay one or more png files with alpha transparency onto the video playback, in such a way that the image file resizes with any video window resize.
Kind of like a HUD display. But I would like to overlay multiple files ontop of the video and add them and remove them at anytime during playback without dropping frames.
The typical size would be for video at 1920x1080 h264.
I was thinking there might be a render frame method or event i could override that exposes the frame image and i could write the bitmaps at that point.
Or perhaps its possible to get the video HWND and write to the graphics object in some way.

2. also i want to be able to take a snapshot with out interupting playback and save the image on the fly to include or exclude any image overlay it might have depending on a user preference.

Note that i know the VLC player has an overlay/logo option but in testing that with the player when i assigned a 1920x1080 png image it did show but the video playback stuttered constatnly and it was playing like at 10 frames per sec.


any help appreciated.

thanks

[Rémi Denis-Courmont]

LibVLC can render into a window of your choosing, and you should then be able to overlay whatever on top with another window - unless old-style overlay is used.

I cannot speak for LibVLC#.

[manity]

@ Rémi Denis-Courmont

Thanks, I guess that is an option, but I wanted to "write" it to the video frame so that when take snapshot then the overlay included.
Is there no "onrender" event of some kind that I can access and extract a frame of the video or write to a frame of the video ?

thanks

[mfkl]

I was thinking there might be a render frame method or event i could override that exposes the frame image and i could write the bitmaps at that point.

There is something like it, not sure if you can achieve what you want with it though https://code.videolan.org/videolan/LibV ... r.cs#L1215
Beware performance is not optimal with this method hence it's discouraged. Read this for more info on implementation and usage: https://github.com/ZeBobo5/Vlc.DotNet/b ... rovider.cs

I'd also have a look at this code if I were you https://code.videolan.org/videolan/LibV ... ow.xaml.cs, maybe you can achieve what you want by modifying it.

perhaps its possible to get the video HWND

https://code.videolan.org/videolan/LibV ... er.cs#L782

i want to be able to take a snapshot

https://code.videolan.org/videolan/LibV ... r.cs#L1453

[manity]

Thanks for all that information.
Given the adverse consequences possible with writing to the buffer I decided I would just try to grab a copy of the frame buffer on a
button click and merge it with an overlaid control image but I admit its currently beyond me to get it working.

I have a basic wpf app running that loads and plays a video in the window onload event in the VideoView control. that works
and I have this rather sorry attempt cobbled together from reading the links supplied (see below)


So this builds and solved my out of scope error I kept getting. then I run it...
But so far it just hits a breakpoint in LocKVideo and pauses for a little and never hits the break point in Display video at all.
then exits with a error "xxxx.exe' has exited with code -1073741819 (0xc0000005) 'Access violation'. xxxx= my solution name

Any chance there is a complete code sample I could use as a starting point that works to copy a frame out of the buffer into a bitmap of some kind.

and how is it triggered ? can I trigger it on a button click or is this callback going to fire with every rendered frame ?
Also is just copying the frames out going to disable graphics hardware rendering and put the load on the cpu like was warned for attempts at writing ?

Any help appreciated to get me started.



private MediaPlayer.LibVLCVideoLockCb testLock;
private MediaPlayer.LibVLCVideoDisplayCb testDisplay;

in private void VideoView_Loaded(object sender, RoutedEventArgs e)
I have
testLock = LockVideo;
testDisplay = DisplayVideo;

_mediaPlayer.SetVideoCallbacks(testLock, null, testDisplay);


follow by

private IntPtr LockVideo(IntPtr userdata, IntPtr planes)
{
Marshal.WriteIntPtr(planes, userdata);
return userdata;
}
private void DisplayVideo(IntPtr userdata, IntPtr picture)
{
}

[mfkl]

Without your full code it's hard to assist.

Any chance there is a complete code sample I could use as a starting point that works to copy a frame out of the buffer into a bitmap of some kind.

This code https://github.com/ZeBobo5/Vlc.DotNet/b ... rovider.cs. It's based on Vlc.DotNet but C# APIs are very similar for this thing.

is this callback going to fire with every rendered frame ?

Yes

Also is just copying the frames out going to disable graphics hardware rendering and put the load on the cpu like was warned for attempts at writing ?

Yes. Which is why it's not recommended (or used in the official LibVLCSharp WinForms/WPF controls)

[manity]

It seems you are saying that even just setting up and getting the callbacks to fire in the LibVLCSharp control results in disabling the graphics hardware acceleration is this correct ?

if yes then

1. The callbacks are available in the base libvlc, does the same disabling of render hardware occur if done at this level with c/c++ ?

2.is there anyway to get access to read only the data and extract a frame (on a user button click) from the hardware buffer even at a low level without affecting performance.

3. As an alternative is it possible to get the standard snapshot method to return the snapshot as a byte array or memory stream instead of specifying a file path and having it save to HDD ?

thanks for your advice.

[mfkl]

It seems you are saying that even just setting up and getting the callbacks to fire in the LibVLCSharp control results in disabling the graphics hardware acceleration is this correct ?

Yes.

1. The callbacks are available in the base libvlc, does the same disabling of render hardware occur if done at this level with c/c++ ?

Yes.

2.is there anyway to get access to read only the data and extract a frame (on a user button click) from the hardware buffer even at a low level without affecting performance.

Not currently, there will be in libvlc 4.0.

3. As an alternative is it possible to get the standard snapshot method to return the snapshot as a byte array or memory stream instead of specifying a file path and having it save to HDD ?

No. See https://www.videolan.org/developers/vlc ... b2d5a59143