Delphi (pascal), video frame, sdl2, directX

[martin86]

Hello,
I made Delphi form application and I have implemented usage of libvlc.dll and sdl2.dll to get video frame.
Sdl is using lock, unlock and display function where you work with PSDL_Texture. According to my test showing video in SDL window use alot of CPU and very small amount of GPU (setting '--avcodec-hw=dxva2' didn't help at all).
If I removed the SDL window and tryed to display video in Panel by calling

Code: Select all

libvlc_media_player_set_hwnd(AMediaPlayer, Pointer(Panel1.Handle));

the CPU usage was low and the GPU was used for decoding/rendering the video. This is great but in this case I am not able to get vide frame since in this case lock, unlock and display functions were never triggered.

Code: Select all

libvlc_video_set_callbacks(AMediaPlayer, @lock, @unlock, @display, @FrameContext[iId]);

So my questions/desire:
- I want to get each video frame so I can display it with IDirect3DTexture9, IDirect3DSurface9.
- As I can see I can't get dx9 texture out of vlc directly?
- so which aproach would be the best to do this on GPU and not on CPU site?

Did I set the hardwer decoding wrong?

Code: Select all

//Before opening an instance: AVlcArgs[0] := ''; //PAnsiChar(libvlc_dynamic_dll_path); AVlcArgs[1] := '--no-video-title-show'; AVlcArgs[2] := '--no-xlib'; AVlcArgs[3] := '--no-audio'; AVlcArgs[4] := '--network-caching=200'; AVlcArgs[5] := '--rtsp-caching=100'; AVlcArgs[6] := '--sout-rtp-caching=200'; AVlcArgs[7] := '--avcodec-hw=any'; AVlcArgs[8] := '--vout=directx'; AVlcArgs[9] := '--ffmpeg-hw'; AVlcArgs[10] := '--extraintf=logger'; AVlcArgs[11] := '--verbose=2'; // AVlcArgs[12] := '--file-logging'; AVlcArgs[13] := '--logfile=vlc.log'; var p_li := libvlc_new(Length(AVlcArgs)-1, @AVlcArgs[0]); //And before play: libvlc_media_add_option(AMedia, PAnsiChar('--network-caching=200')); libvlc_media_add_option(AMedia, PAnsiChar('--rtsp-caching=100')); libvlc_media_add_option(AMedia, PAnsiChar('--sout-rtp-caching=200')); libvlc_media_add_option(AMedia, PAnsiChar('--avcodec-hw=any')); libvlc_media_add_option(AMedia, PAnsiChar('--vout=directx')); libvlc_media_add_option(AMedia, PAnsiChar('--ffmpeg-hw'));

[Rémi Denis-Courmont]

You can't have the cake and eat it. If you use hardware decoding, the frames are in GPU memory and thus not efficiently accessible from CPU.

[martin86]

You can't have the cake and eat it. If you use hardware decoding, the frames are in GPU memory and thus not efficiently accessible from CPU.

Is it possible to call
libvlc_video_set_callbacks(AMediaPlayer, @lock, @unlock, @display, @FrameContext[iId]);
without using SDL window?
And use libvlc_video_set_callbacks to create dxTexture (IDirect3DTexture9, IDirect3DSurface9)?
To minimize CPU usage?

Currently I don't know how to get vlc video frame in
function lock(opaque: Pointer; planes: PVCBPlanes): Pointer; cdecl;

My current code

Code: Select all

function lock(opaque: Pointer; planes: PVCBPlanes): Pointer; cdecl; var AContext: PFrameContext; lockedRect: D3DLOCKED_RECT; pitch: Integer; vlcPixelData: Pointer; vlcPitch: Integer; begin Result := nil; if Assigned(opaque) then begin AContext := PFrameContext(opaque); if Assigned(AContext) then begin AContext.dx9Texture.GetSurfaceLevel(0, AContext.dx9Surface); AContext.dx9Surface.LockRect(lockedRect, nil, 0); pitch := lockedRect.Pitch; vlcPixelData := planes[0]; vlcPitch := AContext.VideoWidth; AContext.dx9Surface.UnlockRect; Result := planes[0]; end; end; end;

[martin86]

Currently my last code looks like this
Before play

Code: Select all

//libvlc_media_player_set_hwnd(AMediaPlayer, Pointer(Panel1.Handle)); [b]This is not set so the lock function is triggered?[/b] libvlc_video_set_callbacks(AMediaPlayer, @lock, @unlock, @display, @FrameContext[iId]); libvlc_video_set_format(AMediaPlayer, 'BGRA', FrameContext[iId].VideoWidth, FrameContext[iId].VideoHeight, FrameContext[iId].VideoWidth * 4); libvlc_video_set_format_callbacks(AMediaPlayer, @setup, @cleanup); end; if (libvlc_media_player_is_playing(FMediaPlayerList[iId]) = 0) then libvlc_media_player_play(FMediaPlayerList[iId]);

And my lock function

Code: Select all

function lock(opaque: Pointer; planes: PVCBPlanes): Pointer; cdecl; var AContext: PFrameContext; lockedRect: D3DLOCKED_RECT; pitch: Integer; vlcPixelData: Pointer; x, y: Integer; begin Result := nil; if Assigned(opaque) then begin AContext := PFrameContext(opaque); if Assigned(AContext) then begin AContext.dx9Texture.GetSurfaceLevel(0, AContext.dx9Surface); AContext.dx9Surface.LockRect(lockedRect, nil, 0); pitch := lockedRect.Pitch; vlcPixelData := planes[0]; //[b]I don't know if this is right[/b] if not Assigned(vlcPixelData) then Exit; for y := 0 to AContext.VideoHeight - 1 do begin for x := 0 to AContext.VideoWidth - 1 do begin PByte(Integer(lockedRect.pBits) + y * pitch + x * 4)^ := PByte(Integer(vlcPixelData) + (y * AContext.VideoWidth + x) * 4)^; // B [b]I am getting exception here. x and y value is random when exception occures[/b] PByte(Integer(lockedRect.pBits) + y * pitch + x * 4 + 1)^ := PByte(Integer(vlcPixelData) + (y * AContext.VideoWidth + x) * 4 + 1)^; // G PByte(Integer(lockedRect.pBits) + y * pitch + x * 4 + 2)^ := PByte(Integer(vlcPixelData) + (y * AContext.VideoWidth + x) * 4 + 2)^; // R PByte(Integer(lockedRect.pBits) + y * pitch + x * 4 + 3)^ := PByte(Integer(vlcPixelData) + (y * AContext.VideoWidth + x) * 4 + 3)^; // A end; end; AContext.dx9Surface.UnlockRect; Result := planes[0]; end; end; end;

I have bolded my question/note in the code.
It looks like I am not getting correct data in vlcPixelData := planes[0];
How to know if the lock function was triggered with good data, if the video frame is ok or not. How to know if I can copy the pixels in to dx9 texture?

[Rémi Denis-Courmont]

Again, you can't have it both ways. If you use GPU acceleration, you can't access the bitmaps from CPU.

[martin86]

Again, you can't have it both ways. If you use GPU acceleration, you can't access the bitmaps from CPU.

As you can see from my post I am trying to get dx9 texture (currently its not important if this will be on GPU or CPU).
Why I get exception in for loop when trying to copy vlc video frame to dx9 texture?

[mfkl]

What you are looking for is the GPU version of libvlc_video_set_callbacks, which is called libvlc_video_set_output_callbacks. You will need a nightly libvlc 4.x build. See https://videolan.videolan.me/vlc/group_ ... 775926b808

[martin86]

What you are looking for is the GPU version of libvlc_video_set_callbacks, which is called libvlc_video_set_output_callbacks. You will need a nightly libvlc 4.x build. See https://videolan.videolan.me/vlc/group_ ... 775926b808

thanks, that looks promising

[martin86]

What you are looking for is the GPU version of libvlc_video_set_callbacks, which is called libvlc_video_set_output_callbacks. You will need a nightly libvlc 4.x build. See https://videolan.videolan.me/vlc/group_ ... 775926b808

I tryed to play video with this 4.0 libvlc but I get nil in my line (p_li is nil)
var p_li := libvlc_new(Length(AVlcArgs)-1, @AVlcArgs[0]);

Are there some examples how to use this 4.0? How to initialize and set everything before playing the video?

[martin86]

Currently with version 4.0 I am not able to play the video. in log it looks like something is corrupted
AMedia := libvlc_media_new_path(libvlcInstance, PAnsiChar(AnsiString('./videoLobby.mkv')));

main debug: using audio output module "mmdevice"
main debug: keeping audio output
main debug: Creating an input for '??'
main debug: using timeshift granularity of 50 MiB
main debug: `file:///I:/VLC/VLC_4/Win32/Debug/%10%A0%A1%01%01' gives access `file' demux `any' path `/I:/VLC/VLC_4/Win32/Debug/%10%A0%A1%01%01'
main debug: creating access: file:///I:/VLC/VLC_4/Win32/Debug/%10%A0%A1%01%01
main debug: (path: I:\VLC\VLC_4\Win32\Debug\ ¡)
main debug: looking for access module matching "file": 5 candidates
dvdnav error: Could not open I:\VLC\VLC_4\Win32\Debug\ ¡ with libdvdcss.
dvdnav error: Can't open I:\VLC\VLC_4\Win32\Debug\ ¡ for reading
dvdnav error: vm: failed to open/read the DVD
dvdnav warning: cannot open DVD (I:\VLC\VLC_4\Win32\Debug\ ¡)
filesystem error: cannot open file I:\VLC\VLC_4\Win32\Debug

if I use libvlc_media_new_location its the same but nothing in log file only in console
Debug Output:
main input error: Could not open MRL »??«.