VLCKit howto : GPU to identify patterns within RTSP video stream

[phlasserre]

Hello,
I'm working on a MacOS application that captures an RTSP video stream, thru VLCKit.
This part works fine (MacOS 10.14.6, XCode 11.0, Swift 5.1): perfect, thanks to all ! (just 3secs latency I'd like to fix, but not too much of a concern for my app).

Now I need to identify some patterns within the real-time video stream, then return a rectangle of the area (possibly several rectangles for multiple zones) where patterns have been detected (ie: return a list of rectangles, with pattern signature for each).
I plan to use the GPU to handle this task.
So I need to get access to the buffers used by VLC just after decoding the RTSP stream, in parallel buffer content are sent to the displayed view (... or not) .
Note: the video stream will not always be visible on the computer's screen, only when a specific window, rendering the camera stream, is made visible.

Questions :
- does VLC / GPU continue the decoding of an incoming RTSP stream in background, even if video stream is not displayed ?
- Could someone provide me with some links (wiki, forum, ...) or recommendations related to this design (use GPU to identify patterns within RTSP video stream)?

Thanks in advance !

[unidan]

Hi

I'm working on a MacOS application that captures an RTSP video stream, thru VLCKit.
This part works fine (MacOS 10.14.6, XCode 11.0, Swift 5.1): perfect, thanks to all ! (just 3secs latency I'd like to fix, but not too much of a concern for my app).

For latency, you can check network-caching parameters, but the less you cache, the more likely you'll get buffering glitches whenever the network is not reliable. It's a tradeoff.

- does VLC / GPU continue the decoding of an incoming RTSP stream in background, even if video stream is not displayed ?

Except if you can activate a timeshift functionnality, it's a live pipeline, thus you need to make sure you're not eating most resources / time budget for this. Potentially you can do your processing only every few frame in parallel and just use VLC's callback to copy the next frame to process. Otherwise, yes it works even when not displayed.

- Could someone provide me with some links (wiki, forum, ...) or recommendations related to this design (use GPU to identify patterns within RTSP video stream)?

Identify patterns within video is more likely a CPU task, but it might be done using compute shaders if they are available on your platform. I'm not sure I know a library doing this however. But if you need this, you probably want to stay in the GPU pipeline and wait for libVLC 4.0 hardware buffer callbacks.

[phlasserre]

@unidan : Thanks for these recos :

For latency, you can check network-caching parameters

I'm playing right now with the network-caching parameters

Code: Select all

let media = VLCMedia(url: cameraStreamURL ) // with : cameraStreamURL = URL(String: "rtsp://192.168.1.50/") for example let options = [ "network-caching": 500, // trying to lower latency to 0.5 second ] as [String : Any] media.addOptions(options)

use VLC's callback to copy the next frame to process

I'm looking into the use of callbacks, but, as you said, just because I don't want to eat most ressources, I'd prefer the GPU to achieve the task (yet I'll test this idea because it is simple enough, especially if my CPU can support this task on top of the rest)

using compute shaders if they are available on your platform

On My MacBookPro, I wish I can leverage the Radeon Pro 560x GPU - also Apple offers the Metal library that allows to create shaders. I need to "connect" VLC's frame buffer to a "compute pipeline" shader that will populate the output buffer. Anyone with a recommendation/examples/or just thoughts : very much appreciated.