How to achieve low latency in H.264 live streaming?

[rk258]

Hi. I'm trying to do near-real-time streaming with libVLC, but have failed miserably so far.

My setting:
- Streaming H.264 120 fps from Android Pixel 7 to Windows 11 WPF application
- LibVLCSharp, LibVLCSharp.WPF: 4.0.0-alpha-20230428-6729
- VideoLAN.LibVLC.Windows: 4.0.0-alpha-20230403
- Current latency: 300-400 ms

I configure LibVLC this way. I had high hopes for low-delay (that's why i switched from version 3 to 4 alpha) but unfortunately it doesn't change much:

Code: Select all

_LibVLC = new LibVLC(enableDebugLogs: false ,"--demux=h264" ,$"--h264-fps={fps}" , "--low-delay" , "--high-priority" , "--clock-jitter=0" , "--clock-synchro=0" , "--network-caching=0" , "--live-caching=0" );

I feed VLC with a StreamMediaInput and provided a stream:

Code: Select all

public int Read(byte[] buffer, int offset, int count) { _ManualResetEvent.WaitOne(); int bytesToRead = 0; lock (_Lock) { bytesToRead = _AvailableChunks.First(); _AvailableChunks.RemoveFirst(); if (bytesToRead > count) { int bytesRemaining = bytesToRead - count; bytesToRead = count; _AvailableChunks.AddFirst(bytesRemaining); } } int readSize = _DataProvider.ReadNewChunkData(buffer, offset, bytesToRead); if (!_AvailableChunks.Any()) { _ManualResetEvent.Reset(); } return readSize; } private void NewChunkArrived(int size) { lock (_Lock) { _AvailableChunks.AddLast(size); _ManualResetEvent.Set(); } }

Incoming chunks are NALUs, with one IDR NALU per second. Incoming NALUs are made available to VLC immediately, if possible always in whole NALUs (if the requested data size allows it). VLC fetches the NALUs immediately, so the list typically does not contain more than 1 element (at most 2).

Nevertheless, the high latency remains. But what really drives me crazy is that I know the latency could be much lower. How do I know? If I set the fps to 130 (the stream still only has 120 fps), then I see phases (up to 5 seconds) of an almost live stream with a latency of about/below 100 ms. This phase then alternate cyclically with phases with higher latency. Of course, this approach has negative effects, so it leads to jumps in the presentation. But it is clear to see that VLC is capable of providing lower latency. But I don't see any way to influence this I would be very happy about any hints

[mfkl]

Does the VLC app (either 3.x stable or 4.x nightly) do a better job?

[rk258]

Good idea! I tried it with VLC 3.0.18 via network stream tcp/h264://192.168.1.2:55555

VLC shows the same latency (maybe even slightly worse) than my implementation. In addition to my first post I made some measurements to make some quantitative statements (i underestimated the latency a bit). For this i recorded a running clock that is again visible in the stream and compared both

The latency is 500 ms. The only option that really helps noticeably is ":high-priority". If i specify 130 fps (instead of the real 120) the latency starts at 500 ms and then goes down to 200 ms. Then a short delay, and it starts again at 500 ms.

At this point we can therefore draw the following conclusions:
- the delay of the pre-VLC processing chain (camera sensor -> Pixel h.264 hardware-based encoding -> WLAN/TCP transmission) is at most 200 ms
- the delay introduced by VLC is at least 300 ms

Is there any way to reduce this 300 ms? A "display the frames as fast as possible and leave everything else to the source" mode would be super cool. But maybe that's not technically possible for h.264 decoding. I don't understand much about it unfortunately

[mfkl]

have you tried 4.0 nightly builds?

[rk258]

I tried but could not get it to work.

I do the same as for VLC 3:
* Media -> Open Network Stream...
* setting Network URL to "tcp/h264://192.168.1.2:55555"
* VLC 4 MRL is "tcp/h264%3A//192.168.1.2%3A55555"
* on 'Play': "Your media can't be opened".

The MRL looks strange, VLC 3 has not made these changes. Is this a bug or am i doing something wrong?

(Version 20230519-0430)

[Rémi Denis-Courmont]

You can't do low latency with TCP, by design of TCP.

[rk258]

I think I got it now

You can't do low latency with TCP, by design of TCP.

Indeed a true statement. I am well aware that UDP is the better choice. But what I didn't expect is that the VLC player adds an additional artificial delay when using TCP. Probably that's what got me confused

And related to my own implementation, the mistake was to use StreamMediaInput. For stream input there is also a VLC-conditioned delay, which cannot be prevented by parameterization. Streaminput is therefore not suitable for low latency scenarios.

But fortunately there is a solution, probably the only one: Feed a UDP stream directly, without the detour of a separate package management / stream:

Code: Select all

var media = new Media("udp/h264://@:55556", FromType.FromLocation); media.AddOption(":h264-fps=120"); media.AddOption(":high-priority"); media.AddOption(":low-delay"); media.AddOption(":network-caching=0"); _Player.Media = media; _Player.Play();

Now the latency has halved to 200-250 ms, which I am absolutely satisfied with

Thank you for all the suggestions. A small idea for the further development of VLC could be to enable low latency also for stream input. For the cases where direct UDP is not possible and a stream source is used instead.