How to achieve low latency in H.264 live streaming?
Posted: 01 May 2023 14:43
Hi. I'm trying to do near-real-time streaming with libVLC, but have failed miserably so far.
My setting:
- Streaming H.264 120 fps from Android Pixel 7 to Windows 11 WPF application
- LibVLCSharp, LibVLCSharp.WPF: 4.0.0-alpha-20230428-6729
- VideoLAN.LibVLC.Windows: 4.0.0-alpha-20230403
- Current latency: 300-400 ms
I configure LibVLC this way. I had high hopes for low-delay (that's why i switched from version 3 to 4 alpha) but unfortunately it doesn't change much:
I feed VLC with a StreamMediaInput and provided a stream:
Incoming chunks are NALUs, with one IDR NALU per second. Incoming NALUs are made available to VLC immediately, if possible always in whole NALUs (if the requested data size allows it). VLC fetches the NALUs immediately, so the list typically does not contain more than 1 element (at most 2).
Nevertheless, the high latency remains. But what really drives me crazy is that I know the latency could be much lower. How do I know? If I set the fps to 130 (the stream still only has 120 fps), then I see phases (up to 5 seconds) of an almost live stream with a latency of about/below 100 ms. This phase then alternate cyclically with phases with higher latency. Of course, this approach has negative effects, so it leads to jumps in the presentation. But it is clear to see that VLC is capable of providing lower latency. But I don't see any way to influence this
I would be very happy about any hints 
My setting:
- Streaming H.264 120 fps from Android Pixel 7 to Windows 11 WPF application
- LibVLCSharp, LibVLCSharp.WPF: 4.0.0-alpha-20230428-6729
- VideoLAN.LibVLC.Windows: 4.0.0-alpha-20230403
- Current latency: 300-400 ms
I configure LibVLC this way. I had high hopes for low-delay (that's why i switched from version 3 to 4 alpha) but unfortunately it doesn't change much:
Code: Select all
_LibVLC = new LibVLC(enableDebugLogs: false
,"--demux=h264"
,$"--h264-fps={fps}"
, "--low-delay"
, "--high-priority"
, "--clock-jitter=0"
, "--clock-synchro=0"
, "--network-caching=0"
, "--live-caching=0"
);I feed VLC with a StreamMediaInput and provided a stream:
Code: Select all
public int Read(byte[] buffer, int offset, int count)
{
_ManualResetEvent.WaitOne();
int bytesToRead = 0;
lock (_Lock)
{
bytesToRead = _AvailableChunks.First();
_AvailableChunks.RemoveFirst();
if (bytesToRead > count)
{
int bytesRemaining = bytesToRead - count;
bytesToRead = count;
_AvailableChunks.AddFirst(bytesRemaining);
}
}
int readSize = _DataProvider.ReadNewChunkData(buffer, offset, bytesToRead);
if (!_AvailableChunks.Any())
{
_ManualResetEvent.Reset();
}
return readSize;
}
private void NewChunkArrived(int size)
{
lock (_Lock)
{
_AvailableChunks.AddLast(size);
_ManualResetEvent.Set();
}
}Nevertheless, the high latency remains. But what really drives me crazy is that I know the latency could be much lower. How do I know? If I set the fps to 130 (the stream still only has 120 fps), then I see phases (up to 5 seconds) of an almost live stream with a latency of about/below 100 ms. This phase then alternate cyclically with phases with higher latency. Of course, this approach has negative effects, so it leads to jumps in the presentation. But it is clear to see that VLC is capable of providing lower latency. But I don't see any way to influence this
I would be very happy about any hints 
