stream lag in HLS

[peac]

Hello,

I'm having issues correctly implementing HLS. When my stream is read by an html video element, it seems to playback below normal speed, and catches up every two seconds (which is NOT the segment duration). I'm under the impression it has something to do with the mpeg-ts keyframes.

The first step happens on a raspberry pi. It has hardware encoding for h264, the load is near 0. This step uses ffmpeg to read the video from a webcam and encode it in h264 using the onboard gpu. It is then passed to cvlc for remote access.

Code: Select all

ffmpeg \ -f video4linux2 \ -input_format yuyv422 -framerate 30 -video_size 640x480 \ -i /dev/video0 \ -c:v h264_v4l2m2m -b:v 600k -pix_fmt yuv420p -vf scale=640:480 -g 30 \ -f h264 \ - \ | cvlc \ --no-audio \ --aout=dummy \ stream:///dev/stdin \ --sout '#standard{access=http,mux=ts,dst=0.0.0.0:8060}' \ :no-sout-all \ :sout-keep \ :demux=h264

when reading this first stream with vlc, i don't get any issue. This first stream cannot be accessed by a web browser though, as nothing in a web browser can actually read a h264 mpeg-ts stream over http. I also need to get this out to a server over VPN only once, as many clients will be connected to watch the stream from outside this pi's local network. So on the server, i hit this rpi over the VPN at address $HOST, to read the h264 mpeg-ts http stream and transform it in a HLS stream.

Code: Select all

cvlc \ $HOST \ --sout "#std{access=livehttp{seglen=10,delsegs=true,numsegs=3,index=$STREAM/stream.m3u8,index-url=/stream/$STREAM/stream-########.ts},mux=ts{use-key-frames},dst=$STREAM/stream-########.ts}" \ :sout-keep

this stream can be accessed here:

Code: Select all

https://global2024.distrifab.fr/stream/6d18-stream1/stream.m3u8

it is served by the fast webserver caddy. Opening this stream with an html video element reveals my issue. It can be observed in a testpage i've made here

Code: Select all

https://global2024.distrifab.fr/video-js.html

and in another HLS player's tester here with the same defects

Code: Select all

https://bitmovin.com/demos/stream-test?format=hls&manifest=https%3A%2F%2Fglobal2024.distrifab.fr%2Fstream%2F6d18-stream1%2Fstream.m3u8

I've tried video.js and hls.js libraries with always the same results.

Playing the m3u8 stream directly in VLC does work without the lagging issue, but VLC regularly says:

Code: Select all

http demux error: local stream 217 error: Cancellation (0x8)

any help or pointers would be appreciated. I'm also on the IRC channel under the same name.

EDIT1: I've just noticed something: When I open the stream with VLC, the first 2 seconds are replaying slow, then it jumps once and then the replay is good. That may corroborate the keyframe issue, as a possible explanation would be that on the first keyframe, VLC is smart enough to understand the right playback speed and adjusts, where the html video element doesn't.

[peac]

So I found the root issue and the root issue is not with VLC, but maybe VLC could help fixing it.

The root issue is that the h264_v4l2m2m codec used by ffmpeg (same goes for v4l2h264enc in gstreamer) fails to encode keyframes correctly. I noticed that when I seen the error in ffmpeg saying "Failed to set gop size: Invalid argument". However h264 is the only format that I can encode with hardware on rpi, and I can't afford the CPU resource on the rpi to software encode.

The encoded stream has malformed keyframes, but the data in the stream is usable, as VLC reads it correctly once the second keyframe is met.

This gave me the idea to read the stream on the server, and instead of directly splitting it into HLS files, first decode it and re-encode it using libx264, using the CPU resource of the server instead of the one of the rpi. And this patch solution works, ffmpeg reads the broken stream, reencodes it properly and chops it into HLS, no more playback issues with this command running on the server:

Code: Select all

ffmpeg \ -i $HOST \ -c:v libx264 \ -b:v 600k \ -g 30 \ -an \ -f hls \ -hls_time 10 \ -hls_list_size 3 \ -hls_flags delete_segments \ -hls_segment_filename "$STREAM/segment_%03d.ts" \ $STREAM/stream.m3u8

However, with only 5 of the 16 streams I need to handle, this reencoding step on the server already takes 60% of the CPU.

Now here's where VLC could save me. I know that VLC is smart enough to understand that the stream is malformed, and still play it correctly, as seen by reading the broken stream with "vlc $HOST". The first 2 seconds are lagging, but then it "calibrates" and the rest plays normally, all without reencoding AFAIU.

Is there a cvlc command I can use to replace this ffmpeg command, that will unpack the stream, do its smart magic, and repack the stream with properly formed keyframes, without transcoding?

EDIT1: I believe the problem with keyframes might be due to the mismatch in metadata. The metadata indicates keyframes should appear every second (30 frames), but v4lm2m seems to generate them every 2 seconds (60 frames). This would cause the playback to proceed at half-speed and skip ahead a second every 2 seconds when a new keyframe is encountered.

EDIT2: For debug purposes I've reexposed the malformed stream from the rpi.

It is generated by this command on the pi:

Code: Select all

ffmpeg \ -f video4linux2 \ -i /dev/video0 \ -c:v h264_v4l2m2m \ -b:v 600k \ -vf scale=640:480 \ -f h264 \ - \ | cvlc \ --aout=dummy \ --no-sout-audio \ stream:///dev/stdin \ --sout '#standard{access=http,mux=ts,dst=0.0.0.0:8060}' \ :sout-keep \ :demux=h264

it is then exposed by the server with this command at http://iostud.io:8090 (can be played with vlc)

Code: Select all

cvlc \ $HOST \ --sout '#standard{access=http,mux=ts,dst=0.0.0.0:8090}' \ :sout-keep

and shown as HLS (with the lag issue) at http://iostud.io:8060/ (right click and play if you're on firefox)

Code: Select all

cvlc \ $HOST \ --sout "#std{access=livehttp{seglen=2,delsegs=true,numsegs=4,index=./stream.m3u8},mux=ts{use-key-frames},dst=./stream-########.ts}" \ :sout-keep

[Rémi Denis-Courmont]

VLC does not currently support V4L2 m2m. Patches are welcome.

(You can stream V4L2 h264 if the camera can do the encoding itself, but that does not seem to be your case.)

[peac]

Hello Rémi, thank you for your response.

ffmpeg is already doing the V4L2 m2m reading and encoding to h264, although for some reason it does not do it correctly. Patching this in VLC is unfortunately way above my skills.

I have 2 questions:
1. How can I understand exactly what of the resulting stream is invalid? (available to read or probe with `vlc http://iostud.io:8090`)
2. Given that VLC understands and somehow fixes this invalid stream, can I use cvlc to repack the stream to a valid one, without using x264 to fully decode and re-encode on CPU?

Thank you

[peac]

After some more research, I believe that the stream's timestamps are messed up, but I'm not sure exactly how to verify this.

When passing `-fps_mode cfr` to ffmpeg (Frames will be duplicated and dropped to achieve exactly the requested constant frame rate) I see the same behaviour as when not passing anything, which is that VLC displays a lag for the first couple of seconds, then understands the actual frame rate and displays smoothly.

When passing `-fps_mode passthrough` (Each frame is passed with its timestamp from the demuxer to the muxer) or `-fps_mode vfr` (Frames are passed through with their timestamp or dropped so as to prevent 2 frames from having the same timestamp), VLC actually displays the video lag all along.

I am guessing that when VLC detects timestamp issues in the stream, it discards the frames' timestamps and displays non-duplicated frames at the detected framerate.

I've studied tens of locked forum threads and github issues, asked on many IRC and Matrix channels, on raspberrypi, ffmpeg, gstreamer, and found no help as of now. Any pointers how to debug this would be greatly appreciated.

My best bet is still to use cvlc to repackage the stream to fix it.

Is there a cvlc command that I can use to read the input stream, and create a new output stream that is made from what would be shown by the VLC player window?

Thank you