Broken screencasting interoperability between vlc and ffmpeg

[sampsa]

Hello,

This question is about broken interoperability between ffmpeg and vlc.

In our system, we send live screencasts using ffmpeg and receive them with vlc. Their compatibility has been degrading continuously as new versions of vlc roll out, namely, vlc is unable to read streams send by ffmpeg (not necessarily vlc's fault, as discussed below).

I am using the latest versions of both ffmpeg and vlc from their git repositories, namely

ffmpeg: N-78646-g10fa50c
vlc: 3.0.0-git Vetinari (revision 21e3fde)

Next, we will be sending screencasts both with ffmpeg (A) and vlc (B) and receiving them in ffplay and vlc. Resumen for the impatient can be found in (C) and the concrete questions in (D).


A) Sending screencast with ffmpeg command:

Code: Select all

ffmpeg -y -f x11grab -s 1920x1080 -r 25 -i :0.0 -vf "scale=960x540" -vcodec h264 -tune zerolatency -preset ultrafast -pix_fmt yuv420p -vprofile main -x264opts keyint=25:min-keyint=25 -maxrate 2000k -bufsize 2000k -f rtp rtp://127.0.0.1:50000

.. and using ffmpeg provided sdp file that is printed to stdout (lets call it "test.sdp")

Code: Select all

v=0 o=- 0 0 IN IP4 127.0.0.1 s=No Name c=IN IP4 127.0.0.1 t=0 0 a=tool:libavformat 57.26.100 m=video 50000 RTP/AVP 96 a=rtpmap:96 H264/90000 a=fmtp:96 packetization-mode=1

Now, viewing with

Code: Select all

ffplay test.sdp

there is no problem, however, with

Code: Select all

vlc -vvv --avcodec-hw vdpau --avcodec-skip-frame=1 --avcodec-hurry-up --avcodec-fast test.sdp

I start getting

...
[00007fcfbcc1dcc8] avcodec decoder warning: More than 4 late frames, dropping frame
...
[00007fcfbcc1dcc8] avcodec decoder error: more than 5 seconds of late video -> dropping frame (computer too slow ?)

After getting those errors, vlc does not recover the stream anymore and is stuck forever. Disabling hardware acceleration has no effect.


B) Sending screencast with vlc:

Code: Select all

vlc screen:// --avcodec-hw vdpau -vvv --sout-rtp-sdp="file:///home/sampsa/test.sdp" --sout '#transcode{width=960,height=540,vcodec=h264,acodec=none,vb=1024,mux=ts,fps=25}:rtp{dst=127.0.0.1,port=50000}'

This produces an sdp file ("test.sdp"):

Code: Select all

v=0 o=- 15742491380871292271 15742491380871292271 IN IP4 sampsa-xps13 s=Unnamed i=N/A c=IN IP4 127.0.0.1 t=0 0 a=tool:vlc 3.0.0-git a=recvonly a=type:broadcast a=charset:UTF-8 m=video 50000 RTP/AVP 96 b=AS:1024 b=RR:0 a=rtpmap:96 H264/90000 a=fmtp:96 packetization-mode=1;profile-level-id=42c01f;sprop-parameter-sets=Z0LAH9oDwEX7wEQAAAMABAAAAwDKPGDKgA==,aM48gA==;

Viewing that with ffplay

Code: Select all

ffplay test.sdp

gives no problems, neither

Code: Select all

vlc -vvv --avcodec-hw vdpau --avcodec-skip-frame=1 --avcodec-hurry-up --avcodec-fast test.sdp

No delayed frames, stuck image, etc.


C) So, resuming:

ffmpeg sends stream that can only be reproduced by ffplay
vlc sends stream that can be reproduced by both ffplay and vlc


D) The questions are:

- Is this a bug either in ffmpeg or in vlc?
- Is the simplistic .sdp file produced by ffmpeg the culprit?
- Does ffmpeg send corrupted timestamps?
- .. In that case, is vlc too picky with the received timestamps? Can such behaviour be disabled?
- Why vlc can't recover from that ("more than 5 seconds of late video") situation? (I must have been ranting about that for 1+ years).

Regards,

Sampsa

P. S. I have cc'd this message to the ffmpeg forum as well:
http://ffmpeg.gusari.org/viewtopic.php?f=11&t=2698

[Rémi Denis-Courmont]

VLC playback is pretty much all or nothing, due to the relatively high computational cost of decoding I-frames. That's why VLC cannot recover, or rather why nobody ever cared to write mostly pointless code to that effect.

[sampsa]

Thanks for the reply.

- So, if vlc comes to the conclusion that it's unable to decode the (i) frames at the required pace it gives up. Makes sense.
- .. but I am convinced that it's not the problem here (ffplay does ok job in decoding ffmpeg rtp muxed streams)
- .. could such behaviour ("more than 5 seconds late video") result because vlc is receiving incorrect timestamps with the rtp packets? .. i.e., the stream vlc is receiving is ok, it could be decoded in time, but just the timestamps are wrong/corrupted?
- Trying to get an earlier version of vlc (2.2.1) to work with ffmpeg rtp muxed audio, resulted in:

[00007faeb0c1dbd8] core decoder error: Could not convert timestamp 2777164513959325

.. but in the latest vlc version this does not occur. I guess someone fixed this and made vlc more tolerant towards corrupted timestamps for rtp muxed audio.

[sampsa]

I end this monologue here (I found the solution).

Correct ffmpeg command is:

Code: Select all

ffmpeg -y -f x11grab -s 1920x1080 -r 25 -i :0.0 -vf "scale=960x540" -vcodec h264 -tune zerolatency -preset ultrafast -pix_fmt yuv420p -vprofile main -x264opts keyint=25:min-keyint=25 -maxrate 2000k -bufsize 2000k -f rtp udp://127.0.0.1:50000

Note in the end
-f rtp udp://
instead of
-f rtp rtp://

So we need to send muxed as rtp, but just as udp packets .. my guess is that when sending with rtp protocol it tries to do rtcp and that mixes things up..?

Sorry for the spam!