Problem with sout/packetizer for recent french DVB-T streams

[kmf31]

When I try to record a DVB-T stream from a HD-channel of french DTV (called "TNT-HD") with vlc using a standard sout option such as:

--sout="#standard{mux=ts,dst=outfile.ts,access=file}"

it happens quite often (depending on the kind of flux, channel etc., see below for details) that the resulting file is either strongly corrupted or a least partly corrupted with macro-blocks etc. and that despite the fact that the very same stream works perfectly without any errors if viewed directly by vlc or if it is dumped into a file using the dump option "--ts-dump-file=outdump.ts" (for the vlc-2.0.x branch) or "--demux=dump --demuxdump-file=outdump.ts" (for the vlc-2.2.x branch). During the record (with --sout=... and not with dump) one gets quite a number of error messages (visible with the "-vvv" option) of the type:

[0x7f9b3c001f28] main stream output warning: trying to send non-dated packet to stream output!
[0x7f9b3c001f28] main stream output warning: trying to send non-dated packet to stream output!
[0x7f9b3c001f28] main stream output warning: trying to send non-dated packet to stream output!
[0x7f9b3c001f28] main stream output warning: trying to send non-dated packet to stream output!
[0x7f9b3c001f28] main stream output warning: trying to send non-dated packet to stream output!

which indicates that apparently there are many h264-packets with zero or invalid dts/pts timestamps which a removed by the sout module of vlc (therefore producing the corruption/macroblocks) while with direct viewing (or dump-record) these packets remain present and one does not see errors.

First of all, personally I have found a satisfactory solution/workaround (for me, also see below) so that this post is not about solving "a problem for me" but really to remind that there is in my opinion a quite serious vlc-bug (or vlc-problem related to maybe somewhat "incorrect" DVB-streams used in french TV) especially since in France since april 5, 2016, the type of signal changed and this bug (for HD channels with h264) is now really obvious for most channels and very often while before april 5 there were still many channels in SD-mpeg2 (without this problem) and also for the former HD channels TF1, FR2, M6 and Arte there seemed to be no real problem. However, even before april 5 also the "additional" more recent HD-channels (HD1, Equipe 21, 6ter, N23, RMC, Cherie 25) seemed to have the same problem but still more rarely than now (I noticed it for N23 for certain type of broadcasts). Before april 5 there were 3 HD channels per multiplex (of about 21 Mb/s => 7-7.5 Mb/s average bitrate per channel) while now there are 4-5 HD channels per multiplex (with about 3-5 Mb/s average bitrate per channel) and I suppose the stronger compression is also partly responsable for the appearance of the problem but this seems to be more complicated than that.


Even though I did most of my testing with vlc-2.0.9 (a version with which I am quite happy and which works better for me for other things) I also verified vlc-2.2.1, vlc-2.2.2 and a rather recent nightly-build for vlc-2.2.3 (a few days old) and the problem is exactly the same in all versions. I have done numerous testings for many things such as vlc-parameters (especially all caching values, also keyframe, shaping parameter or dts-delay for ts-mux for output etc.) which did nothing at all to reduce the problem. I did my testing on a Linux PC (opensuse 12.1) with two Avermdia A800 cards but also using the DVB-T-tuner of my Freebox to which I can access by an rtsp-stream from the Freebox to the PC (=> exactly the same problem also for this case). Therefore I don't think that this is a hardware problem, neither a particular problem for Linux. Furthermore this is not a problem of strength or quality of my antenna signal (which is really very good). As already mentionned with direct viewing there is no problem at all and the PC has no cpu-problem at all with the h264-decoding (I5 CPU with 4 cores, nvidia card with working acceleration etc.).
I have also tested with udp-streaming, for example using:

--sout="#standard{mux=ts,dst=224.0.0.1:2000,access=udp}"

and then either record or view the udp-stream with another vlc-instance which produces exactly the same problem.

Furthermore I have tried different output-mux options, e.g. mux=ps (in this case the audio for eac3 does not work but one can still test the video) or
with the ffmpeg ts-muxer (using "mux=ffmpeg{mux=mpegts}" in sout) and the problem remains present in all cases so this is not a bug in the ts-muxer of vlc but it is indeed before in the sout/packetizer unit (for h264) in vlc.


Type of flux producing the problem
The problem does not appear for every broadcast/channel. For example FR3 (which has a fixed bitrate and does not participate to the statistical encoding because of different regional editions) has no problem at all.
For the other channels it depends on the kind of broadcast:
1) movies or series (typical american movies/series) => no big problems, i.e. "good quality" h264-flux
2) commercials, typically also quite okay but sometimes they may be some problems
3) own production from the french TV channels, local documentaries (e.g. "about police work" etc.), reality TV of all kind, talk shows, => VERY big problems,
4) live broadcast of sport or concerts => also typically VERY big problems,
5) channels with less average bitrate (e.g. ARTE or FR4 with about 3-3.5 Mb/s only !) seem to be more likely to have this problem even with movies/series when there are fast moving action scenes etc.

In case 1), sometimes the channel may add for a few seconds an additional window with some blabla-commercial information inside a movie etc. which may produce some perturbation as well (but more modest pertubation).

"Solution", "Workaround"
In my opinion this is an old bug already discussed in this old topic:
https://forum.videolan.org/viewtopic.php?f=14&t=76283
and with an entry in the bug-tracker:
https://trac.videolan.org/vlc/ticket/6539

Unfortunately there seems to be no official avancement for this bug (maybe due to lack of useful test.ts files for debugging, which I can provide if it is useful). However, the explanations in the above topic and bugtracker entry are quite clear: some h264-packets don't have correct dts/pts-timestamps and vlc discards them injustly.
Therefore I tried to comment out in the file: modules/packetizer/packetizer_helper.h this part (lines 186-192 for vlc-2.0.9):

/*
if( b_used_ts )
{
p_block_bytestream->i_dts = VLC_TS_INVALID;
p_block_bytestream->i_pts = VLC_TS_INVALID;
}
*/

which marks these packets as invalid and to recompile vlc with this modification (the same modification can also be applied to vlc-2.2.2 etc.).
The resulting/patched version of vlc seems to (nearly) solve the problem:
For fluxes with "small problems" before => no problem at all.
For fluxes with "big problems" (reality TV etc.) before => some "small glitches" (short apperances of macro-blocks) but now the recorded file plays rather nicely while before it was simply impossible to view ...

Actually the "small glitches" appear only if I record directly from dvb to file, e.g. by a command
as:

vlc_patched --programs=... (many dvb and other options) dvb:// --sout="#standard{mux=ts,dst=outfile.ts,access=file}"

However, if I produce first a dump file (of the full multiplex with all channels !) as:

vlc --programs=... (many dvb and other options) dvb:// --ts-dump-file=raw.ts

and then do the demuxing in a second step:

vlc_patched --programs=... (some other options) raw.ts --sout="#standard{mux=ts,dst=outfile.ts,access=file}"

then the resulting file is always perfect (note that without the patch/modification also the second variant would produce the same errors !!). Apparently there is a second issue about timing, i.e. de/re-muxing in real time for a stream (=> gives glitches) or on a file (=> no glitches, perfect results).

2nd order workaround
For the last problem of glitches one can try to replace the above intermediary file raw.ts by a fifo (simply create a fifo with "mkfifo raw.ts" in Linux) and start both vlc instances at the same time. Using this the situation with "some glitches" (for very problematic fluxes and complete corruption without patch) transforms to "very very rare glitches" but not 100% absent (one can play here with caching and dump-buffer parameters). The fifo avoids the creation of a big intermediary raw.ts file but it is still a bit close to streaming.
In practice I apply the fifo solution with udp-streaming and then another (3rd) vlc instance to record the udp-streams (this allows to record several channels of the same multiplex at the same time etc.).

I am of course aware that my modification is not at all a bug fix and that there are still problems, especially with the computation for the "rates" (there are strange messages "adjusting rate at ..." or "mux_ts mux warning: packet with too strange dts ..." when using the patched vlc version). This needs real bugfixing ... (I am looking myself in the code for sout, ts mux and demux but for me this is still very complicated ...).

I can provide raw.ts files which itself play nicely with vlc but which produce the problem after restreaming through sout (and no problem if using the patched vlc version). Using the Freebox V6 I can actually create a raw.ts file (using --ts-dump-file) for only one channel (in this case the Freebox does the demuxing of channels in a correct way but still keeps the invalid timestamps) which is smaller and simpler for testing/debugging.
I can also provide raw.ts files directly produced on the PC with all 4-5 channels inside. However, here the file size for upload becomes quickly a practical problem.
I can also provide further detailed debugging messages produced by the patched vlc version.

If somebody is really interested I can also provide more explicit commands lines (scripts) with the different options (my choices for caching values etc.), including streaming solutions etc.

For Windows users who are willing to test the above modification (commenting out the above lines in modules/packetizer/packetizer_helper.h) it would be nice if somebody can cross-compile a vlc-version for Windows for this (I cannot do this, at least I don't think that I can do this; I do not have access to Windows libs/codecs etc.).

[Kvii]

Thank you for your information. I have the same problem
and waiting for a fix from the developers.

[MlTV]

Hello,
Just for information, I had exactly the same symptoms while streaming French TNT HD with VLC.
After applying the hack suggested by kmf31 (commenting out lines dealing with VLC_TS_INVALID in modules/packetizer/packetizer_helper.h) it start to work fine again.

If it may help someone, my config is as follow :
I run an Ubuntu server 14.04.4 LTS
I use mumudvb to read DVB-T streams from USB DVB stics and split all multiplex on multicast IPs.
I read these multicast IPs with emcast piping the standard output to a script that split the stream in small strips and store them on disk (it's a kind of buffer for time shifting)
I use vlc in vlm mode to read from disk and stream via UDP to Set Top Box.
The output chain is like access=udp,mux=ts,dst=192.168.X.Y:1234

I made a quilt patch for the debian package used in my Ubuntu version.
If you want to use it, just apt-get source, copy/paste it in a file in debian/patches, apt-get build-dep and dpkg-buildpackage :

Code: Select all

Index: vlc-2.1.6/modules/packetizer/packetizer_helper.h =================================================================== --- vlc-2.1.6.orig/modules/packetizer/packetizer_helper.h 2016-04-25 13:03:21.915519000 +0200 +++ vlc-2.1.6/modules/packetizer/packetizer_helper.h 2016-04-25 13:05:51.359524722 +0200 @@ -183,11 +183,13 @@ else { p_pic = p_pack->pf_parse( p_pack->p_private, &b_used_ts, p_pic ); + /* if( b_used_ts ) { p_block_bytestream->i_dts = VLC_TS_INVALID; p_block_bytestream->i_pts = VLC_TS_INVALID; } + */ } if( !p_pic )

[kmf31]

Thanks for the feed back.

In addition to the "hack" I also suggest to increase strongly the ts-shaping parameter to 4000 ms, either by the option "--sout-ts-shaping=4000" or modifying this value in the preferences (stream output => muxers => mpeg-TS). This seems to avoid the occasional glitches one might have in scenes with fast movement (as a replacement of the above "2nd order workaround" with pipes but which is still useful as well) but again this is only important when the flux is of "bad quality" (e.g. reality TV, sport broadcast, ...) while with (american) series/movies this is less problematic (however, ARTE and FR4 are sometimes also problematic for this).

Apart from this in my opinion the branch vlc-2.0.9 is MUCH more reliable as far as streaming with udp (and also rtsp) is concerned as compared to the vlc-2.2.x branch (=> stuttering with udp-streaming, requiring the option --clock-synchro=0 and quite large caching parameters, i.e. --network-caching etc. and still there are problems).

The only point where vlc-2.2.x (for x=1,2,3; it doesn't matter) is better than vlc-2.0.9 for me is with direct viewing of ts-files since seeking to new positions is more reliable (no risc to loose audio) but for everything else: record (to file, udp or dump to pipe etc.), demuxing, streaming with udp, client or serveur, vlc-2.0.9 is by far much more reliable.

By the way vlc-2.0.9 has this very nice option "-ts-dump-file=..." which also works with rtsp (!!!!) while the replacement option for vlc-2.2.x (--demux=demuxdump --demuxdump-file=...) does NOT work with rtsp since rtsp does not count as "input-access" but already as "demuxer" itself. Actually using the option --ts-dump-file on the rtsp flux of my Freebox (for a DVB-T-channel received by the Freebox) I generated a nice ts-test file for this problem.

The only point concerning vlc-2.0.9 is that one needs old versions of libdvbpsi (0.2.2) and ffmpeg (0.10.16) to compile vlc-2.0.9 but also ffmpeg-0.10.16 contains patches up to 2015 and is not that obsolete and for french HD-TV (with h264) it works also very well.