Capturing audio from HLS stream - works in gui but not when using cmd line (cvlc)

[bbear1]

For a number of years I have been using a script on a Windows 10 machine to successfully capture audio from a HLS live stream (BBC radio stream) to a file, however I am struggling to get the same setup working on a new Linux machine which I am putting together. Here is my setup:

Raspberry Pi4 with 4Gb Ram, OS: Ubuntu 19.10 64-Bit, Ubuntu-Mate desktop
VLC version 3.0.8 (installed from Ubuntu Snap store)

At first I tried using the same command as I am using successfully on my Windows machine, this extracts the stream and doesn't perform any transcoding, however this would not work on my Pi

So I tried doing a network stream capture using the gui version of VLC on my Pi. This works!!

Next I enabled logging in the gui, set to 2 (errors) so I could see the sout command which was being used in VLC gui. I made a note of the command:

Code: Select all

qt debug: Transcode MRL: sout=#transcode{vcodec=h264,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{access=file{no-overwrite},mux=mp4,dst='/home/ubuntu/Desktop/vlc_test_bbc_gui.mp4'}

Note, VLC gui seemed to default to h264 but I also tried ogg and that worked also.

Next I created a bash script which executes cvlc and effectively runs the exact same sout command which I extracted from the gui log:

Code: Select all

/usr/bin/cvlc http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout "#transcode{vcodec=h264,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{mux=mp4,access=file,dst=/home/ubuntu/Desktop/vlc_test_bbc.mp4} sout-keep" --run-time=60 --play-and-exit --logfile=/home/ubuntu/Desktop/vlc_saturday_test_mp4.log

However this does not work. The mp4 file which gets created is very small (around 150bytes) and has no audio. If I look in the log file there are a number of problems indicated with the following sequence of messages appearing to indicate where it finally gives up:

• Code: Select all

  main debug: ts warning: Broken stream: pid 34 sends packets with dts 934566us later than pcr, applying delay main debug: selecting program id=0 main debug: looking for packetizer module matching "any": 25 candidates mpeg4audio debug: running MPEG4 audio packetizer mpeg4audio debug: ADTS Mode main debug: using packetizer module "mpeg4audio" main debug: Buffering 0% main debug: EOF reached main debug: Stream buffering done (0 ms in 0 ms) main error: buffer deadlock prevented main debug: Decoder wait done in 0 ms main debug: killing decoder fourcc `mp4a' main debug: removing module "mpeg4audio" main debug: removing module "adaptive" main debug: removing module "ts" main debug: Program doesn't contain anymore ES main debug: removing module "record" main debug: removing module "prefetch" main debug: removing module "access" main debug: dead input main debug: destroying useless sout

If I compare it to the messages from the log file from running VLC in gui mode I see some differences:

• Code: Select all

  ts warning: Broken stream: pid 34 sends packets with dts 934566us later than pcr, applying delay main debug: selecting program id=0 main debug: looking for packetizer module matching "any": 25 candidates mpeg4audio debug: running MPEG4 audio packetizer mpeg4audio debug: ADTS Mode main debug: using packetizer module "mpeg4audio" main debug: Buffering 0% main debug: Buffering 5% main debug: Buffering 10% main debug: Buffering 15% main debug: Buffering 20% main debug: Buffering 25% main debug: Buffering 30% main debug: Buffering 35% main debug: Buffering 40% main debug: Buffering 45% main debug: Buffering 50% main debug: Buffering 55% main debug: Buffering 60% main debug: Buffering 65% main debug: Buffering 70% main debug: Buffering 75% main debug: Buffering 80% main debug: Buffering 85% main debug: Buffering 90% main debug: Buffering 95% main debug: Buffering 100% main debug: switching to async mode main debug: Stream buffering done (1009 ms in 1 ms) mpeg4audio info: AAC channels: 2 samplerate: 24000 main debug: adding a new sout input for `mp4a` (sout_input: 0xffff38001020) stream_out_transcode debug: creating audio transcoding from fcc=`mp4a' to fcc=`mpga'

I have scoured the videolan forums, documentation, built-in help and the wiki and cannot figure out why the sout command chosen by the gui version of VLC will not work when I run it in command line mode.

I wondered if anyone reading this might have any ideas how I can fix this?

Note, I have the full logs from my gui run and scripted runs which I can try uploading if it would be helpful

[Rémi Denis-Courmont]

Incorrect quoting.

[bbear1]

Remi,
thank you for your help. Can you please point out where I am using incorrect quoting as I have seen many command line examples in the docs and these forums and some examples use single quotes around the destination file name, and some use double-quotes. I must admit I am very confused as to what is the correct quoting sometimes.

I tried the following:

Using double quotes around just the destination file name:

Code: Select all

/usr/bin/cvlc http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout "#transcode{vcodec=h264,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{mux=mp4,access=file,dst="/home/ubuntu/Desktop/vlc_test_bbc.mp4"} sout-keep" --run-time=60 --play-and-exit --logfile=/home/ubuntu/Desktop/vlc_saturday_test_mp4.log

Using single quotes around the whole transcode to std section::

Code: Select all

/usr/bin/cvlc http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout '#transcode{vcodec=h264,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{mux=mp4,access=file,dst=/home/ubuntu/Desktop/vlc_test_bbc.mp4} sout-keep' --run-time=60 --play-and-exit --logfile=/home/ubuntu/Desktop/vlc_saturday_test_mp4.log

Combining the two:

Code: Select all

/usr/bin/cvlc http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout '#transcode{vcodec=h264,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{mux=mp4,access=file,dst="/home/ubuntu/Desktop/vlc_test_bbc.mp4"} sout-keep' --run-time=60 --play-and-exit --logfile=/home/ubuntu/Desktop/vlc_saturday_test_mp4.log

Moving the sout-keep to outside the single quoted section:

Code: Select all

/usr/bin/cvlc http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout '#transcode{vcodec=h264,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{mux=mp4,access=file,dst="/home/ubuntu/Desktop/vlc_test_bbc.mp4"}' sout-keep --run-time=60 --play-and-exit --logfile=/home/ubuntu/Desktop/vlc_saturday_test_mp4.log

But none of these work.

Can you please point to where the quoting is incorrect?

thank you for your help

[bbear1]

Ok, so I am making some progress, finally I can see VLC capturing audio from the stream but unfortunately it still doesn't write out the extracted audio to mp4 file. The command I am now using in my bash script is as follows:

Code: Select all

/usr/bin/cvlc http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout-keep sout=#transcode{vcodec=none,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{access=file,mux=mp4,dst=test.mp4} --run-time=30 --play-and-exit --logfile=test_mp4.log

Basically, what I changed from my original command was to completely remove the quotes. Note, just about every single example I have seen for vlc transcode has used single or double quotes so I blindly followed their example. I guess I don't understand the purpose of quoting, when its required, when its not, and whether to use single or double quotes.

Anyway, the good news is that I made some progress.

Unfortunately, as I mentioned, vlc still doesn't write out the audio file. Here are some extracts from the log file:

Code: Select all

adaptive debug: Segment #247388983 url=http://as-hls-ww-live.akamaized.net/pool_904/live/ww/bbc_radio_two/bbc_radio_two.isml/bbc_radio_two-audio=96000-247388982.ts stime 1920 duration 640 adaptive debug: Segment #247388984 url=http://as-hls-ww-live.akamaized.net/pool_904/live/ww/bbc_radio_two/bbc_radio_two.isml/bbc_radio_two-audio=96000-247388983.ts stime 2560 duration 640 adaptive debug: Retrieving http://as-hls-ww-live.akamaized.net:80/pool_904/live/ww/bbc_radio_two/bbc_radio_two.isml/bbc_radio_two-audio=96000-247388979.ts @0 main debug: resolving as-hls-ww-live.akamaized.net ... main debug: connecting to as-hls-ww-live.akamaized.net port 80 ... main debug: creating demux: access='' demux='ts' location='' file='(null)' main debug: looking for demux module matching "ts": 55 candidates ts debug: Standard set to Auto main debug: using demux module "ts" ts debug: first packet for pid=0 cc=0x2 ts debug: PATCallBack called ts debug: new PAT ts_id=16727 version=0 current_next=1 ts debug: * number=16727 pid=32 ts debug: temporary receiving program 16727 ts debug: first packet for pid=32 cc=0x2 ts debug: PMTCallBack called for program 16727 ts debug: new PMT program number=16727 version=0 pid_pcr=34 ts debug: * pid=34 type=0xf ISO/IEC 13818-7 Audio with ADTS transport ts debug: - ES descriptor tag 0xe ts debug: - SL/FMC descriptor not found/matched ts debug: => pid 34 has now es fcc=mp4a ts debug: Default program is 16727 ts debug: * pid=17 listening for SDT ts debug: enabling pid 34 from program 16727 ts debug: enabling pcr pid 34 from program 16727 ts warning: Broken stream: pid 34 sends packets with dts 934566us later than pcr, applying delay main debug: selecting program id=0 main debug: looking for packetizer module matching "any": 25 candidates mpeg4audio debug: running MPEG4 audio packetizer mpeg4audio debug: ADTS Mode main debug: using packetizer module "mpeg4audio" main debug: looking for audio decoder module matching "any": 21 candidates main debug: using audio decoder module "faad" main debug: Buffering 0% main debug: EOF reached main debug: Stream buffering done (0 ms in 0 ms) main error: buffer deadlock prevented main debug: Decoder wait done in 0 ms main debug: killing decoder fourcc `mp4a' main debug: removing module "faad" main debug: removing module "mpeg4audio"

I am stuck at this point, doesn't anyone have any ideas/suggestions on how to fix this?

Again, this works when I run VLC in gui mode, it is just when running from script/command line that it is broken.

thank you

[bbear1]

Oops, made a typo, what I mean to ask is:

does anyone have any ideas/suggestions on how to fix this?

[bbear1]

Another data point ...
I get exactly the same issue on my Windows 10 64bit machine - works from the gui, but fails when running vlc from a DOS command shell.
And just to be sure I wasn't using an older version on my Windows machine I did an uninstall first, followed by downloading and installing the latest 64bit version from videolan.org.

For reference, here is the .cmd file which I used:

Code: Select all

"E:\Program Files\VideoLAN\VLC\vlc.exe" -I dummy http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout=#transcode{vcodec=none,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:standard{access=file,mux=mp4,dst="C:\Users\Kevin\Desktop\VLC_DEBUG\vlc_test_bbc.mp4"} --sout-keep --file-logging --log-verbose=3 --logfile="C:\Users\kevin\Desktop\VLC_DEBUG\vlc_saturday_test_mp4.log" --run-time=30 --play-and-exit

And an extract from the log file:

Code: Select all

ts debug: first packet for pid=32 cc=0x4 ts debug: PMTCallBack called for program 16727 ts debug: new PMT program number=16727 version=0 pid_pcr=34 ts debug: * pid=34 type=0xf ISO/IEC 13818-7 Audio with ADTS transport ts debug: - ES descriptor tag 0xe ts debug: - SL/FMC descriptor not found/matched ts debug: => pid 34 has now es fcc=mp4a ts debug: Default program is 16727 ts debug: * pid=17 listening for SDT ts debug: enabling pid 34 from program 16727 ts debug: enabling pcr pid 34 from program 16727 ts warning: Broken stream: pid 34 sends packets with dts 934566us later than pcr, applying delay main debug: selecting program id=0 main debug: looking for packetizer module matching "any": 25 candidates mpeg4audio debug: running MPEG4 audio packetizer mpeg4audio debug: ADTS Mode main debug: using packetizer module "mpeg4audio" main debug: Buffering 0% main debug: EOF reached main debug: Stream buffering done (0 ms in 0 ms) main error: buffer deadlock prevented main debug: Decoder wait done in 0 ms main debug: killing decoder fourcc `mp4a' main debug: removing module "mpeg4audio" main debug: removing module "adaptive" main debug: removing module "ts" main debug: Program doesn't contain anymore ES main debug: removing module "record" main debug: removing module "prefetch" main debug: removing module "access" main debug: dead input main debug: changing item without a request (current 0/1) main debug: nothing to play main: end of playlist, exiting main debug: exiting main debug: exiting main debug: no exit handler main debug: removing all interfaces main debug: removing module "dummy" main debug: removing module "win32" main debug: removing module "hotkeys" main debug: destroying

I seem to have hit a brick wall

If anyone has time to look into this and help me figure out what I can do to fix this, I would be very grateful

thank you

[bbear1]

This looks to be a problem with the latest (or at least the 3.x) version of VLC ..

I went back and looked at my old media PC, a Windows10 machine I have been using VLC to record from this same HLS stream successfully for the past few years. I took the command which has been working on that and tried it on the Windows 10 machine I have been seeing it fail. What I observed is that it failed in exactly the same way.

This is the command which I used: (this doesn't do any transcoding)

Code: Select all

"E:\Program Files\VideoLAN\VLC\vlc.exe" http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout="#file{dst=C:\Users\Kevin\Desktop\VLC_DEBUG\vlc_test_bbc.mp4} sout-keep" --run-time=30 --file-logging --log-verbose=3 --logfile=C:\Users\kevin\Desktop\VLC_DEBUG\vlc_saturday_test_mp4.log --play-and-exit

So, I checked which version is on my old media PC (2.2.1) and downloaded that from the videolan ftp site and installed that. Now, running 2.2.1 it works!!

I also tried a version of the command which includes transcode and that also works:

Code: Select all

"E:\Program Files\VideoLAN\VLC\vlc.exe" -I dummy http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout=#transcode{vcodec=none,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:standard{access=file,mux=mp4,dst="C:\Users\Kevin\Desktop\VLC_DEBUG\vlc_test_bbc.mp4"} --sout-keep --file-logging --log-verbose=3 --logfile="C:\Users\kevin\Desktop\VLC_DEBUG\vlc_saturday_test_mp4.log" --run-time=30 --play-and-exit

The problem is that my aim in all this is to get VLC working on my Raspberry Pi 4, and I don't think it is an option to install 2.2.1 as I have to get VLC from the Ubuntu Snap store (which has the 3.0.8 version). I suppose I could try running make to build 2.1.1 on my Pi but I suspect its going to insist that I install a bunch of old packages in order for it to build. Its likely to be very problematic.

I much prefer to get the latest VLC to work, but as I have reported here, it only works if I open it in gui mode and manually select the network stream and record it that way. As this is an unattended recording which I need to make (using crontab) opening up the VLC gui manually at 11pm every night isn't an option.

Can someone please help?

[bbear1]

I have given up bashing my head against a wall trying to get VLC to capture and transcode this hls stream. However I have found a workaround:

1. Capture the stream to .ts using ffmpeg
2. Use VLC to convert the audio in the .ts to mp4

So far the above seems to be working fine and I can do both steps in one bash script. I have recorded 90mins of the stream and it plays ok

It would have been nice to have used VLC for both steps, maybe I will be able to do so in a later release. Still not sure if this is a VLC bug or not so won't submit a bug report until I get some feedback from the devs

[Rémi Denis-Courmont]

The GUI cannot do anything that the CLI cannot do. This is most likely a bug in your script or its environment.

[bbear1]

Remi,
I have tried various versions of the command, the most recent of which I included in the recent update to this thread. Even copy/pasting to the bash shell directly, it wont work. As I mentioned, the same command does work with VLC version 2.x (well, on my Windows machine that is), whereas the same command will not work with VLC v3.x. I did try to compile the 2.x sources on unbuntu machine (Raspberry Pi 4) but was unsuccessful and had to give up.

Doesn't the fact it works with 2.x mean the command which I am using is ok?

You said that if it works in the gui it will work in the CLI, but could there possibility be some of the VLC environment is slightly different? Are defaults for buffer size, timeout, etc any different between running in gui and cli?

[bbear1]

Well, this is not at all what I expected..

I repurposed a spare Intel i5 desktop PC and installed Ubuntu 19.10 on it. I wanted to do this to see if it was the Pi 4 that was some how influencing the result (I know, its unlikely)

Initially I had the same problems with not being able to record the stream, however I stumbled across a command line which would work ..

Code: Select all

cvlc http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout=#"transcode{vcodec=none,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{access=file,mux=mp4,dst=test.mp4}"

With the above the mp4 gets output ok. So next step was to add to the command string to record for a specfic time period and then exit..

Code: Select all

cvlc http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout=#"transcode{vcodec=none,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{access=file,mux=mp4,dst=test.mp4}" --run-time=60 --play-and-exit

With the above VLC fails to generate the .mp4 (actually it opens the file and generates a very small, unplayable mp4), the following messages are printed to the console:

Code: Select all

VLC media player 3.0.8 Vetinari (revision 3.0.8-0-gf350b6b5a7) [0000558e06d2de10] dummy interface: using the dummy interface module... [00007fbdb802a1f0] main decoder error: buffer deadlock prevented [0000558e06ceb0a0] main playlist: end of playlist, exiting

After a process of elimination I discovered that it is the '--run-time=60' which results in the 'buffer deadlock prevented' error. If I remove this (leaving the '--play-and-exit' in the command string the mp4 still gets generated correctly. I need to be able to use the '--run-time' option though as I need to record for a specific amount of time.

Does anyone know of a way around this?

[bbear1]

I found a possible solution using 'timeout', I got this tip courtesy of stackoverflow..

Code: Select all

timeout <seconds>s vlc v4l2:///dev/video1

Here is my complete (working) command:

Code: Select all

timeout 60 cvlc http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_radio_two.m3u8 --sout=#"transcode{vcodec=none,acodec=mpga,ab=128,channels=2,samplerate=44100,scodec=none}:std{access=file,mux=mp4,dst=test.mp4}"

Now I need to see if it work under Ubuntu 19.10 on my Pi 4

Although this worked on my i5 machine I still would prefer to have used VLC's --run-time option as I am concerned as to whether the recording will finish cleanly if 'timeout' is used to kill VNC at the end