VLC Opus RTP/RTSP support

[changyp6]

Hi VLC dev-team,
Since VLC has supported oggopus file, when will VLC support Opus RTP/RTSP stream playback?
I have implemented Opus RTP server according to http://tools.ietf.org/html/draft-spittk ... d-rtp-opus,
but I can't find any player that can play this Opus RTP stream.

It seems someone had made some dirty hack to make it work, but still no official support in VLC.
http://lists.xiph.org/pipermail/opus/20 ... 01828.html

[Rémi Denis-Courmont]

VLC will support Opus over RTP on client side when live555 adds support. Well, some VLC glue will be needed, but it should be trivial.

[changyp6]

VLC will support Opus over RTP on client side when live555 adds support. Well, some VLC glue will be needed, but it should be trivial.

It seems that live555 has already supported audio/OPUS since 2012-10.22
From this link: http://www.live555.com/liveMedia/public/changelog.txt

2012.10.22:
- Updated "AVIFileSink" to add an index at the end. (Thanks to "qiuchangyong qiuchangyong".)
- Updated the "MediaSession" implementation to accept "audio/OPUS" as a valid RTP payload format (which can be implemented
using a "SimpleRTPSource").

[changyp6]

I have summerized the codes in http://lists.xiph.org/pipermail/opus/20 ... 01828.html
and made a patch againt vlc-2.0.7
Now my VLC can play H.264+Opus RTSP stream.
The patch is attached, if anyone wants to have a try.
If VLC dev team finds this patch useful, please help to merge it, so that everyone can have OPUS rtp support.

Code: Select all

--- vlc-2.0.7/modules/demux/live555.cpp.orig 2013-07-19 00:27:27.906722017 +0800 +++ vlc-2.0.7/modules/demux/live555.cpp 2013-07-19 00:27:32.836579634 +0800 @@ -951,6 +951,17 @@ static int SessionsSetup( demux_t *p_dem tk->fmt.audio.i_rate = 8000; } } + else if( !strcmp( sub->codecName(), "OPUS" ) ) + { + tk->fmt.i_codec = VLC_FOURCC( 'O', 'p', 'u', 's' ); + if ( sub->rtpTimestampFrequency() ) + tk->fmt.audio.i_rate = sub->rtpTimestampFrequency(); + else + { + msg_Warn( p_demux,"Using 48kHz as default sample rate." ); + tk->fmt.audio.i_rate = 48000; + } + } } else if( !strcmp( sub->mediumName(), "video" ) ) { --- vlc-2.0.7/modules/codec/opus.c.orig 2012-09-04 06:09:37.000000000 +0800 +++ vlc-2.0.7/modules/codec/opus.c 2013-07-19 15:11:48.286288317 +0800 @@ -190,33 +190,136 @@ static int OpenDecoder( vlc_object_t *p_ ****************************************************************************/ static block_t *DecodeBlock( decoder_t *p_dec, block_t **pp_block ) { + static int rtpDec = 1; + static OpusDecoder *dec = NULL; + block_t *p_aout_buffer = NULL; + block_t *p_block = *pp_block; + unsigned char *data = NULL; + int buf_size = 32768; + int size = 0, off = 0, err = 0, written = 0; + struct timeval pts; + int nb_samples; + decoder_sys_t *p_sys = p_dec->p_sys; - ogg_packet oggpacket; if( !pp_block || !*pp_block) return NULL; - /* Block to Ogg packet */ - oggpacket.packet = (*pp_block)->p_buffer; - oggpacket.bytes = (*pp_block)->i_buffer; - - oggpacket.granulepos = -1; - oggpacket.b_o_s = 0; - oggpacket.e_o_s = 0; - oggpacket.packetno = 0; - /* Check for headers */ if( !p_sys->b_has_headers ) { - if( ProcessHeaders( p_dec ) ) + if( ProcessHeaders( p_dec ) == VLC_SUCCESS) + { + p_sys->b_has_headers = true; + rtpDec = 0; // if OGG headers are found then is no RTP + } + } + + if (rtpDec != 0) + { + if (dec == NULL) + { + /* Set output properties */ + p_dec->fmt_out.i_cat = AUDIO_ES; + p_dec->fmt_out.i_codec = VLC_CODEC_S16N; + + /* Setup the format */ + p_dec->fmt_out.audio.i_physical_channels = p_dec->fmt_out.audio.i_original_channels = pi_channels_maps[2]; + p_dec->fmt_out.audio.i_channels = 2; + p_dec->fmt_out.audio.i_rate = 48000; + + /* Opus decoder init */ + dec = opus_decoder_create(p_dec->fmt_out.audio.i_rate, p_dec->fmt_out.audio.i_channels, &err); + opus_decoder_ctl(dec, OPUS_SET_GAIN(0)); + + /* Date init */ + date_Init( &p_sys->end_date, 48000, 1 ); + } + + /* Date management */ + if( p_block && p_block->i_pts > VLC_TS_INVALID && + p_block->i_pts != date_Get( &p_sys->end_date ) ) + { + date_Set( &p_sys->end_date, p_block->i_pts ); + } + + memcpy(&pts, &(p_block->i_pts), sizeof(pts)); + + if( !date_Get( &p_sys->end_date ) ) { - block_Release( *pp_block ); + /* We've just started the stream, wait for the first PTS. */ + if( p_block ) block_Release( p_block ); return NULL; } - p_sys->b_has_headers = true; + + *pp_block = NULL; /* To avoid being fed the same packet again */ + + p_aout_buffer = decoder_NewAudioBuffer(p_dec, buf_size); + if ( !p_aout_buffer ) + { + msg_Err(p_dec, "Oops: No new buffer was returned!"); + return NULL; + } + + data = p_block->p_buffer; + size = p_block->i_buffer; + + unsigned char toc, fc, frames[48]; + short sizes[48]; + int payload_offset, nb_frames, nb_channels; + int bytes_read; + + nb_channels = opus_packet_get_nb_channels(data); + nb_samples = opus_decoder_get_nb_samples(dec, data, size); + nb_frames = opus_packet_parse(data, size, &toc, &frames, &sizes, &payload_offset); + written = opus_decode(dec, data, size, p_aout_buffer->p_buffer, buf_size / nb_channels, 0) * nb_channels * 2; // data + /* + fc = data[1]; + written = 0; bytes_read = 0; + printf ("toc %02x (%d %d %d) - frame count %02x (%d %d %d) - offset %d\n", toc, (toc>>3) & 0x1F, (toc>>4) & 0x01, toc & 0x03, fc, (fc>>7) & 0x01, (fc>>6) & 0x01, fc & 0x3F, payload_offset); + for (int i=0; i<nb_frames; ++i) + { + printf ("frame %d - size %d\n", i, sizes[i]); + data[bytes_read+1] = toc & 0xFC; + written += opus_decode(dec, data + 1 + bytes_read, sizes[i] + 1, p_aout_buffer->p_buffer + written, 4096, 0) * 4; // data + bytes_read += sizes[i]; + } + */ + + p_aout_buffer->i_nb_samples = nb_samples; + p_aout_buffer->i_pts = date_Get( &p_sys->end_date ); + p_aout_buffer->i_length = written; + + /*printf("Got %d bytes => written %d bytes (%d samples) TS: %lu.%06lu\n", size, written, nb_samples, pts.tv_sec, pts.tv_usec);*/ + + if( p_block ) block_Release( p_block ); + + return p_aout_buffer; } + else + { + ogg_packet oggpacket; + /* Block to Ogg packet */ + oggpacket.packet = (*pp_block)->p_buffer; + oggpacket.bytes = (*pp_block)->i_buffer; + + oggpacket.granulepos = -1; + oggpacket.b_o_s = 0; + oggpacket.e_o_s = 0; + oggpacket.packetno = 0; - return ProcessPacket( p_dec, &oggpacket, pp_block ); + /* Check for headers */ + if( !p_sys->b_has_headers ) + { + if( ProcessHeaders( p_dec ) ) + { + block_Release( *pp_block ); + return NULL; + } + p_sys->b_has_headers = true; + } + return ProcessPacket( p_dec, &oggpacket, pp_block ); + } } /*****************************************************************************

[Jean-Baptiste Kempf]

PLease send this to the vlc-devel@ mailing list, this is quite useful!

[changyp6]

PLease send this to the vlc-devel@ mailing list, this is quite useful!

I'm new here, I don't know how to send this patch to the vlc-devel mailing list.
Would you mind to tell me? Thanks!

[Jean-Baptiste Kempf]

http://wiki.videolan.org/Git/#Submitting_patches

[changyp6]

I have refined the patch, and fixed a possible memory leak in the last patch.
there was a bug in the previous patch, when server disconnected the client, VLC reconnect to RTSP server, there will be no sound.
This bug is related to the possible memory leak.
The new patch is here:

Code: Select all

--- vlc-2.0.7/modules/demux/live555.cpp.orig 2013-07-19 00:27:27.906722017 +0800 +++ vlc-2.0.7/modules/demux/live555.cpp 2013-07-19 00:27:32.836579634 +0800 @@ -951,6 +951,17 @@ static int SessionsSetup( demux_t *p_dem tk->fmt.audio.i_rate = 8000; } } + else if( !strcmp( sub->codecName(), "OPUS" ) ) + { + tk->fmt.i_codec = VLC_FOURCC( 'O', 'p', 'u', 's' ); + if ( sub->rtpTimestampFrequency() ) + tk->fmt.audio.i_rate = sub->rtpTimestampFrequency(); + else + { + msg_Warn( p_demux,"Using 48kHz as default sample rate." ); + tk->fmt.audio.i_rate = 48000; + } + } } else if( !strcmp( sub->mediumName(), "video" ) ) { --- vlc-2.0.7/modules/codec/opus.c.orig 2012-09-04 06:09:37.000000000 +0800 +++ vlc-2.0.7/modules/codec/opus.c 2013-07-23 18:48:24.283193828 +0800 @@ -75,12 +75,14 @@ struct decoder_sys_t * Input properties */ bool b_has_headers; + bool b_rtp_decode; /* * Opus properties */ OpusHeader header; OpusMSDecoder *p_st; + OpusDecoder *p_raw_dec; /* * Common properties @@ -168,6 +170,7 @@ static int OpenDecoder( vlc_object_t *p_ if( ( p_dec->p_sys = p_sys = malloc(sizeof(decoder_sys_t)) ) == NULL ) return VLC_ENOMEM; p_dec->p_sys->b_has_headers = false; + p_dec->p_sys->b_rtp_decode = true; date_Set( &p_sys->end_date, 0 ); @@ -179,6 +182,7 @@ static int OpenDecoder( vlc_object_t *p_ p_dec->pf_packetize = DecodeBlock; p_sys->p_st = NULL; + p_sys->p_raw_dec = NULL; return VLC_SUCCESS; } @@ -190,33 +194,142 @@ static int OpenDecoder( vlc_object_t *p_ ****************************************************************************/ static block_t *DecodeBlock( decoder_t *p_dec, block_t **pp_block ) { + block_t *p_aout_buffer = NULL; + block_t *p_block = *pp_block; + unsigned char *data = NULL; + int buf_size = 32768; + int size = 0, off = 0, err = 0, written = 0; + struct timeval pts; + int nb_samples; + decoder_sys_t *p_sys = p_dec->p_sys; - ogg_packet oggpacket; if( !pp_block || !*pp_block) return NULL; - /* Block to Ogg packet */ - oggpacket.packet = (*pp_block)->p_buffer; - oggpacket.bytes = (*pp_block)->i_buffer; - - oggpacket.granulepos = -1; - oggpacket.b_o_s = 0; - oggpacket.e_o_s = 0; - oggpacket.packetno = 0; - /* Check for headers */ if( !p_sys->b_has_headers ) { - if( ProcessHeaders( p_dec ) ) + if( ProcessHeaders( p_dec ) == VLC_SUCCESS) + { + p_sys->b_has_headers = true; + p_sys->b_rtp_decode = false; + } else { + p_sys->b_rtp_decode = true; + } + } + + if ( p_sys->b_rtp_decode ) + { + if (p_sys->p_raw_dec == NULL) + { + /* Set output properties */ + p_dec->fmt_out.i_cat = AUDIO_ES; + p_dec->fmt_out.i_codec = VLC_CODEC_S16N; + + /* Setup the format */ + p_dec->fmt_out.audio.i_physical_channels = p_dec->fmt_out.audio.i_original_channels = pi_channels_maps[2]; + p_dec->fmt_out.audio.i_channels = 2; + p_dec->fmt_out.audio.i_rate = 48000; + + /* Opus decoder init */ + p_sys->p_raw_dec = opus_decoder_create(p_dec->fmt_out.audio.i_rate, p_dec->fmt_out.audio.i_channels, &err); + if( !p_sys->p_raw_dec || err!=OPUS_OK ) + { + msg_Err( p_dec, "decoder initialization failed" ); + return NULL; + } + + opus_decoder_ctl(p_sys->p_raw_dec, OPUS_SET_GAIN(0)); + + /* Date init */ + date_Init( &p_sys->end_date, 48000, 1 ); + } + + /* Date management */ + if( p_block && p_block->i_pts > VLC_TS_INVALID && + p_block->i_pts != date_Get( &p_sys->end_date ) ) + { + date_Set( &p_sys->end_date, p_block->i_pts ); + } + + memcpy(&pts, &(p_block->i_pts), sizeof(pts)); + + if( !date_Get( &p_sys->end_date ) ) + { + /* We've just started the stream, wait for the first PTS. */ + if( p_block ) block_Release( p_block ); + return NULL; + } + + *pp_block = NULL; /* To avoid being fed the same packet again */ + + p_aout_buffer = decoder_NewAudioBuffer(p_dec, buf_size); + if ( !p_aout_buffer ) { - block_Release( *pp_block ); + msg_Err(p_dec, "Oops: No new buffer was returned!"); return NULL; } - p_sys->b_has_headers = true; + + data = p_block->p_buffer; + size = p_block->i_buffer; + + unsigned char toc, fc, frames[48]; + short sizes[48]; + int payload_offset, nb_frames, nb_channels; + int bytes_read; + + nb_channels = opus_packet_get_nb_channels(data); + nb_samples = opus_decoder_get_nb_samples(p_sys->p_raw_dec, data, size); + nb_frames = opus_packet_parse(data, size, &toc, &frames, &sizes, &payload_offset); + written = opus_decode(p_sys->p_raw_dec, data, size, p_aout_buffer->p_buffer, buf_size / nb_channels, 0) * nb_channels * 2; // data + /* + fc = data[1]; + written = 0; bytes_read = 0; + printf ("toc %02x (%d %d %d) - frame count %02x (%d %d %d) - offset %d\n", toc, (toc>>3) & 0x1F, (toc>>4) & 0x01, toc & 0x03, fc, (fc>>7) & 0x01, (fc>>6) & 0x01, fc & 0x3F, payload_offset); + for (int i=0; i<nb_frames; ++i) + { + printf ("frame %d - size %d\n", i, sizes[i]); + data[bytes_read+1] = toc & 0xFC; + written += opus_decode(dec, data + 1 + bytes_read, sizes[i] + 1, p_aout_buffer->p_buffer + written, 4096, 0) * 4; // data + bytes_read += sizes[i]; + } + */ + + p_aout_buffer->i_nb_samples = nb_samples; + p_aout_buffer->i_pts = date_Get( &p_sys->end_date ); + p_aout_buffer->i_length = written; + + /*printf("Got %d bytes => written %d bytes (%d samples) TS: %lu.%06lu\n", size, written, nb_samples, pts.tv_sec, pts.tv_usec);*/ + + if( p_block ) block_Release( p_block ); + + return p_aout_buffer; } + else + { + ogg_packet oggpacket; + /* Block to Ogg packet */ + oggpacket.packet = (*pp_block)->p_buffer; + oggpacket.bytes = (*pp_block)->i_buffer; + + oggpacket.granulepos = -1; + oggpacket.b_o_s = 0; + oggpacket.e_o_s = 0; + oggpacket.packetno = 0; - return ProcessPacket( p_dec, &oggpacket, pp_block ); + /* Check for headers */ + if( !p_sys->b_has_headers ) + { + if( ProcessHeaders( p_dec ) ) + { + block_Release( *pp_block ); + return NULL; + } + p_sys->b_has_headers = true; + } + return ProcessPacket( p_dec, &oggpacket, pp_block ); + } } /***************************************************************************** @@ -430,6 +543,7 @@ static void CloseDecoder( vlc_object_t * decoder_sys_t *p_sys = p_dec->p_sys; if( p_sys->p_st ) opus_multistream_decoder_destroy(p_sys->p_st); + if( p_sys->p_raw_dec ) opus_decoder_destroy(p_sys->p_raw_dec); free( p_sys ); }

[Jean-Baptiste Kempf]

Please send this to the mailing list.

[changyp6]

Please send this to the mailing list.

I have sent this patch to vlc-dev@videolan.org
http://mailman.videolan.org/pipermail/v ... 93842.html

[Jean-Baptiste Kempf]

Thx

[changyp6]

Thx

Unfortunately, the patch is rejected.
I don't have time to research on this, hope someone can enable Opus RTSP playback properly.