using libvlc to stream with rtp protocol from memory.

[StewartLewis]

Hello,

I've been trying to use libvlc to create an rtp stream and send simple images loaded into memory. For this I'm creating simple OpenCV Matrices and passing them through the callback api.

Here is what my code looks like :

Simple Mem class that contains the buffer to stream

Code: Select all

class MemVideoData { public: MemVideoData(unsigned char *data, int data_bytes) : video(data), bytes(data_bytes) { } //init ~MemVideoData() {} unsigned char* video; // pointer to video in memory int bytes; };

Callback open, read, seek and close implementation:

Code: Select all

ssize_t media_read_cb(void *opaque, unsigned char* buf, size_t len) { MemVideoData *mVid = (MemVideoData*) opaque; //cast and give context unsigned char *start = mVid->video; std::memcpy(buf,start, mVid->bytes); //copy bytes requested to buffer. return mVid->bytes; } int media_open_cb(void* opaque, void** datap, uint64_t* sizep) { //cast opaque to our video state struct MemVideoData *mVid = static_cast<MemVideoData*> (opaque); //TODO: get mutex on MemVideoData object pointed to by opaque *sizep = (uint64_t) mVid->bytes; //set stream length *datap = mVid; /*point to entire object. Think this was intended to point to the raw video data but we use the MemVideoData object in read() and seek()*/ return 0; } int media_seek_cb(void *opaque, uint64_t offset) { return 0; } void media_close_cb(void *opaque) { }

And this is how I instantiate the whole thing on my main.cpp

Code: Select all

// create a simple red image ::cv::Mat image(720,1080,CV_8UC3, ::cv::Scalar(255,0,0) ); int imgSize = image.rows * image.cols * image.channels(); MemVideoData* mem = new MemVideoData(image.data, imgSize); libvlc_instance_t *vlc; // add Verbose option to instance std::vector<const char*> options; options.push_back("--verbose=1 "); vlc = libvlc_new (int(options.size()), options.data()); libvlc_media_t* media = libvlc_media_new_callbacks(vlc,media_open_cb, media_read_cb, media_seek_cb, media_close_cb, mem); // Add the streaming string given by the vlc app libvlc_media_add_option(media, ":sout=#transcode{vcodec=h264,venc=x264,vb=0,vbv-bufsize=1200,bframes=0,scale=0,acodec=none}:rtp{dst=localhost, port=9555, name=test, mux=ts,sap}"); // Create a media player playing environment libvlc_media_player_t *mediaPlayer = libvlc_media_player_new_from_media (media); // play the media_player libvlc_media_player_play (mediaPlayer); // Arbitrary value, should be replace with user interaction boost::this_thread::sleep(boost::posix_time::milliseconds(60000000)); // Stop playing libvlc_media_player_stop (mediaPlayer); // Free the media_player libvlc_media_player_release (mediaPlayer); // Free vlc libvlc_release (vlc);

The problem I've encountered :

While trying to read the stream with the vlc application I get nothing, small blue load bar and both timers to 00:00

I checked my localhost ports with netstat and I see my port as ESTABLISHED but while checking with Wireshark I see nothing : No data is being sent through the stream.

The libvlc program log :

I had to stick with verbose to level 1

Code: Select all

[00007ff470001760] stream_out_rtp stream out warning: unknown protocol for SDP ((null)) [00007ff470007ed0] main stream out warning: option vbv-bufsize is unknown [00007ff470007ed0] main stream out warning: option bframes is unknown [00007ff47000a230] imem demux error: Invalid get/release function pointers [00007ff47000a230] imem demux error: Invalid get/release function pointers [00007ff47000a230] ps demux warning: this does not look like an MPEG PS stream, continuing anyway [00007ff47000a230] ps demux warning: garbage at input from 509, trying to resync...

Other things I've tried :

Using the --imem-data directly : this gave me a deadlock error and seemed to produced a similar result.
Using the vlm API : I surprisingly managed to stream from a file with it, but not from memory, and I wouldn't know how to link it with the media callback API.
Different ports and launching it as root (I was desperate) .

Any help would be greatly appreciated !

[chubinou]

Hi,

I guess you pass raw video frames as an input, VLC should be informed about how to interpret your data, you may have a look at the rawvid demuxer

Code: Select all

Raw video demuxer (rawvid) --rawvid-fps <string> Frames per Second This is the desired frame rate when playing raw video streams. In the form 30000/1001 or 29.97 --rawvid-width <integer> Width This specifies the width in pixels of the raw video stream. --rawvid-height <integer> Height This specifies the height in pixels of the raw video stream. --rawvid-chroma <string> Force chroma (Use carefully) Force chroma. This is a four character string. --rawvid-aspect-ratio <string> Aspect ratio Aspect ratio (4:3, 16:9). Default assumes square pixels.

[StewartLewis]

Thank your for you reply.

From what I understand, I should be passing the options you mention to the vlc instance right?

This would look something like this :

Code: Select all

std::vector<const char*> options; options.push_back("--verbose=1"); options.push_back("--rawvid-fps=30"); options.push_back("--rawvid-height=720"); options.push_back("--rawvid-width=1080"); vlc = libvlc_new (int(options.size()), options.data());

Or should I be setting these directly onto the media?

Code: Select all

libvlc_media_add_option(media, "--rawvid-fps=30"); libvlc_media_add_option(media, "--rawvid-height=720"); libvlc_media_add_option(media, "--rawvid-width=1080");

Tried both of these possibilities to no avail. I understand what you mean when you say that I should be specifying how to read the raw video, but I don't understand how I should be passing it.

Thank you.

[chubinou]

something like this works

Code: Select all

#include <vlc/vlc.h> #include <memory> #include <cstring> #include <unistd.h> #include <vector> class MemVideoData { public: MemVideoData(unsigned char *data, int data_bytes) : video(data), bytes(data_bytes) { } //init ~MemVideoData() {} unsigned char* video; // pointer to video in memory size_t bytes; size_t pos = 0; }; ssize_t media_read_cb(void *opaque, unsigned char* buf, size_t len) { MemVideoData *mVid = (MemVideoData*) opaque; //cast and give context size_t copyLen = std::min(mVid->bytes - mVid->pos, len); unsigned char *start = mVid->video + mVid->pos; memcpy(buf,start, copyLen); //copy bytes requested to buffer. mVid->pos += (copyLen % mVid->bytes); return copyLen; } int media_open_cb(void* opaque, void** datap, uint64_t* sizep) { //cast opaque to our video state struct MemVideoData *mVid = static_cast<MemVideoData*> (opaque); *sizep = UINT64_MAX; //set stream length *datap = mVid; return 0; } int media_seek_cb(void *opaque, uint64_t offset) { return 0; } void media_close_cb(void *opaque) { } int main() { int imgSize = 720*1080*3; unsigned char* data = (unsigned char*)calloc(1, imgSize); for (int i = 0; i < imgSize; i += 3) //color in red data[i] = 255; MemVideoData* mem = new MemVideoData(data, imgSize); libvlc_instance_t *vlc; // add Verbose option to instance std::vector<const char*> options; options.push_back("--verbose=4"); vlc = libvlc_new (int(options.size()), options.data()); libvlc_media_t* media = libvlc_media_new_callbacks(vlc,media_open_cb, media_read_cb, media_seek_cb, media_close_cb, mem); // Add the streaming string given by the vlc app //libvlc_media_add_option(media, ":sout=#transcode{vcodec=h264,venc=x264,vb=0,vbv-bufsize=1200,bframes=0,scale=0,acodec=none}:rtp{dst=localhost, port=9555, name=test, mux=ts,sap}"); libvlc_media_add_option(media, ":demux=rawvid"); libvlc_media_add_option(media, ":rawvid-fps=10"); libvlc_media_add_option(media, ":rawvid-height=720"); libvlc_media_add_option(media, ":rawvid-width=1080"); libvlc_media_add_option(media, ":rawvid-chroma=RV24"); //24bit RGB // Create a media player playing environment libvlc_media_player_t *mediaPlayer = libvlc_media_player_new_from_media (media); // play the media_player libvlc_media_player_play (mediaPlayer); sleep(50); // Stop playing libvlc_media_player_stop (mediaPlayer); // Free the media_player libvlc_media_player_release (mediaPlayer); // Free vlc libvlc_release (vlc); }

[StewartLewis]

Your sample works wonders! Thank you so much.

for the posterity's sake, I'm noting that my errors were the following :

1) Passing options as "--rawvid-fps=30" instead of ":rawvid-fps=30"
2) Not passing a ":rawvid-chroma" with the correct format.

I have one more question however. Why is it that no .sdp file is needed in-order to grab the generated stream?
I know the answer lies somewhere in these parameters:

Code: Select all

:rtp{dst=localhost, port=9555, name=test, mux=ts,sap}

Or may be it's somewhere "under the hood" of libvlc rtp streaming API.

Anyway, you've been of great help! Thank you again chubinou.

[chubinou]

it generates a RTP/TS stream (mux=ts), which use a known profile

https://en.wikipedia.org/wiki/RTP_audio_video_profile

[chubinou]

btw, it is recommended to use an even port number for your RTP stream (https://tools.ietf.org/html/rfc3550#section-11)

[StewartLewis]

I see. Thank you for all this information chubinou !

[MikOfClassX]

Cool post!

How can we manage audio ?

Mik