Play DVD w/external sound source - Architecture Question

[number42]

Hello!

I hate starting with apologies, but let me clear the air. First, I apologize for being new here. Secondly, I apologize for being new to VLC development. Third, I apologize if I've posted in the wrong category and this would be a topic better suited for another discussion. Fourth, I apologize if this has been discussed elsewhere, but I have searched here, google, and the IRC chat and not found a resolution.

With that out of the way, I am researching a way to use VLC to package an end-user application that will play a DVD along with a separate MP3 sound track, appearing to the end user as if it were a commentary track on the DVD. I am still doing my fundamental analysis on this and am looking for suggestions on how to proceed - I'm definitely not looking for you to do my work for me, I'm just making sure I'm not reinventing the wheel and that I'm going down the right path.

There are rumors that there is experimental code to play simultaneous streams/channels in VLC, but I have not found much more about this than rumors.

Our goals:
* End user friendly (shouldn't require an IT expert to use)
* Cross platform (Linux/Windows/OSX)
* Play standard DVDs on computer's DVD hardware along with an accompanying MP3 elsewhere on computer
* Keeps DVD and MP3 in sync with each other

I have thought of ways of hacking VLC to create a custom solution, but without completely rewriting parts of it I am thinking of the following architecture:

* Create a separate "master" application that will launch a VLC "Video Process" for the DVD with Telnet enabled
* Have "master" app also launch a VLC "External audio" process for the MP3 (thus two separate intances, both with Telnet enabled, will be running simultaneously)
* "Master" app reads sync information from an XML file that would map from one to the other (i.e., 00:43:37 in DVD = 00:44:21 in MP3)
* Have this master process control the two slave processes behind the scenes via Telnet (polling each to see where they are at, seeking each to keep in sync, responding to detected play/pause/rewind requests from user, etc.)
* Modifying the base VLC is totally acceptable - especially in areas where the Telnet command set may need to be expanded to support more options

ActiveX would be a nice Windows-only solution, but obviously won't work since I want to keep it cross-platform.

My questions:

First - am I going down the right (best) path? Is there an easier way to do this?
Second - Is this a better question for the Scripting part of the forum?
Third - has anyone ever tried doing anything like this before? (And if so, is there an example out there, somewhere?)

Any code I write for this I will also publish as an Open Source project if it's of any interest to anyone else.

Thank you very much in advance, I'm very much looking forward to any suggestions you might have.

- Tony

[Rémi Denis-Courmont]

Hmmph. Roughly, you could just open the DVD with the MP3 track as a slave input. This ensures that the two sources will be kept in sync, as they will share the same clock reference. But there are a issues.

First, you will need to ensure that the MP3 has the timestamp reference as the DVD. That's perhaps an issue with the authoring software for the MP3 more so than VLC itself.

Then, DVD menus will probably have to be disabled... or else it's going to be a lot of fun for synching purpose...

Also, software mixing in VLC is supposed to work, but is hardly ever used. Last time I checked, it worked fine but I only tried two audio sources with identical channels configuration and sample rate. I would hardly be surprised if "interesting" bugs showed up in more tricky cases. And I don't need to mention that VLC has a single volume control...

Last, the core currently does not support selecting but one audio track. To play more than one, there is a trick:

Code: Select all

vlc --sout-all --sout '#display'

but this is really a hack, and will play all languages on the DVD. There is no fundamental limitation, but nobody ever bothered to change the elementary stream output API to select more than one audio track.

[number42]

Hi Rémi,

Thank you very much for your input. There certainly aren't many more authoritative sources than yours on VLC and I'm happy for the expert feedback.

First, you will need to ensure that the MP3 has the timestamp reference as the DVD.

Done, and with nearly frame-accurate detail, often creating different versions of the MP3 to match different retail versions of the DVD, as well as creating separate NTSC and PAL versions of the MP3.

Then, DVD menus will probably have to be disabled... or else it's going to be a lot of fun for synching purpose...

That's probably a show-stopper right there (even if you ignore the other issues that you mentioned, such as only one audio control.) Since this has to be end-user friendly (with a custom skin) disabling the DVD menus unfortunately isn't an option.

but nobody ever bothered to change the elementary stream output API to select more than one audio track.

That may actually be a clue right there as to a "better architecture" - possibly modifying the elementary stream output API (as you call it, I haven't looked at the code yet) to find a way to select just a single audio track on the DVD and then possibly use the other mixing option that you mentioned to mix it with the external MP3 source.

I'm proceeding with my Telnet idea (for the moment, unless another potential solution presents itself) as it has the advantage of being able to rapidly produce a demo/prototype, but I will also investigate the option of going into the low-level sound code and finding a way to mix it there for the final solution. Thanks again.

- Tony

[Jean-Baptiste Kempf]

I have more or less then same analysis than Rémi.

If you want to keep DVD menus, then it is going to be tricky, but not undoable, since you need to launch an extra audio when it is launched, as we do it with subs.

[number42]

since you need to launch an extra audio when it is launched, as we do it with subs.

Thanks for the input - this sounds interesting. Could you please clarify what you mean by "as we do it with subs" or is there an area on the Wiki I can reference for that? Thanks!

- Tony

[Jean-Baptiste Kempf]

When playing a file, you can drag and drop a sub, ie a SPU_ES track to the player and it adds it to the playback.

[number42]

When playing a file, you can drag and drop a sub, ie a SPU_ES track to the player and it adds it to the playback.

Thanks again!

Sorry to sound dense - "sub" or "SPU_ES" - refers to a subtitle or subtitle format? (I'm not finding any Wiki info on this, but found some other forum references.) Thus your explanation has something to do with two instances of something running at the same time when subtitles are being displayed?

I'm sorry, I'm very new to VLC (and DVD formats) and just embarking on this journey. I'm not familiar with the jargon yet. My background is in device drivers, bitmapped/3D graphics, raster to vector conversions, and text recognition. This whole "moving picture and sound" thing is new to me.

- Tony

[Jean-Baptiste Kempf]

sub is short for subtitle
SPU_ES is what VLC internally calls subpictures tracks.