Automatic synchronization of subtitles with video

[smacke]

Hello,

I developed a prototype script for automatically synchronizing subtitles with video. Here is a link: https://github.com/smacke/subsync

It seems to work surprisingly well, and correctly synchronizes out-of-sync subtitles to within ~1s of correct more than 95% of the time. I've found this to be useful when a video for a TV episode includes a recap of the previous episode but the subtitles do not, or vice-versa. Since it works via speech detection only, it is language agnostic, and can properly synchronize regardless of the video / subtitle language or whether they match.

The linked project includes a patch to VLC 3.0.6 (at https://github.com/smacke/subsync/raw/m ... -vlc.patch) that makes this capability available to VLC, but since this was a hackathon project, it is mainly optimized for demoing the capabilities of the script, and isn't up to the standards users have come to expect. For example, the autosync hotkey (Ctrl+Shift+S) blocks the input thread, and as a total noob to the codebase, I wasn't able to figure out how to get the various threads to communicate properly, and instead perform inter-thread communication via the filesystem (terrible; I know). Also I didn't pay any attention to memory management, so I'm sure there are leaks.

3 questions:

1. Is this something people would like to see properly integrated into VLC?
2. If yes to (1), any ideas as to the proper design? Call the script as a subprocess from VLC as done in the hacky patch, implement as an extension in lua, integrate directly into the core, or something else altogether?
3. If yes to (1), are there any resources available for understanding how to properly communicate between threads in VLC? I admittedly haven't spent more than several hours with the codebase, but I found it to be fairly opaque and had trouble finding documentation on this front.

Thanks an advance for any guidance!

Best wishes,
Stephen

[Jean-Baptiste Kempf]

Wow, that is very cool.

1. Yes.
2. A plugin, in C/C++.
3. Normal pthreads.

[smacke]

Glad to hear that there is some interest!

Regarding the threading: actually, it's not pthreads that was giving me trouble, it was with trying to understand the various data structures and how thread A should get at data local to thread B, or which thread A should talk to in order to get a particular piece of data. For example, let's suppose I need to get the filepath to the currently playing video. What's the best way to figure out to whom I should talk?

As another example, there are a bunch of getters / setters of the form var_getData / var_setData that take a thread (or a general vlc_object_t? not sure) and a string argument -- is there any strategy for figuring out whether "viewpoint-changeable" is available on, say, p_vout?

Apologies if this is already documented somewhere; either way, any pointers are much appreciated!