Auto-sync subtitles

[rocky]

We all know that sometimes, the subtitles we download for a movie are out of sync with the audio. Like other media players, VLC offers the possibility to "move" the subtitles to sync them with the audio, but doing so is tricky.
My suggestion is to develop some technology that auto-detect the voices, and to add a button that tries to sync the subtitles with them. It sounds difficult, but a lot of work already exists in the field of speech recognition, which could be helpful. The code could sweep through the whole audio until it finds some words that are clearly recognized (not necessarily the first words in the audio), then look for those words in the subtitles file, and sync them.
If effective, the feature would once again distinguish VLC from the competition. Also, it would solve the annoying problem of finding the right subtitiles / syncing them with the audio, and eventually, it could help to provide the option of auto-downloading the right subtitles for any movie.

[DGMurdockIII]

i agree the subtitle syncing feature kinda still need to be improved alot but i do agree that you have improved the subtitle support alot over the past 2 years but subtitle syncing still need more polish

[sccy]

Auto-sync is not a mature technology. Use hotkeys G and H for now.

[rocky]

Thanks for the answers.
I post this reply just to clarify something. The speech recognition software doesn't need to be perfect. All it has to do is recognize with some confidence three or four words in the entire audio. By syncing those few words, the rest of the subtitles would be synced too. And if the software misses every now and then, it wouldn't really matter (we would just have to download another subtitle, or sync it with G and H, as usual).

[asteriskk]

You can also use an online tool such as http://www.sync-subtitles.com to sync subtitles.

[XweAponX]

I suppose with all of the different types of Subs, this would be difficult to implement. There is a script addon that allows you to search for the exact version of the release you are playing - It downloads Subs "on the fly" right into the folder where you have the content stored. I found a windows installer for it, but it is downloadable as a script from some links here:

viewtopic.php?f=29&t=71535

Because with all of the content I have saved, I've found out - I MUST have the exact release for the subs I've downloaded, if you try to use any other subs, they will never sync up even using the hotkey G-H solution. I've also made my own subs for certain media I have, you can do it "by hand" or you can use NCH VideoPad Video Editor to add them in.

[naderghanbari]

Is there any activity on this? I agree that using manual sync feature is very useful but in most cases auto syncing (at least just readjusting the delay or offset for english, french, and german) is very easy. I tried to do it (on a separate project) with some success and I feel confident enough to work on it but I'm not sure whether people actually need it or not.

[Overdrivr]

Hi folks,

I know this is an old post but I wrote a tool that uses machine learning to do this. It finds the optimal subtitle's delay and framerate using a Machine Learning speech detection model. It even downloads the subtitle automatically for tv series (movies soon as well), by using the filename to guess the subtitle to fetch. It works well, at least on many tv series I've tried synchronization is very good. It can fix delays of up to a few hundred seconds (for instance when the tv series has an introduction summary but the sub has not), with an accuracy of ~ 80ms. It's not synchronizing each sentence individually thus far, but that's something that I'll might implement in the future.

It's still recent, so there might be some gotchas to figure out. Give it a try and let me know if something's not working, or help fixing it - the tool is free and open-source

Available here: https://captionpal.org

Cheers,

[unidan]

Hi Overdrivr, it's been a while, very nice project!

There also have been other people doing this, like https://github.com/smacke/subsync
Do you plan to integrate this as a module inside VLC? It would be really nice to have but keep in mind that VLC is a (L)GPLv2 project. It would be ok if compiled as an external DLL but would remove every use case needing (L)GPLv2 with a static build, like VLC for android. It would be awesome if you could provide it under at least GPLv2 too.

I can try to provide some help on the integration if you would like it.

[Overdrivr]

Hey, thanks for the feedback ! Appreciate it

Yes there's plenty of folks who've done similar stuff ! This technique works very well. But as far as I know those apps are mostly proof of concept command line applications that require python and pip install, not actual application you can install and that has buttons for non-technical folks. And I don't think any of them downloads the sub directly from the filename (well, I guess there must be some that does but don't know about it).

I'm not sure about the VLC integration yet. It would require some extensive effort and maintenance (builds for every VLC-supported platform, CI pipelines, conversion to C code, etc) that I'm not sure I can commit to. As it is now, maintenance is pretty simple and everything's already pretty much automated, so maintenance is a breeze. If I can ensure a reliable synchronization on a larger scale (when enough people will be using this regularly), then maybe I'll reconsider.

One thing to consider is that VLC is also used to display video streams (with PopcornTime for instance), but synchronization requires a good chunk of the audio track to work so I'm not 100% certain it would cover this important case.

Anyway, any feedback or contribution on the app is very welcomed

[cbeltrangomez]

When I try to download the windows version of https://captionpal.org/

I got this:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Details>
No such object: captionpal-releases/0.12.0/captionpal-Windows-10-10.0.14393-SP010.exe
</Details>
</Error>
( function(){ window.SIG_EXT = {}; } )()