libVLC Video Synchronization
Posted: 11 Oct 2022 10:42
I need to implement video playback that can synchronize to audio.
My use case is an audio/music application that can synchronize audio playback to other devices on the local network.
The important thing to note is that I cannot synchronize audio to video (but the other way around) because it would mean a complete overhaul of the entire playback engine.
Now I want to add the capabability to play back video (just plain play back without any effects).
Both video and audio live in the same process (obviously on different threads though)
What I have done on iOS:
- Get synchronization info from the the audio thread conisting of: position in the time line to absolute wall clock time
- Tell AVFoundation's video player to synchronization to that info via https://developer.apple.com/documentati ... 91-setrate
This AVFoundation's setRate function is immensely handy for me and from a video playback perspective I am done on iOS.
Now I need to implement the same thing for the Universal Windows Platform.
I have tried to use the way described here https://learn.microsoft.com/en-us/windo ... ediaplayer.
But it seems the built-in video player isn't built for this kind of use case, which is sort of expected. In essence the uwp's MediaPlayer is not accurate enough.
- It lacks the mentioned setRate function for synchronizing video clip time to absolute wall time.
- Trying to play back anything () it will of course play back, but not will always have an unpredictable delay.
- Also (which is really strange): setting its position to (for example) 2 seconds, will actually not set it to exactly 50 frames in a 25 fps movie clip. It is always slightly off.
I have looked into libvlc (and especially libVLCSharp), but I am not sure if I can accomplish what I need with it. Maybe I am missing something though.
What I essentially need is either of the following:
- A way similar to AVFoundation's setRate synchronization -OR-
- A way to start video play back immediately when invoked
Or of course a completely different way I haven't thought of.
My use case is an audio/music application that can synchronize audio playback to other devices on the local network.
The important thing to note is that I cannot synchronize audio to video (but the other way around) because it would mean a complete overhaul of the entire playback engine.
Now I want to add the capabability to play back video (just plain play back without any effects).
Both video and audio live in the same process (obviously on different threads though)
What I have done on iOS:
- Get synchronization info from the the audio thread conisting of: position in the time line to absolute wall clock time
- Tell AVFoundation's video player to synchronization to that info via https://developer.apple.com/documentati ... 91-setrate
This AVFoundation's setRate function is immensely handy for me and from a video playback perspective I am done on iOS.
Now I need to implement the same thing for the Universal Windows Platform.
I have tried to use the way described here https://learn.microsoft.com/en-us/windo ... ediaplayer.
But it seems the built-in video player isn't built for this kind of use case, which is sort of expected. In essence the uwp's MediaPlayer is not accurate enough.
- It lacks the mentioned setRate function for synchronizing video clip time to absolute wall time.
- Trying to play back anything (
Code: Select all
mediaPlayer.Play()- Also (which is really strange): setting its position to (for example) 2 seconds, will actually not set it to exactly 50 frames in a 25 fps movie clip. It is always slightly off.
I have looked into libvlc (and especially libVLCSharp), but I am not sure if I can accomplish what I need with it. Maybe I am missing something though.
What I essentially need is either of the following:
- A way similar to AVFoundation's setRate synchronization -OR-
- A way to start video play back immediately when invoked
Or of course a completely different way I haven't thought of.