media player states and events

[revel8n]

Is there any more detailed information on the state transitions a media player goes through, what information should be valid at the time of a state event, and how the current state affects the media player?

While searching i came across this discussion with similar questions and observations: https://stackoverflow.com/a/68491297

The state graph they created is helpful to an extent, but seems to only really cover the surface level events that occur when you allow vlc to manage most of everything for you.

Once you start including use of video/audio callbacks and encounter more complex video streams, a number of other events and considerations seem to come into play.

At this point most information about a stream has not been gathered by the time the Media Parsed event is received.
This post mentions similar observations: https://forum.videolan.org/viewtopic.php?f=32&t=148553

Some information appears to also not be in sync when the audio/video setup callbacks are made as well, often appearing to not be updated until a valid Vout event is received.

A media instance may also go through multiple transformations of its included tracks as well leading to needing to deal with ES Added/Deleted/Selected events as well.

i have also noticed a case where the ES events have actually caused the media player information to be incorrect, an example being streams with lower resolution previews, that eventually transition to full resolution representation.

In these cases the events suggest that tracks are added for the new resolution, then the prior tracks for the lower resolution are eventually deleted.

Primarily with the video track, the problem occurs with how the prior tracks are deleted, as in some cases it appears to issue a deselect, ES Selected with -1 as the track id, then ES Deleted when the track is removed.
In this instance, the track being deleted is not the currently active track, and it never issues an ES Selected for the actual track that was playing before the deletion, thus leaving the track unselected if you attempt to retrieve it afterwards.

i am unsure of how VLC is handling this case, but i have simply had to either ignore the fact that the track is wrong, keeping the prior one, reselecting it, and/or other workarounds.

This also highlights a difference between methods like "tracks_get" method on media and "get_track_description" methods on media players.
The media methods appear to get all tracks, even those that have been "deleted" from the player, while the media player methods return the current set of tracks.

- Is my understanding of all this correct regarding the apparent event flow i am observing?
- When can it be assumed that the data retrieve due to an event will be valid on the corresponding data object?
- Which events should/can be honored/relied on to be more accurate indicators of the current media and player state?
- Should the currently active track be validated to prevent the track being incorrectly disabled during ES Selected and ES Deleted events?

The width/height returned from video_get_size (and thus the video_get_width/video_get_height methods that wrap it) do not appear to keep up with the previously mentioned transitions from lower resolution preview to full resolution, still reporting the initial size after the transition.
- Which source of information should be considered the most accurate? The values present in a successful audio/video callback, track information from the media object, track information and width/height from the media player?
- Should this information be transitioning with the state of the media and player so that they return correct values?

The audio callback provides the timestamp, while the video callback does not appear to.
- What is the correct way to get this timestamp and correlate it to the audio?
- How do these relate to the clock and get_time values?
- Are there any recommendations when it comes to handling audio/video sync during playback?
- How does this process/information compare to what you might get back from the sout/smem workflow?

Any information you could provide would be greatly appreciated.

Thanks in advance.

[mfkl]

That's a lot of questions, and the documentation does not cover most of them.

I can see 2 options for you:
- look at the libvlc code and try to understand it to find answers to your own questions, since it's not currently documented. Many of your questions depend on the plugin used / the input media type as well, so no definite answer.
- if you're a company, hire a VLC developer for consulting help to speed things up.

[revel8n]

Ok, thanks for taking the time to reply.

As with my other question, a lot of what i have learned so far has been from observing the code flow, so i plan on continuing to do so in hopes of finding a proper way of handling the issues.

The ES selection and audio/video/timestamp sync are probably the main issues, with the former just being annoying as i do not like having such a workaround.

Then, also mentioned in the other thread, how does one go about getting their Gitlab access so that i can make proper issues and merge requests should i happen to find anything of use to contribute back?

[revel8n]

Still debugging through things to try and gain a better understanding, but thought i would ask in case anyone can help clarify.

There are a few places where the current es program is compared in order to determine whether or not to update the corresponding variables: https://code.videolan.org/videolan/vlc/ ... ut.c#L2196

When there are multiple es for a program, should this also be checking whether or not the one being manipulated is the current one in some way, so as to not update the current id selection?

EsIsSelected only checks that there is a decoder present: https://code.videolan.org/videolan/vlc/ ... ut.c#L1657
The decoder is not destroyed until the es is unselected, and a few other places, and there does not appear to be any checks to ensure that the id of the es being unselected during delete is the same one that is actually selected currently.

In cases where the es out policy is ES_OUT_ES_POLICY_SIMULTANEOUS, should there be additional validation for the current es id?
As video defaults to: https://code.videolan.org/videolan/vlc/ ... out.c#L326

Am i interpreting this process correctly?