How to detect dropping frames

[Talkless]

Hi,

I'm using libvc 2.2.7-1~deb8u1 on Debian Jessie via VLC-QT wrapper library to display RTSP stream from some IP camera. It's a h264 @ 1280x720.

When using 200ms buffer, after some time (more than 10s or smth.) video freezes, while these log lines appear:

(please note, these "state: ... " outputs are my addition in the libvlc_callback)

state: 3 event: 268 MediaPlayerPositionChanged
state: 3 event: 267 MediaPlayerTimeChanged
state: 3 event: 268 MediaPlayerPositionChanged
state: 3 event: 267 MediaPlayerTimeChanged
libvlc: picture might be displayed late (missing 0 ms)
state: 3 event: 268 MediaPlayerPositionChanged
state: 3 event: 267 MediaPlayerTimeChanged
libvlc: picture might be displayed late (missing 0 ms)
libvlc: picture might be displayed late (missing 0 ms)
libvlc: picture might be displayed late (missing 0 ms)
state: 3 event: 268 MediaPlayerPositionChanged
state: 3 event: 267 MediaPlayerTimeChanged
libvlc: picture might be displayed late (missing 0 ms)
libvlc: picture might be displayed late (missing 0 ms)
libvlc: picture might be displayed late (missing 3 ms)

Some more time later:

libvlc: More than 4 late frames, dropping frame
state: 3 event: 268 MediaPlayerPositionChanged
state: 3 event: 267 MediaPlayerTimeChanged
libvlc: More than 4 late frames, dropping frame
libvlc: more than 5 seconds of late video -> dropping frame (computer too slow ?)
libvlc: more than 5 seconds of late video -> dropping frame (computer too slow ?)
libvlc: more than 5 seconds of late video -> dropping frame (computer too slow ?)
state: 3 event: 268 MediaPlayerPositionChanged
state: 3 event: 267 MediaPlayerTimeChanged
libvlc: more than 5 seconds of late video -> dropping frame (computer too slow ?)
libvlc: more than 5 seconds of late video -> dropping frame (computer too slow ?)
libvlc: more than 5 seconds of late video -> dropping frame (computer too slow ?)
libvlc: more than 5 seconds of late video -> dropping frame (computer too slow ?)
libvlc: more than 5 seconds of late video -> dropping frame (computer too slow ?)
libvlc: more than 5 seconds of late video -> dropping frame (computer too slow ?)
state: 3 event: 268 MediaPlayerPositionChanged

I doubt that my computer is slow, as with libvaapi_x11_plugin.so in use (as perf states), it's only 8-9% of CPU usage (around 30% without vaapi.)

Anyway, the question is not why frames are dropped (although understanding would be nice ), but how could I reliably detect that programically?

VLC-QT is attached to these events [0]:

list << libvlc_MediaPlayerMediaChanged
<< libvlc_MediaPlayerNothingSpecial
<< libvlc_MediaPlayerOpening
<< libvlc_MediaPlayerBuffering
<< libvlc_MediaPlayerPlaying
<< libvlc_MediaPlayerPaused
<< libvlc_MediaPlayerStopped
<< libvlc_MediaPlayerForward
<< libvlc_MediaPlayerBackward
<< libvlc_MediaPlayerEndReached
<< libvlc_MediaPlayerEncounteredError
<< libvlc_MediaPlayerTimeChanged
<< libvlc_MediaPlayerPositionChanged
<< libvlc_MediaPlayerSeekableChanged
<< libvlc_MediaPlayerPausableChanged
<< libvlc_MediaPlayerTitleChanged
<< libvlc_MediaPlayerSnapshotTaken
<< libvlc_MediaPlayerLengthChanged
<< libvlc_MediaPlayerVout;

And it seems it's not enough. I expected to get at least buffering event.

Could some other events (or technique) help to detect frozen video? I have to detect that as-real-time-as-possible to show appropriate feedback to the user. I can't simply show frozen picture as if everything's OK.

I could.. parse the log messages, but that's really doesn't seem right .

I managed to handle buffering events, but not sure how to deal with this case.

Thanks!

[0] https://github.com/vlc-qt/vlc-qt/blob/8 ... er.cpp#L99

[Talkless]

I can see that lock, unlock and display callbacks are still called, around 10fps (original stream is 20pfs), when actual video is frozen (due to too-low 180ms buffer for h264 stream), which.. does not help .