RTP timestamp, role of RTCP for AV sync with VLC client

[ekvlc]

I am writing a small RTSP server application. I have been able to stream audio and video from a single source and VLC clients plays them perfectly fine but NOT synced. I experimented many things to get AV sync working but no luck yet. It will be great help if someone can provide insight to internal working of VLC by answering following questions.

1. In RTP packets sent by server, does RTP timestamp have to be samplilng instance only? can it be presentation time? if yes, what unit milliSecond/microSecond?

2. If two stream are from the same source (and no RTCP from server), does VLC client try to sync these streams using its internal clock? OR does it adjust its internal clock based on the timestamp received in RTP packets? or it just plays the packets as and when it (VLC) receives them?

3. I just implemented RTCP mechanism for AV sync. I see fairly good AV sync. However, VLC client drops many audio and some video packets with errors "PTS out of range" or "late picture". So my question is - how it uses the RTP and NTP timestamps in RTCP packets from each of the audio and video streams to match with corresponding RTP timestamps in RTP packets?

Thank you very much.

[Rémi Denis-Courmont]

1. In RTP packets sent by server, does RTP timestamp have to be samplilng instance only? can it be presentation time? if yes, what unit milliSecond/microSecond?

The timestamp frequency is specified in the SDP, and the meaning of the timestamp is specified in the RTP payload format definition of the payload type (typically it is the PTS).

2. If two stream are from the same source (and no RTCP from server), does VLC client try to sync these streams using its internal clock? OR does it adjust its internal clock based on the timestamp received in RTP packets? or it just plays the packets as and when it (VLC) receives them?

This is mathematically impossible. The streams will be synched with an arbitrary unspecified offset, i.e. they won't be synched.

3. I just implemented RTCP mechanism for AV sync. I see fairly good AV sync. However, VLC client drops many audio and some video packets with errors "PTS out of range" or "late picture". So my question is - how it uses the RTP and NTP timestamps in RTCP packets from each of the audio and video streams to match with corresponding RTP timestamps in RTP packets?

This is clearly specified in RFC3550.