I expected to find that in the IP camera's RTP audio stream, each audio frame's timestamp would be exactly the previous one's plus the frame length (aka duration). But I've already seen on a couple brands of cameras (Dahua, Foscam) that this isn't true. Eg I'm looking at an AAC stream using frame length 1,024, sampling frequency 8,000. The timestamps differ by 1008, 1008, 1056, 1008, 1008, 1008, 1056, ... If you take several frames and average them, it will usually be about 1024 as expected, but it's not exact from frame to frame. I also noticed tools like ffmpeg (with eg "ffmpeg -rtsp_transport tcp -i rtsp://... -c copy out.mp4") will actually write durations of 1008, 1008, 1056, ... in the .mp4 file's stts rather than the true frame length.
What does VLC do when it sees this? Does it play overlapping audio when frames overlap and silence when there's a gap? does it try to speed up and slow down the audio (I noticed VLC has some "scaletempo" code)? does it delay frames that overlap so they become consecutive? etc. I want to know if it's better to faithfully preserve the timestamps I got from the camera or try to "fix" them (and how).
I tried answering this by experiment: writing out an .mp4 with original durations and one with them all changed to 1,024, then playing them through VLC. They sounded equally awful to me. This camera's microphone is not very good and maybe its audio code is pretty buggy overall. But I'd still like to figure out what players expect so I can make the best of what I have. I also didn't see anything informative (to me) in VLC logs, though I can post them if that'd help.
Any help appreciated—you could save me a lot more trial and error and/or reading through a lot of code.
I also see occasional larger time adjustments, maybe when the camera's clock is stepped with SNTP. I've noticed some cameras use scaled CLOCK_REALTIME values to generate their RTP timestamps on the video stream
; maybe the same is true on the audio stream. I'd like to do something to improve that...