low latency vs caching creep

[deuce]

I have an odd issue with a low latency real time streaming.
I'm consuming a 15-30Mbps SRT stream.
I'd really like to set it at 0ms but have the network-caching set to 500ms since <500ms has inconsistent results.

The stream works fine this way, and 500ms is an acceptable delay, but my problem is that it "creeps" slowly over the course of time up into multiple seconds and I cannot figure out why.
Stopping and restarting the stream gets me back to 500ms but again it starts rising.

My use case requires <1000ms delay.

Does anyone have any idea how to achieve this consistently?

Thanks!