reduce delay ( caching ? ) of network stream

[jacobs1]

Hi, I am using vlc as an rtsp server and client and I need to reduce the output delay.
In the server my input comes from ffmpeg mpegts mpeg4 udp stream, to localhost ( I must use ffmpeg because my actual source is raw rgb data ).
and the output is an rtsp stream ( no transcoding is used in vlc, is that at all a good idea ? )
the client is vlc in a remote machine on the same wifi LAN.
I am getting about 2 seconds delay ( from the time a frame leaves ffmpeg to the time it is shown in the client )

This is the server command:
udp://@:1234 :cr-average=10000 :network-caching=500 :sout=#rtp{sdp=rtsp://:5003/s,port=5004,caching=300,ttl=5}

Client command:
vlc --cr-average=10000 --network-caching=500 rtsp://@192.168.1.111:5003/s

if I reduce the network-caching the the client doesn't show anything.
Are there any other configuration I can change to reduce latency ?