Did you try to set the size directly in the transcode options?
I've used transcode now as follows:
H:\>vlc.exe rtsp://192.168.20.102:554/ch0_unicast_firststream --aspect-ratio=1.333 --sout="#transcode{vcodec=mp4v, width=320, height=240}:duplicate{dst=display, dst={std{access=file, mux=avi, dst='C:\Users\rja\Desktop\aspectratiotest.avi'}}}
"
Two things changed there.
1. Moved --aspect-ratio to before --sout. I find the display window is now set to the correct aspect ratio and if I look at the metadata with MediaInfo while streaming then the display aspect ratio is set to 4:3 as I would like. However When the streaming to the file is stopped the display aspect ratio meta data is set to 1.1111 (10:9) which is the aspect ratio of the source.
2. Added the transcode option set the horistontal and vertical resolution to what I want. The meta data in the file shows this resolution and when played from Window Media Player it appears as the correct aspect ratio but playing the file in VLC player results in the wrong ratio. I think this is because WMP ignores the display aspect ratio setting in the meta data and vlc does not. As I mentioned above the display aspect ratio is set incorrectly when the stream is closed.