Well unfortunately from my own tests and taking a quick look at the source code, there seems to be very little control over the speex codec. For example control over the comp, quality, vbr/abr and other options seems to be missing. I thought I managed to get ultra-wideband when testing before but I'm unable to when I try now so I think I'm wrong (also from looking at the source). Even bitrate control doesn't seem to work as you point out. However I noticed some potential problems with your options. Here's what I use and it works with very small files. E.g I'm able to encode nearly 2 hours in 20 mb. Here's what I use
:sout=#transcode{threads=2,samplerate=8000,acodec=spx,aenc=speex,ab=8,channels=1}:duplicate{threads=2,dst=std{access=file,mux=ogg,dst="c:\oggspeex",threads=2}} :no-sout-video
I don't think you need to specify threads=2 so many times but I did it anyway just in case since I was a bit confused by the options
Also the ab option doesn't seem to do anything. The key thing you may be missing is samplerate conversion. Speex narrowband is designed for 8000hz and if you don't convert it it just accepts it but if it's something like 16000hz or larger it'll be very large. If your source is already 8000hz then there's nothing you can do. 11025 is supported by narrowband but they say your mileage may vary. You may want to look into samplerate conversion in VLC and see if there are options which may give better quality. Also if you have a stereo source remember to convert to mono unless you really want the stereo.
Edit: Actually there appears to be a major problem with what I'm doing. I just compared it to what I get from the speex command line encoder and the file I'm getting from Ogg is a lot larger. It's slightly larger then quality=10 for CBR but it's actually quality=8. No idea why, I'll try some more later. I don't get it, perhaps there's something wrong with the muxing settings
Edit2: Confirmed since if I use raw the file is about right except it doesn't play
P.S. I think you can potentially specify MAX_FRAME_SIZE and MAX_FRAME_BYTES but I'm not sure. Doesn't seem useful to me so I haven't tried