You can substitute "320k" with whatever constant bit rate you want and you can change .mp3 to .ogg, although that appears to create an ogg video file rather than an ogg audio file.
I used the invocation below to force a straight copy of the audio into the same codec (vorbis) and disable video:
$ ffmpeg -i video.webm -acodec copy -vn audio.ogg
Sorry to resurrect such an old topic, but I just found this message (thanks google) and found it helpful.
Thanks.