When you open a file, disk, Capture device or Network stream within VLC there is a menu that pops up. On this Menu are "Advanced options", check the box marked "Stream/Save" and press the highlighted button marked "Settings". This brings up menu called "Stream output"
All of the options for transcoding are on this page. However you will need to know what formats (for audio & video) are compatible with what containers (encapsulation method) and what formats and containers are compatible with what ever other programs you have installed on your computer.
Please note: At the top of the screen there is a long white strip that is marked "Destination Target:" When you are entering information this is changing. This is the internal command line for VLC. For example:
:sout=#transcode{acodec=mp3,ab=128,channels=2}:duplicate{dst=std{access=file,mux=raw,url="C:\Documents and Settings\Owner (this is you)\My Documents\My Music\My.mp3"}}
Will make a compatible mp3 file from a wav file.
I can not in this space tell you what is ideal in every situation, that is why there are options. At the same time I can not tell you which ones don't work together as some of these were never finished (they do work but perhaps not as expected), so that all the accepted variations for all the various formats and containers would work together.
While I believe that any format should be compatible with any container and all containers should be capable of streaming, that's not how it is in the real world.
For most people this is a learning process.
The key issue is:
1. Know your compatible formats and containers.
2. Know your source file types and the containers so you can make intelligent decisions for quality with relatively low overhead when transcoding files or streaming.
3. Know what the programs installed on your machine will accept for both container and format.
http://wiki.videolan.org/index.php/Main_Page
Tons of information on formats, containers, streaming and how to