I have a similar problem trying to convert video from one format to another. I get the message:
"Streaming / Transcoding failed:
VLC could not open the encoder."
Unfortunately I don't have the solution but wouyld add my request to yours for more information if possible.
Clive