I have a vlc application that I upgraded from 0.9.9a to 1.1.3, and now the x11 video output windows no longer work as they did before. This is an application that receives an h.264 codec rtp video stream from a video phone, and outputs it on a Linux system. The previous code using 0.9.9a would create two windows with "VLC (XVideo output)" or "VLC (X11 output)".
Now the windows say "VLC (Colour AsCii Art)". More importantly, the output window looks like the bytes from the movie the Matrix. There are a lot of bright blue and green "t%8SX" characters filling up the entire screen. You can still make out the background outline of the person in the video phone window.
What functions should I be looking at that will have the applications X Window output behaving the same way, within the application?