I'm having the same problem with VLC 0.8.4 on my G4 running OSX 10.2.8 and the Apple 17" Studio Display. Whether I play a VOB file from a DVD or an .avi, .mpg, or .asf file from the hard drive, the video image splits into two halves and the color is very magenta saturated. The older version of VLC 0.8.1 still works fine on the same machine, though.
VLC 0.8.4 runs okay on another G4 (OSX 10.2.
I have and an eMac (OSX 10.3.9) both of which have CRT monitors.
I'm wondering if it has something to do with the Studio Display LCD monitor. Can anyone help us out?