Hi,
VLC 2.0.0 could not playback an RTSP stream correctly on a Windows XP SP3 32bit 4GB machine with a dual-head 128MB video card installed. The initial symptom was that only one corrupted frame of the video was displayed (about a quarter showing the correct video and the rest scrambled), then VLC playback stopped and the application became unresponsive. Changing video resolution did not fix the problem but the nature of the corruption changed. By switching the video output the DirectX, we did get some video displayed but only about 2 seconds worth then the application froze.
We changed the dual head card to a GeForce 7300GS with 256MB, and the problem disappeared completely - video does not freeze, and we did not have to change to DirectX output.
Maybe VLC 2.0.0 is not correctly detecting the amount of video memory available?