I will say you have an interesting and rather colorful problem. Of coarse depending on if we are talking about, the red squiggly stuff in the background, the aspect ratio or the video not staying in side the player frame?
Is the aspect ratio normal when you go to full screen?
Have you tried the non embedded mode of VLC?
Have you tried to reset your video card (color, OpenGL, Direct 3D and overlay)?
Try shutting off the video acceleration inside VLC. Settings, Preferences, Video, untick the "Overlay video output" box and save. Now close VLC and restart. Then try to call a file, within VLC.
As a last resort, try opening VLC Settings, Preferences, Video, Output Modules and using the pull down to select "Windows GDI video output". Then save, restart VLC and try to call a file.
Most of these things are tests to try to expose your problem. So it is suggested to put things back the way it was if it had no change to your problem. Also always call a file into VLC to rule out a possible Windows interface problem.
Considering you are finding that some file types do not mess things up, I do suppose that somehow the file(s) are corrupt. Anything is possible in peer to peer networks. But when you say that DVDs are correct, I can't hep but wonder.
