By far the best looking video's and support i've managed to get from VLC while running Vista was to set VLC's output to OpenGL of all things.
It works great, even better then the Windows GDI output.
This holds true for 32-bit and x64 editions of Vista.
Just my .02$