have the same problem on Vista SP1 x32 and x64 systems, running VLC 0.9.6.
Although it only appears when I change the DPI Scaling ( under Control Panel\Personalization -> left nav bar "Adjust font size DPI") to Large (120 DPI), Custom 150% (144DPI) or 200% (192DPI).
Cause I have my media PC connected to a 46" LCD TV with 1920x1080, I have to use at least a 150% scaling, else the fonts are too small.
If I change the Scaling to Default (96DPI), VLC works fine as usual, I just have to stand in front of the TV to be able to read and pick a file
Hope that helps with the troubleshooting.
Cheers
That did help, thanks. I had been trying to figure this out and couldn't find a solution, probably all over the place and I am going to look foolish signing up for the forum just to post a solution I found playing around... but anyway "Ivo007"s DPI comment helped me (across the room, large flat screen, high DPI to be able to see fonts, same situation I am in), ok, back to the point... I got it squared away. Rather simple, maybe it will work for you... basically go to vlc.exe compatibility tab and check the Disable DPI scaling. After that, it worked like it always had before installing vista 64 a few days ago... I tried to get used to 'taskbar not on top of other windows' thing... but my OCD was going crazy.
so, I hope that helps. I wonder if the VLC team could build into their installer to check windows DPI during setup, and if it is set above norm to disable it automatically (default)... and maybe checkbox in prefs to enable it for whatever reason if someone wants it... now that I think about it, you guys already probably have it in there and I just didn't see it.
so short story for newbies like me:
1. close VLC if you have it running.
2. go to folder where VLC is installed (something like: C:\Program Files (x86)\VideoLAN\VLC )
3. Right click vlc.exe and choose properties
4. click Compatibility tab
5. click the checkbox next to "Disable display scaling on high DPI settings"
6. click OK and go test it.
Hope this helps someone out.