Hi,
first the specs:
3 GB RAM
Windows 7 Ultimate 64bit
VLC Player 1.1.11
Firefox 7.0.1
Recently the following happens to me when I try to watch a new video:
VLC Player starts
Windows switches into Basic UI Mode
VLC Player audio starts but screen remains black
When I close VLC Player, Windows returns to normal UI mode.
After a bit of testing I found out that it's directly connected to the plugin-container.exe of Firefox, especially its memory consumption. It only happens if that process consumes more than ~600 MB (And I have no idea why it needs that much RAM anyway).
When I kill that process via Task Manager and start the video anew, everything's peachy again.