Hey,
Run VLC. Then go to settings:->preferences-> advanced
Then click on the check box -> advanced options
Check the "Allow only one running instance".
Save and close.
Then in command line type "vlc.exe vlc:quit"
Just tried it on WinXP Pro, VLC 0.8.6c