(For next topics, think about giving your OS -can be done by posting in dedicated forum category- and VLC version)
I'm sure a simpler graphical method exists to archieve this (through preferences), but you can try this way:
1 - find the vlc config file ("vlcrc")
On Linux:
either ~/.vlc/vlcrc (up to 0.8.6) or ~/.config/vlc/vlcrc (0.9.0)
On Windows (for what I remember):
%APPDATA%\vlc maybe (or C:\Document and Settings\yourname\Application Data\VLC or ...VideoLan)
On MacOS X:
I don't know where is the preference file on MacOS X, maybe like Linux
2 - edit the file to locate this config (here is mine, maybe yours is set to 1)
# boolean (Allow only one running instance)
#one-instance=0
3 - of course, edit save and close :-)
If the config file does not set one-instance parameter to 1, then maybe the parameter "--one-instance" is given to VLC at launch time.