Well first off, I use Windows XP so maybe not everything I type will apply, but maybe I can still help. Try deleting the “vlcrc” file (deleting this file resets the preferences) located somewhere under your user. For Windows its: C:/Documents and Settings/User (You)/Application Data/vlc/vlcrc, but I’m not sure if it’s in the same directory on Mac OS-X, so maybe someone else can tell you where to find it on you’re OS, I do believe that deleting this file should fix the problem though.
P.S. I assume when you say “the Dock” you are talking about RocketDock. I have this program too, and it works really nice! Anyway, I hope the above information helps, or at least points someone in the right direction to helping you. Well I made an attempt, that’s worth something right?