I am using vlc version 1.1.13 on Fedora 16 with xfce. This happens to me but only if the vlc window is maximized when I try to enter fullscreen. The easy fix for me is to make sure the window is not maximized before going into fullscreen mode then it works fine.