OK, I separated enough code to make a sample, but that works
Apparently VLCJ doesn't like if the video canvas is in a panel, which is located in another panel, and that panel is located in a JFrame (so JFrame -> JPanel -> JPanel -> Canvas (VLCJ)). I tried to cut one panel, and it works (without decorations).
Edit: The bug is (surprise, surprise) in Swing. When the JFrame is decorationless, inner panel (which contains Canvas) ignores all orders to set its sizes (setSize, setMinimumSize...), when they're called from JFrame's ComponentListener's componentResized first call (e.g. when JFrame is being shown).
By ignored I mean that the same calls make impact, when JFrame has decorations... but when it doesn't the JPanel has size of [0,0].
As always, Sun shows how "great" Swing is...