This problem persists in VLC 2.0.1, and has never been fully addressed.
Simple way to reproduce it 90% of the time. Set up Audio Hijack to use VLC to record something off the internet. My typical case is to use VLC to open the following AppleScript as a .scpt file:
---------------------
tell application "VLC"
activate
GetURL "
http://www.publicbroadcasting.net/wqcs/ppr/wqcs.m3u"
play
end tell
---------------------
I use an AppleScript because, at least when I set up this recording, it was the only way to get VLC to open the URL I wanted and play the stream.
Also, set Audio Hijack's schedule Actions to Record, Mute, and Quit Source. I also set it to Quit Audio Hijack Pro.
Schedule a recording. A few minutes is enough. Everything records fine. 90% of the time, when the recording is done, VLC remains visible until I move the mouse over its window. It is still running and playing the stream, though it quits immediately on a mouseover. Subsequent Audio Hijack sessions do not record until I manually quit VLC by pointing to the "dead" window.
This all worked fine until a 1.x revision of VLC. I believe 1.1.1 was the last one to work, and I'll be reverting to that version, since I've lost many shows I intended to record.
Thanks for listening. I think VLC is fantastic, but my main use for it is broken.
P.S.: I wrote that the problem wasn't present in version 1.1.1. I was incorrect. I was able to reproduce the problem in 1.1.1, but not 1.0.5. This is the Universal version, and I'm running Snow Leopard.