I have a very similar issue as well. Whenever using frame forward, the player freezes a lot of the time (usually after a dozen or so frames, but sometimes I've been able to get to about 50 or so before it freezes).
Clicking on the X button to close the player after the freeze closes the window, but the VLC icon remains in the system tray. I can re-open/restore the frozen player window by clicking on the system tray icon, but the player is still frozen at the same frame. I should note that in this frozen state, I cannot resume playback of the video, cannot stop and restart the video, and cannot open a new video file either. Right clicking on the system tray icon and selecting "Quit" does nothing (doesn't even bring up the windows force-kill dialogue). I have to manually open up task manager and kill the process myself. Unlike the original poster, after I manually kill the process I can re-open VLC and playback the file just fine. I have tested xvid and h264 encoded videos, and while both are susceptible to the problem, the h264 videos seem more resilient (I can usually can many many more frames from them).
Here are sample clips of xvid and h264 encoded video's I've been able to freeze consistently:
XviD sample -
h264 sample
NOTE: When I tried making the clip of the MP4 file encoded in h264 using the new VLC record button, it changed the container to .ts and I could not get this clip to freeze. Does this mean it could be a demuxer issue? (as far as I can tell from the modules tree, the demuxer is the only thing that changed between the .ts and .mp4 clips) You can snag the .ts clip
HERE if needed. I also noticed that these small clips didn't freeze as easily as the source video's they came from. Thinking it could be some sort of buffering/splitter issue, I tried frame forwarding immediately after a jump using the time bar and the freeze did seem to occur more frequently in this situation.
I'm running WinXP SP3 with a P4 2.8HT under the hood btw. Let me know if there is anything else I can do to help pinpoint the cause of this problem.