I have also been looking at this.
As fas as I know there is no way of getting IE to quit downloading a certain file type and then use another program to download that file (which would be VLC in yor case)
The exception is for the file types known by WMP such as MP3, WMV where IE insists on using Windows Media Player -- even if you have configured Windows to use something else....
Yet another reason why IE sucks.
Life is a bit easier in other browsers. Support for using external downloaders is built into Opera, and is available with an extension in Mozilla Firefox... I have been writing up some documentation on how to configure this for a streaming server I am writing...
http://www.sage-community.org/index.cgi ... pp#preview