I have this same problem when playing DIVX5.2.0/mp4a(8c).
Also I'ved noticed throughout this anime series, Get Backers (
http://www.mininova.org/tor/859947 ), that VLC spikes to ~619 MB of VM!
Oddly, there are h.264/mp4a(8c) files that playback fine, but still cause the VM of VLC to go upto ~619 MB.
Draw your own conclusions.
P.S. Had to use The Core Media Player, which other than odd functionality... played them fine. Weird thing would default with subtitles, so I managed to disable them.... Now I can't get them back! Oh bother.
Yuo're damned if you do and you're damned if you don't.