Resurrecting this old thread to add a possible fix for this common problem:
I recently had this issue with a bunch of files I had just encoded. My vlc player rarely had problems with playback before so I googled the symptom and this thread came up. Increasing the disk cache as described by
bunnykhatana is one way to remedy the issue, but it doesn't get to the heart of why some files play fine while others won't.
It turns out my files that were specifically causing the issue had encoded with only 1 reference frame. After some googling I found that most h264 profiles should be set to 3. iTunes has a default of 4 for its 1080p downloads. I went back and re-encoded them, this time specifying a reference frame setting of 4, and from the looks of it vlc no longer stutters like it used to.
If anyone still experiences this problem, download
Mediainfo and analyse the metadata of the file that's having trouble playing smoothly. Does it have more than 1 reference frame? Is the profile level 4 and above? Is the profile baseline, main or high? It could be that the file itself is causing the issue due to how its encoded.