If you try to over amplify a file, you can go into clipping, My suggestion is to detect clipping and knock back the volume to a point which would not have allowed the clipping. Clipping is so easy to detect, if you are using 24bit audio then you have numbers from -8388607 to 8388608 to use... just don't allow it to use the last one.. if you multiply by the gain factor and end up with a number greater than 8388607, or less than 8388606 then you just figure out how much you overshot it by and knock back the gain to the maximum you could have to still be always within that range. That way if you had it cranked up to compensate for file with very low volume, then the next track plays which is normal, it will cut it back automatically and save your ears and speakers from annoying clipping.
This won't address the other strange artifacts that happen with VLC even when it doesn't clip at volumes above about 126% but it's the easiest to detect and correct.