which can be modified in the VLC preferences
Yeah, good luck figuring out where and how to modify this in the preferences. It sure isn't with "Always reset audio start level." Maybe "Normalize volume?" Everyone loves trial-and-error, don't they?
UPDATE: I believe that I've tried every combination of audio settings, including "Output module," "Normalize volume" and "Replay gain mode." WAV from DTSHD is horribly overamped and clipped no matter what I try.
UPDATE 2: I think I finally found it! I needed to open the advanced settings and set the "gain control filter" value. I have no idea how or why this is different from "Normalize volume to," but it is.