The easiest way is to get vlc source code, compile it, and add anything you want.
But it's also possible to build a stand alone plugin. You'll have to add vlc's includes and fix issues on your environment.
JoungEunKim did it on windows with visual studio 2005/8 (see here, he uploaded an example based on simple.c, an audio filter :
viewtopic.php?f=32&t=81793#p274912)
Thanks to him i managed to make my own audio output plugin modified to include ac3filter's core (and all it's features, upmix, resampling, DRC, per channel gain, ...). It's more a prof of concept right now, the code is a mess, but it works.
You should read this :
http://wiki.videolan.org/Documentation:Hacker%27s_Guide, and especially the "VLC modules" link.
And don't forget vlc source code, it contains a lot of comments everywhere, it's the most complete doc you can find on vlc.