Xcode is required to create the application bundle, to provide the necessary compilers and of course to edit the nib files. The actual compilation process is accomplished with autoconf/automake. The choice of a text editor is up to you.
For more information on compiling VLC for Mac, check
http://wiki.videolan.org/OSXCompile.
Note that the master repository of VLC is quite broken wrt Mac right now, so you might want to start with the stable repo (available at [url]git://git.videolan.org/vlc/vlc-1.1.git[/url]) to have something working.
Please also note that you'll need Mac OS X 10.5 or 10.6 with Xcode 3.1 or 3.2 for compilation. Xcode 4 is not supported for now.