ahh, thanks for the idea. It was connecting to the hdmi sink instead of my analog. When I would change the device on vlc, it wouldn't work, but forcing it on pulse did.
If I select "Pulseaudio audio output" I get no sound, but using the Pulseaudio Sound Server as the device with the ALSA module works. Is this how its suppose to be, no direct pulse support?