transcode SD video h264 at higher quality or bitrate?

[cocoapepper]

I have been experimenting with transcoding SD video (480i/480p resolution). My test source is a commercial DVD with a nice high bitrate. I can stream this exact DVD bitstream to the clients (about 7mbps) and it looks perfect. When I try to transcode to h264, the maximum bitrate offered by VLC is 3072 kbps. The result looks OK, but there is still some pretty serious artifacting that sort of shifts every second or so. Is there a way to boost the quality and/or bitrate of the transcode higher than what I am currently doing? My dual core CPU only shows about 30% usage on just one core when doing this transcode, so CPU shouldn't be the limiting factor. I have seen some beautiful 3mbps h264 streams before so I know this is somehow possible, I just can't figure it out. Than you in advance!