Sometimes VLC has some problems with transcoding, check the VLCmessage log if any errors occur. Otherwise, just try it over again.
That works perfectly!First of all, choosing DVD with menus won't work, you have to selected DVD without menus, so that only the movie is converted otherwise VLC just shows the DVD menu.
Then try IPod#Converting_a_Video_with_the_Graphical_User_Interface.