Leave out the qp, qp-min and qp-max values because those indeed will take precedence over specifiying a max bitrate, it takes whatever bitrate it needs to maintain quality. If you want something resembling CBR you can specify bitrate like this: vcodec=h264,venc=x264{vbv-maxrate=800,etc.},vb=800,etc....