VAAPI hardware accelerated h264 encoder

[BtbN]

Hi everbody!

I spend quite a lot of time the last days to get this working, and it's still far from perfect, but i think it's working good enough to post it.
I'm not entirely sure if this is the right forum to post it, but i think it should fit best.

The sourcecode is available on github: https://github.com/BtbN/vlc-vaapi-enc

To get this working, you need:
- Linux
- An Intel Ivy/Sandy Bridge GPU running as your primary graphics adapter. If vainfo shows some H264 EncSlices, you should be fine.
- An up-to-date version of libva and libva-intel-driver. By up to date i mean a build of the staging branch from their git. I uploaded a compiled version with a script to source for testing(x64 linux, built with gcc 4.6): http://btbn.de/files/libva-staging.tar.gz
- Possibly also a not too old kernel. I'm using this on 3.6.11 and it works fine.
- A running XServer(In theory it should also be possible to directly use the dri device, but this just fails with the current libva builds)

When using the staging liva, disable ffmpeg-hw, as it does not work with the staging library without patching vlc(which is trivial but has to be done).

I also uploaded a compiled version of the plugin(also x64 linux): http://btbn.de/files/vlc-vaapi-enc.tar.gz

A possible commandline to test might be: vlc "--sout=#transcode{venc=va,vb=-28,deinterlace}:std{access=file,mux=ts,dst=testout.ts}" your-test-file.whatever

vaapi does not yet support interlaced encoding, so if you try to encode interlaced frames it'll just crash with an assertion. Use the deinterlace parameter to avoid this.
Providing a negative bitrate will make the encoder use QB based encoding. So -28 results in an initial qb value of 28.

If you want to look at a sample, i transcoded a the 720p version of Big Buck Bunny with it an uploaded the result: http://btbn.de/files/big_buck_720.mpg
The QB value used for this was 28. It halved the original bitrate, but also does not look that good.

Feedback on this is very welcome!
Would be very cool if this could one day be a part of vlc, but there are too many breaking changes in the libva staging branch atm. Once this hits master or even a release, it could be worth a thought.