I already use Compressor, but I'm told I'd get better encodes using x264? (...) I've looked at ffworks but I find the settings very confusing, and I'm also not sure what settings I'd need to use, depending on the various types of film grain, bit rate, BD compliance etc.
From a DNxHD 175 source, I did an informal comparison between Compressor and FFmpeg (x264) using a similar bitrate and it's true that x264 gave better results.
My personal obsession when compressing video is banding. One interesting option in Compressor is dithering: this is a good way of fighting banding. AFAIK, FFmpeg does not have an optional dithering filter per se, but it offers the gradfun filter that helps avoid banding. With VLC, it can also be applied on the end user side, see
https://wiki.videolan.org/Documentation ... s/gradfun/
For the bitrate question, see
https://en.wikipedia.org/wiki/Blu-ray#Bit_rate