Hi folks,
I was able to troubleshoot and fix this issue with the "Audio Midi Setup" application on the Mac (in your Utilities folder).
On the left side, I selected my output (HDMI) and then under the 'format' dropdown selected '2 channel 24-bit integer 44.1kHz' (instead of the 6 channel that was showing). This brought the dialog back. Hope it will work for you too. cheers
Thank you! I was going ****ing nuts trying to figure this out. It hadn't been a problem until I updated to OSX 10.12.6, which must have changed something related to Midi functionality. I still don't understand why my receiver (a Yamaha RX-A720) wasn't able to pick up the 8 channels and convert to 2, but at least it works now.
For reference, my HDMI-out Midi Setup was for 8 channel 24-bit integer 48 kHz, and I switched to 2 channel 24-bit integer 48 kHz.
Also of note, for anyone else trying this - I needed to restart VLC for the changes to properly take effect.