With the latest 3.0.16 release, VLC plays FullHD and 4K content smoothly on recent 27" iMACs using HW acceleration, however, initially it always opens the video window 2-times bigger than appropriate.
For FullHD, it opens 4K window:
main debug: original format sz 1920x1088, of (0,0), vsz 1920x1080, 4cc CVPI, sar 1:1, msk r0x0 g0x0 b0x0
main debug: VoutDisplayEvent 'resize' 3840x2160
and for 4K it scales the video to all available display space, making it blurry:
main debug: original format sz 3840x2160, of (0,0), vsz 3840x2160, 4cc CVPN, sar 1:1, msk r0x0 g0x0 b0x0
main debug: VoutDisplayEvent 'resize' 5120x2718
Pressing ⌘0 for 'Half size' shrinks the video window to proper size and ensures 1:1 pixel mapping for both FullHD:
main debug: VoutDisplayEvent 'resize' 1920x1080
and also for 4K:
main debug: VoutDisplayEvent 'resize' 3840x2160
Looks like VLC somehow confuses logical display size and physical display size on Retina displays.