This feature would be greatly appreciated. I would ask the devs of VLC Remote to implement this asap when available.
My (desired) setup: VLC on PC, a desktop with windows in the office room, does the heavy lifting, rendering 2160p HDR to a chromecast (ultra) in the living room, where I sit on the couch and remote control VLC using one single app on the phone (iOS in my case)