While documentation can always be better I don't agree in this case it's "quite lacking". vlcj even provides a simple example in its test sources showing how to use this dialog API.
Your follow-up question has no solution via the LibVLC API AFAIK.