Hey Guys,
This is more of a bug report, so someone in the future can bypass the hours of paid work i've wasted toiling over this!
To get Decklink output to work with VLC, you need the SDK (the included Module, doesn't actually do any of the hard work). You need to enable decklink in the configure line when compiling, and also give the path to the decklink SDK. Both of these are easily found, and i've documented these in other threads.
The decklink works fine, and outputs video and sound (this is requirement, you can't seperate the streams), through the card. I am using Decklink Desktop 9.7.7, and the 9.7.7 SDK, however, the SDK still works with Desktop 9.8, but i'm holding off upgrading until a new SDK is released.
Now here's the bug:
Playing a playlist is an issue, you can chain together multiple files that use the same demuxer / codecs just fine, and it will make its way through the playlist as if it's all working absolutely fine, however, if you mix files, this is where the decklink output will fail.
So to clarify: if your playlist consists of files that are of the same codec (e.g. ALL ProRes 442, with the same bitrate audio), it will work, however, if you have multiple ProRes files with different audio, or a ProRes and a h264, the decklink output module will fail as soon as it hits the differing file.
The only way to get back to playing properly is to completely shut VLC down, reopen it, and play different files separately. I hope this helps you!