The problem seems to be there:
vout_directx debug: selecting Matrox Graphics Mystique PCI, \\.\DISPLAY2
vout_directx debug: screen dimensions 0x600
I don't know yet why the dimensions are not valid though.
I'll give you something to test as soon as I find something