I'm using the invmem in-memory codec, and the red/blue channels are coming out inverted. Looking at modules/codec/invmem.c:183-186 in the case for VLC_CODEC_RGB24, I see the code that sets up the channel masks. Apart from being endian-dependent, I think the constants are incorrect: in-memory RGB ordering for little-endian architectures (which most of us are on) requires rmask = 0x0000ff and bmask = 0xff0000 in order to stride memory as R0G0B0R1G1B1...
The case for VLC_CODEC_RGB32 needs a similar fix.
Does anyone think different before I figure out how to submit a patch?
Paul