Stream to memory not working in VLC 3.0

[bdschmidt]

I have been using Stream to memory (smem) for several years in an app. When I upgrade to any flavor of VLC 3.0, it no longer works and instead a Direct3D9 Output window appears. The following is the initiation code:

CODE: SELECT ALL
char smem_options[256];
void *pUserData = 0;
::sprintf_s(smem_options,
"#transcode{vcodec=RV24,acodec=none}:smem{video-postrender-callback=%lld,video-prerender-callback=%lld,video-data=%lld}",
(long long int)(intptr_t)(void*)&handleStream,
(long long int)(intptr_t)(void*)&prepareRender,
(long long int)(intptr_t)(void*)this);

const char * const vlc_args[] = {
"-I", "dummy", // Don't use any interface
"--ignore-config", // Don't use VLC's config
"--verbose=2", // (0=only errors and standard messages, 1=warnings, 2=debug).
"--sout", smem_options // Stream to memory
};

// We launch VLC
m_vlcInstance = libvlc_new(sizeof(vlc_args) / sizeof(vlc_args[0]), vlc_args);
m_mediaPlayer = libvlc_media_player_new(m_vlcInstance);

// Build the URL.
std::string urlStr;
m_url.ToSingleByte(urlStr);
if (m_media)
libvlc_media_release(m_media);

m_media = libvlc_media_new_location(m_vlcInstance, urlStr.c_str());

if (m_media != NULL)
{
libvlc_media_player_set_media(m_mediaPlayer, m_media);

}


This is what the log is saying:

core libvlc debug: VLC media player - 3.0.0-git Vetinari
core libvlc debug: Copyright © 1996-2017 the VideoLAN team
core libvlc debug: revision 2.2.0-git-10950-gfaa5a38a4c
core libvlc debug: configured with ../extras/package/win32/../../../configure '--enable-update-check' '--enable-lua' '--enable-faad' '--enable-flac' '--enable-theora' '--enable-twolame' '--enable-avcodec' '--enable-merge-ffmpeg' '--enable-dca' '--enable-mpc' '--enable-libass' '--enable-x264' '--enable-schroedinger' '--enable-realrtsp' '--enable-live555' '--enable-dvdread' '--enable-shout' '--enable-goom' '--enable-caca' '--disable-sdl' '--enable-qt' '--enable-skins2' '--enable-sse' '--enable-mmx' '--enable-libcddb' '--enable-zvbi' '--disable-telx' '--enable-nls' '--host=x86_64-w64-mingw32' '--enable-debug' 'host_alias=x86_64-w64-mingw32' 'PKG_CONFIG_LIBDIR=/home/jenkins/workspace/vlc-nightly/vlc-nightly-win64-x86_64/vlc/contrib/x86_64-w64-mingw32/lib/pkgconfig'
core libvlc debug: using multimedia timers as clock source
core libvlc debug: min period: 1 ms, max period: 1000000 ms
core libvlc debug: plug-ins loaded: 463 modules
core logger debug: looking for logger module matching "any": 2 candidates
core logger debug: using logger module "console"
core libvlc debug: translation test: code is "C"
core keystore debug: looking for keystore module matching "memory": 3 candidates
core keystore debug: using keystore module "memory"
core libvlc debug: CPU has capabilities MMX MMXEXT SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 FPU
core generic debug: creating audio output
core audio output debug: looking for audio output module matching "any": 6 candidates
mmdevice audio output debug: using default device
mmdevice audio output debug: version 2 session control unavailable
mmdevice audio output debug: display name changed: VLC media player (LibVLC 3.0.0-git)
mmdevice audio output debug: volume from -65.250000 dB to +0.000000 dB with 0.031250 dB increments
core audio output debug: using audio output module "mmdevice"
core generic debug: keeping audio output
core input debug: Creating an input for 'RTSP://204.108.187.254'
core input debug: using timeshift granularity of 50 MiB
core input debug: using timeshift path: C:\Users\bschmidt\AppData\Local\Temp
core input debug: `RTSP://user:ASDFJKLqwerty@204.108.187.254' gives access `RTSP' demux `any' path `user:ASDFJKLqwerty@204.108.187.254'
core input source debug: creating demux: access='RTSP' demux='any' location='user:ASDFJKLqwerty@204.108.187.254' file='\\user:ASDFJKLqwerty@204.108.187.254'
core demux debug: looking for access_demux module matching "RTSP": 14 candidates
live555 demux debug: version 2016.11.28
core demux warning: Password in a URI is DEPRECATED
live555 demux debug: RTP subsession 'video/H264'
core input debug: selecting program id=0
live555 demux debug: setup start: 0.000000 stop:0.000000
live555 demux debug: We have a timeout of 60 seconds
live555 demux debug: play start: 0.000000 stop:0.000000
core demux debug: using access_demux module "live555"
core packetizer debug: looking for packetizer module matching "any": 24 candidates
h264 packetizer debug: found NAL_SPS (sps_id=0)
h264 packetizer debug: found NAL_PPS (pps_id=0 sps_id=0)
core packetizer debug: using packetizer module "h264"
The thread 0x5d70 has exited with code 0 (0x0).
core decoder debug: looking for decoder module matching "any": 50 candidates
avcodec decoder debug: CPU flags: 0x000fd3db
The thread 0x5dc8 has exited with code 0 (0x0).
avcodec decoder debug: allowing 4 thread(s) for decoding
avcodec decoder debug: codec (h264) started
avcodec decoder debug: using frame thread mode with 4 threads
core decoder debug: using decoder module "avcodec"
core demux meta debug: looking for meta reader module matching "any": 2 candidates
lua demux meta debug: Trying Lua scripts in C:\Users\bschmidt\AppData\Roaming\vlc\lua\meta\reader
lua demux meta debug: Trying Lua scripts in D:\vq\global\VQ\VQ\main\Azure\target\Azulon\x64\ReleaseNoOpt\lua\meta\reader
core demux meta debug: no meta reader modules matched
core input debug: `RTSP://user:ASDFJKLqwerty@204.108.187.254' successfully opened
core input debug: Buffering 0%
live555 demux debug: lost 86569 bytes
live555 demux debug: increasing buffer size to 200000
avcodec decoder debug: available hardware decoder output format 61 (dxva2_vld)
avcodec decoder debug: available hardware decoder output format 128 (d3d11va_vld)
avcodec decoder debug: available software decoder output format 0 (yuv420p)
avcodec decoder warning: thread type 1: disabling hardware acceleration
core spu text debug: looking for text renderer module matching "any": 3 candidates
First-chance exception at 0x000007FEFD05A06D (KernelBase.dll) in Azulon.exe: 0x20474343 (parameters: 0x0000000036B51DC0).
freetype spu text warning: DirectWrite initialization failed. Falling back to GDI/Uniscribe
core spu text debug: using text renderer module "freetype"
core scale debug: looking for video converter module matching "any": 20 candidates
swscale scale debug: 32x32 (32x32) chroma: YUVA -> 16x16 (16x16) chroma: RGBA with scaling using Bicubic (good quality)
core scale debug: using video converter module "swscale"
core scale debug: looking for video converter module matching "any": 20 candidates
yuvp scale debug: YUVP to YUVA converter
core scale debug: using video converter module "yuvp"
core video output debug: Deinterlacing available
core video output debug: deinterlace 0, mode blend, is_needed 0
core window debug: looking for vout window module matching "any": 3 candidates
core window debug: no vout window modules matched
core video output debug: Opening vout display wrapper
core vout display debug: looking for vout display module matching "any": 12 candidates
direct3d9 vout display debug: Using Direct3D9 Extended API!
'Azulon.exe' (Win32): Loaded 'C:\Windows\System32\D3DX9_43.dll'. Symbols loaded.
direct3d9 vout display debug: creating Vout EventThread
direct3d9 vout display debug: Win32VoutCreateWindow
The thread 0x6344 has exited with code 0 (0x0).
The thread 0x2cfc has exited with code 0 (0x0).
direct3d9 vout display debug: created video sub-window
direct3d9 vout display debug: Vout EventThread running
direct3d9 vout display debug: disabling screen saver
direct3d9 vout display debug: Direct3d9 Device: NVIDIA GeForce GTX 980 4318 5056 161
core input debug: Buffering 26%
direct3d9 vout display debug: DirectXUpdateRects souce offset: 0,0 visible: 1920x1080
direct3d9 vout display debug: DirectXUpdateRects image_src coords: 0,0,1920,1090
direct3d9 vout display debug: DirectXUpdateRects image_src_clipped coords: 0,0,1920,1080
direct3d9 vout display debug: DirectXUpdateRects image_dst coords: 0,0,1920,1080
direct3d9 vout display debug: DirectXUpdateRects image_dst_clipped coords: 0,0,1920,1080
'Azulon.exe' (Win32): Loaded 'C:\Windows\System32\ExplorerFrame.dll'. Symbols loaded.
'Azulon.exe' (Win32): Loaded 'C:\Windows\System32\duser.dll'. Symbols loaded.
'Azulon.exe' (Win32): Loaded 'C:\Windows\System32\dui70.dll'. Symbols loaded.
'Azulon.exe' (Win32): Loaded 'C:\Windows\System32\actxprxy.dll'. Symbols loaded.
direct3d9 vout display warning: trying surface pixel format: YV12
direct3d9 vout display debug: selected surface pixel format is YV12
direct3d9 vout display debug: Direct3D created offscreen surface: 1920x1090
direct3d9 vout display debug: Direct3D created texture: 1920x1080
direct3d9 vout display debug: Direct3D9 scene created successfully
direct3d9 vout display debug: Direct3D9 device adapter successfully initialized
core vout display debug: using vout display module "direct3d9"
core video output warning: Not enough display buffers in the pool, requested 3 got 1
core video output debug: original format sz 1920x1090, of (0,0), vsz 1920x1080, 4cc I420, sar 1:1, msk r0x0 g0x0 b0x0
core vout display debug: VoutDisplayEvent 'window state' 0
core vout display debug: VoutDisplayEvent 'window state' 0
core spu text debug: removing module "freetype"
direct3d9 vout display debug: DirectXUpdateRects souce offset: 0,0 visible: 1920x1080
direct3d9 vout display debug: DirectXUpdateRects image_src coords: 0,0,1920,1090
core spu text debug: looking for text renderer module matching "any": 3 candidates
direct3d9 vout display debug: DirectXUpdateRects image_src_clipped coords: 0,0,1920,1080
direct3d9 vout display debug: DirectXUpdateRects image_dst coords: 0,0,1920,1080
direct3d9 vout display debug: DirectXUpdateRects image_dst_clipped coords: 0,0,1920,1080
First-chance exception at 0x000007FEFD05A06D (KernelBase.dll) in Azulon.exe: 0x20474343 (parameters: 0x00000003D746E7B0).
freetype spu text warning: DirectWrite initialization failed. Falling back to GDI/Uniscribe
core vout display error: Failed to change Viewpoint
core vout display debug: VoutDisplayEvent 'resize' 1920x1080
core spu text debug: using text renderer module "freetype"
avcodec decoder warning: plane 0 not aligned: disabling direct rendering
core decoder debug: Received first picture
core input debug: Buffering 53%
The thread 0x27cc has exited with code 0 (0x0).
core input debug: Buffering 80%
core input debug: Stream buffering done (1067 ms in 965 ms)
core input debug: Decoder wait done in 0 ms
live555 demux debug: lost 24273 bytes
live555 demux debug: increasing buffer size to 400000
core video output warning: picture is too late to be displayed (missing 66 ms)
core video output warning: picture is too late to be displayed (missing 34 ms)
core video output debug: picture might be displayed late (missing 4 ms)
core video output debug: picture might be displayed late (missing 5 ms)
core vout display debug: auto hiding mouse cursor
core vout display debug: auto hiding mouse cursor
core video output debug: picture might be displayed late (missing 5 ms)

When I revert to a pre 3.0 VLC, I get a section in the logs relating to smem that is completely absent in the above log. I just pasted the relevant portion below.

core input debug: Creating an input for 'user:ASDFJKLqwerty@204.108.187.254'
core stream output debug: using sout chain=`transcode{vcodec=RV24,acodec=none}:smem{video-postrender-callback=8791005323248,video-prerender-callback=8791005324064,video-data=11972362240}'
core stream output debug: stream=`smem'
core stream out debug: looking for sout stream module matching "smem": 22 candidates
'Azulon.exe' (Win32): Loaded 'D:\vq\global\VQ\VQ\TVM6.3\Mnt\main\Azure\target\Azulon\x64\ReleaseNoOpt\plugins\stream_out\libstream_out_smem_plugin.dll'. Module was built without symbols.
core stream out debug: set config option: sout-smem-video-postrender-callback to 8791005323248
core stream out debug: set config option: sout-smem-video-prerender-callback to 8791005324064
core stream out debug: set config option: sout-smem-video-data to 11972362240
core stream out debug: using sout stream module "stream_out_smem"
core stream output debug: stream=`transcode'
core stream out debug: looking for sout stream module matching "transcode": 22 candidates
'Azulon.exe' (Win32): Loaded 'D:\vq\global\VQ\VQ\TVM6.3\Mnt\main\Azure\target\Azulon\x64\ReleaseNoOpt\plugins\stream_out\libstream_out_transcode_plugin.dll'. Module was built without symbols.
core stream out debug: set config option: sout-transcode-vcodec to RV24
core stream out debug: set config option: sout-transcode-acodec to none
stream_out_transcode stream out debug: Checking codec mapping for none got none
stream_out_transcode stream out debug: codec audio=none 0Hz 0 channels 96Kb/s
stream_out_transcode stream out debug: Checking video codec mapping for RV24 got RV24
stream_out_transcode stream out debug: codec video=RV24 0x0 scaling: 0.000000 0kb/s
core stream out debug: using sout stream module "stream_out_transcode"

The absence of this logging information in the 3.0 version tells me that it never even saw/parsed the sout chain. Did something change in how smem is set up in 3.0? Or am I missing something obvious? Any and all thoughts are welcome. Thanks!