App with custom build vlc drops UDP audio stream within 2mins

[phinoppix]

I wrote a console app that will only playback PCM (s16 LE) streamed via UDP on Windows. The other end is using ffmpeg to send down the audio packets.
For this, I cross compiled VLC disabling lots of stuff I don't need (and probably even some I thought I won't need). For the first couple of seconds the audio plays nicely but with some 2sec or so delay (custom VLC client is in Singapore and ffmpeg streamer is in the US).

Why does the audio drop suddenly. I don't see anything unusual. The vlc log only have this:

-- logger module started --
core error: ES_OUT_SET_(GROUP_)PCR is called too late (pts_delay increased to 1000 ms)
core error: ES_OUT_RESET_PCR called

If I use the out-of-the-box VLC sdk, everything works fine it seems. So I'm beginning to wonder if disabled some things that shouldn't be.
Here is my config script:

configure.sh \
--host=i686-w64-mingw32 \
--disable-qt \
--disable-skins2 \
--disable-libtar \
--disable-ncurses \
--disable-goom \
--disable-projectm \
--disable-vsxu \
--disable-atmo \
--disable-glspectrum \
--disable-gnutls \
--disable-libxml2 \
--disable-libgcrypt \
--disable-update-check \
--disable-growl \
--disable-notify \
--disable-sout \
--disable-lua \
--disable-httpd \
--disable-vlm \
--disable-addonmanagermodules \
--disable-vlc \
--disable-x265 \
--disable-taglib \
--disable-chromecast \
--without-libiconv-prefix \
--without-libintl-prefix \
--disable-ncurses \
--disable-xcb \
--disable-xvideo \
--disable-glx \
--disable-sdl \
--disable-sdl-image \
--disable-svg \
--disable-svgdec \
--disable-aa \
--disable-caca \
--disable-kva \
--disable-mmal \
--disable-archive \
--disable-live555 \
--disable-dc1394 \
--disable-dv1394 \
--disable-linsys \
--disable-dvdread \
--disable-dvdnav \
--disable-bluray \
--disable-opencv \
--disable-smbclient \
--disable-dsm \
--disable-sftp \
--disable-v4l2 \
--disable-decklink \
--disable-gnomevfs \
--disable-vcdx \
--disable-vcd \
--disable-libcddb \
--disable-screen \
--disable-vnc \
--disable-freerdp \
--disable-realrtsp \
--disable-asdcp \
--disable-dvbpsi \
--disable-gme \
--disable-sid \
--disable-ogg \
--disable-mux_ogg \
--disable-shout \
--disable-mkv \
--disable-mod \
--disable-mpc \
--disable-crystalhd \
--disable-mad \
--disable-mpg123 \
--disable-gst-decode \
--disable-avcodec \
--disable-avformat \
--disable-swscale \
--disable-postproc \
--disable-faad \
--disable-vpx \
--disable-twolame \
--disable-fdkaac \
--disable-quicktime \
--disable-a52 \
--disable-dca \
--disable-flac \
--disable-libmpeg2 \
--disable-vorbis \
--disable-tremor \
--disable-speex \
--disable-opus \
--disable-theora \
--disable-daala \
--disable-schroedinger \
--disable-png \
--disable-jpeg \
--disable-x265 \
--disable-x264 \
--disable-mfx \
--disable-fluidsynth \
--disable-libass \
--disable-aribsub \
--disable-aribb25 \
--disable-kate \
--disable-tiger \
--without-x \
--disable-xcb \
--disable-xvideo \
--disable-glx \
--disable-vdpau \
--disable-wayland \
--disable-sdl \
--disable-sdl-image \
--disable-freetype \
--disable-fribidi \
--disable-fontconfig \
--disable-svg \
--disable-svgdec \
--disable-directx \
--disable-caca \
--disable-kva \
--disable-mmal \
--disable-pulse \
--disable-samplerate \
--disable-chromaprint \
--disable-chromecast \
--disable-bonjour \
--disable-udev \
--disable-mtp \
--disable-upnp