LibVLC Deadlock on libvlc_media_player stop (win32) [Fixed]

[solud]

I've been experiencing a deadlock on a form in my application that uses two media_list_players to play video on to two panels. The players connect to two different RTSP sources when the form is loaded, and the form accepts feedback on the video displayed before closing. When the form is closing, it releases the media_list_players and the media_players. This form is reopened many times, reusing the same vlc instance but creating new media/players. Everything works wonderfully except for a random occurrence of a deadlock on releasing the objects. It seems to be exactly as described in this post, and ticket 3307 regarding libvlc_media_player stop. For the most part everything closes cleanly, but eventually it deadlocks -- sometimes within a dozen instances of the form, other times it takes a few dozen or more. I've tested on WinXP/7 with the same result, using 1.1.5/6/7 and a few nightly 1.2 builds from the last week.

I've tried different arrangements of cleaning up the objects, thinking it was something wrong with the way I was closing the form. I've played with the order in which objects are released, such as releasing the player before the list_player, stopping before releasing, etc. If it is indeed caused by the order of commands, I've not been able to isolate it. As the code stands right now the objects are released when the form is closing without any call to stop the video feeds and I don't see any difference and the deadlock still occurs.

What is the proper order for safely releasing the references created with libvlc _new functions when using a a couple libvlc_media_list_players to playback RTSP streams?

Thank you for your time

[Rémi Denis-Courmont]

Players are supposed to be independent. There is no order. It could be a bug in LibVLC or in your application callbacks though.

[solud]

I don't use any of the LibVLC callbacks, at least not yet. The video portion of the form is extremely simple.

I don't like claiming there's a bug in someone else's code unless I can show some evidence, however without the use of LibVLC this form has been working for three years -- seems there's no way around it!

On the development station, I get a full crash, after closing the form well over 50 times. I almost fell asleep while waiting for it to crash to get this mostly useless info:

Problem signature:
Problem Event Name: APPCRASH
Application Name: VGAM.vshost.exe
Application Version: 9.0.21022.8
Application Timestamp: 473168a0
Fault Module Name: libvlccore.dll
Fault Module Version: 1.2.0.99
Fault Module Timestamp: 4d62675b
Exception Code: c0000005
Exception Offset: 000a3fe9
OS Version: 6.1.7600.2.0.0.256.1
Locale ID: 1033
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

On the workstations that this software is actually used, the program just deadlocks and I don't get this crash info. I'll try to get a debugger on it to get a stack trace.

[solud]

I've been able to find more information on this problem:

The deadlock is definitely caused by calls to libvlc_media_player_stop() / libvlc_media_list_player_stop() -- both end up with the same deadlock, but media_list_player just calls stop on its associated media_player so I'm guessing it's something in media_player_stop(). This also happens if I just release the players instead of stopping them first.

Every time I see this happening in a debugger, it is always on the 2nd player's call to stop. The first one always seems to return without problem.

I've included stack traces of the last time I saw this in a debugger, but only the ones that reference libvlc functions. If I should include the other thread traces let me know, I have them!

Code: Select all

0 4716 Main Thread Main Thread NtWaitForSingleObject Normal 0 0 2412 Worker Thread CreateApplicationContext NtWaitForMultipleObjects Highest 0 0 2356 Worker Thread CreateHistoryReader NtWaitForSingleObject Normal 0 0 4592 Worker Thread RtlLoadString NtWaitForMultipleObjects Normal 0 0 3788 Worker Thread GdipCreateSolidFill NtWaitForMultipleObjects Normal 0 0 3808 Worker Thread RtlDosSearchPath_Ustr NtWaitForWorkViaWorkerFactory Normal 0 0 3576 Worker Thread CreateApplicationContext VG.VGAM.EventQueueProcessor Normal 0 0 5028 Worker Thread CreateApplicationContext GetPropW Normal 0 0 3992 Worker Thread TranslateSecurityAttributes NtDelayExecution Normal 0 0 4288 Worker Thread Win32 Thread NtRemoveIoCompletion Above Normal 0 0 1500 Worker Thread CreateApplicationContext GetPropW Normal 0 0 1408 Worker Thread CreateApplicationContext NtDelayExecution Normal 0 0 4984 Worker Thread CreateApplicationContext VG.VGAMClientConnection.CheckConnectionState Normal 0 0 5116 Worker Thread _endthreadex NtWaitForSingleObject Normal 0 0 5304 Worker Thread _endthreadex NtWaitForSingleObject Normal 0 0 3772 Worker Thread RtlDosSearchPath_Ustr NtWaitForWorkViaWorkerFactory Normal 0 0 4804 Worker Thread _endthreadex NtWaitForSingleObject Normal 0 0 1416 Worker Thread _endthreadex NtWaitForSingleObject Above Normal 0 0 6492 Worker Thread _endthreadex GetPropW Normal 0 0 5860 Worker Thread RtlDosSearchPath_Ustr NtWaitForWorkViaWorkerFactory Normal 0 [main thread] ntdll.dll!NtWaitForSingleObject() + 0x15 bytes [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] kernel32.dll!WaitForSingleObjectEx() + 0x43 bytes libvlccore.dll!vlc_join() + 0x2f bytes libvlccore.dll!vout_Close() + 0x62 bytes libvlccore.dll!input_resource_Delete() + 0x565 bytes libvlccore.dll!input_resource_TerminateVout() + 0x23 bytes libvlc.dll!libvlc_media_player_stop() + 0x6e bytes libvlc.dll!libvlc_media_list_player_stop() + 0x41 bytes [thread 5116] > ntdll.dll!NtWaitForSingleObject() + 0x15 bytes [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] kernel32.dll!WaitForSingleObjectEx() + 0x43 bytes libvlccore.dll!vlc_cond_wait() + 0x4c bytes libvlc.dll!libvlc_event_attach() + 0x118 bytes libvlccore.dll!vlc_sem_post() + 0x7a bytes msvcrt.dll!_itow_s() + 0x4c bytes msvcrt.dll!_endthreadex() + 0x6c bytes kernel32.dll!BaseThreadInitThunk() + 0x12 bytes ntdll.dll!RtlInitializeExceptionChain() + 0x63 bytes ntdll.dll!RtlInitializeExceptionChain() + 0x36 bytes [thread 5304] > ntdll.dll!NtWaitForSingleObject() + 0x15 bytes [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] kernel32.dll!WaitForSingleObjectEx() + 0x43 bytes libvlccore.dll!vlc_cond_wait() + 0x4c bytes libvlc.dll!libvlc_event_attach() + 0x118 bytes libvlccore.dll!vlc_sem_post() + 0x7a bytes msvcrt.dll!_itow_s() + 0x4c bytes msvcrt.dll!_endthreadex() + 0x6c bytes kernel32.dll!BaseThreadInitThunk() + 0x12 bytes ntdll.dll!RtlInitializeExceptionChain() + 0x63 bytes ntdll.dll!RtlInitializeExceptionChain() + 0x36 bytes [thread 4804] > ntdll.dll!NtWaitForSingleObject() + 0x15 bytes [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] kernel32.dll!WaitForSingleObjectEx() + 0x43 bytes libvlccore.dll!vlc_cond_wait() + 0x4c bytes libvlc.dll!libvlc_event_attach() + 0x118 bytes libvlccore.dll!vlc_sem_post() + 0x7a bytes msvcrt.dll!_itow_s() + 0x4c bytes msvcrt.dll!_endthreadex() + 0x6c bytes kernel32.dll!BaseThreadInitThunk() + 0x12 bytes ntdll.dll!RtlInitializeExceptionChain() + 0x63 bytes ntdll.dll!RtlInitializeExceptionChain() + 0x36 bytes [thread 1416] > ntdll.dll!NtWaitForSingleObject() + 0x15 bytes [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] kernel32.dll!WaitForSingleObjectEx() + 0x43 bytes libvlccore.dll!vlc_join() + 0x2f bytes libdirect3d_plugin.dll!vlc_entry__1_1_0g() + 0x286f bytes libdirect3d_plugin.dll!vlc_entry__1_1_0g() + 0x597a bytes libdirect3d_plugin.dll!vlc_entry__1_1_0g() + 0x1e8d bytes libvlccore.dll!module_unneed() + 0x21 bytes libvlccore.dll!vout_DeleteDisplay() + 0x5e bytes libvout_wrapper_plugin.dll!vlc_entry__1_1_0g() + 0x22f bytes libvlccore.dll!module_unneed() + 0x21 bytes libvlccore.dll!vout_Close() + 0x2128 bytes libvlccore.dll!vlc_sem_post() + 0x7a bytes msvcrt.dll!_itow_s() + 0x4c bytes msvcrt.dll!_endthreadex() + 0x6c bytes kernel32.dll!BaseThreadInitThunk() + 0x12 bytes ntdll.dll!RtlInitializeExceptionChain() + 0x63 bytes ntdll.dll!RtlInitializeExceptionChain() + 0x36 bytes

This is using the win32 debug branch from Feb 22nd, found here.

I'll post more info as I find it. Any suggestions on where to look?

Thanks again for your time

[Rumpel]

I think i have the same problem. I have more then one player in my appilcation and in same cases
if i will stop one of the players.

I always gets a infinite loop by call vlc_join(vlc_thread_t handle, void **result) (src/win32/thread.c) cause this do-while loop never ends

Code: Select all

do vlc_testcancel (); while (WaitForSingleObjectEx (handle, INFINITE, TRUE)== WAIT_IO_COMPLETION);

Code: Select all

void vlc_testcancel (void) { vlc_cancel_t *nfo = (vlc_cancel_t *)vlc_threadvar_get (cancel_key); // sunqueen modify if (nfo == NULL) return; /* Main thread - cannot be cancelled anyway */ if (nfo->killable && nfo->killed) { for (vlc_cleanup_t *p = nfo->cleaners; p != NULL; p = p->next) p->proc (p->data); #ifndef UNDER_CE _endthread (); #else ExitThread(0); #endif } }

in vlc_testcancel() the variable nfo->killed never become true

Code: Select all

/*** Thread cancellation ***/ /* APC procedure for thread cancellation */ static void CALLBACK vlc_cancel_self (ULONG_PTR dummy) { vlc_cancel_t *nfo = vlc_threadvar_get (cancel_key); if (likely(nfo != NULL)) nfo->killed = true; (void)dummy; }

in this CALLBACK but i found this
http://git.videolan.org/gitweb.cgi?p=vl ... ff6b576527

Code: Select all

/*** Thread cancellation ***/ /* APC procedure for thread cancellation */ -static void CALLBACK vlc_cancel_self (ULONG_PTR dummy) +static void CALLBACK vlc_cancel_self (ULONG_PTR self) { - struct vlc_thread *th = vlc_threadvar_get (thread_key); + struct vlc_thread *th = (void *)self; if (likely(th != NULL)) th->killed = true; - - (void)dummy; }

can somebody explain me the change?

[Rémi Denis-Courmont]

That change is not in VLC 1.1.7 so it's irrelevant to the problem.

[solud]

I've just tried the DirectX plugin instead of D3D (using "--vout=directx") -- no luck.

[Rumpel]

Maybe could it be that there is a timing problem?

EDIT:
If i put a breakpoint in debugmode in this function

Code: Select all

void vlc_cancel (vlc_thread_t thread_id) { #ifndef UNDER_CE QueueUserAPC (vlc_cancel_self, thread_id, 0); <---- BREAKPOINT #else SetEvent (thread_id->cancel_event); #endif }

everything is good

[Rémi Denis-Courmont]

I don't know. I don't have the hardware nor the time to debug the Win32 port. Feel free to send a patch if you find a bug in the VLC Win32 thread support.

[solud]

If there is someone who would like to / can help out in finding this bug, I've put up a project file that displays two streams (from included Live555 RTSP server) on to a form.

I've tested this project on XP, fully updated, and got it to deadlock twice within 5 loads of the form. On Windows 7 it takes considerably longer, but has definitely happened to me many times during main development / testing.

This is a C# project, that uses the LibVLC DLL. Included is a nice VLC file that can be used in other .NET projects, though I've not included all functions. Most the contents of this file have been taken from other sources, but parts that I needed myself were added using the VLC Doxygen pages. Here is the link to download the project zip. If there is a better free file uploading service I'll gladly throw it up there if the current one is not trustworthy.

To use this test, first run the "Live555 RTSP Server" included in the Debug folder and then run the test app.
Click on "Play" and a form should appear with two video streams, playing continuously.
Close the form and repeat until deadlock.

More info on the bug: While making this test app, I've discovered that the bug does not occur if everything takes place on the same form (with .Dispose() being called on all LibVLC objects and replaying). I was puzzled when I couldn't get it to deadlock with it just playing on one form, so I created a new form to display the streams and release the objects when closing the form. This is how my main application is designed, since it is possible for more than one of these forms to be active at any one time. The handle to the LibVLC instance is held by the parent form and passed along to the child forms, and it is when these child forms are closing (and the players sent a Stop request) that the deadlock occurs.

FYI: this uses Live555's latest code for the RTSP server (as of 2/28/11) and LibVLC 1.1.7 DLLs

Many thanks to anyone who can help track this down!

[solud]

Has anyone been able to confirm that they also experience the above issues using the sample project? I'd really love to know if I'm not the only one this happens with.

[Rumpel]

So, if i close a media player not all Threads are closed. I don´t know why cause i think i have the right order of function to stop and close a media player

Code: Select all

libvlc_media_player_set_hwnd(pvideo->mp, NULL); libvlc_media_player_stop(pvideo->mp); libvlc_media_player_release(pvideo->mp); libvlc_release(pvideo->vlcinst);

[solud]

I don't think this is related to this thread. Your last call to release the LibVLC instance is not performed in my test app until the app is closed, while the bug I refer to occurs during the normal running of the application. I don't release the LibVLC instance when a form containing players is closed -- it is kept for future forms to utilize.

[solud]

More info regarding the deadlock:

The event libvlc_MediaPlayerStopped is fired before the deadlock occurs and FWIW libvlc_MediaPlayerEncounteredError does not fire.

Has anyone been able to reproduce this error, with or without the sample application I provided above?

[rvs75]

vlc_mediaplayer_stop hang when I use marquees just before. I don't know if it's the same deadlock ?

viewtopic.php?f=32&t=87360&p=289070#p289070

[Rumpel]

I think it doesn´t matter which function you call befor you will stop the player.
But the aspect of solud is interesting. I will test it in my app

EDIT:
at the end of the function libvlc_media_player_stop() , input_resource_TerminateVout(p_mi->input.p_resource) is call and sometime i get a deadlock. Can somebody affirm this

[rvs75]

I try to take a wpf control in a host control over the video to recreate an OSD and the vlc_mediaplayer_stop hang too.

Without libvlc 's marquee or my own osd, I haven't any problems.

[solud]

I don't use the marquee features -- I turn off all OSD aspects with "--no-osd", "--no-video-title-show". I also use "--vout=directx" now, since I find it exasperates the deadlock bug on Windows 7 -- not sure why either.

I agree with Rumpel in that I don't think it matters what you do with the player before stopping it, the deadlock will still happen.

[Rémi Denis-Courmont]

I don't know if what you do before stopping matters. But I think it's quite irrelevant:
There is (reportedly) a deadlock bug. Someone should look at a threaded stack trace where the problem occurs, find the problem and post a patch on vlc-devel. Then it can still get into VLC 1.1.8.

[Rémi Denis-Courmont]

I think i have the same problem. I have more then one player in my appilcation and in same cases
if i will stop one of the players.

I always gets a infinite loop by call vlc_join(vlc_thread_t handle, void **result) (src/win32/thread.c) cause this do-while loop never ends

Code: Select all

do vlc_testcancel (); while (WaitForSingleObjectEx (handle, INFINITE, TRUE)== WAIT_IO_COMPLETION);

(...)
can somebody explain me the change?

Most probably the [handle] thread is stuck and refuses to exit, so the loop condition never breaks. I doubt it's an issue with vlc_testcancel() and the APC stuff. You can easily test that by removing vlc_testcancel() from the while-loop.

[Rumpel]

Okay, i will test it and tell you the result

[bratao]

Just to report.
I´m using in a C++ project and i have exactly the same deadlock.
Both in 1.1.7 and in 1.2 nightly

[Rumpel]

So i removed the vlc_testcancel() call and my app works fine for me at the moment i have no problems. But on other machines with different CPUs i have still the problem with the deadlocks

[Rémi Denis-Courmont]

I told you so. vlc_testcancel() has nothing to do with the problem. It just alters timing a bit which can change the odds of hitting the deadlock depending on the hardware.

[solud]

I have fixed this "bug" in my software!

I no longer believe this is a bug in LibVLC, per se -- more like the way the library is used. I'll try to explain what I've found, though since my experience in this type of thing is limited I may still be wrong (but currently deadlock-free)!

The problem seems to be with Win32-LibVLC's use of WaitForSingleObjectEx(...) during vlc_join(...). If you call this within a thread that has child windows, it seems you risk deadlocking the application. Here's a quote from the docs:

Use caution when calling the wait functions and code that directly or indirectly creates windows. If a thread creates any windows, it must process messages. Message broadcasts are sent to all windows in the system. A thread that uses a wait function with no time-out interval may cause the system to become deadlocked. Two examples of code that indirectly creates windows are DDE and the CoInitialize function. Therefore, if you have a thread that creates windows, use MsgWaitForMultipleObjects or MsgWaitForMultipleObjectsEx, rather than WaitForSingleObjectEx.

With a timeout specified as INFINITE in the following code, a deadlock can occur randomly since no messages will be able to be processed:

Code: Select all

do vlc_testcancel (); while (WaitForSingleObjectEx (handle, INFINITE, TRUE) == WAIT_IO_COMPLETION);

When my application called libvlc_media_player_stop(Handle) from within the main thread, which definitely has child windows, it ran the high risk of deadlocking when it hit vlc_join(...). And from my testing this only happened on the call to stop the 2nd player, I can only assume it was because in the time it was stopping the first player a window message was sent that couldn't get processed thus causing WaitForSingleObjectEx(...) to deadlock.

The solution for me, since I don't think it necessary to modify LibVLC code, was to place the libvlc_media_player_stop(Handle) calls in a separate thread starting from _FormClosing(...). Problem solved!

I tested this on a workstation that would deadlock almost instantly (old CPU, slow) and was able to open and close a form with two players repeatedly for 10 minutes without deadlock. I'll be doing further testing later in the day, and if I find anything else I'll be sure to add more information here.