[SOLVED] VLC (0.8.5) fullscreen deadlock

[jezz]

My friend have had recommended me vlc as movie player for linux (instead of mplayer). I have had successfully compiled vlc (it was some older release), but I have had problem with fullscreen. When I tried to switch to fullscreen vlc freezed. When I started vlc in fullscreen, thenthere were no video output at all, but movie slider was still moving. My friend have similar hw and sw, but he has not this problem. I am testing new releases (0.8.4, 0.8.5), but still without success. MPlayer runs on fullscreen well.

Now I have done some testing and I can post my results. When I run ./configure with --enable-debug, then it is ok, but this message appears:
VLC media player 0.8.5 Janus
[00000265] logger interface: using logger...
[00000341] x11 video output error: thread 2960255920: mutex_lock failed at xcommon.c:2410 (35:Resource deadlock avoided)
[00000341] x11 video output error: thread 2960255920: mutex_unlock failed at xcommon.c:951 (1:Operation not permitted)
[00000341] main video output error: caught in a callback deadlock?
[00000341] main video output error: caught in a callback deadlock?
[00000341] main video output error: caught in a callback deadlock?
[00000263] main playlist: stopping playback

I think that deadlock message is the critical one. In release there is deadlock.

My configuration:
Debian Etch
ii xserver-xorg 7.0.22 the X.Org X server
ii xserver-xorg-core 1.0.2-9 X.Org X server -- core server
ii xserver-xorg-input-evdev 1.0.0.5-2 X.Org X server -- evdev input driver
ii xserver-xorg-input-kbd 1.0.1.3-2 X.Org X server -- keyboard input driver
ii xserver-xorg-input-mouse 1.0.4-3 X.Org X server -- mouse input driver
ii xserver-xorg-video-ati 6.5.8.0-1 X.Org X server -- ATI display driver
ii blackbox 0.70.1-1.1 Window manager for X
vanilla kernel 2.6.15.5 with CONFIG_DRM_RADEON=m
HW: Radeon 9250 128M
Depth 24bpp

I have tried another wm (metacity, no wm at all), than my favorite blackbox; different outputs X11, XVideo, GL; alternative fullscreen method, but without any success.

[jezz]

I have debuged vlc and created patch, that fixes this problem (vlc sources 0.8.5):

--- modules/video_output/x11/xcommon.c 2006-09-01 09:55:12.983813656 +0200
+++ modules/video_output/x11/xcommon.c.new 2006-09-01 09:58:19.398548853 +0
200
@@ -886,8 +886,10 @@
vlc_value_t val;

/* Update the object variable and trigger callback */
+ vlc_mutex_unlock( &p_vout->p_sys->lock );
val.b_bool = !p_vout->b_fullscreen;
var_Set( p_vout, "fullscreen", val );
+ vlc_mutex_lock( &p_vout->p_sys->lock );

ToggleFullScreen( p_vout );
p_vout->i_changes &= ~VOUT_FULLSCREEN_CHANGE;

There is p_vout->p_sys->lock locked twice from same thread. In function ManageVideo is this mutex locked at the beginning and unlocket at the end. When vlc is switched to fullscreen, the callback for "fullscreen" is called and throught this callback is called function Control in my case with parameter i_query=VOUT_SET_STAY_ON_TOP - there is mutex locked twice! So in my patch I unlock mutex before callback and locking again after callback, to prevent deadlock. I know, that this simple solution is not perfect, but works (I am not vlc developer, so I don't know vlc architecture).

[jezz]

As I see, this is same problem as in this post: viewtopic.php?t=14356.

[guilc]

Hi !

Any news about this problem ?

I also have the same bug when entering fullscreen mode, even with the last 0.8.6 rc1 version.

The patch purposed by jezz allways applies fine ans solves the problem.
The 'Set on top' related problem does not solves this, it's NOT the same.

Current tested system : Gentoo ~x86 up to date with an NV17 (Geforce4MX420) card with last (1.0.9629) nvidia proprietary driver.
You can refer to the Gentoo bug : http://bugs.gentoo.org/show_bug.cgi?id=151498

Regards,

[funman]

hello

a ticket has been opened http://trac.videolan.org/vlc/ticket/882

i think the fix isn't the proper way, because i think when you release the mutex in ManageVideo(), and before it is locked again in Control(), another thread can lock it and modify it, and i guess it shouldn't be modified while ManageVideo() is running.

note the _think_, but thanks for providing a patch

however it won't go into vlc before someone looks at it carefully

[guilc]

Thanks for your help !

[jezz]

You are absolutely right. As I wrote, I am not vlc developer, so I don't know what is protected by this mutex. Releasing it can cause race conditions or other problems.
As programmer I have experiences with parallel processing, but without knowledge of vlc architecture, I can only suggest teoretical solutions. Changing mutex to recursive mutex will help, or to implement finer locking (using more mutexes).
May be these ideas won't help, but I hope, that I have contributed to vlc by tracking down this problem - the right solution is on vlc developers, of course.
I am glad, that ticket was opened for this problem, so it can be solved.