HEAP corruption any time I delete a picture pool (picture_pool_Delete)

[theonlylawislove]

I have to do some custom rendering stuff on windows. So, I created a new plugin. I am using VLC for windows (2.2.1). I am using the provided include files.

I created a new direct3d rendering project that is an exact copy of the direct3d provided in the VLC source for 2.2.0 (on GitHub). I had to message a few lines of code due to visual studio lacking some of the newer c++ features, but in the end, I got it working. I can now play using my custom plugin, and the video behaves exactly the same as the default direct3d plugin, EXCEPT, I get an error when the video is finished rendering (play stopped). After inspection, the exception was raised when calling "picture_pool_Delete".

I cannot delete a picture pool. I don't think this is related to my plugin per-say. It just seems to be an issue with the VLC api on windows. The following code reproduces the issue.

Code: Select all

picture_pool_configuration_t pool_cfg; memset(&pool_cfg, 0, sizeof(pool_cfg)); pool_cfg.picture_count = 1; pool_cfg.picture = &picture; pool_cfg.lock = Direct3DLockSurface; pool_cfg.unlock = Direct3DUnlockSurface; sys->pool = picture_pool_NewExtended(&pool_cfg); if (!sys->pool) { picture_Release(picture); IDirect3DSurface9_Release(surface); return VLC_ENOMEM; } // freeing the pool throws a heap violation! // TEST: BEGIN picture_pool_Delete(sys->pool); sys->pool == NULL; // TEST: END

The exception being thrown is...

Code: Select all

Unhandled exception at 0x779F5624 (ntdll.dll) in HDMDWide.exe: 0xC0000374: A heap has been corrupted (parameters: 0x77A11378).

Any ideas as to why I can't free a picture pool on Windows?

[theonlylawislove]

To debug this easier, I tool the source code for picture pool and added it to my visual studio project. I renamed all the exported methods to have a "2" at the end.

Now, within my "vout display" plugin, I am using my new "picture_pool_NewExtended2" method, and I still get the same issue. Also, it seems this exception is only thrown while a debugger is attached. Visual Studio is some heap detection stuff while the debugger is running to help prevent developers from accidently writing to unknown memory, corrupting the application.

My guess is that gcc doesn't have this function to help detect heap corruption, so this issue has flown under the radar. However, I do imagine that anyone using VLC may experience unknown and hard-to-reproduce bugs because of this.

Like I said, I have duplicated the code in VC++, and using some analysis tools and old-fashioned debugging, I will try to find what is causing the heap corruption.

I am not that familiar with C and the alloc/calloc methods (I'm more c++ new/delete), so if anyone can take a look at the "picture_pool.c" file and see if they can spot something that may corrupt the heap, it would be greatly appreciated.

[Rémi Denis-Courmont]

I compile VLC with ubsan and asan, and I don't have any issue with usage or test cases of picture_pool.c...

[theonlylawislove]

Remi,

I understand. All I know is that any plugin compiled for VLC under MSVC++ throws these heap corruption error.

The problem might not be with picture_pool.c. After I duplicated the picture_pool.c in my project, I removed this line.
https://github.com/videolan/vlc/blob/2. ... ool.c#L223

After removing this line, everything behaves normally. Although, I imagine there is a memory leak.

My next step is to duplicate "picture.c" in MSVC++ so that I can now investigate that method.

Also, if you would like, I could give you a simple MSVC++ command line application that uses the compiled .lib/includes for the VLC windows installer that can reproduce this issue easily.

[theonlylawislove]

Remi,

Also, are those tools you mentioned static code analyzers? Or, are they runtime analyzers?

[theonlylawislove]

Remi,

After duplicating the picture_Release method to determine where the problem occurs, I have now narrowed it down to this line.

https://github.com/videolan/vlc/blob/2. ... ure.c#L292

I'll keep digging.

[theonlylawislove]

Remi, I found the issue!

It turns out, in the direct3d module, we didn't have a "pf_destroy" callback being set on the "picture_resource_t" like the other modules are.

https://github.com/videolan/vlc/blob/2. ... t3d.c#L997

I added this code and it worked!

Code: Select all

static void pf_destroy_empty(picture_t *pic) { // do nothing } ... resource.pf_destroy = pf_destroy_empty;

the picture_Release was calling a pf_destroy that was not set. This causes an error on the VC runtime, and frankly, it should throw an error on other runtimes also.

[Rémi Denis-Courmont]

No. In direct3d.c pf_destroy is (implicitly) initialized to NULL. Then picture_NewFromResource() will use the default destroy callback.

There are already assertions in place to ensure that the callback is not actually NULL in the end.

[theonlylawislove]

I'm not sure then, because if I set resoruce.pf_destroy = NULL, I definitely get this issue.

Also, after calling "picture_NewFromResource", pf_destroy is still NULL.

[Rémi Denis-Courmont]

Either this is an ABI incompatibility, or the bug is already fixed in newer (dev) versions.