Film mode deinterlacing

[Technologicat]

Hi everyone,

I was wondering if there are any plans for including an inverse telecine (IVTC; a.k.a. film mode or 3:2 pulldown removal) deinterlacer in VLC? In other respects VLC has become the cross-platform swiss-army knife of video playback, and for playing video files it's great, but considering NTSC DVDs this particular feature is oddly missing. Searching this forum shows* some similar threads over the years, but always with very few posts and no conclusive reply.

* "Telecine" was the most successful search term: search.php?keywords=telecine&terms=all& ... mit=Search

I'm interested in this, because NTSC anime DVDs are practically always hard-telecined. For such material, inverse telecine is necessary to avoid comb artifacts on a progressive display. From what I've heard, the DVDs are often also badly mastered, so the flags cannot be trusted, and a simple cadence-based weave will not work to undo the telecine. Instead, the video stream itself must be analyzed on the fly.

Of the open source video players that support DVDs, Kaffeine and Xine (both aimed for Linux) have a working IVTC implementation. Xine uses the deinterlacer from the open-source TVTime project as a plugin; it provides both video (interpolating) and film (IVTC) modes. I'm not sure of the internals of Kaffeine's deinterlacer, but it switches between video and film automatically.

A third project that implements IVTC is MPlayer, with their "ivtc" and "pullup" filters, but my impression is that they have never been seriously interested in DVD playback (with menus and all). In the other threads on this forum, it was said that on the Windows side, DScaler, AviSynth and ffdshow implement IVTC.

The platform I'm most interested in is Mac OS X, because there no (at least open-source) player currently exists that does IVTC. But regardless of that, I think VLC is in any case the best candidate for this, because the DVD playback is great, and it's cross-platform.

So, what I'd like to ask is, are there any plans toward this direction?

P.S. If not, how complicated is VLC's deinterlacer API, and specifically which files would I want to look at? I don't have much free time, but I'm a programmer, and if it's fairly simple to glue in one of the working IVTC implementations (i.e. if both sides are modular enough), I might be interested in doing that myself. (That is, assuming you guys are accepting patches.)

[VLC_help]

P.S. If not, how complicated is VLC's deinterlacer API, and specifically which files would I want to look at? I don't have much free time, but I'm a programmer, and if it's fairly simple to glue in one of the working IVTC implementations (i.e. if both sides are modular enough), I might be interested in doing that myself. (That is, assuming you guys are accepting patches.)

There are some existing filters http://git.videolan.org/?p=vlc.git;a=bl ... nterlace.c; that you could modify in case you want to test it out.

[Technologicat]

There are some existing filters http://git.videolan.org/?p=vlc.git;a=bl ... nterlace.c; that you could modify in case you want to test it out.

Thanks! Exactly what I was looking for. I'll check it out (pun unintended).

By a quick glance:

• VLC's deinterlace.c seems rather understandable, given the number of existing filters that are available as examples.
• Kaffeine (currently v1.1, http://kaffeine.kde.org/?q=download) handles deinterlacing inside the Xine engine, so they use the same deinterlacer.
• The relevant files in the xinelib source tree (as of v1.1.19, from http://www.xine-project.org/releases) are in src/post/deinterlace. It's pure C, so it should be suitable for this. At least tvtime.*, pulldown.* and speedy.* seem to be required for IVTC. This may take a while :)

There's a possible problem in that IVTC requires keeping state (mainly, cadence position) in the deinterlacer across successive frames, something that none of the other filters do. The filter_sys_t structure doesn't seem to support this. I guess it's possible to keep the state in a local static variable. While not an ideal solution, it should work.

I'm off to figure out whether it's possible to connect glue these things together with reasonable effort. Thanks again :)

[Technologicat]

Updating this thread with the results from my brief study.

Short version: I think this is doable for unstable 1.2-git, but not for the stable 1.1.x series. Unfortunately, as of this writing, VLC 1.2-git for OS X is not ready yet.

A more detailed version follows.

As may be well known, the Mac port of HEAD (1.2-git) is currently not in a usable state. In a post last month*, Kühne suggested staying with 1.1.x on the Mac unless one specifically wants to concentrate on fixing 1.2-git. :)
* viewtopic.php?f=12&t=84214#p278785

From what I could gather, the current status seems to be that the GUI is being reworked (based on comments in projects/macosx/framework/Configure.sh), but the VLC app hasn't yet been ported to the new infrastructure (VLCKit). The main build process no longer produces the old "macosx" interface module, and while audio and the LUA interface work (if launched from the command line as VLC.app/Contents/MacOS/VLC), there is no graphical output. Thus for testing IVTC, the new version on OS X is currently a no-go.

One is left with the 1.1.x series (in the vlc/vlc-1.1.git repository). This compiles successfully on OS X (10.6.5) with a working GUI, as suggested.

Let's take a look at the differences between 1.1.x and 1.2-git regarding the deinterlacer module. Based on comparing the two current HEAD versions of the source file (1.2 vs. 1.1), and the commit log for deinterlace.c at current HEAD from 1.2-git, http://git.videolan.org/?p=vlc.git;a=hi ... c4;hb=HEAD
the major changes seem to be:

• 1.2-git allows the deinterlacer to produce two output frames per one input frame. This is intended for supporting framerate doubling deinterlacers, which currently are Bob, Linear and Yadif (2x). This seems to be a new feature.
• 1.2-git allows the adjustment of output frame timings in Deinterlace(), via the p_dst->date field. This too is a new feature. Currently it is only used for framerate doubling, but the mechanism looks general.
• 1.2-git specifically allows the deinterlacer to drop whole frames during normal operation. (The 1.1 HEAD seems to allow this only in error conditions.)
• According to the commit log mentioned above, the "video filter" part has been removed recently. The diff for this particular commit shows that the function called Render() was removed. Considering the functionality inside the functions Render() and Deinterlace() in the older 1.1.x, it seems the new version of Deinterlace() in 1.2-git has (in terms of functionality) replaced it. This is probably related to the changes to VLC core that were mentioned in the commit log - reading between the lines, I suppose framerate doubling and frame delays should now work during playback?

The new features concerning the timing adjustment and frame dropping are required for IVTC, because the extra fields added during the telecine process must be removed during playback. Each complete 3:2 pulldown sequence consists of eight unique fields inside the 3-2-3-2 pattern, making up four original film frames. This implies one frame out of five must be dropped and the timings of the rest adjusted accordingly (i.e. the duration of each frame needs to be increased by 25%). This is correctly done by Xine.

In conclusion, it seems it's not possible to make an IVTC deinterlacer for VLC 1.1.x, but it would be possible for VLC 1.2-git. This is assuming that I understood the implications correctly, i.e. that the VLC core indeed allows the deinterlacer to change the frame timings in an arbitrary manner and to drop any frame it wants to.

Looking at the code from xinelib, the relevant parts are a bit over 1000 lines in total, and aside from some low-level routines, it looks pretty simple. I could do this if someone else is up for fixing 1.2-git on OS X. :)

[Jean-Baptiste Kempf]

Please, use 1.2.0 for development.
1.2.0 OS X gui more or less works now.

[Technologicat]

Please, use 1.2.0 for development.
1.2.0 OS X gui more or less works now.

That's great news! Sounds like I have a project, then ;)

However, at the moment I'm having problems getting the VLC XCode project for 1.2-git to build - that's why I thought it wasn't ready. I also tried the command-line build method that used to work in 1.1.x. That way the build succeeds, but it leads to the no GUI and no graphical output situation that I mentioned.

Using the "Build Everything" target, the XCode project bootstraps and configures successfully. Upon first build, what XCode thinks are errors look like normal informational messages from bootstrap. Starting the build again then continues from the target "Make VLC", but it seems there's something weird with the build directory setting in the project file (the version in git HEAD). Only config.log gets created in the build directory, which leads to make not finding a makefile and the actual build never starting.

I'll post a new thread in OS X troubleshooting and we'll see how it goes.


Edit: didn't post yet. Now VLCKit builds! I took a closer look at the created config.log. It said "already configured", and configure had actually failed due to this, which I didn't notice earlier. The log suggested running "make distclean" in the source directory to purge the old configuration.

Then I remembered that I had originally first tried the command-line compilation approach described in the wiki. So indeed, it was already configured. I went to the root of my 1.2-git tree, and ran "make distclean" as suggested. Then back to XCode, and "Build Everything". Problem solved.

I still need to figure out a few things, but it's a start :)

I'll post a note in OS X troubleshooting in case someone else runs into the same problem.

[Technologicat]

Status update.

Still having a problem with getting VLC 1.2-git working on OS X. Posted about this in:
libVLC development: viewtopic.php?f=32&t=84336
OS X troubleshooting: viewtopic.php?f=12&t=85644

Tested some new anime DVDs. There are two possibilities:

• Telecine cadence breaks at a cut are the rule, not the exception. (This would imply that anime for DVD is practically always edited in a telecined form.)
• Vektor's deinterlacer loses cadence tracking at cuts, and unnecessarily falls out of film mode until it catches the cadence again.

Currently, need to investigate which one is correct.

Strategy for an independent approach:

Thinking that for a robust solution, some sort of aggressive, comparative field matching may be needed to identify which pairs of fields in the three frames in the history (prev, current, next) are most likely to correctly form progressive frames. The interlace detection techniques from the X and Yadif deinterlacers (in VLC's deinterlace.c) can probably be adapted for this. The field pairs obtaining comparatively the lowest scores in pairwise interlace detection can be considered progressive.

Additionally, duplicate field detection can be used, taking advantage of the fact that NTSC telecine inserts duplicate fields. In out of five possible positions for the three-frame stencil, there is only one position where there are no exact duplicate fields. (This is assuming that there is any motion, so that the consecutive frames actually are different. Also, there are possible implications from the combination of lossy MPEG compression and telecine, depending on how DVD video is organized before compression (frames as-is, frames split into fields, or just individual fields). I'm not familiar with this part, so I'll have to check this somewhere. Though in practice, having a nonzero tolerance for noise probably solves this issue.)

These techniques allow us to reconstruct the cadence position from the video data, in cases where the history (of the last few frames) matches the 3:2 cadence. This is not too different from how Vektor's deinterlacer works (but I should definitely check the sources again). The new idea is to be more aggressive about staying in film mode if at all possible. The "hybrid" technique (pairwise field matching + duplicate field detection) should allow to catch the cadence in at most two or three video frames, if there is motion. This is important for coping with bad cuts.

The mechanism cannot be completely stateless, for several reasons. Four new full-frame interlace detections are needed per each arriving frame. Caching the previous four results cuts down the processing time in half. This is assuming that "new next" is newly arrived, "old next" becomes "new current", "old current" becomes "new prev", and "old prev" is discarded.

Secondly, to detect cadence breaks, the tracked (last known) cadence position must be retained between calls (similar to Vektor's implementation). The known position is taken from an initial successful detection. At each frame, the predicted position (based on the tracked position) is then compared to the newly detected position. The cadence position information is needed in order to tweak the presentation timestamps (p_dst->date) for each set of 5 input frames, to account for the framerate reduction. What to do with presentation timestamps in case of a cadence break is probably solvable by some more thinking on the issue. Leave them as-is until the cadence is caught again?

Notes:

The output from the deinterlacer will have a variable framerate: 23.976 fps when film, and 59.94 fields per second when video (i.e. when the film mode does nothing). The latter case needs to be handed off to a "video mode" deinterlacer (any of the existing ones), producing 29.97 fps or 59.94 fps interpolated progressive. Because the display refresh rate is finite and rather low (especially, not a least common multiple of all the required video framerates), completely smooth motion at a bad cut is impossible. In practice, this is probably not a serious problem.

As a bonus, the aggressive detection strategy lends itself to a local (area-based) approach, like in the X deinterlacer. However, area-based IVTC is probably not a smart thing to do: it leads to a headache with the timestamps, in cases where different parts of the frame need to be updated at different times (e.g. two telecined areas of video in the same frame, running at different points in the cadence simultaneously). This could be handled by generating a new frame with updated data at each point of time where at least one area in the frame needs to be updated, but in practice no display device can handle that smoothly.

But that is probably a pathological case. I assume it is far more common to have a global telecine pattern, so this can very well be left to a (far-)future version. As a stopgap measure, area-based deinterlacing (such as X) can be applied to the de-telecined result, to catch any remaining interlacing artifacts adaptively. This may need to be made configurable, because X smoothes progressive parts of the video, leading to a global loss of picture quality even when there is nothing to deinterlace. On the other hand, this is understandable, to hide the seams between progressive and deinterlaced areas, because these do not match the shapes of objects in the video. It needs to be tested whether this quality loss is noticeable in practice or not.

To obtain output free of tearing artifacts, the output frames should be synchronized to the vblank of the physical display device. This should be done in a presentation layer after the deinterlacer. If the input to the presentation layer has a constant framerate, this can be done by finding a cadence to match the video and display framerates, as XBMC does (search for CPullupCorrection in the XBMC sources for their solution). I'm not currently familiar as to whether VLC does this. However, this strategy will (theoretically) go south with unpredictable, variable framerate input (as produced by IVTC from material containing bad cuts). In practice, I guess it should work most of the time, because the framerate will generally stay constant for long stretches. Thus, any judder will be localized to the bad cuts, if the display cadence finder is intelligent enough to reset itself when it detects a framerate change.

In any case, that's not for the deinterlacer to worry about. The only thing the deinterlacer should do here is to make sure that the same amount of time (clock ticks) goes out as comes in. Depending on the format of p_dst->date, fractional tick tracking may be needed in the deinterlacer state (if the timestamps are based on the 90kHz clock - haven't checked this yet).

To consider: is 2:2 pulldown needed (for PAL movies), or only 3:2 (for NTSC)? Can we use the original framerate (which can be computed from successive presentation timestamps) to detect which possibilities for input the deinterlacer should expect? This would allow it to automatically switch off at 24fps input, and try to detect 3:2, 2:2, or true interlaced video, as necessary.

[Technologicat]

Project update.

Reading http://www.animemusicvideos.org/guides/ ... etb2a.html convinced me that it's the anime that is at fault regarding cadence breaks. I knew anime was difficult to handle correctly, but not that it could be that bad. (See their discussion of PAL conversions. Shudder.)

Some NTSC anime with known interlace types (telecined, pure interlaced, hybrid) are listed here:
http://arbor.ee.ntu.edu.tw/~jackeikuo/dvd2avi/ivtc/
I have some of these in my own collection, so now I know which ones to use for acid testing ;)

About telecine field dominance (a concept different from the field dominance of the video!):
http://www.cambridgeimaging.co.uk/downl ... inance.pdf

Comments from the author of Transcode's IVTC filters:
http://users.softlab.ece.ntua.gr/~ttsiod/ivtc.html
For their implementation, see filter/filter_ivtc.c in Transcode sources (available at freshmeat; current version is 1.1.5). It's shockingly simple, but it does no cadence tracking so it's not suitable for realtime playback. But the field matching approach is good - I was already planning a cadence tracker using this idea.

Currently, I have the infrastructure in place. I'm implementing the actual filter now. It's an original design borrowing ideas from existing filters. I thought about it some more and decided that this was the easiest way. For now, I'll use a three-frame stencil like Yadif does.

I had originally misread VLC 1.2-git's deinterlace.c - Yadif is the only filter that drops a frame under normal operating conditions, and the frame dropped is always the second frame counting from filter startup (i.e. before its frame cache has been filled). But I suppose if it's ok for Yadif to do that, then the core probably works fine with filters that intentionally drop the occasional frame, too. We'll see how this goes.

[Technologicat]

Update update:

Looking at this list http://avisynth.org/mediawiki/External_filters (search for "telecine"), and keeping in mind the first link in the previous update, suggests that it's best to keep this filter relatively simple, and it'll work for a large majority of material.

For the... more creatively telecined cases, re-encoding is the only even remotely sane option. It's impossible to reverse e.g. a FILM > Telecine > NTSC > Blend conversion > PAL sequence during playback, given that it's already nearly impossible for offline analysis.

Fortunately most NTSC anime is simpler than this: either just telecined NTSC, or a hybrid with telecined NTSC and pure interlaced material (computer-generated effects) in the same frame. If the field pair matching does what I think it will (based on what I read about Transcode), I may have a satisfactory solution.

So, I think I won't go into "full field blended" NTSC, as detecting that would make the filter slower, and such material is reportedly rare. That can be added later, if needed.

[Technologicat]

Development update.

The filter's starting to look good. Surprisingly, it started working better when I disabled some of the paranoia that I had designed to make it more accurate. I think I kind of understand why some deinterlacer chips are said to have trouble automatically distinguishing between interlaced and film...

The strategy works as follows. Cadence detection: field pair comparison, utilizing a table of combinations that should come up as progressive at the different cadence positions (these sets are unique). The row with the lowest interlace score wins. Cadence tracking: various heuristics. IVTC itself: based on whether the filter judges the detected cadence position reliable, either the cadence counter + an operation table or Transcode's approach (pick the least interlaced field pair, with appropriate limitations). Frame dropping and timestamp mangling are always timed from the cadence counter; in practice this seems to work. The two IVTC strategies complement each other - the tracking system follows camera pans better, while the "Transcode" approach catches some frames correctly that are missed by the tracking.

The comb detectors from RenderX() are useful. Thanks to the team for those :)

Currently, I think the output is rather good. The motion is relatively smooth, as it should be. There are occasional interlacing artifacts, but then again I haven't seen a perfectly accurate realtime IVTC filter. Whereas Vektor's tracking mechanism has trouble coping with cuts, but is very good at staying locked on, this one recovers slightly faster, but also loses the tracking more often. This is to an extent compensated for by the complementary strategies approach.

To intercept residual interlacing artifacts (from incorrectly IVTC'd frames and from interlaced graphics), the frames are thresholded after IVTC, and as a final step passed through RenderX() if judged as interlaced. This takes care of cuts, but artifacts concentrated to only a small part of a frame (e.g. anime scenes with characters talking and nothing else moving) often go unnoticed by the check. I think trying different comb metrics for the judgement step could be worth a try.

There's a problem on some DVDs with some of the subtitles constantly disappearing and reappearing when this filter is enabled, but I have no idea what could be causing that. It doesn't happen on all DVDs, and it seems concentrated to specific scenes. It seems that the more the filter has trouble staying in film mode (i.e. the more it is constantly switching the framerate reduction on and off), the more the subtitles will flicker.

At first I thought it was to do with the timestamps, then the frame composer (which glues together a frame with top and bottom fields from given frames), but to me everything looks correct now and this problem is still occurring. If anyone has ideas, I'd appreciate input.

Coming up next: testing Transcode's combing metric, to see how it compares to the ones from RenderX(). But after that, I think I'll declare this filter reasonably complete and concentrate on cleaning up the code and comments.

I might be looking to contribute some code next week, if the team is interested in a patch :)

[Jean-Baptiste Kempf]

I might be looking to contribute some code next week, if the team is interested in a patch

... I hope it is a joke

[Technologicat]

I might be looking to contribute some code next week, if the team is interested in a patch :)

... I hope it is a joke :)

I sure hope my code is not a joke ;)

Seriously though, of course you'll get a patch as soon as I've cleaned up my changes.

I implemented the Transcode comb metric and ran some tests last night. It improved the output right away. RenderX()'s comb detectors are good, but for IVTC Transcode's seems unbeatable. Most scenes with localized motion (e.g. characters talking) now seem to be handled correctly. Occasional artifacts do still slip through, though.

To catch scenes with localized motion, I had to give up on subsampling and use all pixels for the detection. Comparing NTSC DVD playback with the filter on and off, CPU usage on the Atom N450 (according to KDE System Activity) is about 36% when the filter is on, and about 20% when the deinterlacer is switched off. So, on the Atom this filter takes about 15-20% CPU. Comparatively speaking, it approximately doubles the workload compared to DVD playback without it.

I noticed that when using this comb metric, it's better to always output the least interlaced field combination out of TNBN, TNBC and TCBN (T = top, B = bottom, N = next, C = current) regardless of what the cadence counter says. Then we just hope that the cadence counter (and the aggressively updating cadence detector) does what it's supposed to, so that the frame drop occurs at the right time (so that no duplicate frames are output, and no unique frames are dropped). In practice this seems to work. This change also helped a bit with the subtitle flicker problem, though I have no idea why.

I'll watch some more anime on this to make sure it works ;) , and then clean it up and send it in.

Where should I send the patch, by the way? Is there a preferred format (e.g. output from "git diff" as-is or something else)? I tried looking on the wiki, but couldn't find any pointers on this.

[Jean-Baptiste Kempf]

http://wiki.videolan.org/Git will tell you all you need to know.
Short summary is: git commit . && git format-patch -o patches origin/master && git send-mail --to vlc-devel@videolan.org patches

[Technologicat]

http://wiki.videolan.org/Git will tell you all you need to know.
Short summary is: git commit . && git format-patch -o patches origin/master && git send-mail --to vlc-devel@videolan.org patches

Thanks!

Code cleaned up and patch sent.