DVD features for VLC 1.3.0

[TheBigTime001]

Here is my list of things I would like to see in VLC 1.3.0 .
1) DVD upscaling filter(hopefully advanced), that can use a hardware-detection script of sorts to determine how much upscaling to perform.
Ideally, lower-end hardware should be able to recieve 480p to 600p images, and medium- to very-high-end devices should receive 720p to 1080p equivalent.
Reason: Increased picture sharpness and definition, less banding and mice-teeth effects, and much richer and detailed colour.
Extra: Maybe to make it easy to reach, there should be a menu item for this concept, if it is ever developed. Un-usable converts should be grayed out in that case.
2) Better recording option and implementation.
When recording, a 5 to 10 second buffer should be performed to prevent loss of either the audio or video stream elements when a pause occurs, such as a transition between .VOB files, for instance, or in the case of a broken but still-loadable media file.
Reason: Most people(me included) probably don't like watching media files where certain elements suddenly cut out.
Extra: I think to distinguish it, you should tack a version number on to it (like Record 2.0).
3) More de-interlace(progressiveness) filters should be added if possible.
There are probably dozens of de-interlacers out there, and if some more of those are compatible with VLC, then they might provide smoother and sharper, more defined de-interlaced pictures. The choices are fine for most people, but some more tweaking there might be necessary for some media.
Reason: All computers today that are recent can both input and output a full progressive stream at 60 fps.

[VLC_help]

3. 1.2.0 has new filters, but there are netbooks that can't do all of those filters realtime.

[TheBigTime001]

What I meant, was that while the Yadif2X and IVTC filters are GREAT for certain films and medias, they don't seem very intelligent at all for media with very unstable progessive frames, like old movies or certain very-high-quality CGI films. If there was a way to mix the Yadif2X and IVTC filters together for better predictiveness of progressive B- and P-frames, that would work completely for those films. If that ever works out, I would simply recommend calling it Yadif+IVTC.

[Jean-Baptiste Kempf]

Did you try phospor?

[TheBigTime001]

I have a suggestion here:
Can you implement a filter that decreases brightness in really bright scenes and also more effectively "de-grainifies" a film? Perfect 480p output would be optimal. Please try to implement a de-interlacer that's as good an equivalent as a component video cable, if you can.

[Jean-Baptiste Kempf]

Try the other filters

[Technologicat]

Hi,

(happened to browse the forums after a long while...)

Here is my list of things I would like to see in VLC 1.3.0 .
1) DVD upscaling filter(hopefully advanced), that can use a hardware-detection script of sorts to determine how much upscaling to perform.
Ideally, lower-end hardware should be able to recieve 480p to 600p images, and medium- to very-high-end devices should receive 720p to 1080p equivalent.
Reason: Increased picture sharpness and definition, less banding and mice-teeth effects, and much richer and detailed colour.
Extra: Maybe to make it easy to reach, there should be a menu item for this concept, if it is ever developed. Un-usable converts should be grayed out in that case.

Sharpness could indeed be helped with a more advanced upscaler. Several well-known algorithms exist for scaling images, with well-known properties. (For an overview, see e.g. http://www.imagemagick.org/Usage/resize/#filter) For downscaling, sinc with a large kernel is ideal. For upscaling, probably something else. The same methods could very well be used for video, too, if they can be made fast enough.

I have no idea, though, if any kind of hardware acceleration is available for anything other than bilinear interpolation (which produces a blurry image). If not, the picture needs to be scaled on the CPU, which at 1080p (output) is a lot of pixels to push.

Maybe the scaling algorithm could be implemented in OpenCL to offload the work to the GPU. However, to my knowledge there is no OpenCL yet in VLC, so getting it integrated may be a lot of work for the infrastructure, before the actual scaler can be implemented.

Mice-teeth sound like a result from an interlaced image that was not deinterlaced properly. Thus 3) should help with this.

Colour should not be affected by pure upscaling. Proprietary algorithms in various AV equipment probably do something on top of the upscaling. I think that if desired, colour processing could be done in a separate filter.

In any case, I'm probably not the right person to do this. I may have a minor interest in improved upscaling now that I recently bought an HD projector, but I prefer to leave the colors as-is :)

2) Better recording option and implementation.
When recording, a 5 to 10 second buffer should be performed to prevent loss of either the audio or video stream elements when a pause occurs, such as a transition between .VOB files, for instance, or in the case of a broken but still-loadable media file.
Reason: Most people(me included) probably don't like watching media files where certain elements suddenly cut out.
Extra: I think to distinguish it, you should tack a version number on to it (like Record 2.0).

(I have no comment on this.)

3) More de-interlace(progressiveness) filters should be added if possible.
There are probably dozens of de-interlacers out there, and if some more of those are compatible with VLC, then they might provide smoother and sharper, more defined de-interlaced pictures. The choices are fine for most people, but some more tweaking there might be necessary for some media.

If you see an interesting deinterlacer in some open-source software, please let me know!

It takes some work to integrate an existing filter into VLC, and it may not always be possible (wrong programming language, interface too difficult to be worth the effort...), but sometimes it is less work than designing a new filter from scratch :P

Of the filters currently in VLC, Yadif comes from MPlayer. I think it was integrated by Laurent. The IVTC implementation is original to VLC - I looked at the Xine deinterlacer, but the code was hard to comprehend and hard to interface with. In this case, I found it easier to design and implement a new filter than to port the existing one. (I also looked at the IVTC in Transcode, but its processing model was incompatible with VLC's. The code for both existing filters served as a useful source of ideas, though.)

Reason: All computers today that are recent can both input and output a full progressive stream at 60 fps.

True, but low-end machines (e.g. netbooks, convenient for travelling...) may not have enough CPU power to do much advanced filtering to the stream. More powerful machines, of course, are fine these days as you said.

[Technologicat]

What I meant, was that while the Yadif2X and IVTC filters are GREAT for certain films and medias, they don't seem very intelligent at all for media with very unstable progessive frames, like old movies or certain very-high-quality CGI films. If there was a way to mix the Yadif2X and IVTC filters together for better predictiveness of progressive B- and P-frames, that would work completely for those films. If that ever works out, I would simply recommend calling it Yadif+IVTC.

As the author of VLC's IVTC, I agree it's not very intelligent :P

It should be slightly more accurate than Transcode's and Xine's IVTC filters (since it selectively combines both approaches, and on top of that supports soft telecine), but it still works on a full-frame basis. It was designed to work primarily with anime, which has more difficult motion characteristics than live-action film, due to the common use of 8fps and 12fps animation in the telecined stream. This may limit its effectiveness for some other kinds of material.

The technical reason is that the possibility of 8fps and 12fps animation causes limitations as to how the filter can make its predictions about the telecine sequence. In such material it is a common occurence that nothing moves between two film frames, so we cannot assume that motion always occurs when the film frame changes. Of course, the placement of the film frames within the input stream is a priori unknown, so they must be detected somehow (this is called locking on to the cadence). But the only thing the filter sees on its input side is the telecined stream, which has no useful flags to help the filter... hence, realtime analysis of the actual picture.

Over 99% of anime is hard-telecined, based on a full-frame telecine. Thus, the filter works pretty well for the kind of material it was designed for. However, I've seen one case of anime ending credits (Claymore) where the left half of the picture (having the actual credits scroll) was pure interlaced, while the right half of the picture was made of telecined animation. The full-frame approach obviously fails with such material. Also, according to 100fps.com, some NTSC music videos liberally mix telecined and interlaced material in different parts of the same frame.


The thing with IVTC is, (a part of a frame in) NTSC video is either telecined or it is not. If it is telecined, then IVTC can help. The best case scenario is a 100% correct reconstruction of the original progressive stream. In practice this is never reached, but over 99% is possible (with the occasional missed frame or slow vertical camera pan). If the video is not telecined, then IVTC will in the best case do nothing (if it detects correctly the lack of telecine), and in the worst case it will damage the picture (thinking it is telecined and running the reconstruction process).

Yadif is a completely different kind of beast, called an interpolator. Interpolating filters will make a pure interlaced stream look better, but they will damage telecined material (they do not always detect correctly, nor do they have the framerate conversion which is needed for IVTC). For 60fps input, the basic Yadif version will output 30 fps, recreating the missing field out of thin air, and potentially discarding 50% of the input data. The Yadif2x version will keep the output at 60fps, using all input data, and recreating the missing fields out of thin air. Both versions are area-based, i.e., they work locally instead of the full frame.

See http://wiki.videolan.org/Deinterlacing for some more info.


So - disregarding the shortcomings of the full-frame approach for now - the problem of combining IVTC and Yadif becomes that of detecting whether a given frame is telecined, interlaced, or progressive, and applying a different filter (or not applying any filter) based on that. While developing the IVTC filter last December, I tested some technical ideas regarding automatic detection of the material type. It didn't work very well. Either I need better ideas, or this is a hard problem :)

I also tested the possibility of having a backup deinterlacer when IVTC fails. The problem is that it is really hard to detect whether a picture is interlaced or not. Most of the time, this experiment ended up detecting incorrectly and damaging progressive frames (that had been correctly reconstructed by IVTC).

Then, there is one further complication - the framerates do not match. IVTC will output at 24fps when locked on to the cadence, and 30fps otherwise. Interlaced video in NTSC is always 60fps. The lowest common multiple of 24 and 60 is 120, i.e. one would need 120fps output to be able to accommodate both exactly. However, it is possible to use the 2,3,2,3,... trick (like in soft telecine) for the 24fps material, and output at 60fps. The current filter was not designed for this - it's doable, but it would need some nontrivial changes.


Finally, I have a question. Can you provide some examples of the kinds of instabilities you mentioned? I'd like to understand the picture and motion characteristics for those, to see what could be done about them.

[Technologicat]

I have a suggestion here:
Can you implement a filter that decreases brightness in really bright scenes and also more effectively "de-grainifies" a film? Perfect 480p output would be optimal.

Hmm, sounds like this would require a combination of what Epson is doing with their "Superwhite" feature* and some denoising.

Brightness and contrast detection, even locally, does not sound very difficult. At least it should be much easier than interlace detection. Making it time-adaptive (to smooth out sudden changes, so that it won't flicker) needs some work, but is fairly simple (e.g. using a simulated, optimally damped spring-masspoint system).

Denoising is trickier, but I'm sure algorithms exist. In the audio processing field, independent component analysis is the best tool that I'm aware of, but it's fairly difficult and it may require lots of computing power, if it even is generalizable to video. I have no idea. Simpler algorithms exist, too, but the problem with cheap denoising techniques is that they tend to generate lots of artifacts.

So, it's probably doable, but I don't know if I'm the right person to do this. As a user I'm more of the please-don't-touch-the-picture type :)

* Mentioned and speculated on in this AVForums thread: http://www.avforums.com/forums/projecto ... hread.html

[Technologicat]

What I meant, was that while the Yadif2X and IVTC filters are GREAT for certain films and medias, they don't seem very intelligent at all for media with very unstable progessive frames, like old movies or certain very-high-quality CGI films. If there was a way to mix the Yadif2X and IVTC filters together for better predictiveness of progressive B- and P-frames, that would work completely for those films. If that ever works out, I would simply recommend calling it Yadif+IVTC.

One more thing regarding old movies.

If your source is in the PAL format, there is no 3:2 pulldown like in NTSC, but some films use 2:2 pulldown. Most of the time this is just PsF[1], so it can be handled by switching off the deinterlacer. However, if a one-field temporal offset has occurred during the telecine process, the fields that make up the same film frame will end up in different frames in the stream (see e.g. [2] about PAL telecine).

Currently, VLC has no filter to cope with this situation. VLC's IVTC only removes 3:2 pulldown (NTSC), not 2:2 (PAL).

I experimented with the possibility of adding 2:2 too when I was developing the filter, but eventually decided against it for two reasons. First, the detection proved too difficult to do reliably (NTSC telecine often triggered the PAL detector...), and secondly, I have no films of this kind to test the filter on. Ideally, I would need several clips (30 seconds or so each) from different films.

Finally, 2:2 is not the only pulldown scheme that is used in PAL. The 2:2 scheme requires a 4% speedup in the audio track to match, because what was originally 24fps film is played back at 25fps by the PAL equipment. Unless advanced time stretching is used for the audio while preparing the telecine, the speedup causes the pitch of the sound to go up by approximately one semitone.

Due to this reason, at least according to Wikipedia[3] there is has been a trend away from 2:2 toward more complicated pulldown schemes like 2:2:2:2:2:2:2:2:2:2:2:3 (Euro), in which the original audio track can be used as-is. This would require a yet different kind of filter to cope with.

References:

[1] PsF = http://en.wikipedia.org/wiki/Progressiv ... nted_Frame
[2] http://www.cambridgeimaging.co.uk/downl ... inance.pdf
[3] http://en.wikipedia.org/wiki/Telecine