Multi-view Pixel Arrangement Plug-in

[Moddy3d]

Hello,

I am currently in the process of writing a plug-in that processes a 8-tile auto-stereoscopic format video into the auto-stereoscopic TV ready format.

In the past I've had experience writing this pixel arrangement algorithm with other libraries like OpenCV and as a plugin for Nuke, however there seems to be a lot of details to writing image processors in VLC I am not familiar with. This is my first experience with VLC, so please be kind ^^

Abstractly, the process is as follows
INPUT: 8-tile view format Video (1920 x 1024~)
1. Extract the 8 individual views and store them as temporary pictures roughly sizes of 640 x 400
2. Scale each picture up to the current screen resolution, or even better, allow for dynamic scaling + proper pixel arrangement
3. Do the pixel arrangement process, basically by masking each individual scaled-up picture in a different manner, and then merging them. (For example, in the final result, column 1 would have pixels from view 1, column 2 from view 2, col 3 view 3... etc, to align with the Auto-stereoscopic tv)
OUTPUT: Full-screen Resolution Pixel-arranged Format Video

Now here is the current progress,
1. Implementing it in the form of a video filter.
2. In the Open() call for filter initialization, I store the size and locations of each tile into the filter object's private data, p_sys.
3. In the Filter() call for filter process, I begin by creating a group of picture_t pointers and allocating new picture_t's.
4. I proceed to copy the corresponding tiles from the original image into these new picture_t's.
Right now I'm at the stage where I need to scale these images but I don't understand how to do that.

I have a quite a bit of questions:
For video_format_t
1. What does the i_x_offset and i_y_offset really do?
2. What does the _visible_ mean? What's the difference?
3. If I am specifying a location on a picture (X,Y), from what corner of the image does it count from?
4. What's this i_align thing?

For Scaling
1. I read the scale.c source but I don't understand it, because it's quite cryptic to me and uncommented, could anybody possibly explain the operations?

For Screen Resolution
1. How do I find out what the full-screen resolution (W,H) of the target device?
2. How do I find out what the current resolution (W,H) of the video in VLC is? This way it could be even better because I can do the scaling from the tiles to the window size before the pixel processing, so the final result would work and display properly at any window size.

Thanks a lot for your time to read, and I will appreciate any sort of advice and feedback!

Regards!

[Rémi Denis-Courmont]

1 and 2. offset and visible value define the useful subrectangle of the picture that is actually meant to be displayed. The rest is padding.

3. A picture does not have a location. A picture is always a whole frame.

4. There is no such field in either picture_t or video_format_t.

1. Scaling is done in hardware or in dedicated plugins. Filters don't do scaling unless they really need to.

1. You don't: there may be other filters behind you, and there may be a hardware scaler; this is not your business. Besides, nothing warrants that you are in fullscreen mode, or that you even do have a display: filters can run in transcode pipelines. You have a fixed input format and an output format and that is all. b_allow_fmt_out_change specifies whether you are allowed to change the output format.

2. Same as 1. If you need to care about the display resolution, you need to write a video output. That's a completely different matter.

[Moddy3d]

Hey Remi, thanks for answering the questions, but I think I will need to rephrase a few questions to make things more clear

3. A picture does not have a location. A picture is always a whole frame.
What I would like to do is to cut out a sub-rectangle from the main picture and save it in a temporary picture. I need to know how to specify a coordinate on the picture and the way VLC interprets the coordinates.

1. Scaling is done in hardware or in dedicated plugins. Filters don't do scaling unless they really need to.
So say I wanted to scale a bunch of temporary pictures up in the middle of the filter, is there a way to call the hardware scaling functions?

1. You don't: there may be other filters behind you, and there may be a hardware scaler; this is not your business. Besides, nothing warrants that you are in fullscreen mode, or that you even do have a display: filters can run in transcode pipelines. You have a fixed input format and an output format and that is all. b_allow_fmt_out_change specifies whether you are allowed to change the output format.
Here is the problem, for this the result to display in 3D, each pixel has to be displayed at the resolution of a pixel on the display. For example on a HDTV, the display resolution is 1920 x 1080, this plugin would need to scale the 8 different views (temporary images) up to 1920 x 1080, then do the pixel arrangement so that the pixels are aligned perfectly to the display's "pixels". For a laptop say with 1368 x 768, this plugin will need to scale the views to 1368 x 768.

Now, I would like to know how to get the current display resolution of the VLC output video for dynamic 3D output (meaning I can change the size of the window however I want and it will output 3D properly) because I will need to do the scaling BEFORE the pixel arrangement process.

Thank you!

[Rémi Denis-Courmont]

3. Pictures (picture_t) do not have coordinates. They are full frames. The question makes no sense.

1. Video output scaling is intrinsically tied to the output mechanism, and typically performed by the GPU.

1. Again, defining the display resolution makes sense only in the video output plugin. A video filter has an input format and an output format and that is all.

[erwan10]

My two cents ..... Why not write a specific Opengl shader at the GPU level ?

In other words, you let vlc load the picture as a GL texture as it already does when using Opengl. Then, you adapt this vlc video output at the 'prepare' level to suit your needs (see modules/video_output/opengl.c). Part of this algorithm may even be better optimized if implemented as an Opengl shader. That may look more like a hack than a valid vlc extension, and may be prone to criticism, but it should work fine.

Also, Opengl is just an example. This could similarly be done with Microsoft direct3D stuff.

[Moddy3d]

My two cents ..... Why not write a specific Opengl shader at the GPU level ?

In other words, you let vlc load the picture as a GL texture as it already does when using Opengl. Then, you adapt this vlc video output at the 'prepare' level to suit your needs (see modules/video_output/opengl.c). Part of this algorithm may even be better optimized if implemented as an Opengl shader. That may look more like a hack than a valid vlc extension, and may be prone to criticism, but it should work fine.

Also, Opengl is just an example. This could similarly be done with Microsoft direct3D stuff.

Hey Erwan, thanks for your suggestion. You talked about this 'prepare' level. Where's a good source (other than google) to read up about import information regarding a video processing pipeline? I'm not very familiar with most of the terms in the VLC libraries so I get confused quite often.

I've been trying to extract a rectangle from a full frame, and it's been a pain. For different videos it's yielding different results. I'm not sure how pitch/lines work exactly. When I mentioned coordinates, I mean I need to specify specific pixels on the frame. How do I do that?

Erwan you mention OpenGL as a way of solving this problem, can you elaborate more on that and how I would begin coding it? I know the pixel transform logic but how do I set OpenGL up to do this when playing videos in VLC?

[erwan10]

Well, what the 'modules/video_output/opengl.c ' file does has got very much similarity with what you seek to do :
- picture_t are received periodically together with subpicture_t (first for preparation then for real display)
- the final picture composition is carried out with the picture as background and subpictures blended on top of it

Actually,this possibility to carry out final picture composition at this level is relatively new. The rationale was to remove as much as possible various picture conversions (chroma conversion, blending, scaling ....) from system memory/CPU and take advantage of OpenGL or DIrect3D APIs to deport such work at the GPU level.

What you seek to do is just another way to compose the final picture out of picture_t. So getting to understand the 'module/video_output/opengl.c' code is perhaps the best documentation !

As for calling it, you can set it in preferences (Opengl video output). And beware, this is a hack that will break normal use of vlc if you just replace the default algorithm with your own. So far, this place is not meant for extending the code right out of the box. Some more work would be needed for a better integration.

[Rémi Denis-Courmont]

The 'prepare' callback has existed since January 2002, so it is really anything but new. It used to be called 'pf_render'.

[Moddy3d]

Hmm... So I've done the image extractions and scaling, and onto the final part where for the pixel arrangement ... Working with subpixels in VLC is such a pain.. hahah.

But I've hit a wall, the scaling operation (lowest quality one provided as sample code) is definitely too slow on the CPU, the pixel arrangement process hasn't even begun yet and it's losing framerate already... Will have to offload everyhing to the GPU like you suggested.

I tried setting my video output mode to OpenGL, and when I load a video, it crashes. Any ideas?

Thanks for your help guys

[erwan10]

Be more specific about the config and the test you're running (OS, versions, ....) ? If you just say "it crashes", the only possible answer is "it can be a zillion things !"

[Rémi Denis-Courmont]

Crashes in OpenGL are almost all caused by bugs in the display drivers.

[Rémi Denis-Courmont]

I don't know how much processing you expect and need to do. But HDTV is typically about 2 Gbits/s worth of data. It's, err, quite a lot for a general purpose processor to handle in any way. Thus image scaling and also conversion from YUV to RGB color spaces are normally performed by the graphic card instead. This is tied to the output process: the scaled and converted video is output to the specified area of an OS window handle, not back to the main system memory. Thus VLC cannot perform any post-processing of the scaled frames...

It seems to me that you need a custom video output plugin that would have the awareness of how to do 3D on your target system. Presumably, the existing 2D video output APIs are not enough anyway.

[Moddy3d]

erwan, I'm running on a Windows 7 64-bit with NVIDIA GEFORCE VFX 550 TI and an Intel i5 processor. Vlc version 2.1.0

Remi, could you point me to a starting point in developing a custom video output module? I notice on the wiki there isn't a documentation specific for writing it. I've noticed in the vlc/modules/video_output/ folder there is a bunch of code. Is that my only resource to figure out how it works? Do you have any ideas what sort of concepts / technology / API I need to learn to write this custom module? Thanks~!

[Jean-Baptiste Kempf]

Start from an existing one.

[Moddy3d]

Alright I've got this to work by just editing the default template_yuv_to_rgb shader. thanks for your help guys

However, I'm still running into a problem where vlc 2.1.0's OpenGL Output module does not work. 2.0.5's work, but it's shaders are based on ARB.

What kind of video cards / driver versions do you guys use do have OpenGL Output module running in 2.1.0??

Thanks

[marco3d]

Hi Moddy3D,

it looks like you've written a plugin for the Tridelity 3D displays, is this Plugin available? Where can I find it?

Thanks