Offscreen Media Service

[Kaffee Tasse]

I am curious if an offscreen media service exists for VLC already, or if it is desired/wanted:

* Offscreen decode/render various media formats in a separate/forked child process(es).
* Frame image composition / overlays / color format conversions / image flipping, etc done by the child process.
* Shared Memory interface to access the service for fast frame-based access and to control the offscreen player.
* Running multiple media in a single or more child processes / services simultaneously.


Advantages compared to 'all in one' players:

* Moving CPU and memory load to an external process.
* Efficient use of multicore CPUs.
* Recoverable service / child process in case of a crash, not affecting parent application(s).
* Centralized decoding 'once per stream', multiple viewers.
* Output is available to multiple applications and/or multiple devices.
* In particular 32bit 'all in one' application may run into memory address issues, when running multiple high definition (4k) media streams simultaneously.


Example Usage Scenarios:

* Applications using their own/existing rendering surface, such as games or custom web browsers for example.
* Advanced frame based video editing, or video mixers.
* Deliver already decoded fast video streams over LAN (local area network) in high quality, without additional load (mem and CPU) to end-clients/viewers, e.g. to workstations, as high definition video decoding may cause.


What I got so far:

I am experimenting with an offscreen service for Windows so far. On Windows I use a fixed size 'MapView' of committed memory for the interface. Tbh, I haven't looked any closer into the VLC source code yet. All I did was using the SDK header files that come with the standard distribution. On Windows, MS's DirectShow interfaces, as well native WM-Format-SDK can also be integrated into this media service, e.g as a level of fallback.


If you think this would be either a cool contribution, or even a part of the existing VLC SDK/interfaces/plugins, please let me know (optionally, feel free to reply to the email of the forum account in these regards with the subject 'Offscreen Media Service').

~Chris

[Rémi Denis-Courmont]

Efficient use of multiple cores can basically only be achieved with threaded decoding. Process separation makes it impossible, not easier.

Also, process separation requires a lot of extra memory copies, so it's actually far less efficient. In any case, it would require a lot of work. Without a wealthy and generous sponsoring company, I very much doubt it will ever be implemented.

[Kaffee Tasse]

Efficient use of multiple cores can basically only be achieved with threaded decoding. Process separation makes it impossible, not easier.

Also, process separation requires a lot of extra memory copies, so it's actually far less efficient. In any case, it would require a lot of work. Without a wealthy and generous sponsoring company, I very much doubt it will ever be implemented.

I think I have to explain this better:

* From an application's point of view (not talking about VLC internal decoding), load and memory usage, as well as count of threads per process are reduced significant when splitting up into processes. In particular mult-core CPUs can take advantage, when single core CPUs might even experience on opposite affect.

* Not a single additional copy of a video frame's memory is required. The 'lock_callback' can assign shared memory directly to the 'p_pixel' pointer, so that even the media-client application can access the original frame memory location from any other process.

[Kaffee Tasse]

The shared mem structure may look like:

Note:
* The context passed to the 'lock_callback' is a pointer to this structure stored in shared mem.
* The m_framePixels member is assigned to the 'p_pixels' pointer of 'lock_callback'.

Code: Select all

typedef struct sharedMemMedia { unsigned int m_cb; mPClient_s m_caller; // some user data... int m_videoWidth; int m_videoHeight; int m_videoDepth; int m_videoStride; // using fixed size frame data for simplicity unsigned char m_framePixels[MAX_RESOLUTION * MAX_RESOLUTION * 4]; } sharedMemMedia_s;

Open shared mem on Windows:

Code: Select all

m_shMemHandle = OpenFileMappingA (FILE_MAP_READ | FILE_MAP_WRITE, FALSE, m_shMemName); m_pshMemMediaInfo = NULL; if (!m_shMemHandle || m_shMemHandle == INVALID_HANDLE_VALUE) { return; } m_pshMemMediaInfo = (sharedMemMedia_s*)MapViewOfFile(m_shMemHandle, FILE_MAP_READ | FILE_MAP_WRITE, 0, 0, 0); if (!m_pshMemMediaInfo) { CloseHandle (m_shMemHandle); m_shMemHandle = NULL; return; }

At the application-end, first create the shared memory region, second launch the service process and pass the name or handle to the service process.

Note:
* We use shared memory of committed memory, physical memory backed up by paging file. Optionally the application can define the shared memory region as non-swapable.
* One media service process can handle multiple streams at once (not shown in the code).

Code: Select all

m_shMemHandle = CreateFileMappingA( INVALID_HANDLE_VALUE, // use paging file NULL, // default security PAGE_READWRITE, // read/write access 0, // maximum object size (high-order DWORD) sizeof(sharedMemMedia_s), // maximum object size (low-order DWORD) m_shMemName); // name of mapping object

These code sniplets show how easy a shared mem implementation can be done. On Linux the principle is not much different, I'd think.

[Rémi Denis-Courmont]

I think I have to explain this better:

I think you need to get a clue about the specific field of digital video playback.

* From an application's point of view (not talking about VLC internal decoding), load and memory usage, as well as count of threads per process are reduced significant when splitting up into processes. In particular mult-core CPUs can take advantage, when single core CPUs might even experience on opposite affect.

That's utterly stupid. From the performance standpoint, splitting a single multithreaded process into multiple processes is stupid. It increases the marshalling overhead.

Process separation is used for fault isolation, bringing extra safety and/or security (e.g. Chrome browser), but it comes at a performance cost. It is not used for better performance of a single system.

* Not a single additional copy of a video frame's memory is required. The 'lock_callback' can assign shared memory directly to the 'p_pixel' pointer, so that even the media-client application can access the original frame memory location from any other process.

How do you convince OpenGL or DirectX to share memory across processes? VLC does not control how the frame buffers are allocated, the video drivers do.

Really, with the possible exception of XVideo and plain slow X11, memory copying of decoded frames are needed if decoding and the rendering run in separate processes.

[Kaffee Tasse]

I think I have to explain this better:

I think you need to get a clue about the specific field of digital video playback.

* From an application's point of view (not talking about VLC internal decoding), load and memory usage, as well as count of threads per process are reduced significant when splitting up into processes. In particular mult-core CPUs can take advantage, when single core CPUs might even experience on opposite affect.

That's utterly stupid. From the performance standpoint, splitting a single multithreaded process into multiple processes is stupid. It increases the marshalling overhead.

Process separation is used for fault isolation, bringing extra safety and/or security (e.g. Chrome browser), but it comes at a performance cost. It is not used for better performance of a single system.

* Not a single additional copy of a video frame's memory is required. The 'lock_callback' can assign shared memory directly to the 'p_pixel' pointer, so that even the media-client application can access the original frame memory location from any other process.

How do you convince OpenGL or DirectX to share memory across processes? VLC does not control how the frame buffers are allocated, the video drivers do.

Really, with the possible exception of XVideo and plain slow X11, memory copying of decoded frames are needed if decoding and the rendering run in separate processes.

After all, Rémi, I would not have expected this out of your mouth. You might find it useful at some point to use a chromium - like shared memory interface for rich streaming media.

Thanks you for time,
Chris

[Rémi Denis-Courmont]

Oh I am all for isolating demuxers and decoders into unprivileged processes. But I very much doubt anyone will ever invest the time and money involved, several man-years not counting maintainance. And, generally, it will degrade performance, for the sole benefit of fault isolation.

[MichaelMc]

For what its worth this was my attempt at a similar problem for VLC 1.10: SVMEM here. Seemed to work well at the time.