Managing 3D sound for HMD

[FL53]

Hi,

I developped an OpenGL 360 video player (to display the video into a head mounted display) but I still need to implement 3D sound management.
Does someone know how I could modify in real time the sound perception using pitch/yaw/roll, quaternion or rotation matrix (and HRTF obviously) ?
Best regards,

Marc

[mfkl]

Hi,

I believe this library is used https://github.com/videolabs/libspatialaudio under the hood, probably the readme gives you a head start.

Good luck

[FL53]

Thanks, I will have a look.

I found "libvlc_video_update_veiwpoint" (https://www.videolan.org/developers/vlc ... 6a74c64af0) in libvlc_video_controls should do the work. I tried to use that creating a new viewpoint for each frame ... but it fails (since the video is not seen as an equirectangular video I think, even if I set projection directly in the track)

[Jean-Baptiste Kempf]

Thanks, I will have a look.

I found "libvlc_video_update_veiwpoint" (https://www.videolan.org/developers/vlc ... 6a74c64af0) in libvlc_video_controls should do the work. I tried to use that creating a new viewpoint for each frame ... but it fails (since the video is not seen as an equirectangular video I think, even if I set projection directly in the track)

You can inject the metadata in the file.

[FL53]

Hi,

In fact, metadata is already injected the file I use (since I can read it in VLC or Windows video player in 360 directly), but the viewpoint cannot be set up ( libVLC log in console [ main vout display error : Failed to change viewport ] ). I do not know how to use this method but ideally I would like to only modify the spatial audio since my rendering is using the full video texture with shaders to generate the 2 textures for HMDs. Maybe I should use another method to inject the orientation of the user for spatial audio ?

[FL53]

I have another bug I cannot fix : with some videos (those with spatial audio plugin used it seems) while stopping media player a breakpoint is reached (happen in spatialaudio_plugin.dll calls), really a strange issue (crashing while quitting is not really "annoying" for user who won't see anything but I used to develop clean code ) ... I keep on searching for heap corruption or someting like that in my code but if this is an already met error I would be grateful if you could help.

Regards

[Jean-Baptiste Kempf]

File 2 bugs for those.

[FL53]

I have posted bug reports for those 2 issues.
I checked some elements before posting to better explain the errors. You'll find here my conclusions:
- ambisonic sound seems to make something wrong into libVLC mediaplayer since libvlc_media_player_stop generates an error. I did not find a way to make it work even using the default instance of VLC and the video and then calling stop method on mediaplayer.
- when the default video output is not used (using callbacks to generate a texture for instance), libvlc_video_upadte_viewpoint is not woking any more (with default output it is OK, I can update the viewport without any problem). Using custom video output, ambisonic sound cannot be updated so. But to do HMD integration, it is really pertinent to use libVLC such a way (using custom output) to "inject" textures calculated from video stream (using loc/unlock callbacks) into the HMD API method. A way to update the viewpoint without taking into account the video output should be considered or a method to update viewpoint only for ambisonic sound should be exposed into libVLC (maybe there is such a method but I did not find it in the documentation).

[FL53]

SpatialAudio module crash :
Waiting to find a solution, I would like to clean my development to avoid the end crash. Is there any way to ask VLC not to use ambisonic audio plugin ? reading the 4 channels as mono channels maybe ?
More information about the crash: in the Visual 2013 console I can see the following lines ...

main decoder debug: killing decoder fourcc `mp4a'
main decoder debug: removing module "faad"
main audio resampler debug: removing module "samplerate"
main audio renderer debug: removing module "spatialaudio"
HEAP[MyApp.exe]: Heap block at 0E379360 modified at 0E37936C past requested size of 4

Viewpoint update issue :
I found the aout_FiltersChangeViewpoint method (vlc_aout.h), how could I use it to update the viewpoint of the current audio filter ? This may be "THE" clean solution for my viewpoint update issue ?

[Jean-Baptiste Kempf]

Share a complete backtrace.