Interoperability of systems is crucial for creative and cultural industries. However, in the area of extended (XR) realities, formats and standards are not yet as mature and established as for standard media like video, images, or audio. Therefore, the main objective of T3.1 was to ensure that TRANSMIXR use cases and technologies are sufficiently supported in terms of formats and standards.
Work in T3.1 started early in the project with an initial overview and assessment of XR-related formats and standards. This gave initial recommendations for design and development. At a platform level, OpenXR was identified as the most promising format for general XR experiences. For web applications, WebXR was recommended. At asset level, it was found that many types of content are already well-supported, however this space seems quite fragmented with several specifications for similar purposes. USD and glTF were identified as promising developments going forward, and this is also reflected in industry activities.
A need and opportunity was identified with as volumetric video not yet supported in glTF. To address this, the project developed an extension to glTF named VVglTF. This was published and submitted to a standards organization (Khronos). It was also used in TRANSMIXR’s experiences Planica and Performance, and in a follow up project of HSLU. USD was evaluated with the nVidia Omniverse platform in the Performance use case. While USD proved very useful at an asset level, Omniverse is not yet mature enough on platform level.
A variety of XR experiences were developed in TRANSMIXR, including both VR and MR. All of them used OpenXR as their platform format, following the initial assessment and establishing it as TRANSMIXR’s “eXtended Reality Media Experience Format”. Meta Quest 3, in standalone and tethered modes, was selected as the main hardware platform. Unity and Unreal were applied as game engines.
At an asset level, the experiences use a variety of different 2D and 3D media in related formats, which results in a rich and insightful overall picture. Volumetric video is used in all experiences. VVglTF is used for offline cases, while real-time streaming is done using the VR2Gather system. Standardized real-time streaming of volumetric video is an area of opportunity for further development.
Other, more standard media (3D objects, animations, mocap, avatars, 2D video, 360 video, images, audio, etc.) are well supported by existing formats and standards, but fragmentation is evident in this context. USD and glTF have the potential to streamline and unify this space in the future.
Gaussian Splatting, as a novel type of 3D asset, has a lot of potential for future work, including in the context of formats and standards. For instance, efficient compression is an active area of research.
Overall, we conclude that the objectives related to XR formats and standards could be achieved and that generally the field is maturing, while challenges and opportunities remain.
D3.2 eXtended Reality Media Experience Format
This deliverable is pending approval by the European Commission and may undergo revisions.
