Creating Virtual Environments with 3D Gaussian Splatting:

A Comparative Study
Shi Qiu*, Binzhu Xie, Qixuan Liu, Pheng-Ann Heng
Department of Computer Science and Engineering, The Chinese University of Hong Kong
Institute of Medical Intelligence and XR, The Chinese University of Hong Kong

Novel View Synthesis

360° GS-based Scene

…
Poisson Texture
arXiv:2501.09302v1 [cs.CV] 16 Jan 2025

surface map
reconstruction extraction

Mesh Reconstruction

Multi-view Images Textured Mesh Scene

Panoramic 3D Scene

Synthesize

360 Photos Panoramic Image

GS-based Panoramic Scene

Figure 1: We develop three different 3D Gaussian Splatting approaches for virtual environment creation, where the corresponding
pipelines are illustrated in green, blue, and orange colors, respectively. More implementation details are introduced in Sec. 2.

A BSTRACT advances in neural rendering have introduced to synthesize novel

3D Gaussian Splatting (3DGS) has recently emerged as an inno- views from sparse input data. For instance, Neural Radiance Fields
vative and efficient 3D representation technique. While its poten- (NeRF) [6] utilize multi-layer perceptrons (MLPs) for implicit
tial for extended reality (XR) applications is frequently highlighted, scene encoding but face computational bottlenecks in real-time ap-
its practical effectiveness remains underexplored. In this work, plications. More recently, 3D Gaussian Splatting (3DGS) [5] rep-
we examine three distinct 3DGS-based approaches for virtual en- resents scenes with explicit 3D Gaussian points, offering practical
vironment (VE) creation, leveraging their unique strengths for effi- advantages: faster and more scalable rendering, high visual fidelity,
cient and visually compelling scene representation. By conducting and straightforward scene manipulation [7]. These features make
a comparable study, we evaluate the feasibility of 3DGS in creat- the 3DGS techniques promising for creating immersive VEs.
ing immersive VEs, identify its limitations in XR applications, and In this work, we explore the use of advanced 3DGS techniques in
discuss future research and development opportunities. VE creation for XR applications, particularly in balancing high vi-
sual quality with the performance constraints of real-time rendering
Index Terms: VR, 3D Gaussian Splatting, User Immersion. on HMDs. As illustrated in Fig. 1, we introduce and evaluate three
distinct 3DGS-based approaches for VE creation: (i) the novel view
1 I NTRODUCTION synthesis approach, (ii) the mesh reconstruction approach, and (iii)
Virtual environments (VEs) enable users to interact seamlessly with the panoramic 3D scene approach. By conducting a comparative
digital worlds across a wide range of extended reality (XR) appli- study, we aim to evaluate the feasibility and effectiveness of 3DGS
cations. To deliver truly immersive experiences, XR applications in creating immersive VEs, identify its limitations in XR applica-
require VEs to be highly detailed, visually realistic, and respon- tions, and provide insights for future research and development.
sive. However, creating such environments remains a significant
challenge, particularly in achieving a balance between rendering 2 I MPLEMENTATION
efficiency, scalability, and visual fidelity. Given the performance
For the novel view synthesis approach, we start by captur-
constraints of XR devices like head-mounted displays (HMDs), the
ing multiview images of the target environment and then pro-
creation and rendering of VE become crucial for user immersion.
ceed to estimate camera parameters and a sparse point cloud us-
Traditional VE creation techniques, such as 3D modeling and
ing COLMAP [8]. Subsequently, we leverage the original 3DGS
photogrammetry, rely heavily on polygonal meshes and manually
framework [5] to train 3D Gaussian representations of the scene.
crafted assets. While effective, these methods are time-consuming,
Once trained, the 3D Gaussian data is imported into Unity to syn-
labor-intensive, and struggle to represent complex scenes. Recent
thesize the novel view of the GS-based scene by decoding the
* e-mail: [email protected] learned information. We use an open-source shader implementa-
This work was supported by The Chinese University of Hong Kong (Project tion1 for the real-time rendering of GS-based scenes in XR.
No.: 4055212); and in part by the Research Grants Council of the Hong In the mesh reconstruction approach, our primary focus lies
Kong Special Administrative Region, China (Project No.: T45-401/22-N). in preserving fine details within the reconstructed scene. Build-
ing upon recent SuGaR [1] framework, we apply regularization
constraints to train more evenly distributed Gaussians across the
1 https://github.com/clarte53/GaussianSplattingVRViewerUnity
 Table 1: Summary of user study results (n = 8). The study comprises
7 male and 1 female participants with an average age of 25. All par-
ticipants have limited experience with VR or 3DGS. We use the SUS
questionnaire [9] to quantitatively compare users’ presence scores
(mean ± std; higher values indicate a greater sense of presence).
Virtual environments created using the approaches of
Score
Novel View Synthesis Mesh Reconstruction Panoramic 3D Scene
a) Novel View Synthesis b) Mesh Reconstruction c) Panoramic 3D Scene
Q1 5.5±1.4 4.8±0.8 2.1±1.0
Q2 4.3±1.3 4.0±1.4 1.8±1.0
Q3 5.1±2.0 5.3±1.8 2.8±1.7
Q4 5.6±1.6 4.8±1.6 2.0±1.3
Q5 4.8±1.8 4.3±1.3 1.9±0.8
Q6 4.0±2.0 3.8±1.8 1.1±0.4
SUS Count 2.8±1.8 1.8±2.4 0.0±0.0
e) Panoramic 3D Scene SUS Mean 4.9±1.2 4.5±1.1 1.9±0.9
d) User Operation from another viewing point

Figure 2: Qualitative comparisons of VE rendering results.

of 3DGS, which has received limited research attention. Our addi-
tional tests find that 3DGS-based mesh reconstruction could hardly
compete with traditional TSDF [3] methods, particularly when re-
scene’s surface. Given enhanced 3DGS representations, we sample constructing distant objects. Despite these shortcomings, mesh-
the scene to extract information about visible points. Subsequently, based representations remain foundational for XR development,
the Poisson surface reconstruction algorithm [4] is utilized to gen- and further research into 3DGS-driven mesh reconstruction holds
erate a triangular mesh based on the positions and normals of the significant potential. The panoramic 3D scene approach inher-
sampled points. We further texture the mesh-based scene given the its the strengths of traditional 360-degree panoramas, making it
cues from trained Gaussians [2], importing into Unity for XR use. ideal for applications like virtual tours. However, it is less effec-
In the panoramic 3D scene approach, our initial step is to gen- tive for XR applications requiring free movement in larger spaces,
erate a panoramic image of the target scene by synthesizing the as noted in the user study results (Tab. 1). Visual artifacts fre-
multiview images captured from a fixed location. Like the Dream- quently occur when users change their viewing positions (Fig. 2(c,
Scene360 [10] pipeline, we use a pre-trained monocular depth esti- e)). Even slight changes in the user’s standing position can signif-
mator alongside an optimizable geometric field to establish a con- icantly impact the display performance, as the panoramic image is
sistent scene geometry structure. To address unobserved areas, constrained to a fixed viewpoint. Future work may focus on inte-
pseudo-views and deformed Gaussian points are generated, leading grating multiview panoramas to create high-fidelity, 3DGS-based
to a high-quality panoramic scene represented in 3DGS. Finally, 360-degree scenes that support dynamic user movement.
this GS-based panoramic scene is imported and rendered in Unity
via a similar process as used in the novel view synthesis approach. 5 C ONCLUSION
This work explores three 3DGS-based approaches for virtual envi-
3 E VALUATION ronment creation, comparing their practical effectiveness through
We render the created VE models on an NVIDIA RTX 4090 GPU, system evaluations and user studies. Our findings highlight the ad-
with a Meta Quest 3 as the HMD for testing. As shown in Fig. 2 vantages of 3DGS in delivering efficient and visually compelling
(a-c), we qualitatively compare the rendered first-person views. All VEs. We also identify challenges unique to each approach, such
three approaches deliver comparable visual fidelity, with some reg- as limited object-level interaction and spatial distortions in certain
ular artifacts such as motion blur observed in (a) and (c), and recon- scenarios. These insights underline the potential of 3DGS for XR
struction incompleteness noted in (b). In terms of rendering speed, applications while pointing to future research directions.
all approaches achieve approximately 70 FPS, demonstrating the
R EFERENCES
feasibility of 3DGS for real-time XR applications.
We conduct a user study with 8 participants to evaluate the sense [1] A. Guédon and V. Lepetit. Sugar: Surface-aligned gaussian splatting
of presence and immersion within the created VEs, using the Slater, for efficient 3d mesh reconstruction and high-quality mesh rendering.
Usoh, and Steed (SUS) questionnaire [9]. As shown in Tab. 1, the CVPR, 2024. 1
[2] A. Guédon and V. Lepetit. Gaussian frosting: Editable complex radi-
novel view synthesis approach achieves the highest SUS Count and
ance fields with real-time rendering. In ECCV, 2024. 2
Mean metrics. It also scores the highest in five out of six SUS ques-
[3] S. Izadi et al. Kinectfusion: real-time 3d reconstruction and interac-
tions. Users report that this approach maintained consistent visual tion using a moving depth camera. In ACM UIST, 2011. 2
quality as they moved through the VE, enhancing spatial awareness. [4] M. Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface reconstruc-
By contrast, the other two approaches only present high-fidelity de- tion. In Eurographics/ACM SGP, volume 7, 2006. 2
tails in specific views, suffering from spatial distortions and arti- [5] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian
facts that negatively impact user immersion when moving around. splatting for real-time radiance field rendering. ACM Trans. Graph.,
42(4):139–1, 2023. 1
4 D ISCUSSION [6] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoor-
thi, and R. Ng. Nerf: Representing scenes as neural radiance fields for
In addition to the evaluations in Sec. 3, we identify several chal-
view synthesis. Communications of the ACM, 65(1):99–106, 2021. 1
lenges during implementation and experimentation. The novel [7] S. Qiu, B. Xie, Q. Liu, and P.-A. Heng. Advancing extended reality
view synthesis approach stands out due to its higher rendering ef- with 3d gaussian splatting: Innovations and prospects. In IEEE AIxVR,
ficiency, consistency, and editability. However, while the learned 2025. 1
3DGS representation captures geometric details, it offers only ba- [8] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise
sic collision and occlusion modeling in XR environments when view selection for unstructured multi-view stereo. In ECCV, 2016. 1
treating all Gaussians uniformly. To enhance object-level immer- [9] M. Usoh, E. Catena, S. Arman, and M. Slater. Using presence ques-
sion and interaction, a more robust mechanism is needed to iden- tionnaires in reality. Presence, 9(5):497–503, 2000. 2
tify and prioritize Gaussian points associated with specific objects. [10] S. Zhou et al. Dreamscene360: Unconstrained text-to-3d scene gen-
The mesh reconstruction approach is a downstream application eration with panoramic gaussian splatting. In ECCV, 2024. 2