Advancements in 3D scanning technology, particularly through the innovative method of Gaussian splatting, are rapidly changing the augmented and virtual reality (AR/VR) market. This groundbreaking technique enables users to capture and recreate photorealistic 3D worlds with remarkable fidelity, offering far-reaching implications for industries like gaming, film, and virtual reality.
The effect of Gaussian splatting compared to regular mesh-based technique. 3D scanning technology combined with this technique can rapidly create photorealistic representations of the physical world. Image credit: Polycam
One of the notable developments is Gaussian splatting, a method that enables the creation of photorealistic 3D models using just a smartphone. This technique represents a significant leap forward in 3D capturing, making the process more accessible and efficient.
Unlike traditional polygon-based 3D modeling, which often struggles with complex textures and lighting, Gaussian splatting uses a network of translucent blobs—Gaussians—to render objects. Each blob stores precise data on color, position, scale, rotation, and transparency, culminating in highly detailed, photorealistic 3D models. When millions of these Gaussians are combined, they form a highly detailed and accurate 3D representation of the object or scene. This approach substantially enhances the fidelity of the captured models.
Machine learning drives this process, allowing for real-time rendering without the need for high-powered hardware.
Tech giants like Niantic, Google, Meta, and Snap have recognized Gaussian splatting’s potential. Niantic integrated the method into Scaniverse, enabling users to create photorealistic 3D captures on their smartphones. Google has applied it to enhance its mapping tools, while Snap has introduced Gaussian splats into its Lens Studio platform. Meta envisions applying this technology to construct immersive, lifelike metaverse environments.
For instance, apps like Polycam enable users to create, view, and edit Gaussian splats directly on their devices. Users can upload images or videos, which are then processed to generate detailed 3D models. These models can be manipulated—rotated, panned, and zoomed—providing an immersive experience. Additionally, the ability to export these models for use in other software platforms broadens their applicability across various fields.
Brian McClendon, SVP of Engineering at Niantic, hails Gaussian splatting as a transformative development in 3D graphics, comparing it to a “JPEG moment” for spatial computing. This ease and accessibility of 3D capturing promises to make photorealistic 3D modeling accessible to everyday users, driving innovation across multiple sectors.
The Use of Gaussian Splatting in 3D Scanning Technology
In 3D scanning, Gaussian splatting is used to reconstruct scenes by distributing thousands to millions of Gaussian primitives in 3D space. These Gaussians are optimized based on point cloud data obtained from technologies like LiDAR and photogrammetry. Unlike mesh-based approaches, Gaussian splatting does not require explicit connectivity between points. Instead, it creates a dense representation where overlapping Gaussians blend smoothly, leading to highly detailed and natural-looking scans.
One of the main advantages of Gaussian splatting is its efficiency in rendering. Unlike Neural Radiance Fields (NeRF), which rely on computationally expensive volumetric ray tracing, Gaussian splatting takes a more direct approach by rasterizing Gaussians onto the screen. This results in significantly faster performance, making it a viable option for real-time applications. The technique also provides a smooth and continuous representation of objects, avoiding the sharp edges and artifacts often seen in mesh-based reconstructions.
Additionally, it is more memory-efficient compared to high-resolution meshes or voxel-based methods, allowing for the storage and manipulation of large datasets without the excessive computational overhead. The ability of Gaussian splatting to capture fine details, reflections, and soft lighting effects further enhances its realism in scan-based 3D visualizations.
Transforming Video 3D Capturing
The use of Gaussian splats is not limited to static objects. Gracia AI, a computer vision startup, is employing the technique to record volumetric 3D videos. For instance, they captured a chef preparing a meal in full 3D, allowing viewers to explore the scene from all angles in VR. Unlike older methods, which imposed strict constraints on attire and lighting, Gaussian splatting enables creative flexibility. Shadows, translucent materials, and intricate details are seamlessly incorporated into the 3D models.
Despite its advantages, Gaussian splatting still faces challenges. Current 3D video files are data-intensive, with Gracia AI working to reduce file sizes from 9GB per minute to more manageable levels. Such advancements could make this technology viable for streaming and everyday applications, including VR training and educational content.
Meta’s Vision and Challenges
Meta has demonstrated the potential of Gaussian splatting through its Hyperscape app, which allows users to explore photorealistic 3D spaces in VR. While these demonstrations are highly polished, scalability remains a concern.
Currently, Hyperscape relies on cloud rendering to manage the heavy data loads, a hurdle that needs addressing before widespread adoption can occur. Nevertheless, Meta’s plans to integrate Gaussian splatting into its Horizon Worlds platform highlight the long-term vision for user-generated 3D spaces.
Implications for Industries
The adoption of advanced 3D capturing methods like Gaussian splatting aimed at various industries. In gaming, the technology promises hyper-realistic environments, enhancing immersion. Filmmakers could recreate intricate sets or historical locations with unprecedented accuracy, reducing production costs. Meanwhile, VR and AR applications are expected to benefit from lifelike simulations for education, training, and entertainment. Virtual reality applications can achieve higher levels of immersion, as users interact with environments that closely mimic real-world physics and aesthetics.
The market potential is enormous, as consumer-grade 3D scanning technology becomes increasingly accessible. For instance, smartphone-based applications that harness Gaussian splatting enable millions of users to engage in 3D capturing, fueling the creation of extensive spatial data libraries. This ease of use and technology accessibility continue mainstream trends in AI and machine learning, where user-generated data catalyzes innovation.
Final Thoughts
Gaussian splatting represents a substantial advancement in 3D scanning technology. Its ability to produce photorealistic models using standard smartphones opens up new avenues for creativity and application across various industries. The technology will mature quickly, and due to its inherent potential to be integrated with AI, we can anticipate even more groundbreaking developments that will shape the future of digital content creation.
If you are interested in this topic, we suggest you check our articles:
- AI-Powered Virtual Cohosts: Intelligent Streaming Assistant Unveiled, Promising to Transform Live Streaming Technologies
- Artificial Intelligence Use in Animation Creation
- Odyssey’s Explorer Tool for 3D Visual Creation
Sources: Niantic, Poly.cam, Wired, XR Vizion on Medium, Snap, The Verge
