What is Gaussian Splatting and How Will it Impact XR?

You may not have heard of Gaussian Splatting before, but I predict that you are going to hear a lot more about it over the coming weeks and months – and probably through this website – if we are doing our jobs properly…!

Gaussian What..?

Gaussian splatting; it’s new on the scene but it’s already making big waves…seriously. The 3D practice of recording and producing photorealistic volumetric point clouds using a technique known as Gaussian Splatting and it is a relatively recent commercial process. As a result of the technique’s reliance on photogrammetry, the generation of rendered scenes of a high quality is extremely rapid, inexpensive, and intuitive.

Inspired by Lee Alan Westover’s 1991 doctoral thesis at the University of North Carolina in Chapel Hill, who imagined splatting as the sound a snowball makes hitting a brick wall, Gaussian splatting started with a basic premise.  Westover described the splatting process like this:

“Consider the input volume to be a stack of snowballs and the image plane to be a brick wall. Image generation is the process of throwing each snowball, one-by-one, at the wall. As each snowball hits the wall, the snowball flattens, makes a noise that sounds like ‘splat’ and spreads its contribution across the wall. Snowballs thrown later obscure snowballs thrown earlier. This mental picture of the feed-forward rendering process inspired the name splatting. In the actual algorithm, the term splat refers to the process of determining a sample’s image-space footprint on the image plane, and adding the sample’s effect over that footprint to the image.”

While the notion was a fine one, the computer hardware of the time hadn’t a hope of fulfilling the dream, and remained a great idea unrealised. But now we have lightning-fast hardware, and a gazillion tons of RAM, and gaussian splatting is back on the menu.

Around 2020, technology researchers were becoming interested in NeRF’s – a neural radiance field – and started to look again at gaussian splatting as a commercial rendering method. A NeRF is a neural network that is capable of reconstructing intricate three-dimensional sequences from a partial set of two-dimensional images. In order to enhance the realism and precision of digital interactions, three-dimensional images are necessary in a variety of applications, including gaming, media, and XR. The NeRF takes on the form of the geometry, objects, and angles of a specific scene. The system then generates synthetic data to fill in voids, rendering photorealistic 3D views from novel viewpoints.  

Researchers started to investigate how gaussian splatting could be used to assist the NeRF process, but the approach was found to be memory-intensive, generated larger file sizes and, like NeRFs, could only render static scenes of a single moment in time. But with increasing computing power, both NeRF’s and Gaussian Splatting have become a major topic in rendering.

How Does Gaussian Splatting Work?

Gaussian splatting begins with capturing scenes from several perspectives from the input material. Structure from Motion (SfM) methods then fill in the blanks to estimate a 3D point cloud that depicts what the scene should look like from all sides. The method then translates every point into overlapping Gaussian splats like hazy clouds. But these first Gaussians just show colour and location, and further rendering is required.

Using gradient descent, millions of rounds of computation cycle through a follow-on training process filling in more information about how the point may be stretched or scaled (covariance) and its transparency. Millions of particles in a file created by this procedure reflect location, colour, covariance, and transparency, but the process doesn’t end there.

During the render process, each Gaussian particle is converted into the corresponding red, blue, and green pixels that comprise each view through a process known as Gaussian rasterisation. The resulting mesh subsequently acquires the texture of the entire collection of triangles; however, the Gaussian splatting replaces the precisely defined triangles with Gaussian smears that directly capture the texture too. The final result is a perfectly rendered image.

Why is it Important?

Of course, we all want increased realism without rendered images and scenes, but do we really need it? The short answer to that is ‘yes’. As users demand ever greater detail in their online experiences, advanced render systems such as gaussian splatting will be required. The key to its success is its ability to fully render any 3D scene. Think of how Google Streetview is currently. You can be in a street, but you can only progress along any road in a series of jumps. But if the entire network was re-rendered using gaussian splatting, then every part of every mapped area would be available, and in much greater detail. While that might be a mammoth task, you can see how the process will benefit smaller projects. XR content will become increasingly lifelike, and that can only be a good thing.

The Asset Store of Unity now includes a Gaussian splatting package, which enables users to immediately begin experimenting with novel rasterisation approaches. Additionally, a specific plugin has been developed for Unreal Engine, which means that gaussian splatting is rapidly becoming a widespread method. Dive in and try it out today..!