3D Gaussian Splatting and Conversion to Its Major Variants

3D Gaussian Splatting (3DGS) is a revolutionary technique that renders photorealistic, real-time environments by scattering tens of thousands of Gaussian distributions in 3D space. Despite its success, 3DGS faces several structural challenges:

"Popping" artifacts during viewpoint transitions.
Visible "thickness" of Gaussians when viewed from specific angles.
Structural instability when initialized from sparse point clouds.

To address these issues, several variants have been developed. In this article, we explain the concepts and implementation methods for 2DGS, Mip-Splatting, and Scaffold-GS.

Technical Features

1. 3D Gaussian Splatting (3DGS)

The baseline method achieving a balance between real-time rendering and fast training.

3D Gaussian Primitives: Represents the scene using millions of overlapping, semi-transparent 3D ellipsoids (blobs).
Core Parameters: Each Gaussian holds position (center), a covariance matrix (defining shape/orientation), opacity (alpha), and Spherical Harmonics (SH) for view-dependent color.
Tile-based Rasterization: Projects Gaussians onto a 2D screen space and uses optimized GPU kernels for high-speed alpha blending.
No Ray Marching: By avoiding the heavy computation required for NeRF, it enables real-time rendering at 100+ FPS.

2. 2D Gaussian Splatting (2DGS)

Focuses on accurate surface representation and better adaptability for meshing.

2D Gaussian Surfels: Replaces 3D ellipsoids with oriented 2D disks.
Ray-Surfel Intersection: Directly calculates intersections between rays and 2D disks, enabling precise depth estimation and significantly reducing "floating artifacts."
Defined Normals: Since each primitive has a clear orientation, it allows for high-quality normal map generation.
Ease of Meshing: Because of the flat geometry and defined normals, conversion to standard 3D formats (.obj/.ply) is straightforward.

3. Mip-Splatting

Designed specifically to eliminate quality degradation caused by changes in camera distance (aliasing).

3D Smoothing Filter: Introduces 3D space filtering techniques inspired by the concepts of Mip-NeRF 360.
Frequency Constraints: Limits Gaussian frequencies based on pixel size, preventing background objects from becoming overly detailed.
Scale-Adaptive Rendering: Applies "blur" appropriately so that Gaussians do not become smaller than a single pixel, suppressing "salt-and-pepper noise" and flickering.

4. Scaffold-GS

Enhances memory efficiency and scalability for large-scale scenes.

Anchor Point Structure: Organizes the scene with a sparse voxel grid and links Gaussians to specific anchor points.
Neural Gaussians: Uses a small MLP (Multi-Layer Perceptron) at each anchor point to dynamically generate Gaussian attributes from neighboring information.
Growing Strategy: Adaptively adds or removes anchor points during training to optimize density based on scene complexity.
Storage Efficiency: Reduces memory usage by over 50% compared to original 3DGS, making city-level scale rendering possible.

Converting from 3DGS to Variants

1. Repository and Environment Setup

For each variant, you must switch to the specific repository and rebuild the environment.

3DGS: Original Repo
2DGS: 2DGS Repo
Mip-Splatting: Mip-Splatting Repo
Scaffold-GS: Scaffold-GS Repo

2. Rasterizer Updates

The core rasterizer (diff-gaussian-rasterization) must be replaced with the version tailored for the specific variant.

3. Code Modifications and Hyperparameters

When running the training scripts (train.py), you must specify new regularization or control parameters that were not present in the original 3DGS.

Conversion Guides (Kaggle):