Gaussian Splat SPZ Support

[keyboardspecialist]

Description
Opening as a draft as some things are still in progress, and this is a long running feature branch.

This adds support for rendering Gaussian splats from 3D Tiles 1.1 with SPZ compression using a proposed draft extension KhronosGroup/glTF#2490

Another PR will remove old Model pipeline code

Affected files

• GltfLoader - Incorporates SPZ extension support
• GltfSpzLoader - Loads and decodes SPZ buffers within a glTF file
• GltfVertexBufferLoader - SPZ loading support
• PrimitiveLoadPlan - SPZ post-processing. Mostly for interleaving color with alpha and ensuring correct model orientation.
• GaussianSplat3DTilesContent - Manages loading glTF with SPZ Gaussian Splats
• GaussianSplatPrimitive - new primitive for managing a tileset's splats, sorting, and rendering.
• GaussianSplatRenderResources - Container for common render settings and shader building
• PrimitiveGaussianSplatVS.glsl - reworked vertex shader to run outside Model pipeline stages
• PrimitiveGaussianSplatFS.glsl - reworked fragment shader to run outside Model pipeline stages

Known Issues

• View matrix projection is currently incorrect. Splats are rendered in wrong orientation in relation to the camera. Fixed
• Handling tilesets with Empty3DTileContent root tiles.

Issue number and link

Testing plan

Will need to gather some single and multi-tile glTF SPZ assets to incorporate into the general testing suite.

Updated sandcastle, possibly a new asset with more rendering controls.

Author checklist

• I have submitted a Contributor License Agreement
• I have added my name to CONTRIBUTORS.md
• I have updated CHANGES.md with a short summary of my change
• I have added or updated unit tests to ensure consistent code coverage
• I have updated the inline documentation, and included code examples where relevant
• I have performed a self-review of my code

[laurynasr]

Tileset extensions may not contain 3DTILES_content_gltf entry even if extensionsUsed has it.

Suggested change

	this.extensions["3DTILES_content_gltf"].extensionsUsed;
	this.extensions["3DTILES_content_gltf"]?.extensionsUsed;

Reality Analysis produces this JSON:

[weegeekps]

@laurynasr can you further describe the situation where you've seen this from Reality Analysis? According to the 3D Tiles specification, listing an extension in the extensionsUsed list is only valid if that extension is used in the tileset or any descendant external tilesets.

We'll fix this here to make it more robust, but please check that Reality Analysis is not producing invalid 3D Tiles files.

[javagl]

listing an extension in the extensionsUsed list is only valid if that extension is used in the tileset or any descendant external tilesets.

Where is that statement made? (The spec says "When it is used, then it must be listed", but not "When it is listed, then it must be used")

A bit more context for that specific case (which is a bit special): The 3DTILES_content_gltf extension originally had the purpose of allowing glTF in 3D Tiles 1.0. When glTF content became part of the core specification in 3D Tiles 1.1, this extension was marked as 'Deprecated'. But... it recently has been un-deprected, because it's still valuable in 1.1 to carry the information about the extensions that are used inside the glTFs (see CesiumGS/3d-tiles#810 ).

(And when listing an extension in extensionsUsed and that extension is not present, then the validator would (only) emit a 'warning', as updated with CesiumGS/3d-tiles-validator#336 accordingly).

[weegeekps]

Where is that statement made? (The spec says "When it is used, then it must be listed", but not "When it is listed, then it must be used")

Perhaps I'm misreading it, but doesn't the "shall" in the statement below indicate that it's a strictly binding requirement, similar to ISO specifications?

All extensions used in a tileset or any descendant external tilesets shall be listed in the entry tileset JSON in the top-level extensionsUsed array property...

It's been my understanding that when "shall" is used in a technical specification the statement is bound strictly to that whole circumstance with no deviations. Ergo, if an extension is not being used, then it should not be listed.

[javagl]

I'm a bit confused (maybe a language issue on my side). The word SHALL or MUST are indeed describing something that not negotiable.

But to my understanding, the sentence that you quoted is a one-directional implication, as in

"If (extension is used) then (it must be listed)"

which does not imply that

"If (extension is not used) then (it must not be listed)"

[weegeekps]

At risk of discrediting myself, I must admit I might be wrong. However, allow me to walk you through my interpretation.

The ISO Forward summarizes the vocabulary around "shall," "should," "may" and "can." It also summarizes the difference between a "requirement" and "recommendation." These are all explicitly defined in the ISO Directives. While 3D Tiles is not explicitly an ISO spec, my impression is that the language used is meant to mimic glTF's which uses the ISO vocabulary.

ISO states:

The following definitions apply in understanding how to implement an ISO International Standard and other normative ISO deliverables (TS, PAS, IWA).

• "shall" indicates a requirement
• "should" indicates a recommendation
• "may" is used to indicate that something is permitted
• "can" is used to indicate that something is possible, for example, that an organization or individual is able to do something

In the ISO/IEC Directives, Part 2, 2021, 3.3.3, a requirement is defined as an "expression, in the content of a document, that conveys objectively verifiable criteria to be fulfilled and from which no deviation is permitted if conformance with the document is to be claimed."

In the ISO/IEC Directives, Part 2, 2021, 3.3.4, a recommendation is defined as an "expression, in the content of a document, that conveys a suggested possible choice or course of action deemed to be particularly suitable without necessarily mentioning or excluding others."

The way I interpret the definition for a "requirement" is that the since no deviations are permitted for a particular objectively verifiable criterion, then that means that if the criteria is not wholly satisfied then it's at a minimum a warning situation if not an outright error in implementation. My understanding regarding why this is the case is that it limits and reduces the opportunities for undefined behaviors within a specification.

Hence my interpretation of our case above, since we use "shall" in the statement:

All extensions used in a tileset or any descendant external tilesets shall be listed in the entry tileset JSON in the top-level extensionsUsed array property...

Then in the absence of the extension being used in the tileset or it's descendants, it must not be included in the extensionsUsed array property as that would be a deviation from the specification.

Again, at risk of discrediting myself, this is how my interpretation based on my own experience and conversations with others. I may be overreading into this directive, and I think it's open to debate how strictly we want 3D Tiles to adhere to those directives. That said, I do think there's significant value in strictly adhering to them for the sake of clarity and reducing undefined behaviors.

[laurynasr]

But 3DTILES_content_gltf is listed in extensionsUsed; it's just not listed in extensions.

[keyboardspecialist]

@laurynasr can you further describe the situation where you've seen this from Reality Analysis?

I only tried couple of times, but as far as I can tell, this is how Reality Analysis produces their tilesets.

listing an extension in the extensionsUsed list is only valid if that extension is used in the tileset or any descendant external tilesets.

The way I understand the spec:

The optional extensions dictionary property may be added to any 3D Tiles JSON object, which contains the name of the extensions and the extension specific objects.

• extensions MAY contain configuration for a specific extension. The extension might not have any configuration.
• 3D Tiles 1.1 made 3DTILES_content_gltf built-in -- so while it MAY still be specified, per my understanding, it does not need to be specified explicitly.

I believe you are correct here. We can see 3D Tiles 1.0 examples where no extensions object exists, but extensionsUsed and extensionsRequired do.

Appearing in extensionsUsed does not imply it exists in extensions.

[weegeekps]

Okay, I think I still have some further questions around interpretation, but I will open an issue in the 3D Tiles repository.

Regardless we need to fix the highlighted issue in the JS code.

[weegeekps]

The promised issue:

CesiumGS/3d-tiles#814

I don't think there's an urgency to resolve this, but we should probably make a best effort once things settle down later in the month.

In the meantime, @keyboardspecialist has added an additional check to the CesiumJS code.

[keyboardspecialist]

Can the Cesium code in the splat-spz-concept branch render 3DGS data normally?

However, using the spalt-shader branch can render 3dgs data normally.

var tileset = await Cesium.Cesium3DTileset.fromUrl(url, { debugTreatTilesetAsGaussianSplats: true, });

This branch and the splat-shader branch target different experimental draft extensions for glTF. By normally do you mean without that debug flag? We're currently using the 3D Tiles extensions to determine whether or not a glTF has Gaussian splats with the KHR_spz_gaussian_splats_compression extension. The Gaussian splat sandcastle on this branch demonstrates loading without needing any extra flags.

See CesiumGS/3d-tiles#810

[RenaudKeriven]

About quaternion errors, a friend of mines pointed me to nianticlabs/spz#31 and a solution he wrote nianticlabs/spz#36. Yet, he told me that for some rotations, there is no way to go under 6 deg error.
He is asking me if we looked at SOGS
https://blog.playcanvas.com/playcanvas-adopts-sogs-for-20x-3dgs-compression/

[ggetz]

@keyboardspecialist Are we using the correct version of the wasm module here, and in the expected way? I noticed the following deprecation warnings in the console which seem to be coming from @cesium/wasm-splats.

[javagl]

This is not in a "Draft" state, but probably also not finished yet. When this is ready for more thorough testing, then I'd probably give it a try.

Until then: One thing that I stumbled over a while ago: There seem to be some assumptions about the structure of the glTF of which I'm not sure whether they will always hold:

cesium/packages/engine/Source/Scene/GaussianSplat3DTileContent.js

Line 381 in 8a362a0

this.splatPrimitive = loader.components.scene.nodes[0].primitives[0];

(The case of two splat primitives in one glTF shouldn't be too far fetched...).

This may be related to another point: I created some dummy/experimental "Spz-To-Tileset" functionality, and noticed that back then, it was also necessary to introduce an additonal child node in the tileset JSON for which I couldn't explain why it was necessary, but it seemed to not pick up some transforms otherwise. (I'd add details based on a dedicated test pass, when the time is right).

Also related: Someone just asked about How to rotate (and scale) a Gaussian Splat in CesiumJS. And I tried out some sandcastle that modifies the tileset modelMatrix (link in my answer), and it looks like these modifications are not properly taken into account.

[ggetz]

@javagl This is PR is in it's final stages, but the support itself is "experimental". Please let us know if you have issues testing.

We do expect additional updates to the functionality in the near future though, so comments like the one you've added above may not be addressed as part of this PR, but are valid feedback which should be addressed.

[ggetz]

I've taken a final pass on the PR, and beside the deprecation warnings above, initial support appears to be in a good state.

@keyboardspecialist Please take a pass on those deprecation warnings, and comments from other which may come up. But, assuming there are no other major changes which would deserved another full review, this PR will be ready for merge IMO.

[keyboardspecialist]

@keyboardspecialist Are we using the correct version of the wasm module here, and in the expected way? I noticed the following deprecation warnings in the console which seem to be coming from @cesium/wasm-splats.

Fixed with the latest commit. We were passing initSync the raw wasm binary buffer where it expects an object with a module property.

[ggetz]

Looking good @keyboardspecialist! Thanks to you, @weegeekps, and @lilleyse for all the work here!

[GISUser1]

GISBox now fully supports the KHR_spz_gaussian_splats_compression slicing feature. Its groundbreaking slicing compression technology removes performance barriers for large-scale applications of Gaussian splats in Web3D, digital twin, and other scenarios.