3D Gaussian Splatting

[n.merazzi]

I'm sure you guys have seen this, but please take a few minutes and type it on Google for anyone who hasn't seen it.

- for Archviz Artists.

for all 3D Artists.

Ondra and Corona team, do you guys think this kind of training can be applied to a 3D rendering engine?

with a single video, you can transform everything in 3D - with incredibly high quality and high-speed frame.

Well... the discussion is open. =)

[zaar]

I think it's shocking that this has no replies. I have no interest in turning my 3d scenes into 3DGS, but I have a huge interest in going out and capturing a location and turning it into a 3DGS. This is something fantastic for arch viz, which is Coronas main audience, right?
It's been oddly quiet on the autodesk, v-ray and corona front about this tech. The UE, Blender and Unity scene is all over it. Alteast from my biased internet bubble.
This is what's going to push me into doing an actual project in UE soon.

If any developer sees this, I would be really interested to hear if the ability to read and render 3DGS would be best implemented on the autodesk/Max side of things, or in the actual renderer. So I know who to nag ;)

[pokoy]

Yeah, was wondering this too, especially since setting this up from actual 3d data might be more accurate, I guess it could help to some degree with slight changes in animations without the need to re-render everything.

FWIW, here's a thread over at Chaos forums asking basically the same question, but don't expect too much:
https://forums.chaos.com/forum/v-ray-for-3ds-max-forums/v-ray-for-3ds-max-wishlist/1193659-3d-gaussian-splatting-in-v-ray

I wouldn't wait for Autodesk, their 'evangelists' are probably trying to come up with a way on how to get rid of expensive 3d dev and somehow utilize AI and be done. Their approach seems to be adding 1-2 modifiers per release cycle, adding stuff no one asked for and fixing bugs introduced while doing so... or not.

[Juraj]

Absolutely amazing tech. I want to use this for some construction help in family garden as quick sort of visualizing scanned pointcloud of environment :- ) As I've seen Unreal will be able to read/use this

[zaar]

Absolutely amazing tech. I want to use this for some construction help in family garden as quick sort of visualizing scanned pointcloud of environment :- ) As I've seen Unreal will be able to read/use this

Here's some inspiration in case you need to put smoke monsters in your family garden:

[zaar]

In a browser:

https://www.reddit.com/r/Spline3D/comments/17v8ii1/update_we_are_adding_gaussian_splatting_to_spline/

Live demo: https://my.spline.design/girlstudio-8b6211e0b6ab456c8764297c6ff3ed45/

If a developer, or any one else who's knowledgeable enough, sees this: is combining splats with a pure ray/pathtracer a difficult scenario?

[n.merazzi]

When I saw that, the first thing that came in mind was to record the context buildings, so We don't need to spend time on it and we will have the most realistic results, but at some point I start to see we have a lot of gaussian splats over the scene that makes it unuseful. Also, It is working on UE, and perhaps Vantage can import the file as Polycam allow you to export the GS as .ply.

That's one of the reasons that I start to study even more UE. If they could make the materials system more friendly would be amazing. hehe.

Juraj, send your results please =)

[Juraj]

Hah that will be some time until I get to it :- )

[TomG]

Some initial tests here at Chaos were somewhat disappointing, though those are but early tests so we can't speak to what may or may not happen long term. For interest, can you share how you see this being used when it comes to integration with path tracing? Lighting, reflections, etc. would already be baked in if adding them into the scene - so maybe as an environment? Or are you picturing creating Gaussian splats from the 3D scene as a new kind of output vs. a still or animation?

[Juraj]

For me for example, I don't expect any modification much on the Splatted object, but I would like the regular geometry to take it into account (reflect the Splatted object, occluded light,...)
So yes, basically environment.

[zaar]

TomG:
In the near term, my interest is on the input side of things. To get locations, and possibly objects, that are difficult to scan or model with traditional methods. It would replace photogrammetry. And I do a lot of photo matched architectural work, so I'm thinking this would allow me to not only nudge the camera after the fact, but also crop and cull unwanted things in the volume. A 3D photograph, that's what I want! :D

Output seems like not a problem already? Just time consuming to render a bazzilions views and process it, just as you would with photographs from the real world. Maybe its' possible to think of some more effective method. But I think input is more interesting.

Juraj: You sell high quality hdris of locations right? Imagine 3DGS with hdri support *drool*

[TomG]

On "replace photogrammetry", thing is photogrammetry aims to remove the effect of lighting so that the scene can be relit in 3D and used in any situation, but splatting is more about capturing the lighting, reflections and baking those in, meaning you'd have to make the scene around it (and all such "splatted objects" would also need the same lighting and reflections, which would seem very unlikely). Is there a way you visualized splatting working differently than that, so that it would replace photogrammetry? Locations maybe, as then it would surround anything you add into it (ie "environments" as mentioned earlier).

[Juraj]

I am sure some smart people are already thinking how to de-light these to create "PBR" Point-Clouds/Nerfs/Splats :- ).

But even photogrammetry is useful in many way with baked light into it, just not as flexible.
And lighting already lit object while fully incorrect from accuracy point, can still be useful and is often used trick in VFX.

So I wouldn't discount something because it doesn't adhere to common workflow.

[zaar]

I am sure some smart people are already thinking how to de-light these to create "PBR" Point-Clouds/Nerfs/Splats :- ).

Exactly, "how many papers down the line?" ;D


TomG:
I understand the difference between 3DGS and traditional photogrammetry that generates a mesh and textures (that one probably de-lights in some way).

With 3DGS I would have some of the same problems that I do with my regular workflow of matching a rendered building into a photograph of a location. I would have to match the lighting on the model to fit the environment in both cases. But there would be more interaction. The model could reflect and possibly be shadowed by the 3DGS surrounding. Preferably also cast shadows on it, even if that might cause non-physical effects of something being shadowed twice by the sun.


Why 3DGS? It seems easier and more lightweight than photogrammetry. Also 3DGS capures view dependant effects in a way that traditional photogrammetry does not. And not to mention handles thin objects such as vegatation and eyelashes (look at the spline demo!). It all seems very nice.

[TomG]

Great info ty! We were wanting to see how people would put this to use, I'll pass it along for any future testing we do with this technology.