In November 2024, Third Dimension AI hosted an event bringing together the top minds in the 3D world to share what Third Dimension has been working on, latest advancements in 3D generation, and real world applications of radiance fields.
Tolga Kart, Co-Founder & CEO of Third Dimension AI, shares the latest on Third Dimension
The event also consisted of a panel moderated by Michael Rubloff of radiancefields.com, with George Kopanas, Google Research Scientist and pioneer in Radiance Fields, Gianluca Corrado, Principal Research Scientist at Wayve, and Sam Howels, Lead Designer at Activision for Call of Duty.
Michael Rubloff (radiancefields.com) moderates a panel with Sam Howels (Activision), Gialuca Corrado (Wayve) and George Kopanas (Google)
The panelists discussed the latest advancements in 3D generation and how these technologies are pushing the boundaries in gaming, autonomous systems and creative industries. Below are some of the top highlights from the panel:
Michael: What are some of the advances you’ve seen in the world of radiance fields?
George: Photogrammetry is the process of taking a bunch of images, putting it into a very, very complicated algorithm and optimization process, and in the end you get a triangle mesh with a texture that kind of represents your scene. The groundbreaking change happened about 4-5 years ago, when this process was drastically simplified and optimized by a paper called neural radiance fields, which we refer to as NeRFs today. The fundamental change was that instead of trying to recover a triangle mesh (which is a surface and something that’s very hard to change, optimize and plug into an algorithm that doesn’t know anything about the world), we changed it to something that’s extremely soft. Everything is kind of a foggy volume that can appear and disappear out of thin air and is much easier to work with. And so with this new technique, coupled with AI training, everything in space is up for grabs.
Michael: Where do you see the future going, especially radiance field representations versus more traditional mesh-based approaches?
Gianluca: I see two phases to this model. On one side, we have 3D reconstruction, which can be very precise at recreating and resimulating and getting you back into something that happened in the real world exactly as it was. On the other hand, there is the generative AI stuff that we’re seeing, which is opposite to this and allows you to generate new worlds, different variants, things that don’t exist, but look relatively realistic.
“The two sides, radiance-based 3D reconstruction and AI-based world creation, have different challenges, but it’ll be interesting and exciting to figure out how these two sides complement eachother.” - Gialuca Corrado, Principal Research Scientist at Wayve
Michael: Sam, what are your thoughts on what large scale generative world modelling would unlock for video games?
Sam: Making high fidelity video games is expensive–it takes hundreds of artists and a lot of time and money to make these experiences that look great to players. Having the ability to get closer to that but with much less resources is one of the biggest things that I’m excited about. You’re going to have artists still, but being able to get there so quickly is powerful because it means we can be more bold and creative with ideas. And the fact that it doesn’t require so much time and money to execute means we can try new things that we might not have been able to do before.
Being able to quickly block out a concept into something that looks arted using 3D reconstruction and generative AI will be a gamechanger to opening up creativity and potential for video games.” - Sam Howels, Lead Designer at Activision for Call of Duty
Michael: Where do you see the future of both 3D reconstruction as well as generation going in the near term?
George: Five year planning is quite a difficult mental exercise! I don’t think we understand enough of these technologies yet to be able to claim with certainty how they will evolve. What surprises me is how rarely we truly interact within 3D worlds today. Usually the final thing that people consume is a video, but 3D implies interactivity, and it’s rare for us to have interactive experiences.
“In the future, I think interactivity will be more and more common for the 3D world. As teams experiment with new applications and experiences, this will dictate what we want to see and how the technology evolves.” - George Kopanas, Research Scientist at Google
Special thanks to the panelists and moderator for the interesting discussion, and to attendees for joining and contributing to the conversation. For future events, stay tuned on Third Dimension's X.
