This week I created an interactive 3D scene of real objects and a real terrace using photogrammetry.
- to capture objects and rooms in 3D
Spline - to render and animate objects in a scene
Luma.ai - to play around with NeRFs
iPhone 13 mini camera - to take the photos
Context
P’s great uncle was a doctor, avid traveler, and meticulous collector of trinkets and curiosities. He left behind many small, fragile objects. Part of our Mexico quest is to find new homes for these items. As we were sorting through dusty boxes, I thought I might try my hand at capturing these trinkets using photogrammetry - a process that converts images into 3D models.
Process
Apple recently released their Object Capture API, which is apparently an upgrade in quality and ease of use compared to previously existing photogrammetry tools. It means you can easily do photogrammetry with an iPhone instead of needing a DSLR and pro lighting. Polycam is one company who have successfully adapted this API for iOS, and they have a 3-day free trial, so I gave it a spin.
At first I thought I would need a lightbox and a rotating platform to properly capture the objects. After a small rabbit hole trying to compare my options and figure out if half this week’s project would be trying to DIY the lightbox and platform, I watched a few videos of people attempting the same - like this one - it looked like a lot of work for a pretty average result. Luckily I next found this video which recommended the simplest setup to be shooting outside on an overcast day. That saved me a lot of headache!
Armed with that knowledge, I found my new studio:
I selected a couple of trinkets:
Instead, I wanted to try a completely different approach using NeRFs, a hot new method of reconstructing 3D models. I don’t pretend to understand too much of how they work other than that instead of stitching together a bunch of photos and textures (photogrammetry), NeRF builds a neural network model from the source data and then renders it, which allows you to do really cool things like view scenes and objects from angles that you hadn’t taken photos from, manipulate lighting, and much more. Here are some cool examples from Nvidia.
So I tried it myself on the terrace, and it worked great!
.
So now to tie it all together. To place all these objects and the scene together, I used Spline - a super intuitive web based 3D editor. I exported the objects from Polycam as .obj files and imported them and their textures into a scene in Spline. Then I hit my first snag - while I could export the NeRF render of the terrace as an OBJ (or point cloud), it had tagged the covered awning as the ‘object’ in my scene, and the rest of the terrace as the background, which meant that when I exported it I only got the awning as a model. This was another case where with further time and tweaking (or maybe different NeRF software or a Google CoLab notebook) I could get it looking better, but I was already past due this week and wanted to get this project out.
So instead, I used plan B, the Polycam captures of the terrace I showed above, holes and all. Since I was going this route, I thought I’d include a 3D model of myself in the mix. It really started to feel like real life The Sims when I had a digital character of myself walking through a real place.
.
Here’s the link to play. As I included all the textures, Spline gave me lots of warnings and it’s very slow to load.. (4.5 minutes???), so please be patient. Maybe open it as a new tab, make a coffee, then come back to enjoy. I’ll try to figure out a way to optimize it in a later week.
- I love fine textures in 3D as much as 2D
Next steps
Find a way to speed up or automate the capture/upload/render process. Each object took a minute or two to capture, a few minutes to upload and process, and another few minutes to crop and export. Say, 5-10 minutes per object. That’s not scaleable for dozens or hundreds of objects!
Play around more with NeRFs
Animate the objects - maybe with a tool like Cascadeur
Add some challenges, levels, points, or other game like elements
Make 3D renders of everything!
Use a drone to capture larger areas, (abandoned) buildings, nature, etc