There are not good workflows for doing this for a whole scene, especially where accuracy needs to be preserved. One shot models seem to be very good at providing 3D from a single angle, but multishot is very shoddy at this point, and without seeing behind things you have no clue about what is actually there.
Also, try meshy and look at how many polygons or triangles you get from any of the model objects. Hundreds of thousands, when you retopologize still goes to the high tens of thousands.
I forgot to mention you could also perform terrestrial laser scanning or SLAM underneath the bridge and fuse those together with the aerial photogrammetry to get a unified model, but this is even more effort and more post process.
I have to wonder at what level you want a truly accurate representation of all bridges so that anyone could see them. Bad actors can ruin a lot of cool stuff.
This is what happens to photogrammetry reconstruction around water and other areas where there are not enough camera angles seeing underneath the bridge to determine the 3-D geometry. This is usually rectified post processed by Google or someone serious about representing these bridges nicely. Often this is a 3-D modelers time, or if you are really enterprising, you can fly underneath the same bridge with a smaller drone and have the camera pointing up so that you can get really accurate photos underneath from which to do SfM/COLMAP. I wish this was solvable by GenAI, but the whole thing of garbage in garbage out really applies here. You don’t know what the structure is of that bridge looks like underneath the roadway unless you’ve taken multiple photos of it, and no two bridges are identical. I’m sure someone could train an AI on every bridge imaginable and we could get something better?
I will believe this when I can actually measure scenes from Gaussians accurately (I have tried multiple papers worth of experiments with dismal results). No one in the reality capture industry uses splats for anything else other than visualization of water and sky heavy scenes because this is where a Gaussian splat actually renders in a nice way. I look forward to the advancements that Nerf and GS but for now there is no foundational reason why they can extrapolate any more data than COLMAP or GLOMAP when the input data is the major factor in defining scene details.
I love the “3D Gaussian Visualisation” section that illustrates the difference between photos of the mono data and the splat data. The splats are like a giant point cloud under the hood, except unlike point clouds which have uniform size, different splats have different sizes.
This all is well and good when you are just using for a pretty visualization, but it appears gaussians have the same weakness as point clouds processed with structure from motion, in that you need lots of camera angles to get quality surface reconstruction accuracy.
Those LiDAR sensors on phones and VR headsets are low resolution and mainly used to improve the photos and depth information from the camera. Different objective than mapping a space, which is mainly being disrupted by improvements from the self driving car and ADAS industries
You are correct; most of these new techniques are using a camera. In my line of work I consider a camera sensor a scanner of sorts, as we do a lot of photogrammetry and “scan” with a 45MP full frame. The inferred 3D from cameras is pretty bad when it comes to accuracy, especially from dimly lit areas or where you dip into a closet or closed space that doesn’t have a good structural tie back to the main space you are trying to recreate in 3D. Laser scanners are far preferable to tie your photo pose estimation to, and most serious reality capture for video games is done with both a camera a and $40,000+ LiDAR Scanner. Have you ever tried to scan every corner of a house with only a traditional DSLR or point and shoot camera? I have and the results are pretty bad from a 3D standpoint without a ton of post process.
The collision detection problem is related heavily to having clean 3D as mentioned above. My company is doing development on computing collision on reality capture right now in a clean way and I would be interested in any thoughts you have. We are chunking collision on the dataset at a fixed distance from the player character (can’t go too fast in a vehicle or it will outpace the collision and fall thru the floor) and have a tunable LOD that influences collision resolution.
I work in the rendering and gaming industry and also run a 3D scanning company. I have similarly wished for this capability, especially the destructability part. What you speak of is still pretty far off for several reasons:
-No Collision/poor collision on NERFs and GS: to have a proper interactive world, you usually need accurate character collision so that your character or vehicle can move along the floor/ground (as opposed to falling thru it) run into walls, go through door frames, etc. NERFs suffer from the same issues as photogrammetry in that they need “structure from motion” (COLMAP or similar) to give them a mesh or 3-D output that can be meshed for collision to register off of. The mesh from reality capture is noisy, and is not simple geometry. Think millions of triangles from a laser scanner or camera for “flat” ground that a video game would use 100 triangles for.
-Scanning: there’s no scanner available that provides both good 3-D information and good photo realistic textures at a price people will want to pay. Scanning every square inch of playable space in even a modest sized house is a pain, and people will look behind the television, underneath the furniture and everywhere else that most of these scanning videos and demos never go. There are a lot of ugly angles that these videos omit where a player would go.
-Post Processing: of you scan your house or any other real space, you will have poor lighting unless you took the time to do your own custom lighting and color setup. That will all need to be corrected in post process so that you can dynamically light your environment. Lighting is one of the most next generation things that people associate with games and you will be fighting prebaked shadows throughout the entire house or area that you have scanned. You don’t get away from this with NERFs or gaussian splats, because those scenes also have prebaked lighting in them that is static.
Object Destruction and Physics: I Love the game teardown, and if you want to see what it’s like to actually bust up and destroy structures that have been physically scanned, there is a plug-in to import reality capture models directly into the game with a little bit of modding. That said, teardown is voxel based, and is one of the most advanced engines that has been built to do such a thing. I have seen nothing else capable of doing cool looking destruction of any object, scanned or 3D modeled, without a large studio effort and a ton of optimization.
Wider implications beyond just the consumer drone market. My company and several of our clients (who are all much larger companies than us) have $100,000s - $X,000,000s of DJI enterprise drones, batteries, and payloads (Matrice 300, 350, zenmuse p1 camera, etc)
I know the bill still has to go through the senate, but this is going to be a sore subject for a lot of American companies who use DJI equipment.
I wonder what camera sensor was used to produce this. A DSLR on a pano/360 rig? Or something purpose built like a Weiss AG Civetta 230MP 360 scanning camera.
I notice a DJI Matrice 300 (larger drone) and a DJI Mavic 2 (can’t be totally sure if it’s a pro model). DJI is well known as the top market leader in UAV space because of its relative low cost ($13K for the Matrice, under $3K for the Mavic) for high utility/reliability (something you care about if you have a payload worth $10K plus). Similar drones made in the USA or EU are 2X-4X the cost and do not come with any other features that DJI hasn’t already thought of.
I guess that being a market leader means you will have a wide range of customers using your product
The data in their nicely colored graph is wildly inaccurate. I have a long range Tesla model 3 (in Alaska) and the range is over 290 miles (it says 310 miles when full, but doesn’t quite get that, even in the summer). This is information you can trivially look up…
I know Tesla gets a lot of flak, but it’s batteries outperform most of the other cars that are listed.
This would be fabulous for switching between brave, duck duck go, and Google. I love the UX, finally something worthwhile in the browser space. Available for purchase at any point?
I recently played Bone Labs on the Occulus Quest 2, and it totally redefines where I thought we were with VR (and with no PC to have to play it). Seriously, just check it out if you haven’t and you have a headset.
I am a huge VR skeptic for many of the reasons that people state here. I have also thought that VR’s killer app was any dangerous professional training situation that you wanted to present as accurately as possible without injuries to your staff (firefighting, EMS response, etc.). If the road to adoption begins with experiences that are decidedly more interactive and immersive than screens and monitors, I am starting to see the (long) path that leads towards that future.
I read a lot of these posts on stable diffusion, midjourney, etc., and I don’t want to fall into the classic “I don’t think this is ready yet” camp, but one thing an Architect will be able to do really well which these tools won’t is make a kitchen that is actually usable. I mean, this thing is basically a corrupted Pinterest of kitchen ideas with doors and drawers intersecting, beams that make no sense, and flooring that looks menacing. And until we have AGI I don’t think our program sophistication will be able to iron out these “uncanny valley” isms that seem built into every aI image generation tool.
Now I know that this tool is just a “starting point” for using as ideas, but I have seen way more fabulous (and sane) kitchen designs en mass on houzz, dezeen, basically any online blog. The technology is cool, but it seems limited. As in we can only get weird amalgams of human concepts. All this to say, I work with architects professionally, and I do not see this as something that will cause their field concern in the near to mid term. These programs cannot make drawings with structural suggestions, floor plans for the layout they produce, or anything else that a person pays an architect for. Just my two cents
It is not that viable. Even large drones cannot carry more than a few kilos for 40 minutes (on one set of batteries). It just doesn’t have good ROI, and is mainly a PR move by Amazon and some VC funded startups. I accept there are some practical applications, but a human on a bike/ on foot is a lot more efficient than buying and maintaining a fleet of mini aircraft.
Also, try meshy and look at how many polygons or triangles you get from any of the model objects. Hundreds of thousands, when you retopologize still goes to the high tens of thousands.