3D Modeling at City Scale! CityNeRF Explained

last year we first saw nerf then nerve and other networks able to create 3d models and small scenes from images using artificial intelligence now we are taking a small step and generating a bit more complex models whole cities yes you've heard that right this week's paper is about generating city scale 3d scenes with high quality details at any scale it works from satellite view to ground level with a single model how amazing is that we went from one object that looked ok to a whole city in a year what's next i can't even imagine but i can easily imagine what should be next for you your next step as an ai professional or student should be to do like me and try the sponsor of today's episode weights and biases if you run a lot of experiments such as playing with guns or any models like this one you should be using weights and biases it made my life so much easier you have no idea and it takes not even five minutes to set up simply install and import it into your code add a line to initialize and another to say which metric to track and voira you will have all of your future experiments in a project where you can see all of the input hyper parameters output matrix and any insights that you and your team have and easily compare all of them to find out what worked best you can help out the channel and give it a try with the first link below it's completely free for personal use and i promise it will be set up in under 5 minutes the model is called city nerf and grows from nerf which i previously covered on my channel nerf is one of the first models using radeon's fields and machine learning to construct 3d models out of images but nerf is not that efficient and works for a single scale here city nerf is applied to satellite and ground level images at the same time to produce various 3d model scales for any viewpoint in simple words they bring nerf to city scale but how i won't be covering how nerf works since i've already done this in a video you can see in the top right corner of your screen right now if you haven't heard of this model yet instead i'll mainly cover the differences and what city nerf brings to the initial nerf approach to make it multiscale here instead of having different pictures a few centimeters apart they have pictures from thousands of kilometers apart ranging from satellites to pictures taken on the road as you can see north alone fails to use such drastically different pictures to reconstruct the scenes in short using the weights of a multi-layer perception a basic neural network nerf will process all images knowing their viewpoint and positions in advance nerf will find each pixel's colors and density using array from the camera so it knows the camera's orientations and can understand depth and corresponding colors using all the arrays together then this process is optimized for the convergence of the neural network using a loss function that will get us closer to the ground truth while training which is the real 3d model that we are aiming to achieve as you can see here the problem is that the quality of the rendered scene is averaged at the most represented distances and makes specific viewpoints look blurry especially because we typically have access to much more satellite imagery than close views we can try to fix this by training the algorithm with different skills independently but as they explain it causes significant discrepancies between successive scales so you will not be able to zoom in and have a fluid nice looking 3d scene at all times instead they train their model in a progressive manner meaning that they are training their model in multiple steps independently where each new step starts from the learned parameters of the previous step these steps are for specific resolutions based on the camera distance from the object of interest here demonstrated with l so each step will have its pre-processed pack of images to be trained on and further improved by the following steps starting from far satellite images to more and more zoomed in images the model can add details and make a better foundation over time as shown here they start by training the model on l1 their farthest view and end up with the ground level images always adding to the network and fine-tuning the model from the learn parameters step to different scales so this simple variable l controls the level of detail and the rest of the model stays the same for each stage compared to having a pyramid-like architecture for each scale as we typically see the rest of the model is basically an improved and adapt version of nerf for this task you can learn more about all the details of the implementations in differences with nerf in their great paper linked in the description below and the code will be available soon for you to try it if interested and voila this is how they enable nerf to be applied to city scale scenes with amazing results it has incredible industrial potential and i hope to see more work in this field soon thank you for watching and if you are not subscribed please consider clicking on the little red button it's free and you will learn a lot i promise and i will be sharing a couple of special videos for the end of the year stay tuned [Music] you

Channel: What's AI

Views: 1,115

Keywords: ai, artificial intelligence, machine learning, deep learning, ml, data, datascience, two minute papers, 2 minute papers, two, minute, minutes, paper, papers, deep fakes, data science, data scientist, 3d printer, 3d video, 3d rendering, 3d rendering software, 3d rendering photoshop, rendering, rendering at 5am, 3d, 3 d, dimensional, 2d renderer, renderring, render, 3d modeling, 3d model, citynerf, nerf, nerv, sharf, google earth, google earth secrets, ai news, best ai, smartest ai

Id: swfx0bJMIlY

Length: 5min 17sec (317 seconds)

Published: Sat Dec 18 2021

Reddit Comments

[u/OnlyProggingForFun]

References:

►Read the full article: https://www.louisbouchard.ai/citynerf/

►Xiangli, Y., Xu, L., Pan, X., Zhao, N., Rao, A., Theobalt, C., Dai, B. and Lin, D., 2021. CityNeRF: Building NeRF at City Scale. https://arxiv.org/pdf/2112.05504.pdf

►Project link: https://city-super.github.io/citynerf/

►Code (coming soon): https://city-super.github.io/citynerf/

►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/newsletter/