3D Gaussian Splatting from Hollywood Films!

have you ever watched a movie and thought to yourself I wish I could turn these scenes into 3D no that's right you can now do it thanks to 3D gin Splat you can literally take a movie shot like this from 15 years ago and turn it into 3D can even bring it to a 3D software like Unreal Engine 5 and do whatever you want with it take any footage where the camera is moving so you have some depth running through the 3D Gan splatting algorithm and just like that you have the film in 3D it's as simple as that in this video we are going to take scenes from The Shining and glorious bastards Harry Potter The Dark Knight isn't that too many no get out of here Prince of Persia and the matrix by the end of this video we will also take some of them to Unreal Engine 5 and create an entire new scene so let's go we began by searching for movie scenes in which the camera moves at a moderate speed in an orbit orbit because we get some idea about the depth of the scene and moderate speed because well we don't want any motion blur more specifically we were looking for shots where there were no moving elements aside from the camera itself we then cut those scenes out and converted them into image sequences now it's time to train and process now while farad is training the ga Splats I'm going to tell you exactly what they are and how they work and by the way we've already done this five times so let's speedrun it you start with a set of images taken from a subject running through an algorithm that creates Point clouds based on that subject then those Point clouds are converted into gaans which have superpower capabilities like holding various sizes color values and transparency values then those gal scans are optimized over and over again until they're absolutely photoal and efficient to view mic drop guys guys the 3D scans are ready let's go and check them out the first one The Shining but before we get into the 3D scene let's take a look at the input images and the input video so this is the video I clipped right after the credit you see I think they needed helicopter that time 20 years ago yeah this is actually the perfect shot because the camera's going in sort of a quarter orbit around the hotel and we have the perfect view of the mountain of the hotel you see the trees the cars everything is super sharp even though the video was taken at a long time ago I don't know where this movie was uh made or when it was shot when was it shot let's find out 19 1980 I wasn't born wow wow 33 years ago so this shot was taken 33 years ago here we are in 2023 turning it into images and turning it into a 3D scene so that we can go ahead and alter the footage ready 3 2 1 oh oh you can never be sure of what it looks like from the first image because it might it might [ __ ] it up the moment 33 years after let's look at it no way dude dude we got the whole freaking shot dude look at this I go I come down I go in go right turn here dude can you go closer to the hotel go closer to the hotel dude this is crazy let me go wow dude look at the mountains look at the trees everything looks perfect dude 33 years later and you're here dude this is such a creepy Hotel though you can't go up because we never captured that we never area yeah no no but you cannot go up you cannot use this because behind the hotel was a part that we've never seen but what's amazing is okay so the shot as you might remember was taken like this right this was the shot but now having access to this in 3D we can go in we can go out like we discussed this earlier can even rotate the camera yes and what's crazy is if you're shooting a film or Netflix series or whatever in Hollywood for yourself on social media you can go ahead and take that 3D scene after shooting your film and go ahead and change the idea for the camera angles that you might have had look at those plats we're going to take it to Unreal baby imagine you do this now with intention so now the Netflix series Disney series Hollywood movies when they go and they want to okay we want to do a credit scene here or we want to do a scene here before shooting anything let me do a 360 so I have every single angle available and mind you we're not even using the original material right we downloaded this video from YouTube which already has how many times has been exported one time for the film The Many Many Times many for that me from the YouTuber so it's it's already destroyed but great result let's move on to the next shot shot number two in glorious bastard are we going to look at the images or the video no the video so this scene is so crazy these two guys barely move when the camera orbits I think half an orbit happens here so you see the movement is very subtle but he he's [ __ ] his pants right right now because his family are hidden somewhere like under under the floor or something or in the wardrobe and what was really important for us with this scene was the movement of the camera it was slow there was no motion blur there was a subtle movement between the two characters but we decided to try it anyways and see what it looks like so let's go okay a promising opening but it's still wait wa you have to move it I want to know until you freaking move it and don't go too fast move I'm moving wo wo wo wo follow the same no no follow the same camera movement for me okay so we failed the face we failed the face here let me let me let me turn one round okay stay here stay here stay here holy [ __ ] look at the hat look at the Hat here look at the Hat here oh this part is almost look at the hat look at the Hat yo let me go behind him yeah here bro look at the Hat I'm just looking at the hat look at him look at him here wow B yeah go up go up go up go up yo you know what this reminds me of the credits or the introduction of a film where you have characters and you just have them stationary in 3D and then you go around them and then you have the names come in between them we have to try that out dude they move their face quite a bit which is why from any angle you look their faces are different look at the items on the table dude this is I'm still very impressed with the technology come on this has just come out last month can I look at this part just look at this this shot was never taken I can go in whoa dude dude look at his face his beard him wow wow if I was recording a movie I would do my credits like this we're not even we're disregarding our own industry if you're doing YouTube content Tik Tok content any sort of social media where you're not putting this on a giant screen for Millions of people to watch but rather on a mobile device I would go to our favorite movies if you're talking about movies at some point you can do this and have the movie in 3D and then play around with the scene it looks so stunning wow imagine taking this to VR and then sit down beside them listen to the conversation we got to do it but for now let's move on to the next one let's go scene number three Harry Potter this is the image sequence that we gave to our algorithm to come up with a 3D scene look at so it's more of a spiraling motion right I got to say your input images here suck ass they they really are noisy they're grainy I can see even in the shadows you can see a lot of noise yeah even on the bottom right here with the bridge in between the castle like that's almost invisible this is a horrible input image I'm wondering how this is going to turn out 3 2 1 what what yo what the hell dude come here come to the hog wall s go to the right side go to the right side I want to see if you got the castle I want to go I want to go to the right go to the right dude we got the freaking Castle I mean listen there's a lot of artifacts but I'm just wondering how the hell it still manag to get the castle cuz the video was oh my God look at the bottom right look at the torch the Flames look at the castle you know what it's done it's sort of done its own denoising pass on the entire image this is a huge space this is I think if you want to walk from one side of Hogwarts the other side it will take you 30 minutes wow so this is a huge SC dude and again we are not using 4K images using YouTube videos YouTube videos compressed and exported multiple times let me see how does the castle look like in Splats oh my god oh no keep it halfway yeah keep it half actually the Splats look so good this is like stylized Harry Potter animation look right shot Number Four The Dark Knight I think this is the part that Ro SP goes and kidnaps someone from Hong Kong yeah and he is standing at the top of the building right now yes so the the Drone or the helicopter goes 3/4 of the orbit and he's standing here so I think like and there's minimal movement from him he does turn his head but the problem with this shot is that the camera Zooms in a lot so you can see at this point onwards you barely see anything just like one small portion of the building but let's see how the results turned out ready yes oh this looks good this looks good what do we got oh I see some artifacts in the air I see some artifacts in the air still there let's see let's see let's see if he go can go one round let's see let's see oh we messed up a little bit this is messed up it's like a crisscross you know those crisscross fries yeah wow here is totally messed up oh Bruce Wayne is here nah this was a bad scan I would say I have an idea why don't we try to do the image sequences in a different way we have another one right we have oh wait wait hold on this this part is not that bad so we got this part looks great yeah certain angles are fine CU here I can see I think it's messed up because of the the the look of the building but this part looks phenomenal yeah this part is great no it's more than just great look at the Quality on the left side here the building look at the material on the building look at the fans the air conditioning units look at the building at the back all the way look at the DI yeah this is perfect this I think this is HSBC building if I'm I think so too yeah wow and look at him turning his head look at the look at the illusion of him turning his head okay now he's looking to the right and now he's going to look to the left and then he's going to look to the right again because that's what he did in the shot you can read the no smoking sign shot number five Prince of Persia let's look at the import AKA far and Far from bad decisions Prince's princess princess princess of Persia what what is the plural of Prince plural form of Prince is princess princess with One S yeah so princess is double s princess no that's a princess this is Princess back so this is the input video right so I thought here we have a very good orbit although a lot of motion blur camera moves very fast and there's two obstacles along the way one pillar on the right side and another pillar on the left side and just look at the motion blue in this shot there's absolutely expectation very low expectation from this shot however however before we start I just want to see so in in the very beginning it sort of looks like an eased out an eased in motion because in the beginning it's sharp and in the middle it's fine it's pretty sharp right you can see this Dome it's just towards the end it gets faster and so there's more motion blur yeah but low low expectations for this as well first the first glance no but usually at the first glance they all look good you can never tell let me just move in a little bit go to the left what oh and you have the obstacle too of course you can see it construct this building that looks like a temple perfectly and he's right you can see the perfect depth between The Prince and the buildings this is freaking crazy how good this looks far yes do you notice the problem here though the motion blur no this the artifacts on top of his head you see that dude this is Assassin's Creed Vibe look actually Prince of Persia and Assassin's Creed are very similar in terms of the vibe one last one which is the Matrix I hope this one turns out well cuz that's my favorite movie so we have two shots from The Matrix one of which is the infamous this is in the opening of the first one right clearly they were using some tricks in order to get this shot to look good at that time right so you can see there is a ghosting effect on top of Trinity look at her hand and her wrist on the left side every time we pause there's literally two of them the motion blur is extremely sharp you know it's not it's it's not the faded out motion blur there's literally two of her just one with lower opacity and that's going to cause issues so we don't know how this is going to turn out so this is one and and on top of that F she's moving as well the only reason Matrix made the cut because we love the movie and we wanted to give it a try else it would fail let's go I'm going to show the other shot and this is the but this is full of moving elements bullets flying I don't even know why we're trying this cuz it's awesome we're just hoping it would turn out well Trinity [Laughter] help come on I it's not even funny I'm just holding my okay as I expected oh no I see something hello someone is coming inside hello sorry we're recording videos YouTube's more [Music] important did you say going to get that I'll I'll entertain them while you're gone H look the shot of Matrix not so good it's full it's it's not usable let's let's just say it's not usable but you still can see your face yeah does that count no it's not usable at all let's go Chey even the perspective of it doesn't work let's check Neo that's unfortunate the other one dude if this didn't work out there's no way Neo worked out yeah probably but let's see there's everything was moving in that shot yeah here we are with the oh wait wait there's some hope there is some hope Trinity hope Trinity hope yeah oh no well you can you see the bullets flying I can see something no there there was one bullet right bullets no few they oh yeah one on the left one on the right right so he's there he's there he's definitely but no this I mean you can't use this there's too many artifacts but you can see his body yeah guys I don't know does this count let us know in the comments does it count doesn't count you cannot use this the previous ones you could use it no definitely this doesn't count I would say that the scene from The Shining was our best yes in glorious bastard was quite decent Harry Potter was quite decent the one that we didn't have any moving element those were the best shots but like things like Matrix that had motion blur and elements moving definitely them work count yes but we not going to stop here right no no no no no no of course not we're going to take this Swan real Engine 5 for this scene in the glorious bastards we want to use the same camera motion because there was a lot of artifacts from the other angles and then we rendered this shot pretty nice but we still got some work to do now what's really cool about these gussian Splats is that we can kind of relight them by just removing the directional light we can go ahead and add a single point light we're going to make it a little warm and put it on top of the table now that looks kind of cool super cinematic now just to show you how flexible these scenes can become let's go and add a chair to the far end of this table nice uh actually let's brand this bad boy far do we have the bad decisions logo in 3D no we don't no all right let's go and make one po F who's an amazing YouTuber made this tutorial on how you can turn any logo to 3D using inkscape and blender let's use that and turn our logo to 3D and bring it to Unreal Engine 5 then place it on the chair and there you go yes sir Boom for the Harry Potter scene we are dealing with the most amount of artifacts but remember limitations and constraints spur creativ [Music] let's pretend that Hogwarts here is a level in a game and we are selecting it I just need to have the camera quickly scroll and bring the castle in the center where I slow down for a moment and then I quickly scroll out from the other side then I just Download the Harry Potter font and use that to create a 3D text that says Hogwarts just like that we have a cool render actually that looks a little too realistic let's sty Iz it a little by bringing down the Sprite size of the Niagara particles that's more like it for The Shining scene let's totally change the camera movement we're going to start from a closeup of the hotel and slide all the way back we're also going to find the original shining poster font and use that to type out the name in 3D add some fog and we got a [Music] render yeah that's what I'm talking about we just remade The Shining all right now that we've taken movies turned them into 3D and brought them to onreal Engine 5 farad what else can we do with this how can we make our life harder I'm thinking video games let's do that cuz you have more control in video games you can rotate the camera yourself so imagine taking any video game and turning it into 3D that you can use yourself probably illegal but you can do it why not take that into VR too Oh you mean like take that environment put the VR headset on and game yes that's like doing yo we got to do it right now okay before we go have you have you have you freaking not subscribed oh my god do it make the subscriptions come true please

Channel: Bad Decisions Studio

Views: 85,920

Keywords: radiance field, gaussian splatting, neural radiance fields, Nerf, AI, 3d Scan, 3d

Id: 8CdLVVny9hc

Length: 18min 25sec (1105 seconds)

Published: Fri Oct 13 2023