Coding Challenge #49: Photo Mosaic with White House Social Media Images

hello welcome to another coding challenge this is another coding challenge that I'm gonna do about President Barack Obama who is still I'm thinking that the time of this recording in this video the active current President of the United States ok so uh what I'm going to do in this one so a part of the data set that was released was every single photo so so first of all I thought I mentioned a previous coding challenge the White House recently released all of their social media data from the last eight years and invited artists and researchers and people to remix it and make projects using that data so one of things I'm going to do here is just look at working with the image data the Facebook download which is four and a half gigabytes has a tons of videos but it also has a giant folder of images so I have that here in this particular I'm sketch folder in the data folder I have a single image which is just the White House official portrait of Obama and then I have every single Facebook photo so I can scroll through these and see all of them so what I'm going to do right now is I'm going to attempt to make a photo beause aook of this Obama portrait and with all of these images from Facebook so there's probably I'm gonna do it in the like post drop dead simple way possible when I post the code by the way I'm not going to post the photo dataset but I will include like a rebe file or something - a link to that photo day said cuz I don't to post some 300 Meg file to github that you have to download ok so let's go into processing and let's just do a couple things first so I'm using for this coding challenge I can minimize the browser and all this other stuff I'm using a programming environment called processing go to processing dot-org it's Java based and it has a lot of libraries for drawing and images and all sorts of stuff like that so I'm going to make a variable called Obama and I'm going to say load image was it Obama jpg and then I'm just going to say image Obama zero-zero so let's just see there we go I need to actually make us a size of the canvas which is like so let's see what is the size of that image it is if I look at it 600 by 749 so let's make the canvas 600 749 and do this we can see there we go so this is just first of all I have a processing sketch that loads this image file and displays it so the first thing I need to do to make this photo mosaic is figure out a way that I'm going to pixelate this image and there's lots of ways I could do that I'm going to do it in a pretty simple way I'm going to sample it down so I'm going to make the image smaller by a factor of something so I'm going to say let's have a variable called scale factor and such a long variable Dave makes me so sad I'll come up with something better later let's make that scale factor 8 for lack of a better idea right now and then what I want to do is I need a new width which is obama dot width divided by scale factor and I need a new height which is obama dot width divided by a height divided by scale factor and then what I'm going to do is I'm going to use the copy function I'm going to make a I'm going to make a variable called smaller and I'm going to say smaller equals create image width sorry I'm like really thinking hard here create an image with that width and height that's an RGB image and then I want to copy from Obama the entire image to this images width and height so I'm using the processing copy function so I wanna I'm using processing to scale the image down I could skip pixels or I could manually scale it like but this is going to be simple and now I'm just going to say image Moeller zero zero so let's see if that worked and you can see that worked now I have that smaller image here I just copied the image to make it smaller because the reason why I'm doing that is now what I want to do is actually want to read the pixels so I'm going to say smaller dot load pixels I guess making this photo mosaic is something I only need to do once so I'm going to add no loop at the end of draw so no need to loop over and over again and what I'm going to do now is I'm going to say for every X and you know I kind of want these that W and H just to be a global variable because that way I can use it over and over again without having to say dot X less than with X plus plus y equals zero Y is less than H Y plus plus now and you know what this scale factor let's just sit let's just call this like factor I know scale SEL for scale and then now what I'm going to do is I'm going to say what is the pixel the color is smaller dot pixels and I need that pixel Spidy the spot of so the pixels is a one dimensional array but I'm thinking of the X and y of all the pixels so what I need to do is look for that index which is X plus y times W I'll link to some other videos about image processing that I have that go over this particularly algorithm more specifically but now I'm looking this is an algorithm for looking at every single pixel in the image all the act every XY location look at the particular color and I'm going to say fill C and I'm gonna just draw a rectangle at X Y and where I draw those rectangles X timescale Y timescale and then that skill so that scale is really the size of each one of these little squares so now you can see I have this pixelated version of Obama where I've drawn a rectangle and there's all these black lines I guess because I probably have us a stroke so if I had no stroke we can see now I just have this pixelated version of Obama so now what I need to do is instead of drawing a rectangle I need to draw an image inside each one of those and let's make this scale factor bigger I'm using kind of round numbers you can see it's not so that I'm things aren't divisible by the right whatever but that's fine close enough let's say add background zero so close enough right so we can see this is still giving us something here we've got a pixelated version of Obama's portrait now it needs to do is the really the hard part so that was the easy part the hard part now is I need to somehow load however many images there are so let's see how many images there actually are if I go here and I do it command I we can see here that there are 200 up there are three thousand nine hundred and thirty-four images so I think I can load all those images into memory no problem let's see hmm the good news is they're mostly like small images so if one thing I'm going to do is I'm going to create an array of all images and the other thing I need to do is I need to now dynamically be able to look in that directory and say give me all of the file names in that directory and there happens to be an example on the processing website I know that it's there actually it's because I made it and I just don't remember how to do it this is what I want I want this function which I give it a directory and then it just makes an array of all of the file names and returns them back so and so I'm just going to grab this code rather than write this room scratch in this coding challenge I'm going to grab this code I'm going to stick it at the bottom here and what I'm going to say now is guess it's giving me file objects not um not strings but that's fine list files right and then I want to give it a directory which is data photos well this work print array files so let's see what I get here okay so let's see if we get this list of photos no okay uh what if I do sketch but it needs a full path so if I wrap it in sketch path which is a function that like figures out the full path of the directory there we go so we can see here it printed every single right now there's some extra stuff in there so for example why did I get like some weird so first let's comment this out let's go to the directory there it looks like there is something extra in the stretcher oh there's a folder called static oops let's get rid of that that was extra by accident so okay sorry that was a little mistake there's nothing to do with this video so now I have all of those files now print line files index 0 just look at that so it gave me that so we can see actually like to string will actually just give me that file up now notice by the way that first file is that DS store which is like the basically the most annoying file ever always messes stuff up but Mac operating systems put these like files everywhere dot da store dot d a store that gives dorks yeah ok but we're going to ignore that file I think everything else luckily it's in the zero spot so it's a quick hack and could just ignore it what I'm going to do is I'm going to say first what I want to do is I'm going to say all images equals a new array with five files at length minus one because I want to ignore that da store file then I'm going to say I equals zero I is less than all images length I plus plus what I want to do is say string file name equals files index I plus 1 right I just want to skip that first dot da store file I really should write some code to check to make sure it's a valid jpg but I think we're going to get away with this here and then I'm going to say all images index I equals load image file name so minivan array with a lot of images in it now this should probably take a while to run so let's see what happens could not find uh so there's also a file called index.html by accident so rather than write code what's this image let's ignore this one some nice images here let's let's let's hope that that fixes it loading loading loading come come come you know what I'm going to do right now is I'm going to use a smaller subset of images let's just use ten images right now just to make sure this is actually working ten images okay so that worked let's use one hundred hundred takes a little bit longer but we could see okay so I'm loading a lot of images okay um here we go so now we have a lot of limit just loaded so what do I need to do with those images what I want to do ah boy this is tough okay so what I want to do is for each one of these pixels that has this given brightness I want to find the image of the clock so I'm not gonna do this by color I'm gonna doing this by brightness so let's let's adjust the way this looks for a second so you can see we can see if it's still going to kind of have the right look floating hundred images okay so this is now the portrait what I want to do is there are technically speaking there are only 256 brightness values so what I want to do is in all those images I want to find the particular image that matches a particular brightness value most closely so the other thing I need to do is and and this is going to be very not sophisticated I'm going to say brightness what I need to do now okay yeah so what I'm going to do sorry brightness equals a new P image array with 256 images in it now the other thing I need to do is when I load each one of these images I think what I'll do is make a corresponding array B let's call this bright images and call this brightness let's just call this brightness I went up an array of brightness values and truth be told there's yeah but look I got to figure this out as we go so um so I have array of all the images and then I want to have a an array of the brightness value for each one of those images so after each load each one I'm going to load pixels and then I'm just going to look through all the pixels this is going to make it take way longer by the way pixels dot length so I need to look at every pixel and I need average equals zero and the brightness is this particular pixels brightness whoo right so what am i doing now for every image that I load and I should sample them down I really should sample them down because we'll make it run much faster but let's just sort of see how this goes every single image that I load I'm going to look at its pixels I'm going to examine the pixels and I'm going to add up all the brightness values and then averages the sum of everything divided by the number of things the length of the pixels so now I have the average and I can say brightness index I equals brightness index i brightness index i breakfast fix i equals average okay so what I've done now is I'm just doing I'm just going to do 10 I'm going to run this and I'm going to say print array brightness get rid of this so let's look at this no pointless nullpointerexception so brightness Oh brightness is a new array with the same number of spots and I should make like image objects I should there's a lot of ways I could refactor this to make it make more sense but this is pretty good okay so we can see now those 10 images each have an average brightness as listed down here so like the first one is an average right into 158 seconds an average brightness of 83 so actually in a way we're kind of done what I'm going to do I'm going to do this in a remarkably inefficient way yes no ah so hard so hard do I have nested into eyes somebody told me no no I think it's okay I use J here where I needed to so I think it's very tricky and confusing but I used I and J in the right way okay so now what I'm going to do is let's do what I was thinking originally I have kind of a better idea for later but this is fine what I'm going to do is now I'm going to say I need bright images so I need to fill this bright images array so Brooke so what I need to do and I said VAR here SP int okay so I need to fill this bright images array so what I need to do is find the image that's the closest in distance to the particular brightness value so I'm going to say right now closest the record the word the record for being closest is 256 right because nothing you know maximum being 255 minus 0 that's 255 so I'm going to now say the I'm going to just use floats here the difference is the absolute value of I - brightness I write in this case I the index is actually bright I want to find for brightness 0 for brightness 1 for brightness 2 and not brightness I look at every single image so I need to say now I need to look at all of the images and the difference is the current brightness the absolute value being the positive value of the current brightness that I'm searching for - the brightness for a particular image and if difference is less than that record then record equals that difference and bright images index I is then the oh yeah yeah yeah all images index J so take the corresponding image take the so what I'm doing is the saying for there's 256 images I need so should be J less than brightness dot length yes thank you chatroom okay so what I'm doing is saying here's here's all the brightness I need to I need to have an image for each brightness value so I'm going to find the one in my all images that's closest to that value I'm going to duplicate some and there's going to lots of problems with this but this should learn and to do that I look for each brightness value I look through all the brightness values of the other images find the one with the smallest difference between the target and then save that so now let's just run this and see that I don't get any errors if I found okay so where am I missing I've got like a bracket out of place another bracket here okay so ups no pointer exception because I forgot to create bright images is a new P image array with 256 spots in it there we go so it I don't have an area so now this is the moment of truth what I need to do is instead of drawing those rectangles filled with that brightness value right this is where I'm drawing the rectangle filled with that brightness value what I want to do is say int index equals that brightness value and then I'm not drawing a rectangle I'm drawing an image all images no no no bright images right this is the array that has all those images indexed by indexed by brightness so draw the one corresponding to that brightness at X scale y dash time scale scale scale so I'm going to dynamically resize the image and what what okay I use index twice image indexed okay here we go and I'll get this hey not so dock to shabby right so we can see here and you can kind of look in and see the level of detail I only had ten images to pick from so it's getting a very like it's not that detailed but let's try again with you know let's try 50 images I don't know how long this is going to take but we can see we've got a little bit better now and let's try 100 images waiting waiting waiting getting better 200 images I'm just going to be doing this forever okay okay great so now what I'm going to do actually is I'm gonna let it do it with all of the images but there's going to be editing in this video not for those of you were watching this live there's no edit point so I'm just curious if it works okay so I'm back and I you don't see the image yet so a problem arose the problem arose which is that I got an out of memory error so the out of memory error that's a that's a problem and it's telling me that what I could do is increase the memory setting and preferences which I did do and I still got it out of memory error so you know loading 3,000 images into memory you know I could is kind of an issue but I have an idea here if if I go back so you know so rather than have to increase the member this will also make it run faster if I go back to doing this with just ten images you'll notice something what I'm actually displaying here is like a really really tiny image so when I load the images it's going to make everything run so much faster I might as well then copy that image resize the image in processing sample it down so that I can Eliza the pixels I have a like a much smaller amount because I don't need to draw a bit smaller thing anyway so I'm going to use that same technique where I'm actually going to do this P image file a P image IMG equals I'm going to load that image then I'm going to say all images equals create image and the image that I want to create I'm just going to make it like a little rectangle so I'm going to make it do I already have that size right scale so I'm going to make an image that's a tiny one that's just 16 by 16 an RGB image and then I'm going to say all images index I same exact thing I did with the Obama portrait to pixelate it I'm going to say copy image and I'm going to use the full size of that image oh actually you know what I should do I should really make it a square I'm not going to add this to the code be complicated to figure out you know what is it what's its aspect ratio but that would be something good to do - just like crop it but I'll leave that for those of you who are so inclined to add that as an improvement to this so this would be exactly the same thing but run much faster so you can see how boom analyte mean so now let's do this with 100 pixels 100 images make sure it's still working there we go now here we go I am going to now put edit this out again I'm going to go back and I'm going to put in all of the files and here we go go okay look I did it actually I did it I had to limit it to 1500 images I'm still having memory issues there I'm sure I could look at this more and figure out some ways to you know load your load images as I one way to do this actually would be to load and unload the images but then only load them again when you need them to to rewrite all this you know I could actually what I probably should do is I know what I should do I should pre-process that whole folder of images and sample them all down so I'm actually just loading the smaller images that I could actually do rather easily but I'm not going to do any of that now because it's time for this video to end you sort of see the basics of this I want to give you some things to think about to improve on this I'll release the code I would love people to I'll put this on github and see what improvements you make number one is is there a way you can cleverly match color not just brightness so think about how would you do that it was pretty simple with brightness because there's only 256 possibilities but the full range of RGB colors that's much bigger with the with brightness itself something you might think about is well how could I have each image be unique or each that array instead of having one image for each brightness value what if each brightness value has a selection of images and every time I find it I pick randomly or I can't use the same image twice for certain brightness values so there's a lot of ways I think you can improve this sort of uniqueness of this and then of course you know this is kind of nice but if i zoom into it you know we're really dealing with these very low resolution images so I'm not really getting much out of it but the I think this would be more interesting to visually as a massive a large scale print right so that you could step away and see it or go close and see the detail so thinking about ways you might present that or represent that whether it was a web browsing experience could be interesting to think about so I hope you enjoyed this Oh bomber photo mosaic and I I hope you make some stuff with it and I hope to see you again in a future video goodbye [Music]

Channel: The Coding Train

Views: 123,696

Keywords: programming, daniel shiffman, creative coding, coding challenge, tutorial, algorithms, processing load pixels, coding, challenges, pixel array, pixel sorting brightness, processing, challenge, programming challenge, processing tutorial, white house data, white house social media data, obama social media data, obama administration data, photo mosaic, photo mosaic java, photo mosaic processing, loadPixels processing, image analysis processing

Id: nnlAH1zDBDE

Length: 27min 1sec (1621 seconds)

Published: Fri Jan 06 2017