[CVPR20 Tutorial] Live-coding Demo to Implement an Image Search Engine from Scratch

okay so research cbpr 2020 tutorial on imagery tree bow in the world this section is about right coding demo to implement an image search engine from scratch I am a u.s. command C from the University of Tokyo okay so here we will implement an image search engine from scratch so it's something like that it's if you have Pro image be a browser see mirror image retrieved by a deep features underneath i permitted by a pure person and you can try this system by accessing this demo site let me try it's info there it's like its web interface if you select this image like in this case just applaud it abroad images and send it then you can find the similar images like this so let's implement this system okay so variety of offices library is available now so it makes it easy to intermittent under deploy an image processing system very easily so I'd like to show you a demo to implement a content based image search engine and this system is actually quite simple so it takes only 24 is in HTML plus forty three lines in fighter without comments and we use chaos which is a one of the most famous deep learning libraries and France it's this is again the most popular web server system and all code approval here and again so website the demo system is here you can try it okay so this is a whole system right side this one is a yeah structure of the directory so we will increment three Python files the fourth one is feature extraction top I and a friend of Pi and some other PI and there's static stuff its feature on images and uploaded images and this is index dot HTML so in Opera in step here we take database images as an input then run of rind of Pi so inside this function this scripture so there's a feature extractor it's by cross with a busy 16 model by using this we extract a deep features like this this is this is image image directory on this if the feature directory then in online face we need these features to here this is some out of Pi on inside about a pipe so there's a Webster or asked here the user access the browser the upload image to this krusk then this rusk call feature extractor it's saying to this one the extract deep feature then this feature is compared with these database features then the similar one is returned so this is the whole system it's it's quite easy right though okay let's start okay so this is BS code it's nothing right thank you so let's start to increment somewhere we need to create make directory study make tree a feature image upload hey so the here static directory and the feature directory and image you've shown a probability actually so it's nothing now then let's put some test images here okay so let's say just five images here to touch in image okay now there are five images something right like that okay then okay sure that then let's see steady employment triangle by trying to PI so in just off Ryan defy vampires it is a friend of the purpose of this or I'm top is read images and accepted it features the walrus check he can read images the image we just select the dot jpg file just just read a static image here so this check age yeah we can see their images are here right then seek or we need to create the feature pass here so it's okay given image this is the image path islam then we just replace the base directory and to feature then replace dot jpg to npy so it's this one so just renaming oh yeah so the path stop here okay the next we will create a deep feature so we need to accept defeat deep pitch then maybe get here then let's implement a deep feature extractor okay then let's create file feature extractor oh hi here so we check structure to apply this is a break okay so we use across very basic function and so fro dr. Ostrow system we use busy c16 this is one of the famous model of the planning process input so chaos provides such VDC 16 library we can just call read it Oh both numpy and brass feature extractor so this is a structure will you create of each extract across I know this cross frost instantiate itself by reading some model then this cross has an function extract its extract function takes an image as an input then return the deep feature a rest increment that a splice model echo [Music] wait Seco imaginate output equal c1 okay so here this model we just read this is a 60 model we can just call functions and from we can use it and it's weight P is imaginary joined features image praetorium the weight by writing these we can just download the data model from the internet already done it then since since this is a recognition model so what our pups here is not recognized recognition now we like to use a deep feature so we just in this line this is this means that we just remove the final recognition layer 1000 cause recognition area will limit that then we can just use final at c1 as an output so this model is a we can use this model as a deep feature extractor anything let's implement extraction this is just [Music] [Music] [Music] [Music] [Music] yeah so this is our kind of a typical prepossessing for Quiroz so we first resize the images and convert it to the numpy array then we add one dimension this is batch dimension because we need this one for processing then this one ring is just subtracting average pixel but it is also a very typical way to use the Browning model then this is predict this predict means we pass the this image to the mode to the model then it returns a 4096 vector especially the budget dimension so this zero mean just removes budget emissions so it's just it it's just the four thousand dimensional picture then we know how is it then we turn it okay so it's a very typical easy to use code and actually work order from chaos chaos library documentation if you see document you can see the similar file there anyway we can use these PGX laughter okay so next thing is let's extract feature you can use few extra fresh new feature extraction extract free eco which extracts equal image open image object let's just take it so here we instantiate on feature extractor then let's just use it we just use it for pier extract just read images and the feature and just show the type on its shape just get back okay this is cancer for stuff okay it seems working this is that and this is crossing apparent area under this is 4,000 a majority so it's fine okay and we can adjust features and [Music] say big here it's a lump I say survive this easy so we already know the feature path you can just save this feature using this path then our five features are saved to feature here right okay so off right but it done so we can run this function this scription we can accept feature so it's nice to travel with five 100 images just okay it's prepared 100 images for offering face and then we use this 100 images for such ok it's to the next phase it's approving okay so let's run it again okay let's start to process images which by the way we need in this tutorial we just steal CPU we don't use GB because we don't train anything so it's fine for CPU okay the next great next file star bar dot type so this is a server so we start to create the server which we define previously from then from fraps let's use frost is the most simple [Music] a CDL ok this is a very basic very basic ask application it's saying nothing if we run it the server is automatically launched its it does not let's check it bison sabor ok oh yeah and the server is launched so if you open this one okay so if we access this one so this server here its function is called and this function we can just cvpr so it's return cbpo ok so this is our simplest server then next to it let's user HTML sing we need to create order can't break it and inside sorry for dusting here I just okay so we use HTML here then I'm sorry but it's typing this takes time so we just copy HTML so this index dot HTML reuse this HTML but it's just a 24 Rhine so a very simple HTML it's actually ginger to temp right so it's temperature and here we defined our form by using this form for we can upload an image and they are named thus query image this is an output area so we can feel something here area anyway so we save index dot HTML so here we come ok and this server returns a previous you find our index dot HTML the tricky ok ok let's access it here so now we render the HTML file I I will show you I showed you yes it's good that's all what happened if we select file we can search something it's fun they upload it but it doesn't work of course so because ok opening HTML is get stuff so get stuff with here but using a form with post stuff so it's a pasta signal here but we didn't do anything here post so it's nothing happens ok so let's implement post post okay then we can image image is sent ok ok then what happens here [Music] okay frosted yeah men may access this website it is a get action so with inside his index function we see this long so it's rendered in private so it's just render template then we if we send on an image there is a post action so what happens here and send it okay so this is a post so we reach here they need to return to receive the query image receive image so we can now here right okay then now we can process query image stuff as an reqtest so since we like to keep we drag to keep using temp right yeah then my piece we can even if the search is done then we can okay let's check it's the same should be saying right so this is image post here and after select something it's still index dot HTML because we show the same thing here okay then let's process query okay so here let's take finish this by dragging this we can access their uploaded query load it [Music] this is just a some strange stuff I'm posting grenade so what happens we read this query image and save it on our he wrote it let's check it rest wrong here we FA a product this one for example and send it you can see here right right so here the Ascended image is saved in the robot file ok ok then let's move to the next tier wait Eco loaded image plus ok so we when rendering template we select this pass to the HTML then we can show backs they are rich and savory images select some GB for example then send it as you can see so this image is saved here and then we consider select this image as an pass pass here so we can this index.html this is this fine this image path ok ok good then we have all images or data so we can just run such sighs this program okay next so read English feature we define features [Music] feature well [Music] sure pass just name a JP alright features okay this means that we lead features images feature parties here dot n py then we can really name it as an image pass then just save it here and this line is just converter a list of numpy array tuna belly ok so this video image feature here then run cash and such here we go feature ensure extra this is already filled image here so we can simply simply use the same feature extractor here this is l2 business [Music] sorry okay the top starting then scores equal East's ID and in the past e or ID in I guess okay okay okay so here we need this is a query image so we accept quite a feature vector then we compute the difference between query and all feature vectors then normalize it so meaning that this is a distances to the or this database next rows from the query then we thought so we we find the closest I came to the query and we select top study result then just ask wise format the thing then return it let's check what happens if we select something send it it's types title sorry in your algebra 3 again elect select something and send it okay then oh it's something strange to me I know oh yeah sorry so it's ok it's working here here this is scores so here since we have the same image so this is the nearest one in the same image so this distance is equal 0 and next one is distance and this is path to the image database image so it's something like that so now we can get scores the final step is path this is HTML back yeah then finally this HTML renders is result [Music] hey elect image send it okay so finally it's working okay so since it's dark and since database side has it's the same image so this guy is same the top one distance equals 0 and these are similar images it's it's a dog and this is not similar to dog okay if we select like that's true I do like parson type of things this is a person so the result are like button right so if we select bird hey stop okay so in summary we can use just a 100 lines of code are we can we use chaos and frost it's very easy to implement an image search engine from scratch like beer like wave server stuff okay so thank you for watching

Channel: Yusuke Matsui

Views: 7,914

Rating: 5 out of 5

Keywords:

Id: M0Y9_vBmYXU

Length: 31min 39sec (1899 seconds)

Published: Fri Jun 19 2020