Django and ChartJS data visualization reading from csv file | Data Science project

[Music] so hello everybody today we're doing a project called expert assist we're doing a project called expert assist where a user is a data science project where user will be able to upload a csv file and can do visualization of the data from the file it will take it to inputs it's going to be first let me write here input input 1 is going to be the file and the input 2 and put two is going to be the uh what the user wants to visualize okay do you want to okay okay this one is a mistake anyway it doesn't matter so we're gonna what is what visualization you want to show and maybe stupid question i don't know okay anyway so what i have so far is just um this text please upload your files so i'm gonna do an input form where you can uh where the user will see here to import to input the file and uh a button to upload it and it will show a input text where what do you want to visualize so if you want so the column in this case it will give the column name okay so firstly what i have to do is here to create a form okay is to create a frame in how the django understands so we should say here method to a method post is to get an input and encrypt and type is going to be multi-part form data okay i'm sorry so we firstly do this cscsf token and we're gonna say here input we're going to write here three inputs okay let me write the first one it's going to be a file it's going to have a name document and a an id document and it just says that it's it's required okay and same we're gonna we're gonna do we're gonna do another another input which is gonna be a button okay this button is gonna be um i'm sorry it's gonna be id submit and just gonna call it upload and let's go and see what we have here okay choose a file and upload so let's make it a bit prettier so here in the title i say expert assist firstly give the title i'm going to take the put strip link so i already have prepared to put stripping okay and and and and and so let's do some some stuff here okay so basically let's do let's make it a bit prettier so or let's but by now let's leave it like this and let's just firstly do to upload okay let's do some some style here in the upload button so i think i there is a class in button called uh button primary and i think it gives okay cool anyway i don't wanna touch anymore so let's let's work with this one only be able to upload this file and show us where the file is uploaded so we go in the views we go in the views uh um file and here we're gonna we're gonna uh read be able to take the file save the file name okay firstly i say context i don't know i like it to have this context okay and i'm gonna say if request equal dot method post so if the user submitted this this um this form so i i create a variable uploaded file which is gonna do like request dot uh files and here i call uh request files and i call or get the input by the name and id in this case it's gonna be document okay they are called the same and i'm just gonna say print uploaded file print just this uploaded file name but uh let's see what what is gonna show so far to us okay so i have a file here i have this file and i'm just gonna upload i'm gonna see what happens okay it took the file it reads and it prints here the file the file name okay cool so now firstly we wanna we want to upload only csv file this pro this application should be able to read just csv file what i'm gonna say here i'm gonna firstly check if this uploaded file dot name ends with is a method which checks the end extension no if it checks the ends of the file okay ends with dot csv okay and if it ends with csv i need to save it save the file save the file in media media folder okay so what i have to do firstly i need to import the the system storage file system storage to be able to save this file so i go in django core i'm sorry in django core in files storage i say import file storage and i'm gonna say here oops inside this one save file inside the if condition save the file within the file storage format okay and what what else do i do basically i just get the name of the file to be able to save it in a media folder okay so i'm just gonna call this attribute this variable i'm sorry name and i'm gonna say i'm gonna call this other variable and i'm gonna say save this file with the uploaded file name uploaded file name dot uploaded file so when what why is this like this so every time you're gonna upload where is it every time you're gonna upload a file it will add some extension here so it's not gonna over collapse so every time you upload the same file uh it's gonna be saved in different names you're gonna see it what's gonna happen okay so after we get the name we firstly need to to uh know where to where to save the file okay so basically in django we can save the files in media is better i think and is more convenient so media it's used to save those files the uploaded file same as we have here templates and statics and is media as well to save the media whatever you upload so in this case i don't have statics because i don't think that i need aesthetics but i need the media so after i create here a file um sorry a folder called media i need to do some configuration here in the settings so the django compiler will know where to look for same as this here static urls i need to do with media so media root and i i use a a operating system ios import s because to get the part and and to get the part of the media and to to be able to uh understand where the media is okay so here i'm just going to say import media i'm sorry import ios so in this case it knows that media root is in this file directory in the folder directory of the project called media okay base there is the base of the application in this case base of the project and i need to do media url and i do the same as in the static okay i say media so everything is correct here we're done with this part we go again in the views and in the views we need now to save know where to save the file now we need to to save the file basically firstly i need to take the whole directory of this um of this project for example in in my case is users blina the scope youtube video i'm sure you have different okay so i i need again importance okay so i i'm sure you have different so to not over collapse or or to have problems so i i just get that the directory of this project what i say i say i os get cwd is to get the directory current directory okay i write here current directory of the project okay so while we have this directory i need not to save the file okay i need to save the file in this directory i'm going to call it file directory theory file directory is equal to it's just gonna be um some matching or how to say putting uh all those strings in one so it will know uh the compiler of the the compiler of the django and python and i will know where to look for okay and i say here what i said is this the file directory is in this directory so is users blina desktop youtube whatever plus in the media folder and look for the name of the name of the file okay so this is the name of the file okay so after it saves this file uh it saves this file in this directory i'm just gonna say nothing okay let's just run this like this and we're gonna see what's happening so we see uh that we're gonna be able to see here that the in the media that there is gonna be the file that we're going to upload okay let's see i'm not sure i i think i forgot something so we see that in the media that the file was just uploaded what i need to be able to understand this file and everything about it i need the forms okay i need here to create forms.pi forms and i need to create here a class i'm just going to say firstly from django import import forms and i'm going to create a class called upload file okay upload file form which is going to be a text forms and this has a title this file has a title friends dot chart field title it should have max length let's say 50 or i don't know 80 doesn't matter and it it's going to be a file we also can use this one for example if we want to do our project with a user login so every user that logins to to we can see in the database that this user uploaded this file this user upload this other file okay but this is another another thing we're going to do something else and later we can improve this project here but the video today is to read this file to check if there is some missing data and to do do visualization of the file okay so i go in the views now and and what i'm gonna do now i'm going to uh firstly so now i read the file okay so i wanna know i wanna get a message here what is going what is happening or direct me somewhere else okay firstly let's create another template which is going to be uh let's call it result results okay and in this results we're going to put what uh what uh what we read or what what happened with the file okay and i'm just gonna call it dash dashboard result okay so uh i need to go in the urls and and do here something like this and i call here results and i need to create now a method in the views which is going to be a results method deaf results take a recall request and return request results.html so let's say here when the file was uploaded successfully okay uh let's say uh redirect me so i go here and i i'm sorry i go here and i need to import redirect okay and here i say redirect yeah return redirect and i say results okay i call this results method oops return cool so what if another thing is what if a user tries to upload another csv so the program will collapse or will have a compile error so we just say that understand if if if a user tries to upload another format of the file send the message how we're gonna send the message basically it's super easy in python and in this case django framework you have the message uh module already okay so we're going django contrib import messages and here i'm gonna say uh in this if if the file wasn't upload successfully i'm gonna say messages and i put a warning in the request that what file was not uploaded please use tsv file extension okay so after this everything seems to be cool now we need to put those these messages in the index okay so in the index i'm going to do here let's do it inside this one and i'm going to call it block messages and block and uh i say here so i i i say here i put here the message okay if something goes wrong if message if message if it's true and if if there is a message so go in this message for a message in messages okay i'm sorry if you go if there is a message i say what does it mean if messages so if there is a messages here go in this message and print all those messages okay and for so what message foil file i read this i i am able to save this file so now let's be able to read the file and print what is inside the file for that i am going to do a method called read file which this method is going to be an abstract method which is just going to read the file okay what is in the file for this one i need okay before declaring this one i am going to use a panda because it's easier so i need to install the pandas i need to install panda in this project i'm sorry pip install panda okay i need to install panda and then after that i am going to use panda to uh read the file because it's way easier so i'm going to say here to import panda okay import pandas i'm sorry panda as pd and and let's do like this so let's read the file okay file name so uh uh let's just say my file is equal to pd.read t s v and here i give the file name which comes from here file name and i say a separator is going to be everything that you can recognize okay uh some people just do that like this so they know that the project the file is going to have comma but in this case i don't know if it's gonna be comma if it's gonna be dot and comma if it's gonna be space or whatever so i can just create an array here by saying that to recognize those signs recognize those signs let's say two dots dot and comma comma and this straight line or underline red recognized as a separator and if i use this kind of way to define separator i need to use the engine equal to python okay so in this case the panda knows that okay i read this one so it's not going to have any warnings or errors because there there's going to be in my opinion it's better to do it like this i think you will be able to read any kind of file and with any kind of separator that knows okay columns yeah columns are separated like this okay so let's see this data i read it in the in data frame because again i think this easier is gonna it's gonna be a data frame okay and um what what i'm doing here right now let's let's let's recap okay this method accepts the file name which in this case let's say it's gonna be like project dot csv and and i'm creating a variable my file so i'm saying read my read this file open open the csv file and this file that is open that read it is going to be saved in this variable in this object okay so i need to to read the date to read this file in a data frame format so i will be able to split this in an array of keys and values later so i give an index i give index none okay and what i'm gonna do i'm just gonna print data let's see what is happening okay so i i read the data and i print this data in is going to be a data framework before here before um redirecting me to another frame template i'm just calling this method and then and then here i should be able to see the to see the content or what so uh so okay i'm doing firstly the migration okay so i am doing the migrations here and me great because i forgot and it's a really important step okay and another thing is i need i uh need to use pandas okay i'm sorry not panda because with pandas uh i can do a media is it pandas is like is used for media category media transcoding category so in this case i'm using media because it's a file upload okay so that's why you just do um pep install pandas okay and it's already installed okay so now let's run it python manage dot pi run server and we should be able to see the data the data in the file okay so it's directing me in another template it read the file and it's printing the file this is what the file includes it has a thousand a thousand uh rows and it has 20 21 columns okay so i want to save this row number and column number and and say to the user and say to the user here you i found this row as and these columns okay so i'm going to i'm sorry i'm going to do a global i'm going to create global variables rows columns uh data i'm going to make data uh a global so other methods will be able to access this data okay and and i do also my file so i can uh later access it when i when i'm going to visualize or for any other matter okay and i create another value called missing values okay i'm sorry i create another variable i don't know why i'm mentioning values okay so i created those let's say variables objects okay and now i i uh count uh i count how many missing uh values are here if there is here or here or here and i tell the user that i found this much rows and this much columns okay so what i'm gonna do firstly i get the row of the rows and columns okay i'm doing it easy i just get the length of the data framework so i say i'm sorry here for the data frame okay so i say rows is equal to the length length is the size okay is the size of this data this data in this case is saved as a huge data frame okay as a huge object object or let's say uh array in arrays like jsons where there is objects inside objects okay so i'm gonna say in in data frame another thing is you can you have lots of tools okay so x's uh recognize diagonals i think this says diagonals so diagonal zero or the why i'm saying here print okay i'm sorry here uh x's um saying uh recognize access the index 0 is rows the index 1 is column okay here this one is a diagonal like this one is uh one and this one is zero recognize rows and columns okay and uh data access i'm going to put down some um some documentations that i use to understand how things work okay how you recognize access how you recognize like how in this case how you recognize rows how you recognize columns so now find the missing data how is super easy there is a method there is a is null method you can use this is no method to find the missing data but missing data is going to find only one is like this none and empty but what if there is a file is that is going to have like those um question a mark or is going to have zero for the missing data is going to be zero or something like that so we gonna be able we doing now to be able to recognize all those kind of missing data uh ways okay so i'm going to create firstly an object calling missing a missing value which is going also let's call it missing signs and it's going to be an array which will have a question mark if there is a question mark if it's 0 or if it's two hyphen or two minuses this is going to be missing data science of course you can add more if you want if it's gonna be like i don't know if it's like just like this or if it's just like space or something like this i'm doing like this with this much okay i don't wanna i don't wanna do more so i say null data i create now an array which i'm going to save in this null data array the data the data frame so count is null okay count hum save how many nodes you're seeing in a column so i'm going to say is null method if if there is no method in the data frame and i give column okay i how any column here i just say any x's equal to x is one okay check one by one the axis those um uh columns if there is any missing data okay so here in new data is going to find the missing data in this all data frame and this one is going to find for example one here two here three here for here i don't know and and what i do now i just say missing values this global variable that i created okay i just say missing values is equal to the length of this null no no data null data uh array that that i'm saving all those new data okay so this is all good i saved the number of rows number of columns and missing values now i go in the results and i want to show because when i'm uploading here is empty so i want to show the user that i found this much okay so i say here message and i write here i found plus a row i'm sorry i called the rose yes plus another thing is so here i'm having a warning is uh in this case in in this matter rose is um string i'm sorry a row is a number so i need to do it as a string so i'm just going to say here recognize these rows as a string so whatever number it is it's going to take us a string and it's going to add all this string in one object so i'm going to say and columns found this let's say he's going to be a thousand rows and and okay and let's say here plus a string columns columns plus columns and missing data are two and i say i'm sorry string yeah and i call this variable here or this object missing values okay this is also a number this is number this is number and this is number okay cool i'm saving this uh long string with an information in a message object so i i i am going to pause it in the content so i am able to call it here in the result so what i'm going to do is i'm going to say messages dot warning and i say request and i put this message that i just created and what else i'm going to do is uh is uh right now i can access this message in in the results content okay in the results html so uh what i do now now let's go in this result and we're gonna we're gonna put this message it's going to be similar like here okay a block of message so i'm just going to copy and paste it and it's going to be messages okay with this one and it's going to show me what what he found what this algorithm or this application found let's go again upload the file so i upload this project data and it's saying i found a thousand rows 21 column and missing data are zero cool now let's do something else so when he says here when he says here so he's uploading a file let's ask what do you want to uh let's do an input and ask what do you want to uh upload okay give the attribute name or give them give the column name okay like in this is the first line represents the column name so the user should write the column name okay so let's go and and and and do firstly let's go here in the index and we add here another form another input and we're going to say input type text name i'm going to call it a tree view id so id and is required i'm sorry required okay and now i'm sorry it should be should be able to see it okay i i wanna make it a bit prettier so i'm just going to add some bootstrap here firstly i'm going to put here a div class i call it rogue row content and all this form is going to be inside this div okay and and all those i'm going to put in a separate column okay div class a column okay so i'm going to create three of those in the first one i put the file in the second i put the name in the third one i put the upload button okay and and and here i'm going to add a paragraph i don't know let me just give an id what id and i i say a tribute id so let's see what we have okay cool so let me just add some breaks or some margins so it should look everything in one line cool or one more break two let's have two breaks here okay okay okay let's just leave it with one break okay i don't care just i think it's because of this break here okay so [Music] what else now i i i i so what what i'm going to do now is uh to read this one when you write something to be able to recognize it okay so what i do i i read this i recognize this attribute in the views so in the interviews here i'm here i'm getting only the only the file file name and then i'm saving i'm going to get the attribute how let me call it the same name so i don't get confused later what is that for so i say request a request post get and i give here the id the name uh what i'm getting okay and i'm just going to check if i am really getting this attribute okay so now let's run it again let's choose a file project data and i say x12 in this case so uh x12 here we have the attribute okay so it's getting the attribute name good so now i read the data and everything so now i i wanted to hear some visualization i'm going to use chart chart gs so with charges i'm able just to take some pre-prepared charts and and just put it you can use lots of there is i don't know line there is pie there is a lot of them so i am just going to use some pie something like this okay so firstly let's go in the results and then we need to import we need to import here we need to import the chart and i have the link so i'm just going to pause it and now i have the chart and everything before putting the chart okay let's put firstly the chart and then we're gonna do we're gonna do later the the data calculate the data of the chart prepare the data firstly i'm going to call this one just the div i don't know what name of the div let's call it like a row class row and let's do it container so it's going to be in the middle and i'm going to say another div which is going to be the which is going to include the canvas okay canvas id and i'm going to call it pi chart and it's going to be this canvas so uh i i have this script which i already took it for let's just go chart chart yes pi so uh here it is like this is donut here you have it type pie and and chart so what we do so basically here is the data set okay so i i create mine i say here a script i need to create inside the script and it's going to be this uh my in this case a variable my chart and and it's just going to get the document by id okay so let's let's do it okay firstly i have created a variable a list with arrays all the list i'm sorry a color an array with colors okay which i'm going to use to do some visualization here i'm going to say uh very much my pie or i just can say directly like this new chart and this new chart created in the a document id in the canvas okay so i'm going to say document get element by id and i'm going to call this document pi i'm sorry pie chart okay and here is a type a type has is a pie and here in the data i'm going to write my data is it should have labels okay so labels are the data in disk is going to be an array with with the data okay is going to have data sets which is going to be background colors and those other stuff okay let me just close those together in one okay so data sets and is going to have again another label when i'm just going to leave it this label empty those are from the chart okay they are already prepared and they should look like this so i just using it background color background color take the color from this color array that i created and data is going to be the data that i'm going to get from the views that i'm going to prepare now okay options options is like options to put a title and i'm going to put a title which is going to display true just play display true and and text expert assist v solarization this is going to be my data and this is where it ends everything so now i need for here and here i need the data i'm going to go back and use and prepare the data how so i have this data frame okay i need to split this data and prepare okay so split the data into keys and values values based on the attribute on the attribute input okay so what i'm going to let me first stop running okay so let's uh go and do it so what i'm going to create firstly i'm going to create a dashboard dashboard array is going to include those objects i'm going to do a for loop which it will go inside the data frame the data array frame which we have and it will go in the specified um in the specified column okay so i take this attribute id oh i forgot here i need to create global attribute id should be global so from this method i can access it in this method and i'm going to say this attribute and go a do a for loop and then save all this uh look so i'm saying here go in this data framework in this data frame and in this specified column so go row by row and save all those uh all those elements in this dashboard all right okay cool and now i uh this dashboard array i need to do it in um a directory so you know directory in python is like this so where is keys and values so in that way i will be able to separate to split keys and values so what i'm gonna do i'm going to do another object my dashboard equal to i create this stick you can see it also here deck is for okay you don't have some information okay in the directories or i say counter counter is to you to count i'm using this model so i'm going to be able to to collect the data easier okay and i can just say now here print my dashboard so just you later gonna be able to see when i do like this okay okay we're gonna see what is this my dashboard stands for and let's do keys and values keys and values so keys is going to be in my dashboard get the keys so easy in the values in my dashboard get the values super easy and now i have the keys and values i just want to create an a list of them because here is javascript and javascript understands only these labels when i put the data data should be like this like this something like this okay so it understands this chart understands only one is an array that's why i need to do this step so i'm going to create list keys and list list values where i'm going to save all those list values where i'm going to say all those information from those two directory from those two arrays okay so i say for x in keys in case and the list of keys dot append which means insert the x same for epsilon in values list values append oops here should be list keys my apologies have hand offend uh epsilon in this case and what left is what else left is uh just to put this one in a context how we put it we just call this one this keys and what list values oops this one should be into that and these values and this context we pause it here okay so cool now we did some pretty some really good stuff now let's take those list keys put it here in the results so in the results we just post the the data okay so labels here in the first in the labels i'm gonna print or let's say it recognize it takes the keys and in the data it takes the values okay and save is to import it better okay i think safer i don't know so now let's see we're gonna be able to to see a really nice dashboard i think let's go and then so please upload your csv file okay i'm going to upload this csv file this is my file and i'm going to give this column name c12 and when i say upload i see a really nice chart what can i do i here it gives me i found a hundred a thousand rules 21 columns missing data are zero what i can do is i can change this one i can just put it uh put it uh like you can just change here instead of instead of doing like pie you can use doughnut you can use other other stuff and here is for example the column a uh 100 those are the values this uh values a 123 is being appeared in this column 332 and stops like that so this is the this is what we did this is the project we did and worked on it it's really cool a nice project thank you for watching my video and i hope you like it and it helped you

Channel: Middle Ground

Views: 3,324

Rating: 4.9436622 out of 5

Keywords:

Id: Y7OAk7DiLJs

Length: 50min 15sec (3015 seconds)

Published: Mon Mar 08 2021