Get it Right in Black & White Episode 12 - Quantitative and Categorical

and we're live oh my camera doesn't seem to be coming through in the video uh-oh hang on i want to leave this meeting and come right back all right i'm back and my video is working yeah google me grabbed my video but now it's live on the stream all right let's see we are live welcome everyone oh wait testing testing looks like my sound was the sound was a little clipping there we go all right welcome everyone to episode 11 of get it right in black and white quantitative and categorical we've got a full house today uh we've got kostov here adil is here eric is here anita and myself welcome everybody are things going all right great great thank you all is well all is well um yeah the submissions for this week were quite cool um looking forward to step through them and maybe we can bring up some any questions that that have arisen through trying to do this exercise so here we go here's the forum submissions from last week um oh there was a typo in there make the menus work with species that was the goal so let's see what happened um let's see oh can you all see my screen actually i can't see your screen there we go sorry now you should be able to see it oh great yep all right so kostov uh had some had some great um discussion and i was so impressed uh that you were able to use the little snippet embed feature of his hub that's so cool so we can talk about the specific line numbers yeah this was really helpful to have the snippet in the chat as well thanks for the tip on how to do that nice yeah my pleasure i'm happy to see the feature being used it's a feature that's been there for a while we haven't really used it much but it's supe it comes in handy when we're you know asking questions about the code so the question was um you're trying to add the click event listener to the option element yeah see this is actually what i did too the first time i tried to get the menu working and i was also confused as to why it wasn't working but it turns out that what you need to do it turns out that what you need to do is actually add the event on the select element not the inner option elements and once you do that it seems to work but yes yeah did that work out for you custom yes yes that worked out so i was instead of use applying the listener on the options element and applying on the select element it worked yep excellent so yeah i think event.target.value works um and this is a great lead to some of the stuff we're gonna do today but the follow-on question was okay now that i have you know the column name back from the menu how do you get the text of it like the the label text and it's a great question um there are a couple of different approaches but my preference is to keep uh sort of metadata about the columns in a data structure that you define yourself and this is also something i'd like to do in today's live coding this way you can associate the value the label and the type for each of the columns which lets you then distinguish okay species is categorical and you could do this by introspecting on the data like checking the type but i like to you know type it out explicitly so that we have this data structure and to be clear you can generate this data structure by looking at the first row of the data so it's a nice way to decouple those two different activities to have an intermediate data structure like this and then to answer the question you could look up the text by constructing a map which is a data structure that's essentially a dictionary looking up key value pairs so yeah um yeah this was really nice i really liked the way you structured your metadata as a separate variable and looking into it has also found a subtle difference like when you said map i think previously we also used the map function which had a lowercase m right this map this had this uppercase m for sure so this uppercase m map that's the dictionary and the lowercase one is like uh a functional functional programming construct is that right exactly that's exactly right and i can see how it's confusing because they have the same name but yeah map is a data structure built in from es6 from i think it's from the introduction of es6 so it's a fairly new data structure if i want to know about it i just typed javascript map mdn which is the best place to look for documentation so this page here tells you all about uppercase m map what it does how to use it so it's it's different from regular objects in a number of ways but essentially it's the same and it has some nice methods on it hey felipe is here welcome felipe but yeah map you can create a new thing you can set keys and values for that thing and then you can check does it have a certain key all of this stuff is available on maps whereas javascript array map is a method on arrays which you can find out more about here it's it's indeed a functional programming construct where on an array there's a method called map you can pass in a function that takes as input one of the elements of the array and returns something and then you get a new array of just the returned values so yeah that's the distinction between the lowercase m map and uppercase m map here and i think we'll we'll end up using this data structure in today's live coding as great well this was really helpful nice so yeah here's the the submission oh i like the styling on the menus very nice and it works fantastic beautiful beautiful very nice work awesome let me just take a quick look how it's done nice see here's the trigger nice work so it says if x name is not species so for any others column except for species we use scale in here just like we did before but otherwise use a point scale and set the domain to data.map x value which isolates all the unique values and then point scale spreads them across the visible space so that's why this works nice job let's see what else do we have well assad also took a stab at this and went a little bit beyond as well uh this is quite impressive so we got sepal length pedal width using linear scales species works using the point scale and nice job adding the padding see it's not all the way down to the bottom and then also added an option to use a square root scale this is amazing or a power scale what is this so yeah different types of scales in use here it's brilliant so if we want to see what the scale does we can say okay pedal width linear and then petal width square you can see it's a square root relationship if i remember my high school math is this a parabola or maybe not i don't know but yeah very nice work i have question about this one sure are there circles on top of like many circles in one place it looks like that yeah it looks like there is yep you could see this the edge on this one is is sort of soft i don't know if it comes through in the stream but the edge on this one is really rough which indicates that there's a bunch of circles in the exact same spot yeah and this makes sense because there may be a bunch of data points that share the same value what's the point of square scale ah great question i think here is just an experiment so there's not really any any point per se except to figure out how to do it however the the place where square root scales come in handy is defining the radius of circles based on data such that the area of the circle is equal to the quantity in the data yeah i think we'll get that we'll get to using it like that some in some future stream where we use size but that's i think the main use case for a square root scale yeah and then there's a log scale which is not here but that's useful when the data follows a power law distribution and i think we'll use we'll get into that again also in the future but yeah great work here from uh well assad let's see how he did this just adding a type property to the various columns very nice band square square power and then in the scatter plot hook into that value and he even made a separate function called getscale which is very nice very nice refactoring so if it's in the the code is very readable if it's linear then use scale linear if it's log then use log if it's square then use square root otherwise use power scales or band scales so yeah great work let's see what else yeah there was some interesting discussion using the options array approach works oh eric has one look at this eric's here with us right now let's see what is going on here oh my gosh they're blue that's awesome and it does filtering look at that oh this is brilliant brilliant so it filters by the species you know each particular species eric you want to talk a little bit about um your experience with this sure yeah i uh i guess misinterpreted the assignment so i i was just wanting to look at um all the data but stratified by species this is awesome oh nice yep oh yeah flex box can be um kind of tricky and the way i did it last time was sort of a shortcut in a way to just overlay them on the top but yeah when you need more complex layout you need different different tools css grid is an amazing tool to use nice that's awesome and i see that the colors change as well now that they're now they're all blue but if you go to one of these it's it's red nice nice yeah this is awesome um so yeah let's just take a look at how it was done options so we've got all setosa color and virginica which is this menu here and then that's passed in to the scatter plot and then here's where the filtering happens nice so this is yeah this is great it uses the dot filter method on arrays and if if species equals the z value of d which is passed in through these accessors then include it very cool very cool and i like how you split it out into functions with dot call that's a nice approach very nice yeah it is kind of crazy yeah that's the crazy thing about javascript um and one of the reasons why i love it because it's it's got elements of functional programming and it's got elements of object-oriented programming and you can make it yeah you just showed that map yep slower and uppercase and like what yep it's it's a great playground to be in there's tons of tools in the toolbox so to speak yeah yeah it's getting complicated yeah it is it is it's a journey it's a journey of learning new things using them and this functional and object oriented programming is kind of confusing really and you don't know where which one starts and which one is right yeah and the other thing that's also interesting is how i don't know if this is a style right yeah it does get confusing when a variable has the same name as a function or a property has the same name as a function somewhere else yeah yeah it's totally a trade-off picking variable names but um the one piece of feedback i would i would have for this in terms of like visualization design is that when you when you switch between setosa and versacolor the dots animate and that's a cool effect however it communicates to the viewer that each dot is like the same in a sense you know what i mean it's like when a dot when a dot moves across the screen it makes sense if it's the same exact the same iris flower as it was before like this kind of animation makes perfect sense because when the circle moves its identity remains unchanged it's the same flower that it was however when when this animation happens it's actually kind of misleading because the flower is a different flower that's represented by the same circle as it moves across the screen you know what i mean well i think it's just a matter of you know noticing that that's an issue and we can go in and fix it and here's how i'll fork this so i can play around and i don't think i ever even talked about this but because we didn't need it but there's a second argument that you can pass into dot data which is a function that defines the identity of the thing and d3's internals will use that to distinguish between um you know things that are the same and things that are different so here's here's what we can do uh we need some some kind of identity across the different iris flowers first actually before we do anything else so here's here's what i propose when we load in the data there is no column that is a unique identifier for each individual flower it's just not there but we can easily make one by saying d dot id equals you know the simplest way to do it is just to have an id counter so like id counter plus plus and oftentimes there is something in the data that is a unique id but since there's not here we can just use the index in the table as the id and so just to see what this does it gives us oh sorry d dot id i had a typo wait why is that not working somehow it's broken did i break it i didn't really do anything huh i don't know i'm having um what i like to call a twilight zone moment or i have no idea what's going on so i think i'm just going to delete this and try again i'm getting a little sidetracked here it is the original one is not loading now which is weird huh i don't know maybe there's some outage or oh there it comes there it is maybe the data was just taking forever to load or something i don't know let me try again i want to do this because it's a very useful thing to know okay now it's working it must have been like a data latency issue or network problem so id counter is zero d dot id is id counter plus plus console.log d just to see what we're dealing with here it should be integers see this one id 3133 id 134 the point is that it's unique to each of these different iris flowers so now that we have that and also we might actually get past i as a second argument to parse rho i don't remember but if we do then we don't need to do the id counter oh we do check that out that's nice that's even easier we don't need to do the incrementing ourselves so d dot id equals i done we're just going to have integers as the ids now in the scatter plot when we do the data binding we can pass as a second argument a function that takes as input one row and returns d dot id and hopefully with that one single change it should actually solve the visualization design issue that we were talking about okay it sort of does there's there's one there's one that's not behaving correctly but it's getting close and this is this is more accurate in a sense more honest in terms of what's happening although there's well i mean i've seen this sort of thing so many times that like oh it's sort of muscle memory at this point i don't know why that one is misbehaving though i'm getting not a number from somewhere is it let's try console.log d dot id just to see what it is yeah so these are coming from our index.js there's not a number coming from some other console.log somewhere else but i think the ids are good maybe maybe they need to be strings or something so if we do empty string plus i they're going to be strings now maybe zero is coming back falsey and that's causing problems no that doesn't do it the problem is that there's a little there's one of these red ones yeah good idea you know this is buggy too because now it they should animate they should animate smoothly yeah it sure does it looks like they're they're exiting and then entering again which is not supposed to happen good idea ah it does not yeah see we're doing this i think i know what's going on we are missing the id in this transformation here so good idea to look at this place we can just solve it by adding the id to the resulting objects and this makes sense i mean it decouples the the data transformation from the rendering which is good but it adds a place where we can forget something well let's see if it works now it still exits all the time which is not what we would want oh there's something else marks overall oh there's different paths that it takes so let me add it there too now this is working as i would expect so when you do it like that the right thing happens now the right thing happens as well see okay great this gets to the bottom of it and i think this is like the you know correct quote in quotes correct design where if you change these columns it makes sense that because the identity of the dot correlates to that iris flower but then when you do the filtering they don't animate anymore they sort of disappear and then reappear and if you wanted to be complete about it you would make it so that the exit mirrors the enter in terms of the animation so instead of animating in it would animate back to zero radius and then disappear that i think would be the ultimate best solution but this is uh this is pretty awesome as it stands so i think i'll leave that as an exercise for you eric if you want to take that out yeah yeah this this is great learning so thank you awesome awesome yeah my pleasure and the way to do it just a little teaser would be to um put the transition on the exit okay and have something like grow radius but oh here's actually how you would do it dot transition t dot call shrink radius something like that oh cool yeah can i have a quick question here sure when you have things entering and exiting at the same time which runs first in terms of animation well they that's the elegance of using the same transition they run at the exact same time i see and what would you do if you want things to exit first and then enter again well you would have to invoke the entire code multiple times with different data um that's the only way to trigger things that happen at different times or now that i think of it i mean you could add a delay to the whole thing so so that the transition is delayed by a fixed amount of time if you wanted to yes i see so if i want them to exit first i would add a delay to the enter so they would exit and that would that would be enough time for them to exit and then enter will will be involved oh i see what you're trying to do yeah yeah and it almost pains me not to do it right now because we're so close and we're discussing this let me just let me just do it real quick thank you so on exit we use exit.call so that we can use the transitions and then um we can call shrink radius and then dot remove calling the dot remove on the transition will remove the dom elements only after the transition finishes which is something that only d3 does really well if you try to do transitions with other libraries like react it's such a pain to do this sort of thing but d3 does it very well now we just need to implement shrink radius which should be pretty easy just copy grow radius and make it transition r to zero like this let's see if it works it does but the delay is problematic the delay is like too much i don't think we need any delay at all as a matter of fact the delay is the fancy thing for moving x and y but for exiting and entering i don't think we need any delay at all hmm not sure what's happening here is shrink radius taking the enter selection as an input shrink radius is just taking well it's it's called enter here but the name doesn't matter oh wait a minute it's adding the transition here oh well if it's if it's adding the transition there we don't need to add the transition here as well and we don't even need this call so here's what we can do just exit dot call shrink radius easy peasy let's see if that works oh i think i think we're not making the radius we're not growing the radius on update and shrink radius really should remove the elements as well let's see if that works yeah there we go that's how it should be yeah this is exactly what i was hoping for okay cool problem solved and it was a interesting journey to get there okay great so i think now what i'd like to do is you know present my take on how we would implement this just so that every all the pieces are very clear but before we do that i think i'll take a five minute break we've been going already for a little bit so um let me check if there's any questions in the chat oh there's a good question about let is um is i is let id counter the same as var id counter yeah essentially just i mean let us the modern version of the syntax all right i'm going to take a five minute break and when we come back we will live code the solution to this problem of handling species see you soon you all right it's been five minutes i'm back um i was just noticing that the problem we were just working on is not fully done i'm not going to solve it now but just just to show you if you wait for the transition to finish before you change the menu everything works perfectly however if you change the menu twice before the transition ends or rather if you change it three times before the transition ends you can end up in a state like this that's not correct and i think what's happening here is that the exit transition is starting and then you change the selection which causes the filtering to change and everything to update but that exit transition is still going on and when it finishes it removes those dom elements and so to really solve this um eric if you wanted to take this on as a challenge because this started from your work um if you really wanted to solve this the thing to do would be to cancel the exit transition whenever you render the data again which may be a little tricky to figure out but that would be the ultimate solution okay give it a go yeah give it a go see if you can make it happen because this is i'm glad we got to this point though because this this reveals some of the trickiest aspects of working with d3 and this would be a good solution a good a good little puzzle to solve to really solidify the knowledge of everything great thanks yeah so good luck good luck with that okay let's dig into today's um live coding by handling species using scale point and manually adding metadata about the columns like type for example so column name and type that's what i'm thinking of as metadata before we dig in though i want to talk about these terms quantitative and categorical these are terms that i got from tamara munsner's book tamar munsoner's textbook called visualization analysis and design which is a great book really great book highly recommend and it's used in a lot of classrooms you know people teaching about data visualization use this as the textbook i use it for example when i teach every fall and that book defines terminology that can be used across you know discussions of visualization design so on and so forth i mean there's a lot of different terms that have been used over the years and she sort of defines it like okay these are the terms i'm going to use so i'm going with her set of terms to define different types of attributes in the data and the term attribute that's another term that means column some people refer to them as columns i often refer to columns as columns because that's what they are to me but in a data table sense columns can also be called as attributes and attributes have types much like variables have types and the types of attributes that we're dealing with here when we come up against this problem of handling species in our drop down menu is categorical categorical attributes have different type things in them meaning not numbers they are not numbers they're strings or identifiers for for things that have identity so the identity of the values in a categorical column categorical attribute are different and in the case of this iris data set we've got a bunch of quantitative columns that are numbers that you see here on the left but then we've got one column that is special and needs to be treated differently than all of those other columns that are numbers namely species and the reason why it's fundamentally different is because it's not numbers it's strings it's a different species of a virus setosa is one species versus color is another and there's one more but the point is that they're strings they're different things they need to be mapped to the visual space in a fundamentally different way and so that's what we'll get into here any questions so far about this all right i think i'll move on okay so now we'll actually handle species and we're going to use scale point which is kind of like scale linear but it deals with categorical attributes you can give it different strings and it will it will spread the unique values across the space which is how we want to do it all right so let's dig in the place where i'm going to start from is animated scatter plot with menus this is what we created last week and just to give a quick recap of what we did we added these menus with these animated transitions so you can change x and y to be the various quantitative attributes of the iris data set but when you select species it just breaks it crashes and the way we did this is we introduced a menu component which is using the d3 reusable charts pattern but for a menu added event you know event infrastructure to that thing and then in index.js we add a a listener for the change event on these menus which will change the x value accessor of the scatterplot instance and re-render it with svg.call and in scatterplot whenever it gets rendered it redefines these these scales x and y scale and it's here in the definition of the scale that we're going to have to make some adjustments so to start i'm going to fork this one i'll call it scatterplot with menus that handles um let me come up with a nice title including species thank you that's perfect perfect thank you okay so what we want to have happen is that when you select species here in this menu and for y for example it should spread out across the y coordinates the different values but it doesn't do that right now let's see where we can jump in to solve this issue to me it makes sense just to start scratch that to me it makes the most sense to start here at the definition of these options for the menus and what this does here is it defines the entries of our menu essentially that's the id for the thing this is the text the label the display name that appears in the menu i think the the most sensible approach would be to introduce another property on these objects called type and the type of the species column i would say using tamar munzener's terminology is categorical and the type for all the others is quantitative so just for completeness sake i'm going to fill all those out run prettier yeah unfortunately because it gets to be long i think that's okay it's explicit so now we know when we select a given option what type it is which is information that we need in order to change the type of the scale that we use so now that we've got this in hand we need to tell the scatter plot you know what type of column it's dealing with we could do that in a number of ways but i think this the most straightforward way would be to add another accessor you know at one of these getter setter functions on the plot called x type because we already have x value it's working perfectly fine we don't want to over complicate that but i think we can add another one called x type and the invocation would be something like dot x type and then we need to figure out what the type is for the column so we have column which is you know the name of the column and we can use it to construct the accessor like this but we also need to use it to get the type somehow i'm not sure how and because i'm not sure how you know this is a perfect place to introduce a level of indirection i'm going to i'm going to call a function called get type yeah just just one question i was trying to do this uh at home but i was not able what i tried uh i first i rewrote the get data i put outside in a model and and then i try to get the json that's inside your your github uh together with the the data there is in the site where i got the data there is a json there that has all these options already the type really yeah so the gist in here yes and i try to use the json but i'm not able to get the data out of the scope when i try to to put this in an array or in a dictionary uh i lose i lose the data inside the loop i have the data right when i get outside of the loop the they don't vanish so how could we use this so we don't have to hard code like the type string or the type number right from from this right and it's so funny i actually forgot that i put this here um let's see when did i make this this was like years ago that i put this up here at that time as well i was thinking about metadata for columns because it makes sense to be able to manipulate the metadata along with the data set which is totally possible so i mean we can use the raw url here and we could potentially use d3.json to fetch this file and use it but for simplicity's sake i'm just gonna you know paste it into the code but because we have this data structure you know uh we could potentially do it like that and years ago i was thinking about like oh i'm gonna develop a data publishing format where it's supposed to be a csv file and a little json file like this that describes each of the columns don't give up on that right [Laughter] it's not a bad idea i mean it would be super useful to be able to have like a standard that you could just plug in so i'll keep that in mind but yeah the way to do it would be d3.json to fetch this file and then use promise.all to fetch the csv and the json at the same time and then run some code after both of them have loaded but no matter where this information comes from we're going to have to implement this function get type for the column i just yes yep sure and so let me just put that right here get type is a function that takes as input the column and returns the type of the column now the name column might be confusing because conceptually a column is described by one of these objects but what this really is is the column value meaning like the name the name of the column so it's just a string it's it's not the entire object if it were that'd be simpler but it's just the string sepal length so you know honestly to make that more clear i'm going to call it column name or column value i like to think of it as column name that way column [Music] column attribute i don't know in a way i wanna i wanna i want this to be name like this but it's a refactoring that would me we'd need to update the code elsewhere i don't know maybe i'll just leave it like the way it was but we just have to understand here here's what i'll do i'll add a comment say column is a string corresponding to the value property on metadata objects can i can i ask a question about the problem we're trying to solve here sure thank you so am i right to understand that we have an array of objects and we have a value a unique value for one of the properties of those objects somewhere in that array and we want to reach the other property of that same object within the array that's exactly right yeah we have this as in we have the string corresponding to the value properties in these objects that exist in this array and what we want to get is the type which is a strain that comes from a different property of those objects but we have a array of objects right correct and we we need to get the value of the object inside the array okay okay yeah we have to given the column name for example species this function should return the type namely categorical yeah and so one one way we could do this is you know iterate through each of these entries and then when it matches when the value matches we'll have access to the entire object and then we can just access the type of it and that would look something like options.find which is another method on arrays d dot value equals column this would give us the object and then we could just access dot type that should work yeah cool yes and and that will enable us to get to jump from any property to any other property within the same object exactly in an array of objects correct perfect however the dot find method on arrays will check you know the way it works dot fine the way it works is it checks each and every one it says okay let's take a look at this one and it passes that object into this thing and then this function runs it returns true or false so it says is dot value equal to column it checks the first one it says okay d dot value is battle width is that equal to column which is you know whatever column we selected if it is then it returns true and then array.find returns that one that matched but if it returns false array.find goes on to the next one and it passes in the next one and says okay is that one the same no it's not the same go on to the next one check it again if if it is the same then the define method returns that object so if it matches species for example it would return this entire object from this expression here that's why we can say dot type we could we could just as easily say text to access the text so because it iterates through each and every one i generally don't like to do it this way because it's big o of n algorithm wise that means the algorithm takes n steps where n is the number of columns in this case it's it's not that bad there's only like five so it's not really it's not really an issue so this would work perfectly well but i kind of do want to show the way i would do it which is to create a lookup table using the map data structure but any questions so far i just had a quick question around this find method yes so if like it's not present in this current table lookup example but say we had multiple columns or multiple columns with the same name so species was appearing twice so what find like return both those objects or will it just return the first uh occurrence of the of the selection selected column or the column that we're trying to find that's correct it will just return the first one okay okay yeah that's what fine does it just returns the first match that it encounters so if you did have species as the value for multiple of these which you wouldn't want to that would be a bug but if you did uh it would just return the first one and not the second one is there uh find all uh method um not the case here but uh just yeah this is what i would do i don't think there is but there is a radar filter which is essentially does the same thing yeah okay yeah exactly yeah so if you wanted to do that you would say options.filter and you wouldn't want to say dot type because the result would be an array this would find all of the matches it would so filter is essentially find all that match perfect okay clear now i just want to before moving on to the implementation with map i just want to make sure that this is actually working i have changed the filter for find oh thanks so to just check if it's working i'm going to say console.log get type of column this will let us just check if it's working and this is on the the x menu so when i change x it works it says quantitative excellent what about yeah so if i if i use species it outputs categorical perfect so it works quantitative quantitative quantitative quantitative categorical if it's species so that part works that's great x dot type is not a function but yeah we'll deal with that later but first because we're on this topic and this is such a common thing to have to do i just want to say this is not how i would actually implement this how i would actually implement this is to create a map i'll call it column 2 type is a new map data structure and this is built in to the browser you don't have to import any libraries or anything it's an es6 feature and map has a number of methods like set and get so column two type meaning the way i named it like that because it's a lookup table from column to type you know from column which you use as the keys to type which are going to be used as the values and so what we can do is loop through all of these options maybe something like options dot for each is a way to iterate through these and we could say for each of these options we can say we want to set the value and the key will be option.value and the value that the key maps to which is confusing because the value that we're using the word value here but the the value that the key maps to will be type so how do we put these together this is where we can say column to type which is an instance of the map data structure dot set and set takes as input two arguments the key and the value okay i think this uh solves the problem that i was having when i imported from the from the website the json ah because when i try to to set the what i did i create a external array like before the column to type and try to push the value inside this array but outside this scope when i try to get back the array it's empty inside the four each if i if i create a array outside and try to push the values inside this array when it's outside it's empty so i think this set may solve this yeah and it's important to connect the dots between the data structures because what you need to implement this get type is a a dictionary essentially a map a key value mapping where you give it the key gives it gives you the value but if you're starting from an array you have to do something else like dot find which traverses each element of the array which you don't want and you can't you can't say like options at index column which is you know you could use an object as a map as well but um but yeah i think this does solve the problem and once we have built up this map we can access it uh but first just to really comprehend what's happening let me say console.log column to type so we can see what this map and ended up looking like it's a map instance that has a bunch of entries and this is just how chrome presents it to you in the console it means the key is pedal width and the value is quantitative and this double arrow here means like it maps to you know if you give pedal width to the function dot get it will return quantitative and it'll do so efficiently it doesn't have to check each one internally it implements probably like a hash table kind of a lookup scheme where it's big o of one instead of big o of n in algorithmic terms it doesn't have to check all of those it just gives you back instantly the one you that you want so if we say column to type dot get column inside of get type this should work as well and to test it we can change the column and observe that yeah okay it still prints out the right thing quantitative and if i switch to species it outputs categorical okay this is working this is working however this is not the best way it's not the simplest way to do it the simplest way to do it is to take advantage of a way of calling the map constructor where you can actually pass in an array of arrays and each of those inner arrays has only two elements the key and the value that way of doing it is much more concise and all you know although it is a bit more cryptic and the way that we would do that is is we can pass in options.map again the functional programming construct that lets you give it a function that function accepts each element and the return values end up as a new array we can return an array that has just two elements option.value and option.type and this should work as well let me see if i got it right yeah seems to work just fine so just to review what i did there options.map maps over all the options and it returns an array that has the value and the type the first thing being the key of the map the second thing being the value in the map and this is just an alternative syntax for doing the exact same thing as this other thing does namely looping through all of the entries and calling dot set and to simplify this even further we could use es6 destructuring to destructure value and type from the argument and then we don't have to say option dot and option dot so all of this is a roundabout way of you know exploring the ins and outs of constructing maps um but since it comes up so often i wanted to dig into this this level of detail because you know i think it's it's really important to fundamentally understand how to use maps in today's javascript world i see there are some questions let me see what is this and is it live yeah it is it's live why are you not using typescript well i'm not using typescript because typescript is a pain to use in my experience um then you don't have to write comments yeah i know yeah it's just it's just a pain is this microsoft monaco no it's not it's a viz hub it's a thing that i made and you can use it too anyone can use it you can fork this stuff and get a link to it what are we doing here i love these questions these are great um it's a new visualization tool people can use yeah i mean this being vishub yeah i created about two years ago d3 has been around a while as larry points out thank you larry yeah d3 has been around a while but the apis have changed so that's why i'm doing this tutorial now to like use the most modern way of doing it okay thanks for those questions okay now we can move on to the next phase namely implementing x type but before we do that i just want to make sure are there any questions so far about what we've done here so far please go ahead that song oh she just said it's so complicated yeah i know there's a lot of details there are a lot of details to remember but um and the first time you see stuff like this i realize it can be overwhelming that's why i wanted to like yeah this happens all the time first time i see something it's like wow i'm not understanding it and once i get into it it's like oh it wasn't that bad yeah and console.log is your friend i mean if you were on your own you can say console.log options.map value type and see what it is and and use use console.log to interrogate what you see here anything that's confusing to unpack it and understand what's what's happening in that intermediate stage see it's an array of arrays and i realized when i described it it's a bit abstract but when you console.log and you see it you can understand oh it's an array of arrays with the keys and the values and also if you're on your own and find the stuff confusing it's good to consult mdm which is the de facto standard documentation for the built-in stuff in javascript such as map and it says right here it describes the map constructor creates a new map object and it says right here that you can pass an iterable into the constructor which is what we're doing and it just it documents that right here it says what is this iterable it's an array whose elements are key value pairs in the form of arrays so here's an example that maps numbers to strings so yeah if you if you're ever looking at this stuff on your own and feel oh my gosh this is so confusing use console.log and and and just do google searches for the data structures that we use like map because the documentation is great and also the same goes for d3 the d3 documentation is really good so if you see a d3 method that you don't know like just google it find the documentation um i do it all the time but it takes such a long time it does it does there are a number of um there are a lot of rabbit holes that you could go down and get distracted really time consuming and just takes a lot of time yeah everything it does yeah it does work yep i would say it's well worth it it's an investment in your future yeah i agree that's why you and you have every other week yeah i changed and that makes it difficult because for example i'm so behind i'm supposed to catch up yeah i wish it was every other week well actually i was going to announce it at the end but since you brought it up i did change it to be every other week not every week so going forward is going to be every other week thank god i updated the meetup page why because yeah it's for me too it's a bit hectic to do this every week anyway are there any other questions i just wanted to ask what what would happen if you have two entries with the same keys so you could like while you were trying to iterate you're passing two keys two exact keys to in your map object that's a fantastic idea well if we if we go back to this other variant that i did here with this variant it's easy to understand what the answer to that question would be if you understand the semantics of dot set if you call dot set multiple times with the same key it will change the value and so in this case let's say we had two of these where species was the same what this algorithm would do is call dot set passing the key species and it would set the value to be this one here however the next time around you know in the for each when it gets evaluated with the last option it's going to call dot set again and so when it calls set the second time with the same key the way that maps work it's going to overwrite the first version of it and so we're going to end up with a map that only has one value for species and it's going to be the last one encountered in this array so that's complete opposite of find in a way exactly it's the opposite of find yes yes that's a great insight the way find works is it checks each one and it returns the first match the way this works is it sets up entries in the map for each of these entry entries in the options array one at a time and if it does encounter the key multiple times it overwrites it so essentially it's putting the last occurrence of the match in the original array as the the value in the map that you get thank you yeah my pleasure and i'm not so certain what the behavior is when you use the constructor like this but it may well be the same yeah it may well take the last match although i'm not 100 sure so you know what we can do we can actually test it out and this is the beautiful thing about coding too you can use the code to ask questions about the code for example um oh there's a breakage column to type is not defined i'm sorry forgot to uncomment uh sorry i'm just a little disoriented but yeah let's do this little experiment to figure out what the answer is all right so when we use this variant that uses four each goes through all of them i would expect it's going to give us the last one and if we select species it outputs categorical which is the last one see the first one was quantitative and we actually have the same key here multiple times now let's answer the other question of what if we use the map constructor like this is it the same so if i select species it outputs categorical so the answer is yes it is the same that takes the last one great question great question i love how it digs in a little deeper yeah thank you okay now let's go ahead and solve the next piece of this puzzle now that we know what the type should be when we select it we need to pass it into x type and you know while we're at it let's pass it into y type as well why type is get type of column oh there's some error there's a syntax error i don't know what that was about so i'm going to also call why type here okay it's fine nothing was there was no problem so now from the outer sort of view of things we're invoking it where we want to be invoking it we're setting why type when we change the y menu and we're setting x type when we change the x menu now the task at hand is to implement that method in our scatter plot so let's go into our scatter plot code in scatter plot dot js we've got a bunch of these getter setter accessors and let's just make a few more we've got x value and y value i'm just going to copy paste these and change value to type so x type is going to be x type change y value to y type and we're referring to the variable y type and x type those don't exist yet so let's make those at the top of the file like this now these are available to us when we render our scatter plot let's just make sure because this is where we're going to want to use those let's make sure it's available with console.log x type here so we get undefined initially which makes sense because we're only passing it in when we change the menu but when we do change the menu we get quantitative for these and if we type if we select species we get categorical which is exactly what we want okay this is great and i think what we can do is say if x type is categorical using this um ternary syntax we create a linear scale otherwise now this is where we can use category exactly yeah yeah the good call so i got the order reversed so it's going to be scale linear if it's not categorical but if it is categorical then it's going to use scale point just use prettier to format that and if it is a scale point we can set the domain to be data.map x value and what this what this does is it returns an array of all the different x values including duplicates so it's going to return like versus color versus color versus color setosa setosa setosa but then when you when you pass that array into dot domain the scale will internally figure out that there are duplicates and it will deduplicate it so the domain will only end up having three entries the unique values of the species column and then dot range um actually should be the same because we want it to span across the same space in pixels so this should work let's see if it does if i change x to species there's some breakage let's see what it is oh scale point is not defined yeah i forgot to import it we just need to import that from d3 along with this other stuff now it should work okay great check it out it works hooray we solve the problem yep everything's working the quantitative stuff is working and when we hit species it transitions to this which is the behavior of a point scale by the way it just takes the values that it sees in the order it receives them and identifies the unique ones and spreads those across the space across the screen it's exactly what we want and i must say i am impressed by the way that d3 axis handles the transition see how it fades d3 axis is brilliantly implemented because if it is given another linear scale see how it animates the numbers it animates the ticks but if you pass into d3 axis with the transition a different type of scale it does this nice fade animation which is just brilliant brilliant but anyway there's a little bit of cleanup work to do here but um any questions so far yeah just um one question here is eric um just curious uh how hard would it be to actually show all the data that are in each of those categories as dot plots so in other words in other words some of those as you mentioned earlier are on top of each other but you would really have sort of a frequency is this what you mean by dot plot yep yep so so so what you're trying to show is uh almost the distribution across those categories so that you can see where the peaks are so that's hidden right now yep but it's a sort of a natural transition but as i'm thinking about it just trying to get a sense from your expertise how deep one would have to go and actually to do that not that we do it here but i'm just curious yeah it's a great question go ahead i i just want to suggest an idea for doing that and and let you see whether it works or not so we yes that's right um to implement that what you described eric the first thing i would do is just look at it from from a bird's eye view and say like what should the architecture of this be should it be one visualization or should it be a parent visualization and a child visualization where the child is one of these reusable components like the scatter plot but for a single dot plot got it you know what i mean and then i would i would get it to work that's probably how i would do it i would get it to work for a single dot plot and like you said costa that requires a step of binning where you take these um well in this case you can just use the the different values that are present here i think it increments by 0.1 that's the resolution of the data so for each one of these unique values you would want to bin them and count how many occurrences there are for each of these unique values and that there's a feature in d3 called d3 bin i think or d3 histogram if you google search d3 histogram you can see this in action so that's a d a data processing step so first you need to do that binning so that for each of these numbers you know how many occurrences there are and then based on that after that binning happens you can visualize that as a dot plot got it and then that would be one of these components and then what you're doing essentially is called small multiples dot plots where you would want to actually have three dot plots one for each species and so then you would need to change this scatter plot code completely so that for each of the values across the x-axis here it would iterate through those and for each one invoke that reusable dot plot instance so you'd have three instances of a reusable dot plot that's one way to do it okay that was that that sort of just curious how you'd approach it so um not that obviously we tackle it but it's it yeah so that's how you would do that but there's a uh if you step back a little there's another question what is the best visualization design to show this type of data and small multiples dot plots is one option but there are a number of different options there's actually one option that's very low hanging fruit for us that we could do right now and i love doing this because it's so simple on the circles we could set the fill opacity to 0.2 in the css um i don't think that actually worked maybe it's just opacity yeah there we go that worked so if we set the opacity on the circles and we subdivide them by species you can see it's doing it right there you see what i'm saying yeah yeah so this is a simple modification we can do to make the visualization communicate the information of density like how many overlapping dots there are which is the same thing that would be communicated by the dot plots yep yes so this is bird's eye view yep okay so the bird's eye view meaning what i meant by bird's eye view is you step back and look at the design space of the visualization this is one other option there are so many other options i mean you could have small multiples box plots for example you could have small multiples histograms like bar style histograms you could have small multiples violin plots so many different ways i mean once you get this data structure and you want to visualize it there are so many options but you can frame it as small multiples meaning you want to implement one instance of it and then just multiply it across the different values that's really cool nice yeah and r by the way ggplot in r does all this stuff okay it's brilliant the way it does it so yeah stuff to look into thank you yeah my pleasure all right let's finish this up here it's almost done but we've only done it for x and not y and this code itself can be cleaned up whenever i see duplicated logic like this dot range dot range copy pasted it's exact same thing i asked myself how can we get rid of this duplicated logic in this case it's fairly simple because this expression if you put parentheses around it returns a scale it could be a point scale it could be a linear scale but it returns the scale and so we can essentially factor out the call to dot range so that it gets applied to the returned scale whatever type of scale it happens to be then i just run prettier on that and this is what we get yeah i would prefer this just because it has less duplication so to be clear this set of parentheses creates the scale you know a different type of scale depending on whether x type is categorical or not it sets the domain but it does not set the range but whatever scale comes back from this expression we call dot range on that scale whatever it is so it's just a simpler way to do it now that we've got that we can just do the exact same thing for y i'm just going to copy paste it change x to y all over the place uh so instead of x value is going to be y value [Music] but with the y scale we need to be careful about the range because it's different its height minus margin.bottom and margin.top so i'm going to take this definition of the range and use it here and get rid of our original y scale definition and this should do the trick for both x and y let's see if it does x species works y species works excellent and one last little thing that i don't like about this is that there's no padding it goes all the way up to the edge maybe it's just personal preference or stylistic but i like always to put a little padding and the way where we can put that is right here scale point.domain.padding that's a function on these scales i'll say 0.2 yeah so if we look at it now we get this nice padding it doesn't go all the way up to the edge which i just find kind of i don't know distasteful it has space to breathe the labels you can read all the labels now so virginica used to be off the screen and just as the final step i'm going to call dot padding in the case of y as well so now it works for both x and y and the the labels get cut off yeah let me just change the margin to address that problem because what i want to do is finish today with a complete product that works for all the cases that you select but where's my margin margin oh it's right here margin left let me set it to 150 pixels and see if that's enough for species okay that's a little too much maybe 120. all right there we have it any questions so one question i had was with those drop downs there um i couldn't find where one um does this or if it's even possible to change the font and the size of those drop downs oh is that a css thing that's a great question you know the html select element and the options it's notoriously difficult to style with css so if you're working on a product where you need styled menus the best approach might to go might be to go seeking out some third-party library that implements a drop-down widget that you can style with css okay yeah explains the hunt yeah they're right it's it's a hunt yeah and like you just it's very difficult to style these the way you want um but again it's a whole other level of complexity to like evaluate the different libraries and pick one and figure out how to use it however with this d3 reusable chart pattern you could implement a menu component just like this having the same api you know having the same methods and everything but internally it could use that third-party menu library that would be the approach that i would suggest okay thank you but that said i was actually surprised to see that in one of the submissions the menus were styled let me see if i can find which was that you cost off yes custard how did you do this i think i used normal css to do this this is beautiful it has that custom font that's awesome yeah i just wanted to make the font consistent with axes and menu so i just use this select uh html and apply this property here brilliant so you know there you go all right i was trying to figure that out yeah there you go so it looks like it looks like you had to style the select the label and option to get it to work yep so i think the labels were for these uh access uh labels and the option and select work for the drop down right yep so there's your answer eric should work thank you and i was actually quite surprised to see this working because um i've struggled in the past to try to do this but it looks like this works yeah i was just playing around with it i totally forgot that it worked for my case that's awesome and i wonder if it would behave correctly if you try more advanced css like setting the the background color or the you know the roundedness of the edges i think you might run into a wall beyond which you can't customize but worth exploring for sure right but yeah this is very nice and i i have to say it's really nice how these animate or how your labels animate really nice work all right well i think that's all for today i thank you so much yeah my pleasure thank you all thank you very much thank you yeah thank you very much and uh yeah like i said it's i changed it to every other week so just mark your calendars it's not going to be every week and i will leave you all with an exercise for this next two weeks i want you to find a compelling data set look around online for different data sets and yeah try to find a compelling public data set and just search around you know i would suggest coming up with some idea of like data that you want to see visualized that you've maybe seen in the past or thought about deeply like some sort of existential like question about society or i don't know climate change the keeling curve if you the killing curve would be a good one but yeah find a compelling data set that interests you and the way to do it is just search for the topic see if you can find any visualizations that pertain to the topic track down their data source uh you know follow links until you can get to a downloadable data file like an excel file and then export it as csv or just try to find a csv or a json file load that up you know fork the scatter plot we made with the menus load in that data update the data parsing logic and i want you to actually explore the data using this scatter plot with menus because now that now that the scatter plot has the transparency uh we can see the density of things and now that we made it handle species it can handle any column that's categorical and uh i want you to write up the key insights that you discover in the readme of the viz so fork this modify the readme.md file to write up like bullet points of like this is cool um this other thing is cool you know interesting insights that you find about the data not about the vids or the coding but i want you to like actually explore some data and uh yeah share your results in the forum i've made an entry for for today episode 12. here it is all right so now it's goodbye for real thanks for joining and i'll see you all in two weeks yep have a great weekend thanks take care all right bye bye take care thank you thank you bye

Channel: Curran Kelleher

Views: 503

Keywords:

Id: tbvJbgBgvPM

Length: 110min 42sec (6642 seconds)

Published: Sat Jun 12 2021