Create a Data Studio community visualization using D3

hi there this is Michael I wanted to share a few tips with developers who are thinking about creating a data studio community visualization it's fun to build a community visualization and is approachable for mere mortals such as myself I'd consider myself a coding data analyst rather than the coder who also does data analysis however there are quite a few moving parts and a good workflow may not be all that obvious so this is what will build a simple time series chart built with d3 which I hope will be instructional d3 could be a great starting point for you to build your own visualizations but first there's a little bit of initial setup to be done by the way I'll post links in the description to the code you'll be writing JavaScript code and that code has to be hosted on Google Cloud Storage which is part of Google cloud platform otherwise known as GCP if you don't have a GCP account it should be pretty easy to sign up and there may even be an initial credit to get started just a note you might be incurring minimal costs because of your use of storage but mine were two cents last month so not too bad at all and of course you'd also get access to other awesome tools like bigquery and cloud functions when you have GCP setup you'll need the command-line interface or CLI it's a simple installation just search for Google Cloud SDK and it should be very straightforward data studio will then use these CLI programs to upload files to storage what you're looking for in particular is the gsutil CLI this program allows you to manage storage through the command line for example if you type in gsutil LS GS analytics buddy vis you'll get a list of these these folders inside my bucket this is my GCP storage section and so let's see what we have here so this should be reflected in the gsutil so it shows that it's all set up properly so I recommend that you setup GCP first and then create the storage buckets so you'll need to one for development and one for production you'll need the location of those buckets when you set up the community viz in my case I set up folders called tests dev and test prod in the analytics body viz bucket so we'll be using this one and this one and so they're empty at this point right that should be it now we're ready to get on with the main stuff the document you'll want to refer to is here and what we'll do is we'll go through the doc together so basically read everything in that document and watch the videos it's it's it's a great resource and it has helped me a lot alright let's let's take a look so it says that you need NPM which is typically part of node so if you don't have node again just search for install node and there's installers for every kind of the operating system let's just make sure and PM version 6.1 one that's good node 8.15 alright so yeah I think we're ready to start building obviously once you have GCP set up it will be much easier and quicker when you work on a new vase so to help you get started the data studio team created some great tooling to make your life easier and the tooling that I'm talking about is installed here with this command so let's just run this we'll copy this and go back to the terminal oops MPX so this will just install all the dependencies and and yeah we'll see once that's done what it gives us okay so first you're asked for a for a project name let's just call this d3 test all right so you remember that we had to set up or that I recommended that you set up the buckets first this is where you'll need them so analytics buddy is slash test let me just make sure that that's right analytics buddy vis test phase okay okay and the fraud bucket test products okay well this is basically all you need at this point it'll create a project for you and we'll just wait for this to to finish and it even helpfully tells you how to access this so you CD into your your project and then you hit npm run start to begin working on your biz okay so let's do this CD t3 test and actually before we do that let's just pull this up in the code editor which in my case is Visual Studio code alright so this is the code that the tooling generated for you you get two ways to build your visualization essentially local development and in data studio itself let's look at the source directory the main file is index.jsp that an import some external files DSCC and DSCC scripts initial various and local message and there's also a variable local set to true so this means that you're in local development mode so what does this mean well it basically means that when you run npm run start you'll see your visualization locally so this is localhost 8080 and you have this the simple bar chart with age and and the metric so where does this data actually come from well let's go back to the code and so we know local is true and then this draw this function takes in local message and local comes from this file local message so let's take a look and yes so this is just a JavaScript object message tables default and then it has dim ID and then you can see the the age brackets and the metric so why actually have this way of developing locally and why does this object or rather the data look you know why does it look like this well actually this structure is the same that data studio would give you once it's actually uploaded to Google Cloud storage and when you use the visualization directly in data studio so what you're trying to do essentially is have the same development locally and make sure that the data is in the same format that data studio would give you so so this visualization here uses age and and the metric well in our case we know that we want to have date let's go back here we want to have date as on this axis and then the metric here so how do you know what the data should look like well the data studio team thought of this too and you can use this this command here I'm going to cancel out of the the dev server NPM run update message and it should be clear once I run this command what this actually does remember that at this point we actually haven't gone to data studio itself at all this is all local stuff right now so let's do this and then what npm run update message what it does is it basically builds local files and then it deploys it to your storage bucket and then so what you do is you copy this and then now we're actually ready to go to to data studio so this is just a very simple visualization I mean a very simple report the only thing I've done here is I've added a Google Analytics connector to the Google store demo account so to add a visualization you just click this button here and then you paste in the the path to your visualization and see if this works add great and actually to get closer to to what we have here so these are these are age brackets let's also use age here drag this in and then what you want to do is you go into View mode and now you can see all of the data so if you if you recall local message looks like this and yes so so the idea behind update message is that you get the the shape of the data into you get the data into the shape that you want and you can then click into into this here and do command a and command copy and then you can go to local message and hit control a and then paste that in yeah and if you want in our case date then you could do this and you see it updates this this right away and so here let me grab this control a copy a fee and then save this great so now you have the data in the shape that you want and and now we can start developing locally go back here knowing that this local message is the same as we want to have up here and actually the only other thing I want to to do is to also test the the publishing and then so the way this works is I want to make sure that I can see this visualization see npm run start i want to have this also data studiolo so let's make sure that this this works to cancel out of this and the command for that is npm run build dev so this builds it locally and then you push it up with npm run push dev so this will take a couple of seconds and go here go back to your visualization and reload actually it's going view and here's the visualization and and the chart looks a little bit funky because I changed it to be date as opposed to the age bracket but now we basically know how to do local development to see where the data comes from and if our visualization requires certain dimensions and metrics then we have a way via update message to get the shape in exactly the right format that we need all right so we're finally ready to build our visualization and so what we'll do first is we'll take out the code that we don't need so we'll just have this empty function and let's go and see what we need so this line graph as I said earlier is built with d3 so we need to add this to our project very simple npm install d3 and then once that's installed we can refer to it locally require d3 and so the next thing that we'll do is we'll just grab the code and paste that in Hue format this a little bit okay so let's let's take a look at this so the data for this visualization is actually in an external file this CSV file has two columns date and close the date is in a particular format like this or we'll have to deal with it and then the metric is here so d3 very often uses this function here at d3 CSV which is a an asynchronous function which loads in that CSV file and then it calls the the callback function once it's finished loading so d3 CSV basically returns an array of objects and each each of those objects has a key of date and close and then the the values so date will have this value and enclosed will have this value and and then once this is in there it'll loop through that array and so yeah each key each each array has has an object with a date key and a closed key and it does some parsing so it parses the date and we see it's in this format here and then this makes sure the plus makes sure that it's coerced true to a number so really our skill is to turn this structure into something like this so an array of objects and each object has a date and a close and and the other thing to note here is that you cannot actually use d3 CSV in in data studio because you're not allowed to load files that are hosted somewhere else and and that's actually okay because one of the benefits of using data studio is lurking that you can just use a built-in connector like Google Analytics or Google sheets you don't want to have to import export and then upload CSV files all over the place so the first thing we'll do is we'll just take get rid of this wrapper and just keep the callback function so to speak so I'll delete this and and yeah so now how do we get data this into an object that looks a little bit like this and for simplicity I'm going to keep date and close as before rather than rename everything say to date or or metric all right so the first thing I'm going to do is I'm going to rename this this input parameter to data in to distinguish it from this data I'm just going to be lazy and leave the data here and I'm going to comment this out so that I remember what it's supposed to look like so first thing I'm going to do is I'm gonna reference the the object here so data in will look like this so tables default we're going to do tables dot default alright so this object or this variable will have this array of objects and so now what we want to do is loop through that so I'm going to call this data remember that I'm lazy and I just want to keep reusing data and we want to have a shape like this so I'm going to loop through table list map and and so each object in that array I'm going to call it row and then what I want to return is is an object right so map gives me an array and then and each array element will have this object so I'm going to reuse the same naming so date and close so date and we'll use the parse time function and let's see how we can pick out the data that we need a row so each row and I'm looking for dim ID which itself is an array so I'm going to take the first value there and then the other one is close and that will just be row metric ID and this is an array itself so we'll pick out the first number there and and that might be it so the one thing will that we'll have to pay attention to is the parsing so remember that this visualization here has the date in this format whereas our data looks like this so we'll have to change the the parse time function a little bit so it's the year capital since it's two four digits and it's the month and it's the day without any any hyphens and save that and let's see if this actually looked good locally npm run start ah not too bad uh obviously this is a little bit screwed up and the display is a little bit funky but our axes look good and inspects and no errors so not too bad so why does this look more like a random walk than than the line shot that we're expecting to see so there's two things going on there so first of all the styling but what data studio does is it likes to aggregate data so in this case aggregate and sort data so in this case it actually sorts the metric in descending order and what we want obviously is sort by date so we'll need a little helper function to rearrange the the data array so that it's sorted by date and not by the metric so that's that's simple to do we'll just use a helper function we'll call it function sort date by ascending it takes two two values a and B you know it's basically and comparing and we'll return because we know the the object that we'll send in has a date key so we'll do date - B date and then what we can do is on here we can sort this and pass in that that function sort date by ascending save this will rebuild automatically and yeh so this this look looks much better already and but we have to fix the display so let's see how we can do that so how do they use get the blue line here and actually what they do is it just uses some very simple CSS and this is the the rule for that so we'll grab this and with community visualizations you can actually use CSS and there's a file here which is empty now so we'll just add the the CSS rule save that and then we go back here yay the section looks great this is this is what we want now the next thing to do is this is just local let's see we can get this to display actually in data studio so we'll cancel out of the local dev server and then what we need to do is we need to set local to false because now we wanted to deploy it so I hit save and then NPM run build dev so first we need to build it locally and then we do push dev so this sends it up and obviously the the viz will be the same so let's go back to the data studio and we should be able to just reload this yay cool excellent so let's just make sure that we can use a different metric so the nice thing about data studio community visualizations is that it's very dynamic as soon as you choose a different metric it should update the chart and in fact it does but there's a there's a weird behavior that you may be able to see and I'm going to show it to you now it actually appends a new chart each time you change anything in data studio so we'll have to fix that this visualization is basically static right it sort of runs once and you know loads the CSV file and that's it but in our case that data object is sent in all the time so we have to make sure that each time this this function is called to you know to get rid of anything that was there before so that's that's pretty easy to do so before SVG we'll basically just use a d3 helper function so we'll select the body this is basically our route and then we do a select all we want to select all SVG SVG elements if they're there and then remove them from the Dom that's it oh sorry removing from the Dom so if there's any SVG elements we'll remove them and then once that's done it'll recreate this so save this and then rebuild it and then we'll push it up again refresh this okay great and just to make sure that this is working now let's change the metric back to pageviews oops pageviews great and the chart updates and we don't have that issue anymore now the other thing that we'll have to take care of is the display so right now the the line width and the color of the of the of the visualization is fixed right but we want to have people use you know the styling options so that they can choose the line width and and and the color so let's let's work on that next okay so the final thing I wanted to show you is how to enable users to customize the appearance of this chart and specifically I want them to be able to change the color of the line and also the width of the line as you'll recall we basically hard-coded the appearance of the line in CSS but what we want to do is is use the the panel here the styling to to add options here so that people can set the line and the width and for that we have to look at another file which is indexed JSON we haven't looked at this yet but it it is a config file that also gets uploaded to to your Google Cloud storage and data studio then uses that for their community visualization so and actually if we look at this some of these look a little bit familiar so we have labels concepts label a dimension a metric minimum maximum minimum maximum so let's see whether shows up so it's like here concepts a dimension date and a metric pageviews and you'll see we we said min max so minimum of one maximum of one so this means we can't actually add anything here so this is where that config file manifests itself in data studio so we want to get something into the style so I'm not going to spend too much time talking about how to set this up except to point you in the right direction so you want to look at reference and then manifest actually I beg your pardon config and you see here this is what we had before an array of the of IDE label elements and for style we have this so let's go back here this is empty now and I'm going to just paste in what I what I have here and you can sort of follow along so I have an ID I call it style options and then what I'll call this and I have two elements one for line color and one for line weight and it has some default values so this is the default color and then the default value weight so let's just say this and I'm going to re-up load this two two data studio I just want to make sure that this config is is up there so I'm going to go back to my visualization click on that style and you can see now that I have these options here so chart line options I have a color picker and then align with picker obviously these are not linked into the chart yet so this is what we'll do do next all right so we'll go back to index J s and you see that the that the class line is that is applied here the attribute so I was going to comment this out and so we'll have to figure out how to add this now to to this visualization okay so the styling information will also be sent in via this this data object so in local message this is the data part and these other fields that get sent in just going to minimize this there's actually style object and it's an empty object right now so let's see what it looks like once we have styling applied and then you'll recall that there's a way that we can see what the data looks like using NPM run update message so let's try that NPM run update message so this will upload that sample viz with with a text information will refresh this ok view I'm gonna click in there a copy and then we'll send this in here into local message a V say that and what you see here is that now we have style information whereas before it was empty now it has line color and it sends in the value here we haven't said it yet so it's still the default value that they are the same and then what's get what gets sent in is its value too because the default value is 2 all right so we already know how to deal with the data part so now we just have to figure out how to how to pull this out and use it in our in our code all right so it's style line color and style line width okay so I'm just going to paste in the code for that and I'll walk you through it so I'm declaring a variable style it's an object and it has the keyline color and the value that it takes is it comes from data in which is from here and then it looks for the style and then the line color and the value and if that's undefined or rather if it's not undefined then it'll take the the actual value if it's not set then it will take the the default value and the same for the line weight so if this if the line weight value is not undefined then we'll take the value and if it's undefined then we'll take the default value which again we can see here values declared so we'll take that if some of this isn't declared then we'll take that okay so the last thing that we need to do now is well how can we take this data and apply it to to our code here so I I commented out the class and what we want to do now is just basically add this as attributes and we'll use the stroke you can see you again stroke stroke with Phil and and we'll set these directly now so the stroke we're going to set this to style dot the line color and then the other one is a true Phil oops and we'll set that to none that's not something that we'll all set right now none and then the last one is is stroke width which will pull from style dot line weight okay take get rid of that so I'll save that and I think that should be it Oh actually and will will take well take this out too because we don't need this anymore I'm gonna save that and now we'll build it and then we'll push it up there okay drum roll fingers crossed let's refresh this great Hey so this this is our chart and now we can click on it style and change the color and then the width ooh excellent so now you have have a chart that uses the power of data studio to customize it and obviously you can think of other customizations for your particular chart so to recap this is a relatively simple example of a line chart that uses d3 and we talked about the fact that you need Google cloud platform so GCP you need JavaScript node or NPM locally you need a code editor like Visual Studio code and you basically build the visualization in JavaScript and and as I said you you basically build the date I mean there's no external loading of assets or data so so that's another thing that you'll have to pay attention to but on the other hand you can connect directly up to up to these great connectors whether it's Google Analytics or Google sheeps sheet hope this was useful for you and thanks for spending all this time if you made it this far

Channel: Michael Whitaker

Views: 3,761

Keywords: Data Studio, D3, community visualization, Javascript

Id: 7axX1R_ttOI

Length: 43min 1sec (2581 seconds)

Published: Mon Dec 09 2019

Reddit Comments

[u/hat_like_dad]

Great job! Really fantastic. Can’t wait to try.

Any cool ideas?