Elixir Application (part 1)

when you want to write some elixir code starting a mix project is not a bad way to go it provides structure for your tests dependencies and configuration typically an elixir application lives inside of a mix project application in elixir refers to a module that is used to wrap the supervision tree that module will provide callbacks for starting and stopping the application as a unit the code inside of a mix project doesn't have to be part of an OTP application but it will still be convenient to use mix project for its structure mix is a tool that fetches dependencies runs tasks compiles project code and knows how to run an application it does quite a few different things in this presentation we're going to use mix to create a project go over project components introduce the application module and start building a real elixir application where we'll apply gen server and supervisor concepts instead of basic structural examples though this project will be doing something a little more interesting we're going to rely on gen server and supervisor modules if you're not familiar with those please check out the previous videos in this series that introduce those modules and go over their usage and syntax for our project we're going to start a package delivery company we will need to deal with things like distribution and routing which means we need to solve a pretty important problem in the delivery industry and that is to figure out where everything is located to calculate the distance and delivery routes we're going to need some geolocation data the data set we're going to work with is going to be fairly simple the Three Valleys we're going to use are the postal code the latitude and the longitude so the first requirement of our delivery company is to get geolocation data and build a reliable geolocation lookup service that will help route the packages to the closest distribution center and later on help set up an optimal delivery route but before we could do all that the first step is to start our project to create a project we're going to use mix what a build tool that comes with elixir to start on your project type mix new L hex delivery and let's get into that directory and see what we got this is what's inside the directory that was generated all the code is going to be in Lib with the only thing here so far is the module that was created by default based on the name the test directory is going to have tests using X unit and the project file mixie XS it's going to have everything mixed needs to know about the project including dependencies compilation and how to run the app one important note is that what we created is a mixed project note that we are using mixed project module here at the top project and application are related but not interchangeable terms the application refers to the elixir OTP application meaning the code for the entire Supervision tree the project includes the application plus the dependencies tests and all the configuration details take a look at the two methods defined in this module we have the project method and the application inside of the project method we have the name of our application the version of our project the version of elixir we'll be using and a list of dependencies that's defined as a separate method here build embedded option is going to be true in production what it means is that all the code that's necessary to run our application is going to be included in the compilation in test and development it's more convenient to link to some things that exist locally rather than include them in a compilation every time start permanent option is also only going to be true in production it means that when our supervision tree gets shut down the virtual machine will crash which won't be convenient in testing development environments inside of the application method we have a list of applications that need to be started before our main application we also need to designate a module that will wrap our main supervision tree which we'll do at a later step before building our OTP application let's get some geolocation data we can get poster geolocation data by heading to census.gov once there click on geography select maps and data find our geographic index files in the right column and select the latest year once there you'll find our postal code information on the bottom when you download it you'll get a seven column tab delimited file with the first column being our postal code and the last two being the latitude and longitude let's place the census file inside of a data directory in our project now we need to write a module that will parse this data and load it in a format that we'd like this will be a plain elixir module named space inside of the postal code directory the first thing I'd like to do is to get a list of data rows by loading in our file and then taking that file string and then splitting it by the newline character at this point it will be a good idea to verify that our data rows loaded and properly the best way to do that is to write a test this file is named after our module where the word test appended to it notice that the extension in this file is exs meaning there will always be run as a script without being compiled we're going to alias our module which defaults to whatever comes after the last dot Union though we can refer to it by the name data parser without the rest of the prefix our test is going to be incredibly simple in fact technically it's not even a test since we won't have any assertions to begin with all we're going to do here is to call parse data method on our data parser module grab the results and just print out the very first element just to be sure we don't get any errors and everything works as expected Ronni mix test will run all our tests so you can see here we get our tab delimited row which is what we expect and we also get a warning saying the variable header isn't used which we're going to fix right now we can ignore our header variable by starting with an underscore meaning we don't intend to use it since we're dealing with a tab delimited file the next step is to map over every row and splitted by the tab character giving us a list of columns once that's done we can run the same test from before to confirm that our partial still works it looks like it does since we won't need all valleys from our file we can map over each row again pattern match on all seven columns and just grab the data that we need a we'll end up with is another list of postal code latitude and longitude when you run our test we get a no match error which is probably the most common aerial scene elixir what it means in this case is that whatever we specified on the left side did not match whatever was on the right since we did not provide any exact values in our list I suspect this is simply because we provided a wrong number of elements looking back at our code we can see that we specified eight elements instead of seven when we run our test again we still get a no match error this time nodes because we couldn't match a one element list that just consisted of an empty string this is because our tab delimited file had an empty line at the very end to fix this issue we need to filter our rows to be sure that we're only working with the data that we expect will change our map method to a filter method and at a case statement filter method of all no hang on to the rows that match a pattern or a condition that we provide in our case all we care about is that the list has seven elements in it everything else is going to be false having our data in the proper format we can add the map method that we had before and after that's done we can run our test again our test passes with a few unused variable warnings from the filter method 20 to fix after that's done we can move on to the next step what we are going to do is transform each row from a three element list to a two element tuple which we could then easily load into a map we also need to clean up our latitude and longitude values by getting rid of all that extra space and converting them from a string to a proper floating-point number once we do the same for longitude you can take that result and load it into a two element tuple with the first element being the postal code and the second element be another tuple of latitude and longitude let's run our test and see what we get looks like we got our two-element up over the postal code is the first element and the latitude no longitude is the second the only thing we'll have to do now is to load our list into a map I expect our test will still work but now since we actually have the data in the proper format we can write a proper test at this point would actually have proper geolocation data in the format that we want to test it I'm actually going to plug in a familiar postal code and expect to get a latitude and longitude as my value here I'm also confirming that my value is going to be a two-element tupple in addition to that I'm going to make sure that both latitude and longitude a proper floating-point numbers when we run our test it passes which is great looking back at our code I feel like it's starting to be a little bit difficult to follow especially if you haven't looked at it in a while I would also like to see if there is a way to improve this test performance in any way we can start doing some cleanup by replacing this fairly trivial iteration code with the ampersand anonymous function short code it's always a good idea to run a test just to see if our improvement broke something moving on we can take the code inside the filter and move it into a separate private method doing that has an added benefit of naming the method just so it becomes clear what the code is supposed to be doing in this case our data row method is going to check if our row as a Dana row and the question mark at the end of the method implies that the return value is going to be a boolean so now we can just replace this code with our new method and see if it works it looks like it does so let's keep going we can do a similar thing with our row parsing method here just start a new anonymous farce data call his method move all the code from our map method into here delete the code in the map and replace it with our new parsed data columns method and again it's really important to keep an eye on our tests as we're doing this notice that the code for parsing latitude and longitude is identical even though it's trivial it's still a good idea to move this into its own method just to avoid this kind of duplication let's check in on our tests thank you going the last method we can abstract is the row format transformation method where we actually load the data from a three element list into a two element tuple with the same idea in mind we can create a new method called format row take all the contents from our map method above and pretty much move everything down here we also need to parse the latitude and longitude in the same way and just replace the contents here with the format row method when we run our test we get a failure now you can see that the problem happened on line 32 and it looks like we have a missing parenthesis so unfortunately when we gained in line numbers we lost in clarity so what I intend to do now is to add a few comments explaining exactly what this method does and move out the latitude and longitude function calls outside of the return value so let's add our comments or we explain the methods intentions and give an example what they return the value is supposed to be next let's move out the parse number function calls from outside of our return value into the wrong variables above then change the value in the return just to use the variable names hopefully now our test passes and it does one thing we can do in terms of our code performance is to change from greedy enum module and to lazily evaluate its string module this should make the processing much more efficient because each iteration will not have to wait for the previous one to finish before it gets started I expect our tests will still pass and will be slightly faster well looks like it's 0.3 seconds faster or about 25% in this case so after all that we ended up with a pretty efficient geolocation parser method that's not too bad to look at next we're going to write a gen server module that's going to load and store this data so that we can access it dynamically here I'll be passing an empty map to the init method which isn't necessary but I find it helpful to specify it what data type our gen server state is going to hold we can provide a name for our process which we can then use as a reference the benefit of that is that we don't have to worry about tracking the process IDs it can always refer to it by its name the drawback of naming a process is that we won't be able to run multiple processes of the same type we'll call our data parser module from inside the init method we'll add an alias for it so that we can refer to it without its prefix the data parser module is going to be called each time we start or restart our postal code stored gen server process the init method has to respond with ok and the state that our gen server process is going to hold which is our geolocation data in this case when the postal code is the key and latitude and longitude tupple as the value the gen server process will not start if there's some kind of an error during parsing there is also a default timeout on the init method which you'll have to keep in mind if you're working with a bigger data set so now that our data is going to be loaded in we can write a lookup method to get geolocation data by the postal code notice that we don't have to explicitly pass in the process ad to this method since we can refer to it by its name now we're going to write a synchronous callback method that's going to get our geolocation data by the postal code it needs to be synchronous because whoever is going to be calling this method is going to expect the latitude and longitude as the reply remember that the Jin server callback responses must be in this format with the second element being what our caller is going to receive and the third element is going to be the new state-of-the gen server process which in this case just stays the same we can try out this code right from the elixir console type ie X - S mix which will load the code from the current directory and start the console next we'll start our postal code stored gen server process and notice that we'll get an errors which try to start again because we can't have two processes or the same name and when I pass in the familiar postal code to get geolocation lookup method I get back my latitude and longitude it works and then quit the console with ctrl C of course it's still a good idea to write a test for this method which we'll do right now we'll call our test file store test at get geolocation test method and alias the module since we haven't wired up our application will still have to explicitly call start link on our module after that we can call get geolocation method with a familiar postal code and expect back a floating-point number for both latitude and longitude once that's done we can run our test looks like they all pass so that means we get to keep going I would like to add a postal code supervisor module that's going to monitor all worker processes related to postal code and geolocation data will put the supervisor file into the postal code directory inside the live inside the init method we'll specify our children so far we just have one worker to monitor which is the postalcode store module next we'll call the supervised method passing the children and specify the strategy we can try out our supervisor from the console if everything works as expected we won't have to explicitly start our gen server store process since the supervisor would have done that for us when we run it that's going to fail but that's because we were missing the word store from the module name once we add that back everything works we can quit the console with control C and keep going we'll add an alias to the postal code store module so that it's a little bit easier to look at next we're going to create the main or the top-level supervisor that's going to start when our application starts up this supervisor is going to live in the top level directory in Lib and is going to monitor all other supervisors including our postal code supervisor now I want to note that we could have had our store worker directly under the main supervisor here I mostly wanted to demonstrate a supervision tree with some depth keeping the postal code workers under their own supervisor will allow us to keep any problems that might cause isolated and it's also good for code organization once our application starts growing in size now we need to go back to the very first module that was generated by default and turn it into an application with a callback method that's going to start our main supervisor and that's all we'll ask it to do once everything is wired up properly we can expect all the gin server processes under the main supervision tree to start up as well mix is going to call the start callback method by default going back to the console we can expect to use whilst ocultist or get geolocation method without having to start up anything explicitly however we get an error saying that our process isn't running so what's happening we can even confirm that our application is running by typing an application that's start with our application name and that should fail with an error saying it already started what we need to do is go back to our mix file and specify the module that we want to use as our out ocation the module will provide here will be expected to use the elixir application module and will be responsible for starting the entire supervision tree in this case this is just the module that was generated by default that we then turned into an application and now that everything is properly wired up we can go back to the console again and run our postal code store get you location method without having to start up anything explicitly okay now use this method throughout our application that includes our tests which means we no longer need to start it up explicitly here we can take it out run our tests and still expect everything to work so what we have so far is our application that starts the main supervisor that monitors our postal code supervisor that in turn monitors our postal code store worker that contains all our geolocation data in the next video we're going to be adding the navigator module which we'll use to calculate distances between postal codes and we'll add the cache module which we'll use to improve performance over our distance calculations this video showed how to start an elixir application when over gents our run supervisor concepts we learned so far and touched on some code refactoring testing and debugging I hope you enjoyed it thank you for watching

Channel: omgneering

Views: 12,660

Rating: 5 out of 5

Keywords: elixir, application, otp, gen_server, genserver, supervisor, performance, refactoring, screencast, mix, project

Id: EDu3EwTbrFM

Length: 21min 17sec (1277 seconds)

Published: Fri Jan 27 2017

Reddit Comments

[u/zachgarwood]

I like this series. Keep it up!

[u/GigglesClifton]

More OTP goodness, love this stuff. Thanks, Michael.

[u/Spaceman-_-Spiff]

This is a great series of videos. Thank you!