KotlinConf 2018 - Building Server Backends with Ktor by Ryan Harter

[Music] [Applause] [Music] welcome my name is Ryan and I work at a little company called pic site where we make creative consumer apps if you know me from Twitter then you most likely better than me no my son Milo who's somewhere in the back so feel free to say hi to him somewhere unless he's napping so anyway so today I'm gonna talk about doing some server development in Kotlin more specifically with a library from jetbrains called Couture and so what is couture so the tagline on the website for Couture reads easy to use fun and asynchronous so what is couture I like to think of Couture as a library that lets you build composable DSL based web services in Kotlin so before we get into some code and this talk is going to be a lot of code we're gonna go step by step building a project that I've been working on but before we get to that I want to talk a bit about how Couture works specifically about its composability so the main actor in a Couture application or server is the application object all right this is what accepts requests from the servlet engines and returns responses when you build a Couture web app you first set up the engines and Ketu our applications have interchangeable engines and servlet drivers that allow them to work with several different servlet engines the standard servlet driver allows it to work with auto scaling environments like Google's App Engine in addition to these drivers you can create a k2r application by installing features that add functionality to the server in this adds functionality like routing serialization HTML templates static content authentication and much more many of these features are built-in but you can also get them from external dependencies by adding simple gradle dependencies or you can even create your own custom features so now I want to talk a bit about this project that I've been working on and it is a server project so you're in the right talk so I've been working on this server project that I chose to use Kay tour for pretty early in Kay tours development just so that you know K tour is almost released it's 0.9.5 right now and one of the big reasons for that is that it's largely based on Co routines and so we need Co routines to come out of experimental before Kate or can be publicly released so anyway so this server project that I'm working on is meant to work with an app right I have an app and we sell in-app subscriptions now according to Google's documentation when you work with subscriptions or in-app purchases or anything like that you're supposed to verify that purchase against your own server that way if somebody has a pirated version of the Play Store or some other way that they can circumvent actually paying for your in-app purchases you verify on the server an environment that you control to make sure this is a valid purchase now what the server is going to do is reach out to the Play Store api's via the web and it's going to make sure that there's a valid transaction in the Play Store API that matches up with the purchase information you got on the device now the last piece that I needed in my system was a web interface for administering this so I want to be able to have a web console where I can manage or cancel subscriptions all of that sort of thing so this the the main point of this is verification of purchases but the other nice thing about this is that it easily allows you to adapt to a subscription sharing model so in my app pigment it's a coloring book a lot of people like to color on their iPad right their tablet but maybe for their phone they have an Android device if they subscribe to premium con on the iPad on the phone they should also have that premium content and so a system like this will allow us to simply add some sort of user identifier and now when a user logs into the app we can see do they have a subscription not only on the Android device but on any device so that's the system overall now let's get into some code so here is one of the most basic servers that you can write using Kate or so to look at this we have that application that I mentioned and we actually make an extension function on the application there's a configuration file that that you deploy or configuration depending on if you're running locally and development or on an app server it's really simple and just for simplicity sake I'm gonna keep it out of this talk we're just going to be looking at code but that configuration is what points the Couture server at the particular extension function you want to use so with after that extension function we have routing and this is a feature that we're installing this is some shorthand but this is the main container for your code so we're gonna make a post route at the verify path and that post route right now is simply going to return hello world so let's run this and see what happens we're gonna use curl since we're using a post route we can't just use the browser because that only does get requests so we're gonna use curl to make a post request with some empty data and we get back hello world and hooray we've made in five lines of code our first K tour server so to make this useful though we need to do more than serve static responses and what we really want to do for a JSON API is return type two responses so that everything's types pay type safe will return type responses what we want to do going back to the server code is be able to have a response object whatever that might be for a certain for a certain API call in this case it's a simple response object that has a status that's a string and you can see I updated the server up there to respond with that response object this is ideally what we want so let's see what happens we plug that in we run it again with curl and nothing we get back nothing we could you know play around with running curl in verbose mode and and figure things out but wouldn't be nice if the server would just tell us what went wrong so here's where we get to caters composable functionality what we do in order to get this type of information back is install a feature called status pages and this is the standard feature installation notation so status pages is a built in feature and what it is is it's a way that you can catch exceptions in this case we're gonna configure it to catch the throwable which is just any exception that might be thrown and you can have it respond however you want in this case we're going to respond with just the text the message of the error and a response code internal server error you can set up status pages so that there are different responses for internal server errors or authentication errors anything like that and return whatever you want and we'll see a bit later on how easy this makes the rest of our code so now that we've got this feature installed we still have a very simple web server here we run the request again using curl and we get a message back can't transform this request content to response so now we know what's happening the server doesn't know how to translate that response data class that we said that we gave it into anything so let's go back and we're gonna collapse the status pages since to make things a little easier to read and as you might imagine we're going to install another feature in this feature is called content negotiation now this is nothing new that jetbrains orkut or invented content negotiation is the standard mechanism by which a server and a client negotiate to determine what type of content they're going to communicate with and you'll recognize this with different HTTP headers like content type and accepts so when client sends a request there will be an accepts header and that'll tell the server what type of data what mime types that client can accept and then when the server responds with data there will be a content type header which will say this data has the content type application JSON or text plain whatever that might be so we do a little configuration here and what I like to use for Jason I'm an Android developer I should mention is I like to use a library called mocchi some friends at square created this library and it does a really good job especially with Kotlin to serialize json in a type safe way so what we're gonna do to enable that is simply add the JSON class annotation and tell mocchi that at compile time I want to generate an adapter and this is all if you don't use mascha if you use JSON or jackson or something else then it's all the same you can simply add ji-sun or Jackson converters just like this so now here's our simple server so let's run this again and see what happens so now we run our request our post request to our server and we get back Jason just like we wanted what we gave the server was an object an instance of an object and we got back this jason so perfect that's exactly what we wanted and again here's how simple this server really is so we're just going to collapse that so we can move on and so that was about sending type data in the response but this is a JSON API so we also want to get type safe data out of the request so let's take a look at type 2 requests so if you remember what we're doing here is we're verifying in-app purchases from a device so what data do we need here's the documentation from the Google Play API the developer API this is how you connect to the developer console and in order to get subscriptions which is specifically what I work with we need a package name a subscription ID and a token and this is when you make a purchase what you get on the device so that's pretty simple let's just create a request object that contains those fields as I mentioned before I also want to be able to share subscriptions so I'm adding in a user ID which in this case is an arbitrary string if the user doesn't login it could be a firebase authentication token from an anonymous user or really anything else that will uniquely identify that user across platforms so we need to update the post route here so to do that we're going to take that request and the way that we do that is with this call object so you'll notice that a lot of this in Cato are looks like a DSL and that's exactly what it is it's a it's a DSL to make your web services and this inside your route is a route object and what you have on there is a call and this call is a wrapper for the request the response the environment of the application and a few other items that sort of travel throughout the pipeline as a request flows through the system on that call object on that call object you can receive items out of the request and by using those brackets we make this a typed request and we tell it that the type that we're expecting is requests now if the request can't be deserialized if mocchi can't make a request object out of this this is going to throw an exception which immediately you might think this is bad but if you remember earlier we set up status pages and so the server is automatically going to handle that exception and return an internal server error with the message of that exception in in development that's exactly what we want we want to share the internals of our system in production we might change that to be a generic you know something went wrong we'll look into it message once we have that request for now we're going to use that call object again to simply respond with that same object just so that we can go one piece at a time so this is our server pretty simple before we run this we need some JSON and so I'm just going to put some JSON in a file this is what it looks like you can see we have a user ID package name Product ID and token and this is actual data from from my app pigment so this is going to be a valid response so now we make a new curl request we have to set the content type and that's what tells the server that hey I am sending you JSON data we use the D parameter to point it at that file send it to the server and we get back that data this looks super simple right the server's just responding with whatever it gets but if you think about what's happening here the server got the data it deserialized it automatically by me just calling this call dot receive method the server took care of deserializing it based on what the client told it it was and it gave us an actual object we returned the object and the server took care of converting that to JSON for the client so this is fun but not super useful so let's build it up some more so what we need to do is take that request and do something with it and that brings us to using Qatar with external components and I'll show you in just a minute what I mean by external components but first let's talk about what we need to do so we get the request object out of the server request and we need to find a valid subscription either cached in the database or we need to look for it remotely on the Play stores API if we find it remotely but it wasn't cached we're gonna cache it locally and then we simply return that subscription we found or we return a 404 saying we couldn't find a valid subscription based on this information you gave us so what are these external components well we have an API this is going to be the Play Store API in the the library that I'm working on this is actually extended a bit so this is a generic interface and my version of this will connect to the Play Store API the Apple App Store the Amazon App Store the unity store it'll connect to all of that and try to find an appropriate purchase wherever it can that's how we do the shared subscriptions now I wasn't gonna do this until I don't know yesterday afternoon but I realized that this all of the code in these slides is so close to a fully working example that I wanted to share with you the code for my Play Store so if you want to download these slides and and take them in making a work working verification example you can do that so here we've got the Android publisher that we create lazily the code for that is in the getting started guide on the Play Store API documentation so I didn't include it and all we do here is create a request to the publisher API we give it that information it gives us back a response we convert it to a subscription now if you look at this what you'll notice from all of the co-routine talks that you've been going to over the last day and a half that we're using co-routines here as I mentioned before Couture is an asynchronous server and it's all based on co-routines initially I thought why is that important it's sort of a cereal thing right a user makes a request and then they get a response and they're used to waiting for that it's not a big deal but what this does is allow Couture as a server to be extremely performant because every request that comes in if we have to make a request to the API this external system which is going to take n number of seconds we don't know how many we don't want that server to stop waiting for this request before it can go to another request or to have a use a separate thread for every single request that comes in so by using co-routines we can spread out that load while while this request is being made to the server other requests can be handled on this thread and it allows us to use the efficiency of co-routines which is sort of a lightweight threading model as I'm sure you've learned on the server so that instead of having to scale this server up and add 10 servers to handle a hundred thousand users we can hopefully use three or four servers so anyway that is our API the other external service we talked to as a database I'm not going to go through implementation here there's examples to work with using JDBC h2 different different server drivers I actually use the App Engine datastore in mine because I run my app in App Engine but this is pretty standard there's a few methods to get subscriptions out of the database and wanted to save subscriptions to the database and you'll notice all of these are suspending functions we do the same thing with co-routines so how do we use this in our app that's really up to you for this example I'm gonna just make some local variables that are you know a totally real API and an in-memory database and you can serve these from from shared resources from Singleton's from using dependency injection however you want to get them it doesn't really matter at the end of the day and then we're simply going to use this in our request so we try to find a subscription in the database if it doesn't exist we try to get one from the API if we get one from the API we save it to the database and that helps us not bypass the Google Play stores I don't know hundred thousand requests a day limit they ask you to cache these for at least 24 hours and then for this last part if the subscription is null we return not found which is a 404 with a message subscription invalid and if we found it we simply return the subscription so let's take a look at that we're gonna use some valid JSON this is the same Jason that I used before so it has the user ID the package name subscription ID and token and when I submit that request you'll see I get different data back because the server the the Play Store API gives us a little more information and this is one of the main drivers for this as a as a project for me because you'll notice one of the pieces of information you get here is the expiration date of a subscription Google quick sidebar google has this requirement that since they don't pro rate refunds if somebody cancels a subscription you have to serve to the end of the subscription but they also don't tell you on device when the end of the subscription is you might think that's simple I'll just if it's a monthly subscription I'll do it to the end of you know a month after their last renewal but you also don't know are they in a trial are they in a promotional period or any of that so this Web API allows us to understand when the expiration date is have they cancelled but they're still in that when did they actually start everything all of that so back to the talk if we try this out with some invalid Jason this is I think for this Jason file I simply added a character to the token to make it an invalid token you can see we get back that message subscription invalid and that's a 404 which curl will not show you so that's it so we have a REST API here it has one post request but more can easily be added so let's move on to that admin interface because this is something that we're all going to need in our web services to make them easy this is our web server right now let's collapse that down make things a little easier to read what I want to do for an admin interface is I want to be able to see a list of all of the subscriptions right so I'll have a get request at the subscriptions URL and you might think we've already done something like this I'll just get all the subscriptions out of the database and simply return them so if we do that what happens when we run it in the browser is we get the JSON data because all we've told the server is how to convert objects into JSON so that's what we get at this point and that's correct but not super useful so let's go back and take a look at how we can change this as you might imagine we install a feature in this case we're gonna use free marker free marker the Apache free marker template language is a well-known Java templating language and we'll configure it to pull our templates out of the resources directory in our project so that's pretty standard configuration stuff there and then what we need to do is update our response here so all we do is we set free marker content there we give it the subscriptions and a map of data and then we tell it which template to pull from so you can see super easy let's look at the template real simple right so we're gonna go through this line by line and eventually we'll have a working system that's not true all of this is just standard HTML so we're going to ignore most of this and just look at this part this is free marker templating you can find more information at the free marker template website this isn't a free market template talk but just to point out we passed in subscriptions we can iterate over that here make a row in our table for every subscription and we use the mustache dollar sign mustache to pull out the data of each subscription into a cell it's pretty simple so let's go back to the browser and refresh that and see what we get there we go we have a full template so this is all driven by the server and aside from some standard HTML it was a couple of added lines of code but now we have a problem you can see how all of the problems sort of build up like a snowball where the whole world can see our entire list of subscriptions so we want to add some authentication so going back to our server can anybody guess how we add authentication yeah we install a feature this is I take it I'm getting repetitive exactly so we install a feature and it's called authentication as you might guess and built-in Cato has a whole bunch of different authentication mechanisms because I'm lazy i'm gonna use basic authentication which i'm sure you all know is extremely secure and a great way to secure everything basic authentication just means i don't have to make forms and that kind of stuff but Couture does support out of the box for Mothe nth occation basic bearer authentication I think ooofff all sorts of things they have a real long document if you want to integrate with with Google's OAuth that even tells you how to set up OAuth from Google side which seems to change every other week but so anyway we make a basic authentication type and you notice we give it a name which seems a little weird but I didn't really understand when I just started with Kate or why would we why would we give a name to our Thoth in ocation mechanism and the point is you can have different routes that have different authentication requirements that may be authenticate against different systems or may be authenticate as an admin user versus as a regular user you need to use some different authentication so we're going to set this up realm it's part of basic authentication has to do with what's put in the headers the value is not super important for a project that's not even written in an IDE but written in keynote and most of these authentication mechanisms have this validate block where you're given credentials the credentials are different based on what type of authentication for basic authentication you get a name and a password and you do with it whatever you want in our case I wrote this I didn't write I typed off service which just returns some static values for this case but basically you you do whatever you want with these values and the idea with this validate block is if you return a value then that means authentication succeeded if you return null authentication failed and the server using your status pages that you installed before will handle that accordingly what you return is pretty close to not at all important from the respective khutor there's an empty interface called principle which you just put on any object you want and that becomes the object you can return from here and then you can pull that out later and that means that you can store if you have you know multiple tokens or if you have a user object that's backed by different things or composed differently Kait or doesn't care so let's collapse this and look at how we use it in the routes and KTR decided to make this really difficult all you do is you wrap the in an authentication block and we're done this is now an authenticated route we added here the name of the authentication we want to use if you only have one type of authentication omit that and you're fine but we do want to use that data right I mentioned that there's a user object in here so again very simply we're going to just expand that out so we can look at it and then we'll just get that user out of the call object the call object if you use authentication will have an authentication object property on it from which you can get the principal and this principal is whatever you returned from the authenticate block so in my case it's a user that has I think it has a user name and password in like a first name field or something not super important and then we're just going to add that to our template then we go back to the browser refresh the page and we're presented with the login dialog and this is how chrome handles basic authentication its standardized this is why I did it because it's easy so we type in some credentials there we hit sign in and we're logged in and you can see up there in the top right that it now knows my name because I gave it a user object so that's it right we have this admin dashboard that doesn't really do anything it shows us stuff but the last thing that I want to talk about is forms just to give an example of how to make things more interactive on the web if you're not using Ajax which I'm not in this case because I want to keep it simple I'm gonna use simple forms so let's take a look at what we want here this is what we have right now we have a list of subscriptions but I want to manage those subscriptions so I'll update it here and you'll notice that I added these buttons over on the side and these buttons are just going to allow me to cancel or delete subscriptions so pretty simple you might think of a few ways to implement that I'm going to do it with forms in which case I'll show you each one of those buttons is going to be a form in and of itself so to do that we're gonna add a route under our authentication it's gonna be a post route and we're going to receive parameters so when you submit a form you have parameters just like in a get URL you have the you know name equals value and the same thing in a form only that's in the in the post body so we get those parameters this parameter is effectively a map it's a map of lists of strings with some added functionality to make it easier to query so what we do is I'm gonna get the ID of the subscription out of the parameters and if that doesn't exist since this is a required parameter I'm gonna throw an illegal argument exception again throwing exceptions into the wild here but we have that status pages set up so if we get an illegal argument exception on a HTML request our status pages are set up to know how to handle that and tell the user gracefully what went wrong right so we'll do the same thing for the action this is going to be delete or cancel and then pretty standard Kotlin code we act on the action if it's delete we delete that item from the database if it's cancelled we get the item out of the database and then we put the subscription back in the database but we do it updated so with canceled flag and then we simply respond with a redirect back to the list of subscriptions to update the templates again standard HTML but we're going to be using some some FTL tags free marker template language tags we make a couple forms it's pretty simple it's a post request to the subscriptions action we have some hidden inputs for the ID and this is what's going to get submitted with the form and a button that submits it and then now you can see we go we click cancel on that subscription and the page is going to refresh and it was very small but what you could see there is that the cancelled status on that user was changed to yes and the button to cancel is disabled because we redirected to that list with our updated data so that in a nutshell is a fully working Kate or system that we did in a little over 30 minutes I have plans to open source the system if you want to be doing you know purchase validation in any of your systems that will be coming later this fall as all you know software release promises go but if you want to learn more and if you want to take a look at Kate or you can do that on the web Kate or IO github this is all done in the open on github Kate REO Kate or the slack channel is reasonably active in that you'll get responses if you want them the Colin Lang slack there's a Kate or channel if you want to talk to me I'm easiest to reach on Twitter it are harder and if you want the slides you can get them at Ryan's dot link slash Kate or and with that I would like to thank all of you for coming to learn why servers hurt Colin you

Channel: JetBrainsTV

Views: 32,314

Rating: 4.9617682 out of 5

Keywords: jetbrains, Ryan Harter, kotlin, kotlinconf, kotlinconf18, kotlinconf 2018, ktor, DSL, coroutines, google, app engine, API, TEST

Id: V4PS3IjIzlw

Length: 32min 40sec (1960 seconds)

Published: Mon Oct 15 2018

Reddit Comments

[u/the-real-guanabanana]

This was actually one of my favourite talks. It shows how simple it is to create a web service.

Very clear and concise presentation!