API-Driven Development // Paolo Lim, Flow.io [FirstMark's Code Driven]

[Music] a company called slow and so you know this talk really is about how we build software up flow or how we find how we find our flow through API driven development and so flow is a essentially a b2b software company where we provide a REST API to help ecommerce retailers sell globally and what this specifically means is that we enable all of these companies to be able to do of the specific things they need via API calls to fulfill to fulfill stuff from here going to customers around the world and so you know obviously this is a technology talk so I'm going to talk a little bit about what we do and so for just a little bit of background we run a micro services infrastructure built on on the back end in Scala in play and there are events streams all behind that the front and this are react no js' and we you know all of our infrastructure runs on docker containers hosted in AWS and there's a little bit small thing here about other word flow where did that come from and basically it really means like you're in the zone so how can we taylors get in the zone of selling internationally and for our API driven you know development what does this mean so that developers can be like more productive and really get into the zone really build the product without managing you know a lot of the craft that typically goes along with it and so a micro service architecture if you run a micro service architecture in your organization you probably see some of these things so they're so like a lot of common boilerplate across all of the call across all of the various services like different themes different people tend to do different things differently you know the previous company the previous company I worked with all lot of us from flow had a background of being from flash ale retailer gilt groupe we ran a micro service architecture we are truly like one of the first places that went from a monolith to our to a micro service and one of the hard learnings over there are precisely or precisely this it's hard it's very difficult to maintain a lot of teams will do things differently like someone will write some code in one service another team takes over of that service they have no idea what's going on and therefore they'll just end up rewriting it to do the exact same thing documentation is hard it's hard enough already just to write it that's hard it's even harder to keep it up to date and you know meaningful alerting and reporting around you know common metrics around all the services is very difficult also just because everything is a little bit it's a little bit different so developers get stopped doing all of these things rather than building the actual stuff - you know doo-doo-doo-doo the stuff that they were paid for to solve the problems of their retailers in our case or you know to build whatever it as they were signing up to build so here's at a high level what our what our architecture looks like over here in the over here in the lower part here those are our some of our micro services are about they're about 3035 of them now in production this one just simplifies it a little bit so you know each one of the each one of them is a micro service backed by what typically would be a Postgres database and a lot of them will emit events into these event streams which we have in AWS Kinesis right now and so an example of this is when when we resume typically what we ask from our what we ask from our clients is that they'd send us our catalogue first and so capilla they send us our catalogue with their products we call them items into our title goes through our catalogue service at the mitts events and any of these other things that need to act on them will listen to the events as they happen or you know when they are in the Kinesis stream this proxy layer this proxy layer over here serves as a common layer that our clients integrate with so whether that's whether that's our own flow console which we develop internally or the SDK that our customers use it's all the same they all go through the same proxy layer and so Oh API models and resource is just a little bit here all of our all of our stuff is documented on API dot dot me that's a personal pet project of our co-founder and CEO Mike Brissac and we you know all of the stuff that we work with in the back end is Scala Scala and play as I mentioned we have a lender to enforce consistency across the API as if one person will build something differently some other person will work and at different will work in a different micro service we want to make it look like it's actually just one service so that clients can consistently go onto multiple models or resources they should all look the same and and we you know to produce a proxy to produce a proxy thing there's a there's essentially a builder that gets all of those um that gets all of those definitions merges them together and it's basically served from from the part over here that proxy they they hit that and it actually proxies to one of these other micro services and so this talk really will just walk you through what we what we actually do here so here's a definition of here's a definition of a catalog item this is how we document this is how we document the model here all of the fields or the description sir types and here are the resources for the end points I don't know if you guys can see this but basically you know there's like you know you can put an item you can post an item you can there multiple get end points that do slightly different things and a lot of this structure is consistent across the different models in our API and as I said earlier when we unify when we build a single API definition from multiple ones and multiple services prasena our proxy layer now can be smart enough in hand and enforcing that and so when our clients integrate when our clients integrate with us they may see things like you know they may see things like validation errors around like missing fields over here hey the form requires all of these all of these fields you only passed and you only pass the name error and that and so typically what might what might happen is that it's very difficult to keep this up to date but this one is actually super up to date and you know it's exactly what we're what we have on the back end and so the documentation is also auto-generated so you don't get into this you don't get into this face where a lot of the a lot of the things are you know it's very difficult to write the right documentation we wanted to make it a natural part of our of our workflow so the way this is generated over here just comes from whatever you put over here in the descriptions over here and so everything stays consistent with what's on the canonical model as it is right now versus what's in the documentation site and so this one relieves a little bit of frustration from our clients where you know I've integrated with a bunch of API so before where you have to you go to their documentation site you try something uncurl it doesn't exactly work the same way they said Atwood and so you know we wanted to solve that and this is our way of making sure the documentation is essentially up to date has to be a test to be generated it doesn't you know and it's not an impediment for words towards towards developer productivity and so with the auto generation thing we can also do stuff like generate rubia R will be SDK which our early clients used a lot of our early clients were smaller more were smaller companies that that use will be in rails so you know this is something really essential that we we got so anytime they anytime our API updates the gem can automatically did can automatically be generated and then you have a very clear distinction of you know what some what's available what what changed and and how do you communicate with flow right now and so another one is a our node SDK our front-end beam actually uses as auto-generated node SDK for our front-end as us talking earlier about what what the stack is our front our front end uses the exact same auto-generated code so that it can consistently talk with a back-end API layer and so I was talking a little bit about events before and so event the event models are essentially a JSON payload that we put into Kinesis and they're also defined in they're also defined an API doc we write them like this they're also auto-generated and then what can we end up doing with them not sure if you can see as you can see this but essentially the this one is this one is a processor that we have a consuming service so this service say is interested whenever a new catalog them is uploaded by one of our clients or one of our the data has changed the event will drop into Kinesis and something can something that's interested can easily consume it and you know it's a I think that code is a pretty straightforward and all of the services do look like that and a lot of the tests also can be auto-generated a lot of the things that we worked on really just look like that just making sure that the event is able to flow through from end to end you put in a fake event and do the mock you it's able to get processed and the effect of that is that it's in the database in the end so a lot of this is pretty standard and so we also generate a lot of dub a lot of the things that talk to the database now so you know it's um there's another thing where we use auto generation using the same API framework where we're able to say okay here are the here the columns in my items table and here's some meta information about what I want to generate and then I could generate that generates Postgres schema generates a scala dowse so a lot of this we found at the start we were just copying and pasting everything so this is better to come up with a generator to do that and everything looks consistent across across multiple services so so you know a little bit of time but so far essentially most services tend to be tend to be structured the same way we think it's a very pragmatic approach especially since we're uh we're an early stage series a startup where our screen was supposed to be scrappy we're supposed to be building it a product going after customers and so you want to you want the development process to be as predictable as possible but still not hinder you know developer creativity when they actually are building the core meat of the product so you know someone's actually working on on currency conversions or doing things like tracking a package a lot of that you can concentrate on just doing that and none of the boilerplate around it and so you know this made it easier for people to like move around themes a lot of the services sort of look the same a lot it's made it easier for troubleshooting I don't know if any one of you heard of truck number sometimes called the bus number but basically it's like the number of people needed to be hit by a bus before your company's good so so I I think I think this actually it should actually be higher truck number because more people can be hit by a bus before our company's screwed and and so and the generators also improve over time we add new stuff to it we think things like having the generator actually add an index will make things easier or generating a lot of the helper classes in our Scala in our skeletal so make things easier so you know you can improve those over time and they're all allowed to super-easy because everything just goes through the same thing gets almost everything is consistent I think it's also helped us eat our own dog food it's not something that they here we provide this API to customers but in the backend what we're really doing is you know doing all of this other stuff that that they don't have access to we don't really like that so we prefer to see like if it's something that if it's something that that works for us and we love internally our clients will probably work love it and we found that a lot of the developers are generally happier also in the end because they're able to concentrate on the meat of what they want what they want to do which is actually solve the problem rather than rather than you know all this craft around you know how do you set something up how do you set the new service are probably set the queue all of those things and so you know I know I'm so I'll open the field good questions [Applause] sure yes that's correct there there are times when you know it's a little bit of a little bit of a loaded question some sometimes I feel that a monolith would have been easier for certain things but when you get to let me get the things like especially with issues issues of scaling certain services like some services need you know need dedicated things that other services don't then you know we're pretty happy we chose to go the micro-services route and so individual developers can also feel that they are that they are more empowered they really own something in their specific area that more or less should and shouldn't really bring down the entire than their infrastructure which is what micro services was for you create you create this breakout this well-defined module within the larger system and then someone would be able to really own that and I think for for you know we for the senior developers and the team that was really empowering because they're not trying to figure out like well if I work on this will it break someone else's code and so we're able to either it very quickly with what we with what we have I think I think a lot of the things that people generally say when they miss when they when they miss a monolith I think what they were what they really mean is that well I don't have to rewrite a lot of a lot of the boilerplate that's already there because someone else already wrote it and and what we try to address over here is all that all that overhead that is just really copy and paste and just essentially like such a few values that type of stuff should be that type of stuff should be auto-generated so that people can just be productive in doing in doing what they do okay yeah so let me go back over here so the the proxy layer here is the one that and four so on that enforces authorization we actually have one of the other services that we have around here is an authorization and user service and that one has your it has the user that's a user tokens and whatnot so what happens is when a request comes in with a request would come in with an authorization Heather hits proxy the proxy layer talks to this other talks to this other service and whether or not like if this person for is this you know this requested this API key allowed to talk to the service that it's trying to go to if the answer is guessable forward it'll for the requested our internal infrastructure to that service if the answer's no it'll return a 401 yes the proxy is home go and I think that is one of the that is one of the bigger like question marks early on in the process about like you know are we going to regret building this and I think the answer right now is is no because we're able to do things like you know add rules in there to actually validate the to actually validate the Skee must do things like up casting and you know some of our older partners mostly like shipping cares and logistics companies want things in you know in em in ugly envelopes or they want things converted into something else and we can easily do that there without you know without polluting any of the individual services okay when we when we had our first clients I'd say you know a lot of this a lot of this bottom half over time like we really consolidated on asynchronous events streams over Ceyda past year but our first clients when they appeared here a lot of this layer wasn't built out yet and a lot of these services we're talking to each other by our rascals but the interaction from that top from the top half of the slides through the proxy layer through this micro through these micro services were there when our first clients came online okay thank you [Applause]

Channel: Code Driven NYC

Views: 1,238

Keywords: software, development, engineering, code, programming, startups, product, design

Id: x4VLgRBrcXQ

Length: 21min 0sec (1260 seconds)

Published: Wed Jul 05 2017