GopherCon 2021 - Day 3

[Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] do [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] foreign [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Applause] [Music] do [Music] [Music] [Music] do [Music] [Music] [Music] [Music] do [Music] [Music] [Music] [Music] so [Music] do [Music] [Music] do [Music] [Music] do [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] good morning everybody welcome to day three of gopher con final day unfortunately i know right we've had some good stuff so far we've had some really good stuff so far and some more good stuff to come yeah today's going to be no different than the last days maybe maybe some more exciting stuff even maybe a little warmer too yeah i mean we're starting on a high-ish note it's gotten up to just a blistering 60 degrees i'm starting to sweat we've also convinced aaron to introduce a slight bit of color i hit everyone i hit you all with some blue and some maroon that's some teal is there teal in here that's why they're still looking out for me a little bit of orange stripes in there johnny you stood you upstaged me though with your white and light blue i mean it wasn't hard though right i'm setting the bar pretty high i'm just i'm just saying you know we don't need to really just you know pick a shirt any shirt [Laughter] you know wardrobe suggestions welcome you know i i'm i go pretty deep as you can see but you know wardrobe yeah next year we promise to get wardrobe and makeup yeah yeah absolutely yeah uh so first of all i want to thank all of our sponsors for making this happen please please go check them out on the websites they're doing all kinds of giveaways talk to them on discord talk to them in gophertown many of them have vms's virtual meeting spaces that are running in parallel with parts of the conference uh definitely go check those out there's you know they're running workshops and little game shows there's a lot of fun stuff going on if you're in the market for a new job lots of them are hiring there's some exciting stuff we have tom lyons on stage did one of these we're hiring so crowdstrike is hiring for sure so almost all of the sponsors are hiring so now's the perfect time if you're looking for a new gig go talk to them if you want to attend any of the virtual meeting spaces you do need to have a virtual ticket for that just go to gophercon.com and register there you can also see that the schedule for the talks today and those virtual meeting spaces on the agenda page we've got go time coming up with vote for say overseas the episode i've been most looking forward to all week they're gonna do a family feud style game where they ask you the community a bunch of questions and then they make the go team compete to guess what the most popular answers to those questions are and accurate or not accurate they are the answers if you want to participate in that survey go to go time dot fm slash s for gopher say all right and uh who do we have up yeah let's get to it uh our first talk of the day is matild reynald she's going to talk about quantum computing today which i mean we've had some i'm going to feel really dumb yeah oh yeah yeah i'm going to just be sitting in the corner crying while i watch her talk it's just way over it's not even like here over my head it's like up here over my head so when you say quantum is that like quantum leap like with scott bakula and you know with ziggy and stuff am i dating myself here yeah yeah anybody anybody know what i'm talking i mean we've got dad jokes i've never heard of such things what is this yeah i mean i mean it's gonna it's gonna be interesting we've got some novel uses of go we've seen some at this conference but and we're talking about quantum computing this is a new frontier i think for go that has been not explored maybe at all so i'm really excited for for this one yeah me too yeah yeah matild is a i believe newly graduated phd student um at the swiss federal institute of technology so we're really excited to see what she's got in store for us today um yeah so with that let's get to it matilde take it away hi everyone my name is matthiel and i am super happy to be here and to welcome you to my presentation taking the quantum leap with co so before we start just do quick worry about me i am currently a phd student living in switzerland and this is why you will see throughout the slides and pictures about the alps and the mountains and all this kind of stuff regarding my research interest the range through many applications and topics and for example like privacy enhancing technologies or flight cryptography i also also enjoy looking at machine learning when needed and today we're going to talk about the topic that is very dear to my heart which is post quantum cryptography so it might sound scary and overly complicated but um i will try to make it as fun may i say and simple as as possible today really we're gonna try to answer three main questions so of course we're gonna start with um an introduction and i'm gonna explain really what is going on right so everyone's talking about quantum comes from computers but why does it what does it mean what is this quantum threats really that people are getting so worried about and after explaining a bit what is going on i will motivate really the need for what we call plus quantum cryptography so explain how this threat is going to impact you and your project and why you should be interested in this presentation basically finally i will not leave you hanging with an existential crisis and i will conclude with the presentation of um one library which comes as a solution to address the threat and spoiler alert this library was developed and is available in google okay so quantum right quantum computers on the right you can find a picture of the quantum computer and other of them being um i think kind of pretty they are a true technological marvel um because they can perform some kind of competition ridiculously fast faster than any kind of super super computer could even dream of um it will benefit many many uh research uh regards so like i think about chemistry i think about finances healthcare and so on and so on so this is this is a good start but the issue is that because of these computational powers that um let's say enable people to compute things very very fast they threaten the security of the publicly cryptographic um algorithm that we know and use nowadays something about rsa pgp all those algorithm will not protect our sensible information anymore so that means for example that you're sending like an encrypted message to someone um a person an attacker who has access to a quantum computer can just run the run a very powerful attack and just crack your message open like if it was not even encrypted and reading we didn't clear this this is disastrous right we don't want that and as a side note i think it's very interesting that because of the time frame i don't uh i wouldn't talk about it the not only the probability cryptography is impacted right uh symmetrical cryptography and hash functions also um how to say suffer from a drop in security yet they are not completely broken the way publicly cryptographic algorithms are um so this is going to be out of the scope of this presentation but nevertheless if you have questions about that be happy to answer them the public and cryptographic algorithm that we use nowadays are not going to protect our messages anymore and so what do we even need public key cryptography right can't we replace it and the quick answer is we need probably cryptography so those of you who might will be working on security related field might directly interact with encrypted data some others might be sending encrypted emails at work relying on the bpm solution digitally signing some documents looking at certificates and why not who am i to judge buying some cryptocurrencies such as bitcoins all those technologies are only a sample about everything that we're doing in our online life whose security really relies on public key cryptography and there is no replacement for that we need public cryptography you might be now having a follow-up question which is okay okay we need publicly cryptography but we've been using this trick for many years of um slowly but steadily increasing the the key sizes right it was at first 128 bits then 256 and so on and so on can't we just keep doing that unfortunately we can't because the way we foresee quantum quantum computers to to scale and the powers we we think they'll have we calculated i mean the researchers calculated that for example if you want to use rsa you will need one terabyte keys for your system to be secure this is not memory friendly this is not efficient you do not want to go through the trouble of handling one third like keys so okay we need the liquid cryptography and we need solutions that are not going to come at the cost of unpractical uh unrealistic key size or other consequences and those alternatives are called the quantum safe alternatives or as you might have heard of the pos quantum cryptography so post quantum cryptography has this very nice property of uh in addition to being secure against the classical adversaries that we're used to they are also secure against adversaries that have access to quantum resources so meaning that even if this isis asset adversary was trying to run this very fast operation um against um public post quantum cryptography it would still be secure just as classical cryptography there exist different flavors of post quantum cryptography so we can think about lattices hash based functions isogenys and so on and so on all of these flavors have different properties so some are really going to target um efficiency in terms really of speed and performance uh while the other are going to look and minimize the output sizes so really work with very small signatures very smoky blue keys and so on and so on so my guess is that according to the requirements of your project or the technology that you're trying to make to make quantum safe there is a techno um how to say solution that is going to work with your um objectives right if you're trying to be very bandwidth efficient then i would suggest maybe to have a look after that i would say rainbow and all this kind of stuff which is a multivariate um technique so maybe something that is also in the back of your mind is okay so now i'm talking about the threats i'm talking about uh we foresee things is there something that exists now and why should we talk about this now so to answer the first question um what exists well for now quantum computers are the sage of prototypes right um and why we should talk about it now is because we know that we are super slow to integrate new tools and new technologies into whatever we're working on um so even git i think it's i think it's a nice example because um they i mean they have the manpower to make things move if they wanted to um so gitz knew back then that xiao on a hash function was broken um but they took their time right to um to change it to sha 256 and it took so much time that actually some bad guys um just to prove a point um implemented an attack i triggered the ripple and completely destroyed it so that not my that might not be um super let's see impressive just to destroy um destroy a repository but if you think about it if you think about okay i have a database of my password which is being encrypted well you don't want anyone to go and look at that um and and encrypt them and look at your password right you would not want that so this is why it's important to anticipate really the fact that there are going to be quantum adversaries and go the fastest we can while still making reasonable decisions so in front of a few uh in the slide there is a very simplified version of um an integration process so of course it starts with some research people designing new tools new algorithms and the standardization process so community looking at what has been proposed assessing their properties and deciding agreeing on a few of them which are going to be deemed as standards so this step the standards are going to be announced by the end of this year and i think maybe beginning of 2022. um afterwards after the standards are being announced there's going to be an army of developers that are going to look at deploying on the system bringing them from theory to practice um so i'm thinking about supporting different languages hardware software and so on and so on to make really them accessible to anyone and anyone being the project owners which might then decide to create a fork test the technology and after a certain point maybe eventually um complete the transition and use this new tool this new cryptography as the main component so this is what we want with quantum cryptography right we want we want slowly we want it slowly to replace what we've been using until now and the issue and why there is this sense of emergency is that we don't know how much it's gonna take before scalable quantum computers are built huge companies are putting a huge amount of money um into this research and i think it's gonna go even faster than anticipated and if um it is the case that scalable quantum computers are accessible to people with malicious intentions before the transition is complete well you're exposing yourselves to the risks that we just went through such as for example exposing all of your passwords online when should we care about that a quick and very marketing like answer me maybe yesterday we should have heard yesterday if you want my personal opinion i think you i mean you should care about it from like a developer point of view as soon as standards are being announced because at this point you're gonna be able to see okay i can work with that so let me see um visibility so i think this is um this is better answered more honest so okay so maybe by now i've convinced you that um the quantum threat concerns you and thus you need cross-control um cryptography in your project but of i mean i can understand that you don't want to have to go through the trouble of implementing it testing it yourself and so on and so on we went ourselves through the trouble and we developed a post quantum cryptography toolkit so you can access it via the query code on the slides um and i guess the slides of themselves are going to be available online so or you can email me or ping me and i'll be happy to send you the link so we made our toolkit um open source right so you can really see what's happening inside them you can try and understand what we did and we implemented two algorithms the first one being an encryption scheme whose name is kyber the second one is a signature slash authentication scheme called diligent both kyber and diliphium are part of what is called the crystal suites which is a finalist in the nice standardization competition that i mentioned like this standardization competition that i mentioned that i mentioned a moment ago so what it means is that people have valued relieve their security they have looked at it assessed it rigorously um also they say that it was not coming up too much of a performance issue and so on and so on and of course as i said uh what a surprise we developed these toolkits entirely in google we had a specific focus on security right so if we're talking about security we were like let's go all the way so first step is to look at the algorithmic security and this has been done by looking at the proofs the following carefully at the pseudocode that the algorithm of designer put online and this is as very like how to see easily checkable we can we can how to see we can run like this little um how to say a known test answer vectors test and all of that so this check um in the second step we were also thinking okay it's cool to be algorithmically secure but there is also another point which is of concern which is practical security and practical security is enabled when really looking at the code right and how do you evaluate if one code is um secured this is up this is very suggestive right i think there is no tool yet to ensure that one code is very secure whereas the other is um very not secure so what we did is that we enable practical security by implementing counter measure against a very specific class of attacks that are called side channel attacks so it might get a bit technical at this point but i think it's very interesting to look at these kind of attacks because they are um super interesting and um for those of you who are developing security algorithms it's very important that you know that they exist um and how do they work so you're playing a poker game with your friends and um really one key idea of poker is to keep your cards secret because if you're showing your cards to your opponents they're obviously gonna win and even if you manage to keep your card secrets you might still be kind of leaking some kind of information so for example if i look at the two um other two guys playing poker my guess is that this guy spamming himself maybe he doesn't have that much of the good game he is expected to whereas the guy smiling old teeth might have i don't know like um um a sweet or i don't know what and um so this is the same the same way that sciento attacks work you can gain information about a secret so in in this case the cards by looking at all these little signs external signs right so it can be um the emotion it can be like how is your adversary breathing does he look stressed and so on and so on and if we go back to our setup now that we can understand that the laptop is not going to try and hide some cards put a key and he's not going to be leaking some information through his smile or strands but maybe through the time it takes to compute some operation or with the power it consumes and those are physical measurements um that can be in i mean how to say well measured by an adversary and correlated back to the key and if the key can be inferred well you algorithm it can be the most algorithmically secure algorithm that exists once you know the key you're allowed to do everything you want so this i mean there is not much security left so this is very something very important to consider this um side channel attacks right so just yeah to rewrite uh to reread what i said what we did is that we looked through all the attacks targeting both kyber and deletion the algorithm that we implemented and one by one we implemented the counter measures so we ended up with something that was um very secure on top of just saying post quantum security and i think this is a this is a great add-on finally i mean um as the last point to uh conclude this uh quantum topic i think it's interesting to look at the cost really so we will need post quantum security but what is going to be the cost of that that security so what we did to assess the overhead is that we compared our algorithms against two algorithms that we picked from the gu official crypto library so we we picked the same level of security of course in the classical setting because if you um followed you should know that um what exists what we've been using ecgsa and rsa are totally broken when facing quads adversaries so we don't want to go that way so for the same level of security in the classical setting we um compared both the performance in terms of speeds and the size of the outputs right and we can see actually quite surprisingly that uh regarding the performance our library performs way better than um the algorithm the algorithms from the go library so in a very light blue is our the runtime theory operation of our library and in dark blue it's the time it takes period operation of either ecdsa or everything so of course the smaller the better right and okay so far so good we um we have post quantum security and we can performance what's more to ask right and i wish i could stop here but actually there is a trade-off to this um nice increase in performance which is the bandwidth so the bandwidth necessary to run kyber and lithium is actually quite big it is orders of magnitudes higher than what we're used to so if we compare for example diligent against ecd ac i mean you cannot miss it right um so this is something to keep in mind so this is not going to be as disastrous as choosing a one terabyte key right but still it will have an impact that it's good to know in advance because i wanted to conclude on um on the brighter and more most positive node i'm going to now present a little um test that we did to make sure and to further evaluate things like this overheads and uh what we did is that we took wire guard which is a vpn solution which is available um in google and is open source so we looked at the code we identified where is the public key algorithms um being used and we replaced them by the alternatives offered by our library we run some tests and benchmarks and what we noticed is that the other head is not that big right there is a theory and then there is the practice so every two minutes or so you can expect another head of two extra ip packets being sent over the network and regarding the time there's going to be a little delay of half a millisecond i don't know about you but i can't notice half a millisecond delay so i don't think i would even be able to tell which cryptography is being used um if you're just giving me the software which is actually the goal right you don't want to have to sacrifice anything in your application to have postmanton security on these very good notes on this very um let's say experimental demonstration of the practicality of our library i think it's a good step to uh to conclude the takeaway message relief that i've been trying to to uh today is a bit explaining what what quantum is and that it's neither bad nor good uh like um all the technologies are it's neither black or white there's always this bit of grayish thing that depends on the context and um if you care about security well there is all those things that you can't really say do anything on with this big tech and how to say building softwares except if you're working for some of them but at your level and for your projects you can start having a look at what exists and how you can incorporate it into your infrastructure right and um when still i would repeat my personal opinion of caring about this kind of thing as soon as standards are being announced and um more very if you're interested in having something which was developed in go and has a very strong level of security i would also have a suggestion which is very personal i would suggest you to use our library and at last but not least i think um if you want to help well there are many things that can be done this um practical security process that i mentioned it's a continuous process so attacks are going to be implemented we will have to find defenses and implement them so we would be very happy if this could be done in a very interactive process so we welcome any kind of um poor request or contribution on this side to make our library even safer which is the uh of course the end goal thank you very much for your attention and if you have any question i'll be hanging in the discord in the discord chat so yeah drop me a message i wish you a very good day [Music] hi my name is rinky sathy i am the chief information security officer for twitter praetorian always brings the most amazing talent when we need to engage with them on a project and what's always been interesting too is they probably have had the most number of women that i've engaged with out of any other pen testing company i've worked with and that's always meaningful to me i'm always championing more women in tech and more women in cyber security and how we need to get more diverse as a cyber security industry and so i really appreciate that about praetorian thank you so much mithild for that very very fascinating talk and as expected i feel not so smart and uh yeah if you've got questions about quantum and go i know i do please jump on the discord uh if you're not on there already you can get to it through discord.go4congress.com the field is actually in the discord right now answering your questions about quantum so definitely go check that out yeah don't forget some of our other channels in discord too obviously peanut gallery people have not been shy in the pinot gallery to give us well should we call it feedback commentary johnny and i were trying to be pretty forgiving yeah this morning on your new attire they are good they have not there there is a screen grab of me though where i don't look too ridiculous i'm smiling right you know like my eyes are open that's always a plus not nearly as good as the screen grab of johnny yesterday yeah though this one yeah yeah nirvana he does look like the other one where you were actually laughing oh that was a good one that one was your new profile thing for sure yeah and there's you know we've got dad jokes we've got a mainframes channel there's been some other new ones by popular demand and the pinot gallery so yeah you know make a difference change the world in the peanut gallery you know get in there you can also heckle us obviously which is we didn't really always telephone in particular aaron aaron yeah he needs it leave johnny and i out of this bring it on i wore my thickest armor today so let's bring it on everybody do your worst help him out with his self-esteem bring me back down to earth is that what we're trying to do yeah well um but seriously though we've got some virtual meeting spaces coming up too we've got gophers of microsoft they've got an ama session coming up which would be pretty cool you know you can go there do do your worst there as well they can they can take it um and then we've got buff coming back uh they're gonna do a weightlifting seminar just kidding yeah they're gonna do come into it they're uh the one today is called buff helps you go faster um that'll be super interesting too protobuf stuff grpc stuff as well yeah so what have we got next well um when it comes to distributed systems um eventually consistency i know right one of our follow-up various objects uh eventual consistency must never be into the conversation right so um lots of ways uh to go about it but um we're gonna have a crdts into the conversation as well and who better than our next speaker to tell us about how crdts can play a role in eventual consistency and go so arash shabina is a principal engineer at the bread and he's here to put some knowledge on us go ahead and take it away arash hello and welcome to my talk my name is arash bina i am a principal engineer at bread i've been spending a lot of my time over the past few years building decentralized or distributed applications and distributed platforms and most of that has been in go recently at brett we used a sort of a new approach or approach that i wasn't um familiar with a lot uh using crdts to build a service and um i thought it was very interesting and i wanted to tell you a little bit about it at brett to give you a little bit of context we have a multi-tenant platform and we provide um the capability to our clients to create um highly configurable payment products and offer them to their own customers to their buyers and then during the process of the checkout we allowed them to do credit decisioning fraud checks and various different types of sort of workflows and when the checkout happens we allow the clients to uh to sort of service those payment products uh using the platform the talk is mainly divided into four sections i'll talk a little bit about consistency what that means in that distributed systems but i won't dive too deep into it i just want to mention where crdt sort of fit into in that spectrum then i'm going to introduce crdts what they are and how we developed them in the third section what different categories of them are um and how you know what are the advantages and disadvantages of each of the categories and at the end i'll talk a little bit about um you know our experience overall how how was it building the system uh this way so as i mentioned i won't get too deep into uh consistency here but you know the the type of consistency that we're all very much used to is a sort of a strict type of consistency where we either provide that with some sort of a data struct database for example different replicas of a service they all talk to the same database and that's how they coordinate sort of on and agree on a sort of a strict a sequential way of events and it's important because at that point all the replicas can sort of agree on the exact time that events happened and so if you ask any of the replicas what the state of the system is they can all sort of agree and they return the exact same response hopefully as a in contrast to that we have a causal or eventual consistency where the you know we allow the state of the different replicas to be mutated and so when the when um you make a call to a replica and mutate the state that replica of the state will be different than the state of other replicas in in that sense we allow the system sort of to get out of sync and then we have instead of uh coordinating that in a sort of a you know either with a database or in in some sort of a leader sort of follower type of architecture instead of that we we sort of allow these sync events to happen between the different replicas and what that does is that it it allows the replicas to sort of converge just in their states to converge you know in a weaker sort of a eventual consistency consistency system and uh there's only a probability that that convergence will happen and it's not a it's not necessarily a guarantee if there are two events and one of the replicas you know interpret that event as the first event happen before the second event uh and another replica interprets that as the second one happens before the first one it's possible that um the state of those replicas sort of diverge and so there are different techniques that you can use for example to you know fully sync between the replicas um you know compare them and then choose one of them and tell all the replicas that this is now the state of the system and so there are different techniques that you can use and in contrast to the sort of a weaker weaker eventual consistency there's the eventual strong consistency where through a specific type of you know merging function which is a lot more deterministic so if you have two events you know the result of the merge of those two events is always the same thing and through something like that you can sort of guarantee a stronger eventual consistency where you know that all the replicas will eventually get to the same state and this is really versatile that is fit and that's the type of guarantee that they provide so what are the crdt saucers stand for conflict-free replicated data types um they're essentially it can be anything but for most common examples are a structure that provides um methods for sort of querying and mutating um you know the state of of a replica and um the second part of it is a what we call a monotonically increasing function uh it's a merge function and what that does is that it has that deterministic sort of behavior where um you know you have a crdt you have a replica with a state and another uh replica um census state to this uh replica um the the sort of the result of those marriages would always be the same thing um and a very simple example of this is for example a max function imagine if you have multiple replicas each of them are holding a counter if we are sinking between them if you always pick the maximum of the values um you know after a certain point in time all the replicas will end up with the maximum value among all of them and so that's how they sort of this converge and that's the that monotonically increasing function um the other thing that's important about crts is that they every crd would have a full perspective a full view of all the whole system now their perspective different replicas perspectives could be different at any point in time but um sort of the behavior of the system overall is that all those estates converged the same thing um eventually over a period period of time um so strategies are essentially a way to build a system and replicate uh you know data uh without centrally uh you know coordinate um between between these replicas um and so instead of that we do periodic sort of synchronization so there are two major categories of crdts the state-based crts and operation-based series in state-based crts which are relatively simpler than operation based what we have is that each replica essentially holds the full value or full state of the system of all replicas so for example if you have two replicas the first replica knows about the state of the system at replica 2 and vice versa or at least has an idea about what the state of the system is at other replicas now what it thinks it is between different replicas could be different at any point in time but the hope is that eventually they all come to the same conclusion to the same state um the other thing that's important about the state-based crt is that is that during uh synchronization the full state of all the replicas that a replica knows about is transferred or communicated with other replicas so if i am you know replica one and there are three replicas in the system for example i will communicate my not only my state but what i know about the other two replicas state to everyone else and then the merge function takes care of that sort of the merging between them as opposed to that there's the operation based crdts uh which are essentially uh instead of communicating the full state of the system and storing the full state of the system i would each replica only holds their its own state and instead of communicating the full state it only communicates what that mutation was so for example if i have a counter and we we increment the counter by a value let's say by one uh if that happens in in a replica instead of sending the the actual value of the counter it sends the operation that happens so for example i incremented the counter by one and the other replicas are supposed to you know receive that and do the same thing down replicate that operation um one of the important aspects of operation basically is that that message needs to be sent be received essentially the message that is sent by one of the replicas needs to be received by all replicas they don't have to be received at this exact same time but we want that message to be received by everyone so that everyone can do the same operation so in order for everything to be a little bit become a little bit more familiar i was thinking of starting from a sort of a very simple crdt concept and then we're going to build on top of it and make it a little bit more complicated so the simplest thing that i could think of as a crdt is a boolean flag and let's say we want a flag and the value of the flag we want it to be true if the flag has ever been set to true so if if a flag has been set at true once then we want that uh true value to be carried over and not be flipped back to false by a false event so the way we can define the merge function for this crdt is that you know as the value if the value of my flag is true then i'm going to keep that value and if it's false then i'm going to you know accept the incoming value uh and if it's true then i'm gonna flip the value of the flight to true so let's look at it from the perspective of you know different replicas let's assume that we have a system that has three replicas each of those replicas holds a flag value in this case the flag starts from zero point is false and then we we can see how that that value is mutated over the system now in this case i want to mention that although these three different replicas are have a flag in them and have they have a value these are effectively the same flag it's just been replicated three times these are not three separate flags and so when i asked one of the replicas what the value of the flag is my hope is that all of them can give me the same answer because they're effectively the same flag now that may or may not be true at different points in time but we will see that eventually we want all of them to become the same so let's say a client makes a call to one of the replicas randomly here like replica 2 and sets the value of the fly to true so in this case if at this moment in time if a client asks replica 2 what the value of the flag is it returns true but if it asks the same question from one of the other replicas it will return false so obviously there is an inconsistency in the system at this point now let's allow the system to synchronize so you know each replica picks one or two of the other replicas and sends a message you know broadcasts its sort of state it communicates it's a state and in this case replica 2 is sending a message to 1 and 3 is sending it to 2 and one and so you can see because the replica two communicated it states replica one now replica one also has the value of the flight to true but no one really communicated to replica three here and so the value of the flag stays false if they allow another sink event to happen maybe one of the replicas of water two they communicate back to replica three and so now the whole system has the value of true so at this point in time if you ask any of the replicas they all return true now i want to emphasize that this happens exactly because of that monotonically increasing merge function right because we are keeping the true values um and we are not flipping the flag back to false if we receive a false event um and because of that eventually the system becomes consistent uh if you learn to not have that rule and flip the value back to false um you know the the system does not converge uh and so it it's possible that um they may all end up with let's say the value of false but that's not the actual um that won't be the true value of the flag which in this case should be true let's look at a sort of simple code example here and like i said we'll gonna build on top of this progressively uh but i have a flag here and it holds a boolean value and so you know it's a zero value essentially when we create a flag is going to be false now i have a merge function so if if another flag sends its you know state to the receiving flag the receiving flag looks at its value and if it's false then it accepts the incoming value and if it's true then you know it it sure sort of returns without making any changes and so if you look at the main function i create a sort of a zero value of a flag i update it to true with a true with an incoming true sort of state and then i will attempt to update it with an incoming false state if we run it as as we expect you know this is a very simple example it was initially false then it became true and then it stayed true okay so let's look at something that's a little bit more useful in this case let's assume that we want to implement a counter and the value of that counter um is um you know the the we want the counter so let's say count the total number of calls that are made to a service um now in this case because the calls will be made to different replicas so the the total value the true value of the counter would be the sum of the values of all the replicas however because we want those calls to be made to different replicas and because we want to be able to ask any of the replicas for the value and be able to get an answer then all the replicas as i mentioned before will hold the values of the other flags so rather than having just one value we will have a slice that keeps the value of other replicas and when you're measuring them the way to do it is to use a max function as i mentioned um which is an increasing function to sort of update our own state and always keep the sort of the max value for all the replicas and so if someone asks for the actual value of the flag we can add all the values together and return the answer so in this case let's say we have the three the same three replicas each replica has a slice and the green sort of shows the the element that a replica increments if if a call is made to it and asked to increment its counter so in this case let's say replica two receives two calls to increment the value of the counter by one and replica three receives three calls so replica two's value is now two and replica three's value is three so at this point in the system obviously if you ask different replicas for the total value of the flag a replica of one would return zero replica two radio trend two and replica three would return three but the true value that we know because we made five calls should be five so let's say a sync state happens and replica three communicates to replica two first and sends its state so now replica two knows about the three calls that were made to replica three and then replica three replica 2 makes a call to replica 1 and updates the state so now replica 1 has the correct state of 5 replica 2 has the correct state but no one really communicated to replica 3 in this case and so the value for from the perspective of replica three the total number of calls is still three and now let's allow another thing to happen in this case replica one for example communicates through replica three but it could be replica two and so at this after the second sink all three replicas agreed that the value of the total value is five this sink in in this case a specific example happened in two steps but depending on how the replicas communicate obviously it could happen with just one step or it could happen in more steps so we don't really know you know there's randomness here depending on our communication layer and so although the system will eventually become consistent it may uh there's no guarantee that it would become become consistent in in one step or multiple steps a simpler example of this let's assume that we are implementing this in just one replica but three instances sort of with three elements in in our counter so if the counter has a slice i can initialize the slice or create a new counter and initialize the slice and then i can increment the counter but when i'm incrementing i send which replicas value i want to increment and then i increment that and when i'm calculating the value i add all the elements together and create that and so if i'm the main function you know i create a counter i increment r1 and then r2 and r3 and in this case because i'm using one slice uh you know i would have the full value uh the full value here and so if uh if i increment r1 r2 r3 and then at r1 the total counter value would be four and so if i run this you know you you see that um it it gives me four now we can implement what we actually discuss about three replicas and create actually three separate replicas and each of them holding a slice and in this case i need the counter to know which replica it belongs to so i add an id to to the counter and when i'm creating the counter i i would send it which replica it is for and when i'm incrementing because the counter knows the replica that it belongs to it can use the id so i don't need to send the id in the increment function anymore the value function stays the same and i implemented the sync function so essentially now because we have three separate instances three separate replicas we need to be able to sync between them otherwise the values will always stay you know different so i implemented a sync function and so a counter can send its state to a receiving counter and the way these two gets mer get merged is through the max function so if the receiving replicas value is lower than the incoming value you know it it picks the higher value and in this case i mutate the receiving counter but obviously the cent counter stays um the same so now if you look at the state of the system so i increment three counters uh i increment one and so from perspective of counter one um the total value is one but at this point the uh the sort of the total value of the counter from counter two and three obviously is still zero um so i increment counter two now counter two has a one for its element uh and then i send the sync message from counter two to counter one uh and in this case counter one gets updated and now it has the correct value of two i increment counter three and three sends a message to one so counter one now has the correct value for the the whole system now i increment counter one it becomes four and counter one then communicates with counter two and then counter two communicates with calorie three and then i print the state of the system the code um so we can see that you know uh as counter one got updated and then a sync happened between counter one and two now different from different replicas perspectives the counters values you know are different and then it goes through um you know incrementing the counter at uh replicas three and a sink happens and then increment in the counter the camera at replica one and the sync happens and so eventually at like at this point in time at the step three uh replica one and two think that the total value is four but replicas three things that it's one and then after another sink all of them agree on the on the same counter value so um so that was a g counter which is a growing encounter um but what if we wanted to implement um you know account a counter that we could decrement this value um if we just have one slice if we implemented the way we implement the g counter and decrement but decrement the values you know because we're always taking the maximum um the decrement um events will be lost because we're always picking the highest number so the way to implement implement this a pm counter which is a positive negative counter and we can decrement it is to have two separate slices so we use one slice for the maximum increment values and we use another slice for the maximum decrement values let's look at the code for a pn counter so we have three replicas you know same ids but now here i have added a negative counter in addition to the positive counter or as negative slice in addition to the positive slice sort of when i create one i initialize both slices and when i'm doing the increment value i am basically incrementing the positive counter positive elements in the positive slice and when i'm decrementing the counter i am still incrementing but the values in the negative slice and so when i want to calculate the total value of the counter i add all the positive values together and subtract from that the total value of the negatives and when i want to sort of sync or the merge function looks at the positive elements separately finds the maximum for those and then it it looks at the negative values or the negative counter and it it still picks the maximums and that way it merges the two states together so to make it a little bit more automated and interesting i implemented a sort of a gossip function here and the way it uh the way it behaves is that each replica picks one of the other replicas as uh you know the replica that it wants to send a message to and in this case sends a message to the other replica and sends his state and then this states gets merged so there's a sort of a simple pick function that a replica can that the gossip uses to pick another replica randomly and i implemented a utility initialize function so i can initialize everything in there and then the main function basically seeds the randomness for you know for the the functionality to pick a random replica initializes all the nodes and then really it's the it's the part where we are incrementing the counter so in here uh replica one i'm incrementing the value of the replica one and so you can see the positive counter for replica one becomes incremented to one and then i do the same for replica two in this case two gets uh incremented and replica three three gets incremented and the negative values are obviously zero because i haven't decremented yet and now i decide to increase decrement the value for replica one and in this case instead of decrementing the positive counter for the replica i increment the negative counter right so in this case you can see that from the perspective of replica one the value is zero but from the perspective of other replicas the values are one and actually none of the replicas are correct at this instance because i've incremented three times and decremented one so the actual value should be two to make the gossip function um more automated i basically i create a ticker and that ticker takes every uh one second and then at every one second it does a the gossip functionality and then it prints the state and after five seconds i want the application to be done after the initial increments and the decrement this is the state of the replicas and then after after the gossips happen every one second the the values converge and so all of them agree on the same value now it's possible that if you run this multiple times because they pick random values now in this case you can see for example after the first gossip that these two values are different so after the first gossip for example the counter two also knows the correct value but counter one is not updated yet so if you run it multiple times you you might get different results so i modified that same implementation for a pn calendar but to make it a bit more interesting and see how the system functions in more of a live sort of system i what i did was i created two other tickers so there's an increment ticker that takes every 600 milliseconds and what it does is that it picks one of the one of the nodes randomly and it increments its value and then there's a decrement uh ticker which picks one of the nodes randomly again and decrements this value but that one runs every one one second and then the gossip ticker takes every 200 milliseconds so it's possible that it may tick before an increment or after or before a decrement or after or multiple times between those because it it's a shorter time uh so you can see that the the sort of the go function here is a little bit more involved but uh fundamentally uh when one of the tickers ticks it prints that it incremented that value and then when the gossip happens the communication happens so let's look at what happens when it runs so you can see in this case that initially the system you know starts at the zero zero zero and then an increment event happens in this case at replica two and the value of the replica two is increased to one but as you can see replica three also shows one and that's probably because after the increment happens immediately after before the state was printed the communication between uh replica 2 probably sent a message to replica 3 and updated this state and in the next step the replica one also has the correct value and then we decrement replica three which becomes zero in an increment um replica one and so you can you can see that it's difficult to predict exactly unless we print all the exact timings of the gossips and increments and decrements it's difficult to predict what result we should expect the important thing is that you know the the values don't diverge the values keep converging to the same values sort of for in all all the replicas and at the end before i uh kill the application you know replica two and three i agree but uh two doesn't agree probably because the application was killed before the last sync or you know before the full convergence happened okay so that was the last state-based rdt i want to talk about and so i just want to conclude the part about the state-based crdts and take a look overall at the advantages and disadvantages of them on the advantage side um i think you know the state-based crdts are a lot simpler to implement as you will see um in comparison to the operation-based crptes but on the disadvantaged side obviously every replica needs to hold the values at full state of all the other replicas so on the for the space complexity um you know you need more space than if you're only holding uh the value of you know your your own replica essentially on the communication side as well uh each replica communicates the full state uh to all the other replicas and so because of that um the bandwidth for the communication is a higher bandwidth you need you need more data is transferred between the replicas okay let's look at an operation based crdt so for operation base i wanted an example of you know let's implement a set and the way it has set functions is that you know you add an object to the set it stays in the set until you remove it from the set and so similar to the pn counter we can't simply remove something from the add the when we add it to the set uh because that function will not be increasing so similar to the pn counter i will have two separate uh sort of sets one is an ad set and one is a remove set and then i want to add something i added to the ad set and when i want to remove something i added to the remove set and the the net value of the set is sorted out it looks at the items that are added objects that are added and if they don't exist in the remove set then it concludes that those objects exist i wanted to make it slightly more interesting what if we wanted to be able to remove objects and re-add objects that we have already removed so if you add an object and then remove it that object will exist both in the add and the remove sets and so there's no way really to re-add uh something that i've already removed um so in this case what i did was i i created a sort of an arbitrary rule um so when something is added i will remove it from the remove set but i'll also timestamp everything and so if there's a case where two concurrent uh sort of events have happened one is an add and one is a remove i allowed at addition to win over the removal and now that's a completely arbitrary rule because i wanted certainty to function specifically this way but you could for example build another one that the remove wins over the the edition and so that clocked the increasing sort of clock that is what provides that sort of um monotonically increasing merge function so if you look at the implementation for that i reduced the number of replicas to two just because this is a longer example and wanted to be a little simpler but so we have r1 and r2 for the ids and now here i've implemented a clock and the timestamp is essentially the version value for all the replicas which is a slice of integers and the clock holds an id that id is the id of the replica that the clock belongs to and it has a timestamp now each clock can increment sort of give the next value for for the clock next timestamp and then i have comparison functions less than and more than and so when a clock is sent a timestamp it can look at the timestamp and make a comparison and say if the timestamp is um you know ahead or um you know before the timestamp of the clock it is possible that it's neither of those so and if it's not before and not after then i conclude that those events from the perspective of the clocks happen concurrently and that's where i will allow the the addition to win the timestamp returns a sort of a time stamp for for a clock and i have a new function to to create a clock and then i have an update function where when when a clock is sent a timestamp uh not only can determine if it's before and after but it also needs to sync its own clock with that some time stamp because we want always to have keep the maximum version of of the you know individual clock values so in this case it syncs its own internal time stamp with that so the set is actually this the crdt itself so it has an add map and it removes that remove map and i have a mutex here so that i can lock it when i'm sort of mutating the values of the maps the new set just initializes the maps and then the value as i described uh it looks at the ad set and it looks at the remove set and if something is added and not removed then it concludes that it exists in this set otherwise it doesn't so i introduced uh i introduced a node struct here so that it makes the functions a little bit easier and a node essentially is a is a struct that includes a clock and a set so the node can add something to its set and to do that i just add it to the ad set and delete it from the remove cell as i mentioned earlier and uh update the clock of of the node and the remove uh just adds it to the remove set so initialize nodes just creates clocks and sets and adds them to the nodes and here's the operation part of the operation basically where i define an action and the action is just a you know an integer it's an either an add or a remove and the operation struct has a node id that's the id of the node that's coming from it has an action that happened and it has a timestamp and the object itself what was um added or removed i have a send function method that's implemented on the node and so onenote can send an operation to another node here all the function does is that it initializes sort of it creates an operation and then it sends it to the node to be processed and so the process is really the the core of the logic and so initially i want and no matter what the results are of the merges i want the clock to be synced to the incoming timestamp then i look to see if a clock you know at the time stamp has happened after or the clock is before um the timestamp of the event that was sent and if that's the case i just update the internal sort of state based on the incoming operation if it's happened after it means that or the clock is clocked clock's timestamp is after the incoming timestamp then that means that i don't need to do anything and all happens is that in the in the sort of differ function i just update the timestamp or i sync the clock and if the code sort of reaches here it means that it was neither uh less or uh more so they must be concurrent and so in that case if it's an ad i added and remove it from the remove set and if it's a remove i just basically added to the remove set so if you look at the main function here i try to manually control sort of the additions and the sync so you can see what is happening in the system so initially i add a just a simple object to the first set and so for replica one i have added a and its remove set is empty i get the time stamp and i send it to replica 2 and so now replica 2 also has that object here i remove it from replica one and then i sync it to replica 2 and so it gets added to replica 2 as well then i re-add it back to replica 1 and what that does is that it removes it from the remove set and then at the same time before i allow the sinks to happen i remove it from also replica 2. so it was already removed from replica 2 but that what that does is that it creates two events one is at replica one that's an addition and then one is at the replica two that's a removal and they happen at the same time because they uh they they sort of the clocks haven't been synced yet so then i sync both of them and because of that specific rule that i want the additions to win over the removals um both of them should end up with the same object and if you run it that's exactly what we expect so the add happens the sync happens the remove happens the sync happens and then the add happens and the remove happens but it hasn't synced yet and then i let it sync and now both of the replicas end up with the same value okay then the last uh series that i wanted to sort of mention here briefly and i don't have an implementation for this but we actually use this for some of the things that we implemented in our service is a contextual series in a sense that with this you can really implement any arbitrary object and all you have to do is define an increasingly sort of that monotonically increasing function and the way we define it on internally was that we look at the state of these objects and we determine we define as sort of an increasing sequence for those so imagine you have a ticket and you have a ticketing system and you have a ticket and the ticket can be only like opened and modified and closed um now if you have two uh simultaneous or concurrent states one is a modify and one is a close you can say that close should always win over the modify right and then that's how deterministically you can merge those together now that gets a bit more complicated if for example you have you allow multiple modifications to happen at the same time and so in those cases you have to determine um you know which modification should be applied last or which modifications should be applied if they are happened if they're modified the tickets have been modified one of them for example represents a name change and one of them requires represents a description change um then uh you have to sort of have fine tune those and decide um how we want to um how we want the modifications to win um one one simple way to do it for example is that you can always allow the last sort of modification to win but in that case um you may not see some of the earlier events so it really depends on how you want to design the system so on the operation based crdt sort of conclusion obviously the advantage is that um you know you have a smaller storage footprint uh because each of the replicas only hold their own data and on the communication side they only communicate um you know the the operations which are probably very small compared to the whole state of the replica and also you can see that you can probably build more sort of general purpose use cases uh with with operation-based crdts on the disadvantaged side um the system is more complex there are more moving parts um and so you have to write more code as we know when we write more code we also introduce more bugs in the system so um they are trickier to sort of implement and test and make sure that they are um they are correct and if there is a bug in the system it will be more difficult to sort of find and address and the last thing is that they require more reliable infrastructure because we want each operation to be uh you know received by all the all the replicas um and we have we need to have a sort of a reliable infrastructure that can do that um delivery the way um we implemented the crts in the service we started with concrete implementations so because initially we didn't know what we wanted behavior or what is the common behavior for all the crdts so we implement them in a concrete way but as we were we implemented a few we noticed that uh you know there are common sort of a pattern common behaviors between all of those and so we created this interface where each rdt can give its id kind of this type has a merge function that can merge itself with an incoming crt you know uh some of them need a time stamp and so you can set the timestamp this way and also you can see rdd can prepare itself for broadcast and that means that when i want to communicate my state or the operation or whatever it might be the crt knows how to package itself into a message that gets put on the sort of a transport layer to the communication layer and sent out and this really helps to simplify the other layers of the system the communication layer or perhaps the storage layer depending on how you implement it so there are a few uh sort of miscellaneous smaller things that i want to mention here on the story side obviously each replica keeps its own storage they don't use a common storage and so you can implement that different ways we ended up implementing a simple sort of a memory based which each replica holds the whole sort of state in memory and also file based and depending on your needs depending on how critical a system is you can you can use different types of storage the other thing is that for tests to be quicker and not have specific dependencies for example you can use memory based and you can use file based for for other use cases on the deployment side this can be deployed really as any service as long as the replicas can communicate to each other in our case we deployed these as side cars because we use a kubernetes deployment and so the services that require access to these crdt values they have a sidecar that is essentially our service that provides the crdt functionality another very uh fascinating thing that was the topic of the clocks when we were doing research on this there's a very interesting implementation i've referenced it at the on the last slide where there's a specific implementation where the clocks have three components a physical clock a logical clock and sort of a vector clock and what that does is that it attempts to keep the vectors keep the clocks close to the physical clause as you know in the distributed system it's uh impossible to keep all the physical clocks exactly in sync and so typically they might be a bit out of sync by a few milliseconds pro perhaps there are some certain implementations that for example use a specific hardware to keep the everything in sync but in this case it's a it's an algorithm that you know they have used and that's how we um sort of that's that was also our implementation for the client and that's what we used you need a gossip layer and um essentially the communication layer can function different ways uh but we use the gossip layer in which each node doesn't broadcast the whole um state to all the nodes it just picks some of the nodes randomly and you can obviously fine-tune that on how many nodes uh or how often it wants to communicate but then through this gossip all the nodes receive the state one of the interesting implementations that we learned from was uh hashicorps sort of the member list and um it does a lot of thing a lot of things happen in that library for example you have cues that you know between the gossips messages can be queued in there and then it communicates the whole queue and it takes care of for example the maximum length of the messages and a lot of things like that and some of the messages piggyback off of each other so that the whole system is more efficient the last thing i want to mention is that i mentioned that in the state-based crdts the whole state is always communicated to other ones but there are some interesting articles as well where they implement the merge functions in a way that you can also communicate delta um updates on the testing side uh so the main the main thing that you want to test on the crdt's uh based on the unit testing side is really the merge function and the value functions because that's where most of the logic lies and on the on the value functions are typically a lot simpler than the merge functions but most of the functionality is in the merge function so you really and it's something that's relatively easy to to unit test so you really want to make sure that the merge functions are you know correct as a central to everything on the integration testing side as you can imagine there are a lot of moving parts the integration testing is a little bit more difficult but we wrote several several sort of utility functions that helped us to uh to build the integration test a little bit easier and faster so one of those is for example there are functions that can create multiple replicas and establish communication between um we have functions that allow you to set the values for the replicas directly if you want to test something that for example before or after it goes a sink and the value of a replica is is something sort of that you know what it is you need to be able to manually adjust the clocks to be able to sort of be deterministic about the result of a test function you want to be able to perhaps create time stamps that uh you know happen just before or just after another time stamp you sometimes you want to be able to manually control sort of the communication between the nose for example you want one node to specifically communicate to another node and so you can you want to be able to manually control those in some of the tests and then there are utility tests where you can insert essentially the consistency between all the nodes so as the communication happens and the states change at the end you want to be able to sort of assert that the state of all the replicas are the same and then you want like cleanup functions that can clean up the replicas and such so in conclusion you know one of one of my colleagues asked me if if if this was if it was worth to implement this service this way and i think that's a fair question it's a lot more complicated than sort of the regular ways we're used to writing services and although it's a fair question it's also an impossible question to ask out of context because you know you want to choose the right tool for the right job and so if you have certain performance for example requirements if the eventual consistency is an option for you and all those conditions are right then yes it might be the right tool and it might be worth implementing um if it's not if the function of the service is a lot simpler if it can't be easily implemented um you know using the sort of a regular ways that we implement services then perhaps no it might not be worth implementing that way but overall um trds can provide great performance um because um you know every you can ask any of the replicas and it can very quickly uh respond back um and also that coordination on the right and mutation side sort of does not have to exist and it's sort of it happens asynchronously on the disadvantaged side there are many moving parts to cruts and they can be complex to implement um and so um the testing is is more difficult and those are some of the things that we have to consider when we want to build the system this way and um these are some of the references uh there were many more but these were the ones that i think we used we looked at most often and so i wanted to have them here and thank you that was my talk i hope this sort of give you a different perspective or perhaps if you knew about it it added a little bit more to um what it takes to implement series and i hope it was useful to you all right well thank you so much arash uh that that was amazing actually i mean i really enjoyed how practical that talk was you know crdts can be conflicted you tell it's our last day you know like the the jokes start getting cornier i was going to say crdt is going to be a complicated subject but you know they're also yeah technically they're contract free yeah yeah but you know he it was really practical it's talking about deploying it and all the pros and cons and the pitfalls and everything and you know that's really cool to hear because some of us don't want to read it in a white paper some of us want to deploy it and you know use this stuff to improve our systems and add reliability and you know all that good stuff so i he also kind of mentioned it's complex to implement so i hope that complexity gets kind of removed as better libraries and higher quality software reusable software you know comes out and it lets us take advantage of this stuff more easily i mean i think anything in distributed systems is complicated too yeah it doesn't even matter how many times you've built a distributed system they're all hard yeah yeah and we've been chatting about that a little bit behind the scenes with even like how all of this stuff works in the video when yeah you know it's like oh yeah press a button and we're live and you're like you know you start learning about how some of these things work like oh it's um it's amazing it works most of the most of the time yeah right yeah i always love starting to work for companies and seeing like how the sausage is made you know and you're like whoa so that's how that works you put that in there you sell this to people well then you always see like those pieces where you're like why would you do that then you hear the context behind it you're like uh i'm so sorry i'm so sorry you get blinded right and so when reviewing people's code you almost have to have empathy for that right where you have to think like they were traveling down a path and was making decisions as they were discovering things and it's so much easier to kind of sit on the outside and yeah oh well why would you go that direction yeah and then you know you you think you know better you you don't i like to tell people like solving the problem is the hardest part you know like looking at something be like how do i make that look cleaner or more performant yeah that's much easier your your solution is influenced by the fact that you even saw the other one yeah you know you got to work with older code or older systems or stuff people built 10 years ago or whatever that's harder you know than building something greenfield you got to work with like you said you got to work with people who made that decision 10 years ago you got to work with that decision i think we should just be able to come into a company and throw away everything and start over right now and use all bleeding edge software uh speaking of the fields in green so i do have a question for you all though if if the earth is made up of seventy percent on the carbonated water does that make it flat let's say the latest on the uh the dad jokes channel folks oh yes we're here all day literally yeah whether you like it or whether you like it or not but on a serious note though um if you are having issues with the microsoft live stream we do have a a backup um there is the youtube um gopher academy on channel that you can uh um switch over and see if you have better luck there like we said uh not everyone while you're there subscribe i want one of those smash this series i'm not mad i'm just disappointed we got we have less than one day of conference left we need 75 000 just shy yeah i mean we picked up a couple yeah it's just a couple of you it's not a big deal everybody and if you were to make a cool bot we're not just again we're not saying to do it but if you did but if you did but if you did yeah yeah would probably yeah um so we do have some vms's um stolen going we have a buffer they're going to help you get buff with protocol buffers that is so they do have open office hours so go check them out we also have course hero with an open office hour so you do need to be registered to to attend those sessions however registration is free just uh you know do your thing go for gun.com and you are in like flynn and then later today we have the go time podcast coming up uh they're going to do a family feud style game where they're asking you the audience a bunch of questions and then they're going to make the go team compete to guess how you answered correct or incorrect feel free to join that survey to make the responses more fun you can do that at go time dot fm slash gs short for gophers say we also have lightning talks coming up again today with mark bates so that's always fun i don't believe we have any more room so you missed this year but i encourage you to submit next year or any other conferences that do lightning talks it's a great opportunity to kind of test out your your content and see how you know the questions people have developed they're interested in it it's a low pressure environment your toe and yep exactly speaking yeah yeah yeah i mean we all three of us do lightning talks we've we've done a decent amount of public speaking i don't know if this counts as public speaking but we've gotten up on stage it's public gesturing we're jesters i'll gladly take talking in front of a few thousand people than sitting in front of a camera talking like nothing then you get like we're having a conversation and somebody in your ears like we got eight minutes to fill when you're like oh great what are we gonna talk about we're already on camera but regardless yeah you know we do we do these lightning talks too it's it's a great low pressure environment to try out some new content tell the world about an idea that you just kind of thought of a little bit you haven't really gotten deep into it yet you know obviously if you're a new speaker to get used to speaking in general that's another way another reason you might want to do lightning talks or even even if you even if it's a topic that you know well one of the one of the best ways you can sort of figure out how well do you know this topic is can you distill it down to seven minutes can you get the core you know idea across to somebody else right in a much shorter time uh constraint than normal right so that forces you to really reduce it just to the important bits so just one slide trust me yeah seven minutes that's a great point seven minutes is tough to get you know anything substantive down to seven minutes i mean content and code simplicity is actually really hard yeah one of the go tenants right simplicity is hard you know what you know it's not hard to do in seven minutes though we could go over your wardrobe color scheme we think we could do that within seven minutes i bet we've got are you ready to screenshot right so outside the studio to warm up erin puts on a coat and a hat anybody want to take a guess what color that hat is to be fair the coat is green that's a new one for me i like dark though i was you know i was outside my comfort zone when i bought that coat because it close-up is green you know and i'm like i don't know if i can pull this off but that's why i went with the dark looks gray it's on brand for me yeah yeah so who do we have next on that note so our next speaker i'm actually particularly excited about i've got a bit of nostalgia here thinking about our first year uh with gopher khan and showing up at the marriott hotel just you know 700 of a 750 of our closest friends uh so next up we have alan shreve who is the founder creator of ngrok and he spoke at the very first gopher con we actually have another speaker later today who spoke at the first gopher count and so he's here to talk about how we can write less code by writing code that writes code oh let's grock it so take it away alan hello gophercon uh i'm delighted to be here today to talk to you about one of my favorite topics in software engineering which is code generation we have a lot to get through so let's jump right in what we're going to be going over today we're going to start off by talking a little bit about what code generation is and why you might choose to use it then we'll kind of get into the meat of the talk surveying some of the practical applications of code generation some of the popular tools in the ecosystem to do code generation and talk a little bit about how to use them and why you might use them also like jump out a little bit and survey uh some of the other ecosystems as well and code generators that expand beyond go then we'll we'll work on actually writing a code generation uh code generator together um and uh we'll wrap up by going through some some best practices here so let's talk a little bit about um what code generation is but before that i just wanted to introduce myself uh i'm alan trev i'm the founder and ceo of ngrok um if you're looking for me on the internet i am at incontrivable so code generation um very succinctly code generation is about building programs that produce other programs as their output for newcomers to to programming that can sound like a very intimidating definition and it really it really shouldn't be a code generation at its heart is is actually quite simple um when you think about it like you might be saying like well if i'm generating a program as an output like isn't a program just text like isn't it source code if you kind of put aside like the meta discussion about like what is a program like um and focus more on like the the concrete implementation of it um our our programs are source code so building a a program that writes other programs is really just about producing the text for that other program and even you know when you're you're really just getting started uh you're you're very used to writing programs that produce text outputs so if you're if you're just getting started and really getting into this that's a really easy place to to find kind of a point to hold on to to really get into this so this is like uh the simplest version of a code generator right it is a program that produces some uh output which is text that would compile as as a program but there are very few code generators that that work like this that you just like run and you get a program as an output instead when we're talking about like real world and practical code generators they tend to have two main sources of input into them one is the source code that they're using as kind of the the reference input and then some kind of configuration so a code generator will often read source code and then use some configuration to decide what to output a really common source of that configuration would be command line parameters of what to generate but you know just as easily a configuration file or or some combination of the two is is a very popular choice not all code generators actually read source code for input sometimes it really is just a reading from some kind of configuration in that case the configuration serves as both kind of the the source material and the configuration knobs on it a good example of this might be something from the rails ecosystem where you're generating kind of scaffolds and templates as a really quick way to get started without having to write the boilerplate this is a good example of a piece of code that runs to very quickly generate a stub for you to to start filling in there so before we actually get into uh surveying some of the different types of code generation i want to talk about why we choose code generation we're going to come back to this at the end trying to understand why we do code generation to start with is a little bit challenging without having seen some of the examples so we'll do this to start with and then we'll circle back at the end there but at a very high level like one of the reasons that we one of the main reasons we choose code generation is to write code that we as humans don't want to write there are kind of a bunch of second level objectives um in writing uh in in using code generation tools and writing code generators um these would be things like code reuse or getting better performance supporting multiple languages and you can do all of these things as as humans writing code as humans but it can get very verbose and tricky to write rather than writing something that will write the code for you so like i said we'll circle back to these and kind of tie them back to the examples that we're going to go through let's get into some of those examples here we're going to go through a survey of practical code generators we're going to focus like starting just on code generators in go and then from that we'll we'll kind of jump off to some of the other code generation tools that you're probably familiar with in in other popular ecosystems getting started um we want to talk about uh pretty much the de facto like intro code generator for any uh tutorial and go that that introduces you to code generation and that is the the stringer tool so what stringer does is given um an enum uh or as much as of an enum that you have in in go right a set of like typed constants you often with that constant want the ability to render that constant as a string where the string value you want out of the constant is the name of the constant itself and of course you can write that that routine yourself but as you like add and remove constants it's you know it's it's just annoying to to really keep that in sync with uh the method that turns it into a string so instead uh we can write a program that will do that for us um we don't have to though because uh we've there is a first party tool that will do that for us and it is called stringer down here at the bottom i've got an example of using stringer where you specify the type in a particular file that you want to generate string methods and then over on the right you're able to call that string method to to return the name of uh in this case ibuprofen if you actually look at the code that it generates um this is the code that it generates you don't really have to understand it it looks a lot more complicated than you would expect if you had written it as a human and that's because it's it's space optimized in a way that humans wouldn't normally write it and it comes back to one of the things that we were talking about earlier about code generators allowing us to write code that is more performant than we as humans might write ourselves let's get into another code generation tool this one is called go mock or mock gen there are a number of tools in the ecosystem that look like this this is a is one that we use at ngrok when you're working on testing code you want to mock out all of the dependencies so that you can test just the piece of code that you're interested in and not have to worry about you know calls to external systems like databases or working with the file system and the way that you can do that is to call all of those other services through interfaces and what that means is that when it comes to testing time you can substitute in a mock object for those interfaces that will instead not talk to a remote service and will instead do something else of course you have to write that mock object though um and that is often you know a lot of work to to just continually write these new mocks especially as you change interfaces you have to continually update your mocks to match so what you can do instead is use a tool like mock gen which you point at an interface and it will then generate out an object that matches that interface for you the nice part about it of course is that it also generates a whole bunch of methods for you that allow you to define what it should do when those methods are called really quickly as well as what to expect so when you're writing a test harness like we have over on the right ins what you can do is as part of your test harness instruct the mock what you expect to be called in the running of the test so that you can guarantee that the the object was exercised the external dependency was exercised correctly if you take a look at the code that's generated behind it this is just a snippet of it there's actually quite a bit of code that's that's generated to make this happen it's one of the reasons that you obviously don't want to write it as as a human you can see like if you if you squint like really quickly you can kind of get a sense of what's going on here where where the the code generator has defined methods that record the calls that are going in and record what you expect so that at test completion it can match what was called and what you expect it to be called let's continue on this is a tool that goes into the the category we talked about about getting to better performance when you're using marshalling and unmarshalling functions of goes json encoding and decoding routines in the standard library those routines actually do that through the use of reflection that means that they are dynamically inspecting the object at runtime to determine what json to output but doing that kind of reflection can be slow and so what we can do instead is we can write a tool that will look at the structures that we want to serialize and deserialize and it will generate ahead of time optimized routines for the serialization and deserialization of those objects since those don't need to be calculated at runtime it allows us to create uh more efficient marshalling and unmarshalling routines the ffjson tool is one example of this there are many that look like it where it generates those routines for you they're actually quite large which is uh rather common in performance generated code so i've omitted that from the slide here but that hopefully gives you kind of an idea of where that code like fits in it it fits into those functions that are known by the json package to indicate an overridden method that provides a specialized implementation of json marshalling moving on to another tool here this is the first one that we're going to look at that doesn't take go code as an input so the previous tools we've looked at take the go source code as an input and produce go source code as output this tool is a little bit different this is called sql c one of my favorite code generators and what it does is it reads sql code as input and produces go source code as output for those of you who have written sql code in go you know that it can be quite frustrating for two reasons one is again it's quite repetitive especially when you're working with large result sets you need to iterate over many rows and the code that you write to like walk over those rows objects and check all the errors is very repetitive and the second is that it lacks a lot of type safety uh it's very very common to change the columns and database um or change the type and forget to update your code have them mismatch even just change the order you query them out in a in a sql query and have them not match and run into runtime errors so what sql c does um is it takes a definition like the one that you see on the left of this slide here um which is your ddl of the table it constructs uh an understanding of what the types are in the table the types of the queries and then generates a code that allows you to write calling code like you do on the right hand side so you take a look down at the code to actually call getauthor you're just passing in a context and an id and it's running that query for you without all of the ceremony that i'll show you in the next slide that you typically have to do which is around querying that row out and scanning it into the individually named columns anyways we we use sqlc all the time at ngrok and it has made working with the databases tremendously easier let's move on into some of the other ecosystems here um this is the c preprocessor the cpu processor preprocessor is also a code generator it is a program that runs before you invoke a c compiler and it is essentially doing processing to determine what is the output program that is then going to be handed off to the c compiler this is an extraordinarily simple use of the c preprocessor to just switch between two different statements that will be included in the output program depending on some definition that's set on the command line but actual c preprocessor directives can be quite advanced including all sorts of other files together and making complex decisions even before the the program has been compiled jumping a little bit back into the go ecosystem i want to start branching out into other areas of code generation that aren't necessarily thought about as code generation gofund is a really good example of this or maybe diverge a little bit from from the the things that we've talked about already gofund is a really interesting code generator um in that we we don't normally think of it as a code generator but it is it produces a a program as its output after reading a program as input it doesn't take a whole lot of configuration and interestingly it doesn't add anything new to the program itself it simply restructures uh the the text of the program itself but in the end at the end of the day it is still in fact a code generator branching out even further here one of the very popular code generators especially used in the go ecosystem but that is multi-ecosystem multi-language is proto-c so proto-c is part of the protocol buffer grpc ecosystem and it is a compiler the proto-compiler that takes as input an idl and interface definition language that describes structures that you want to compile and service definitions of rpcs and from that it can generate marshalling and on marshalling code as well as the actual server and client side stubs to call remote services that meet that interface definition and server side stubs that can be used to handle calls for those those apis it allows you as the application developer to focus on just the application code that you need to use to call service or or implement a service and the really interesting thing about it compared to kind of some of the other code generators that we've talked about so far is that it outputs to multiple different languages the the code generators we've looked at so far output to one language or another but proto-c and what we'll look at afterwards generate code for multiple languages this is a really interesting property of of some code generators is that they allow you to write at in some ways at a higher level than you know the the languages themselves by allowing you to generate across many different languages so in some ways when you're working with some code generation tools you're often working around limitations in the compiler or limitations in the language definition but these are really working across multiple ecosystems which is is quite fascinating uh i'd be remiss uh after mentioning prodigy did not mention swagger and open api um when you zoom out at a very high level um they look very similar um at the end of the day swagger and api open api are about defining uh an rpc layer it just happens to be rest and http and json instead of protobuf but you're still from it generating the same kind of client libraries and server stubs and documentation and other things that you can generate out of that definition language so a lot of interesting parallels between those two um and and finding the similarities there let's start and start talking about some other code generators that that maybe are not as well known as code generators a really good example of this is go test go test is a code generator although from using it you you may not know it because the output the code generate the generated code output is ephemeral so when you run go test what's happening is you've written your tests in your go program but nothing is calling those functions but when you run go test what it does is it's actually reading that source code of all of your tests identifying the names of those tests and taking some input from command line parameters about which tests you want to exercise and then it writes a new program a new piece of source code that invokes those tests it compiles it which is the test harness and then runs it interesting uh way to look at this is all of the go tool chain has options to really show you what they're doing under the hood go test works like that as well if you pass it the dash x flag it will show you the the kind of raw instructions that are running uh underneath and if you dig through the output there you can find um the place where it actually like links together that test program and ultimately runs it to run the code that it generated there sticking in the testing world we'll also talk about another code generation step inside of testing which is the cover tool interestingly the cover tool of course being part of go test it's doing the exact same code generation we just talked about uh in the last slide here but the coverage tool also does another piece of code generation which is quite fascinating the coverage tool's goal right is to tell you which pieces of your code ran when a particular test was run and there are many ways to do this um you can instrument the runtime to let you know when a particular statement was called but another way to do it is what the go tool does here which is that it takes your original source code program and then basically creates a new copy of the program with a whole bunch of statements injected that mark when particular statements were run or when different blocks of code were run this is a really interesting solution it's one that you you know in in solving this coverage problem you can implement that in the runtime you could implement it at compile time and which you choose to do you know really depends on uh what is most practical and in this case the authors found that this solution of rewriting the input source code was a more practical way to solve the problem than trying to add new hooks and instrumentation to the runtime to measure the same thing let's branch out into some other ecosystems and talk about other types of code gen here for those of you who do browser development i'm sure you're familiar with tools like webpack and es build that take many different javascript files or css files and combine them all into one these tools are large and expansive and have many more use cases than just that but that is certainly one of those popular ones and ultimately what it is doing is it is taking javascript as input or css or many different files and compiling them all together and generating a new program as an output that is sometimes optimized or minified or combined but has kind of the same raw source code as the you know it doesn't really um cause a behavioral change and it is more of an optimization change so another example of a different type of code generator that's that's still moving between two different places but without changing the the meaning of the program in the same ecosystem babel is a similar program here which is reading uh one version of javascript and compiling it to another version of javascript this is really common when you're you want to use like the latest javascript features but you want to support older browsers so what you can do with babel is write your latest javascript code and it will compile it into a code that works for all of the older browsers and it does so by doing code generation understanding what needs to be done and then rewriting the code to make that work and outputting a new program it can do that for for other types of transformations as well moving between like react.jsx files and actually the raw javascript the browsers need to interpret but fundamentally also a code generator um this is an example of what that what that looks like for babel where over on the left hand side we're using a fancy new spread operator but older browsers that all the browsers don't understand natively so it needs to be compiled into something that they do understand there [Music] that type of transformation between like one uh form of a program and a different version of it or a different language is sometimes called a transpiler like a translating compiler another example of that is typescript typescript in in the same ecosystem is taking a typed version of javascript right and recompiling it into javascript but allowing you to add type annotations in front of it so again like taking a piece of source code translating it to another one this also falls into the the transpiler kind of bucket of of code generators here as we've zoomed out into like these different kinds of code generators things like transpilers things like the go cover tool and babel and the c preprocessor and gofund you start to wonder like what is code generation really like those are very different programs serving very different purposes and they're like a long way away from where we started with something like stringer that's generating additional code to compile into your go program and you start to wonder like what what is code it seems like very broad like and there are other things that seem like they should fit into that category like isn't the go compiler itself a code generator it reads your go source code as input and it outputs assembly or machine language um as its output and that's another program so that too should be a code generator there are ones that there are programs that do that at run time like v8 or the hotspot g hotspot vm that are interpreting um machine code and rewriting machine code on the fly and doing that translation and outputting a new program ultimately that's cogeneration as well there are other examples like rust macros c plus plus templates are those code generators marked down like that's not a turing complete uh translation but you're still moving from one one type of mark markup to another is that a code generator you start to wonder like is everything a code generator are are you a code generator am i a code generator like i interpret instructions and then produce code as an output so doesn't that kind of make me a code generator probably and that's a fun little meta tangent to be on not an ultimately practical one so pulling it back um down to like why we're using code generators and focusing more uh on the ones that we started with generators that produce source code that we will then uh use in our compilation steps um our future like go compilation steps we can find a number of places where the tools that we've looked at fit into the benefits that we were talking about up front i will tell you that these categorizations are fuzzy right um saying that like protobuf like is only about multi-language support does a disservice to it it also improves type safety and code reuse a lot of like what benefits you get out of code generation really depend on your alternatives like what you would do instead so it's kind of hard to pin down what categories all these things fall into but hopefully that gives you a good idea of why where these tools fit in and the benefits that they're providing and why we're looking at them for code generation so we've looked at a bunch of tools that already exist in the ecosystem for code generation so now it's time to write one of our own um i really want everyone to walk away from this uh talk thinking and understanding that code generation is an easy thing to get started with it's an eminently practical thing to do and to do that we're going to write a code generator together we're going to motivate it with a very practical use case that we we have at ungrock actually which is user facing errors so what i mean by user facing errors is i want you to imagine that you are implementing a piece of uh like a web service application something really simple where like you have to take in an email address and as part of like working with that email address before you work with it you obviously need to validate it very common thing to do in in programs right maybe you need to make sure the email address isn't too long that it can fit into your database maybe you have to make sure that it looks like an email address by matching it against a regular expression things like that and when you're first starting to write code for this you will typically use the standard library methods for working with errors like format.erf to print out errors when the input isn't valid however when you're you're kind of working in organizations and trying to create uh repeatable and reusable systems um you have users who who use your system and they will say things like my email is invalid why doesn't it work um and this is a very frustrating thing to hear right because you don't know why it didn't work right someone just told you that it didn't work but it's not clear like what problem they ran into and so one of the things that we've we've really tried to do is to make it a lot easier for users to communicate to us about why a particular error happened and why they ran into the failure that they did so one example of how you might do that is to add some kind of unique string to all of your errors right something that is very easily identifiable in here i've added you know error email to long or error invalid email and these are unique strings that users can latch on to and see ah this is like a very unique error code that indicates my problem that i have when you do this kind of thing uh folks who are using your software will often like use will take that uh error and plug it into google or they'll send it to support something like that this is a very common practice if you're working with you know large-scale software systems like a database or an operating system that tend to have defined error codes for all the different errors that they can encounter however jumping back there one thing is this is very ad hoc right uh this is just a string that you've embedded inside of another string there's no indication to anyone else who's going to write the next error message that they need to create a unique error code so how can we start to get more structured about this one way is we could you know create our own function which takes the unique error code as a parameter this starts to get into a place where the libraries that you create help the developers who are building the software to understand that they need to follow a convention to create these kind of errors and that's a good start but these are still strings right we can still do things that are unsafe with them it's still easy to not be regimented about doing this correctly so how can we take this to the next step the next level one example would be to actually create unique functions for every single error that you want to create in your software and have those functions output the string errors that we were seeing earlier with the appropriate formatting and the unique error codes when you do this it also allows you to return error types that are specialized that can include the code so that you can potentially inspect them for observability purposes or have higher level pieces of the code actually work with those errors and switch on the particular error conditions so writing each of these functions though would be really uh time consuming right if you have hundreds or thousands of errors in your program you don't want to define a new function every time that you you want to write one of these errors so um what's what's a way that we can get around that with code generation one example would be to define your errors in a reusable format like a yaml file for example is is a place to start with and that's what we're going to work with here this allows you to define upfront the set of errors that you want to to use it's very easy to add a new one and out of it you can create those functions that we were working with interestingly right it means that you can also create functions for any number of languages that you want to work with and have a unified error directory of of all the errors that your software produces so what does this look like um we're going to write together uh the code generator that reads that yaml file and outputs the error definitions that we want to use in our software so this is the very simplest version of it we're not going to go into the entire thing but what i've done here is i've defined two types that i expect to get from reading that errors yaml file which has an error definition which has the name of the error the message of the error and the variables that i can interpolate into each error message parsing out those variables isn't too hard but that's that's what we expect this load defs function to do and then at the end all we do is we execute a template with all of our our air definitions in it so to look at what that looks like let's take a look at the template this is the template that outputs that errors yaml file it's a standard go template which outputs the type that we're going to use for all of our generated errors and then it ranges over every one of our error definitions and outputs a unique function for each one of them where each of the parameters into the function is guaranteed to be correct and and type safe reading this template definition um can get a little hairy it's a little difficult to parse um really quickly so just to kind of like walk you through what that looks like this is really the meat of it where we're we're doing this for each error in our definitions file if we're substituting in the name this is where one of those names substitutes in which is the name of the function and also the name of the struct that we create the parameters then get substituted in right um which is both uh the type and uh the name of the parameter um in some cases and just the name in in the actual uh formatting of the message um and then finally the message itself since there's only one parameter in each of the errors that we were looking at the loops also disappear you'll see that there's some extra commas in there but don't worry about that that's still that's still valid and what that means is that at the end you call these functions that are type safe and generated with your unified error directory generate them for all the the different pieces of software that all the different languages that you use and still have a unified definition for them all so that was a really quick way to get started with code generation a really quick example of how with just a few lines of go code and a you know a standard definition in a language like yaml or json can be used as the source for that code generation and now i want to talk about some of the best practices when you're writing code generators you may have seen some of those in the example that we looked at but i want to go over them in more detail here so there are a number of different best practices we're going to walk over each one of them relatively quickly here so the first one was one that you probably saw in the code generator which was a comment that was at the top of the file um and it said something like this like code generator by stringer type pill this is the generation comment that the stringer utility that we talked about earlier leaves and it says do not edit this is pretty important it instructs humans not to edit the code if you don't include it it is possible that it makes for a poor developer experience when developers look at code they assume that they can change it or modify it to their needs but then it becomes out of sync with the source that was used to generate it the best practices for creating this comment should indicate for humans not to write the code it should redirect them to the source of the source of truth where they would want to to work to actually make a change to the generated file and then ideally also help them understand the command that they're supposed to invoke to actually run that code generation step sometimes if possible if the definition comes from a single unique file including the source file itself that was used to generate it can be a really helpful addition as well moving on and in a similar vein of really focusing on developer experience isolating generation files serves a purpose both for developer experience as well as making it easy to write the code generator i suggest that when you when you do code generation that the files you output use a distinct suffix or put in a separate directory away from human generated code isolating code files in this way really help developers quickly understand what is machine generated and what is what is human written and that has a number of benefits one of which is you know when you're working with tooling uh like grep for instance it's really easy to filter out files that match a certain pattern in a certain directory when you're looking for you're often looking for things that are human generated and not not code generated so that's one reason that that kind of good developer experience there's a side benefit when you're writing code generators that every time you run the generator you need to remove all of the old files so making it really easy to remove all the old files becomes a benefit in in the construction of these generators as well let's talk about the actual like implementation of the generator so in our errors example what you saw is we used a string template file uh or our string template to actually serialize out the code that we were generating it's very tempting when you start going down the code generation rabbit hole to find a couple packages in the standard library like go ast and go printer and say ah and abstract syntax tree that's what i want and to try to construct the code that you want to generate as a set of ast nodes and then use the printer package to output them it's often not the right thing to do um and and almost never the right thing to do when you're getting started um string templates usually are a better choice to get started with that's not always the case there are some cases where you do want to be working at the ast level like transformers like gofund are a really good example but the thing to keep in mind is that code generation can often be difficult to read we walked through that you know very simple template like one at a time because it's difficult to parse for humans so trying to keep things as simple as possible to reason about is paramount for uh the maintainability of your generators right so the simpler you can make them the better and that often means that just starting with string templates for for writing them out is the best choice to get started with nice part is you can write use those string templates without having to worry so much about the format of your code because you can always go fund it afterwards which should always be the last step of any code generation tool you write in ingo i would be remiss in a talk about code generations to not mention the go generate tool this is the canonical way to instruct the go tool chain to run a code generation step before you compile before the go generate tool existed it's very common right the interesting part about code generation steps is that they exist outside of the normal tool chain of compiling a go program right the go tool chain is go build go install but if you have to run some kind of like stringer thing beforehand or a mock generator it's not clear that as a developer you're supposed to do that so uh what we added to the language was a directive that the go tool chain understands means that it should run a program before it actually runs its compilation that it can run a particular step before it runs the compilation step and there's a command to invoke that will go find those directives and run them there's an excellent blog and documentation about it it is kind of functionally equivalent to make but in a very ghost specific way and one that works cross-platform without having to make sure that you have any other tools installed so if you're building these things into your go programs definitely the the preferred way to make sure that those generation steps are easy to access for other go developers last kind of point i want to touch on is the question about should i check in my generated code um there's no one-size-fits-all answer here there are pros and cons to going in each direction on the plus side when you check in generated code as someone who is contributing or working on the project when you download that code you clone it you can just invoke the go tool chain to build or install that program without having to worry about first trying to figure out what are the other steps that you need to run first to generate the code that you want to work with in some cases your program like if you didn't check in the generated code your program wouldn't even compile without the generated code being checked in in there so that's one thing to to certainly trade off it also right it avoids like adding generation steps to the build process where those can be like kind of more human steps if they're things that change very rarely on the other hand it does make doing pull request review quite challenging when you're looking at incoming changes and there is you know like one line that actually needs to be looked at but you see that there are like 10 or 50 files that have changed because of uh generation that can make reviewing those kind of changes quite challenging so to address that problem there is a file that you can use called git attributes which allows you to annotate a specific set of globbed patterns as being generated and some source control managers like github for example will pick up on this to know that there are a certain set of files that are um generated and will automatically collapse and kind of hide them so that your human reviewers know that they're not things that they need to look at and can instead focus on the pieces of code that do matter which are the ones that were written by humans all right so just quickly recapping here and going over what we learned code generation is at its heart about programs that output other programs there are a wide variety of use cases for it it can help you solve a tremendous number of problems and make writing more code with less lines of code really easy and the other thing that i hope everyone took away is that code generation is eminently easy it is easy to get started with and it is not really that that complicated to get started with and you can do it too to really start scratching your own itches and and solving your own problems it's been a delight to be here at gophercon i'm so happy to be here and to share um about codegeneration with you and i'm looking forward to seeing you all at the future gophercon hopefully in person one day thank you so much all right thank you so much alan for sharing all of your knowledge best practices everything um you know this is a super important topic in the ecosystem we see some of the biggest go repos out there that are doing code generation as a core feature you know it's it's mission critical for them um which i think it you know just goes to show sdks yeah so i did see a few questions in peanut gallery so do make sure to join alan uh in his channel in discord uh if you've got those questions you know he'll be there he'll be there answer him so do go do go join him um yeah so one other thing that i noticed was he mentioned start with string templates and not go ast which is reminiscent of don't use go routines yet you know the first first run through your program or i'm gonna i'm going to adopt containers therefore i'm going kubernetes i'm doing everything all at once yeah you know you can get pretty far with a basic implementation you don't need to go kubernetes you know you don't need to go go ast yeah and i mean i love kubernetes yeah it's interesting that it's like all or nothing most of the time right when people adopt new technologies yeah yeah so one thing i want to point out is we're we're like super light on the heckling so either people are like asleep or we're growing on them they may be starting to like us i have a guess which one of those two it is well there was one thing uh dylan in the peanut gallery kind of nailed the purpose of the channel right on the head he just basically said trolling is the primary person purpose of this channel it is yesterday i discovered that the context of what peanut gallery means actually is lost on a lot of people and that makes me feel really old so [Music] if you're not aware peanut gallery comes from back in theater where the cheapest seats up front um where most of the hecklers and stuff would be they served peanuts because they were like a cheap snack and they would often get thrown at at the performance in disapproval so okay that's why it's peanut gallery okay so there's no peanuts cannot be thrown over the internet okay so discord is the next best thing what we can do is we can throw the show over to mark bates with the next round of lightning talks indeed and i'm particularly excited about day three because it's the last day that i have to say nice things about mark and i really we can't thank mark and all of the presenters enough for putting together all of these fantastic talks and please please please apply to do lightning talks in the future hopefully in person next time yes uh take it away mark hello everyone uh and welcome to day three of the gopher kong lightning talks now look i said on wednesday that we had an incredible slate of talks and i provided i said on thursday we have an incredible slate of talks and you know what i delivered guess what i have today for you an incredible slate of talks it's remarkable how every time i get to present the lightning talks it's just full of amazing people and amazing new talks uh as i always say this is an honor and a privilege for me to get to bring on fresh new voices and fresh new ideas to the community because that is how we all grow as a community and to me the lightning talks are one of the most important things we have as a community allows these new speakers to get a foothold in the door and get their voices heard so here's what we're gonna do today i would like to go over the rules of the game here before i bring up our first contestant as always each speaker has exactly seven minutes to give their talk and to convince you that their point of view on their subject is the right point of view fyi my point of view always the right point of view just kidding um these people are talking about stuff i have no idea about i'm really excited um so here is they have seven minutes at one minute mark if there they will hear this sound perfect and should they hit the seven minute mark unfortunately they will be yanked off stage with these curtains there we go boy my finger's huge when i do that isn't it um everybody at home on a big screen tv is like whoa cut your fingernails oh actually i just did so they're not saying that anyway with that said enough about my cuticles uh let's bring up our very first speaker andrew are you there yeah i'm here thanks very much it's nice to see you where are you coming in from i'm normally in edinburgh in in the uk but today i'm coming from downtown los angeles oh so right next door just about yeah i was excited to come join you in person but it didn't work out so maybe next year maybe nice to see you brought the rain to los angeles then i think that was my fault sorry that was your fault well anyway take it away andrew love to hear from thank you so much um it's definitely very exciting to be here and and presenting today my name is andrew i've been coding go for nearly four years now i think and fine has been my focus that whole time so today i wanted to show you how we can use the fine toolkit to get up and running with an app and deliver it to the store in just seven minutes although now that's six and a half so i just wish me a little bit of luck for anybody who hasn't heard of it fine is a graphical user interface toolkit for building platform agnostic applications so applications that are written with go and fine will build and run on just about any graphical platform i'll talk about the specifics a little bit further down um and so these are going to work the same on on every platform that you run them on although they will adapt resolution and match light and dark modes to the current theme and so forth so it feels about right for the person running the application and the fine project aims to be the easiest way to get up and running in cross-platform application development it tries to be really easy to learn and also it encourages your code also to be maintainable so other people can pick it up really easily so first of all i wanted to run a really quick setup now i know most people will be familiar with this already but right from the beginning we want to create our project here i'm calling it my project i will go into that directory and run go mod init for my project so now we're up and running with our go project we want to add fine as a dependency so we've run go get find dot io slash find slash v2 that brings the latest version which is part of the v2 um group of versions into your application and really that's all of the setup necessary it does use some c behind those scenes so if you haven't previously set that up you might find that there's a dependency or two that you need to grab there's more details on our website but i won't go into the specifics just now if you would like to test that your system is working with fine or if you'd like to explore what's available you can run the find i o slash find v2 cmd find underscore demo there which will make sure that everything's working correctly the window will appear and it will show you all of the things that a fine application can do so we're up and running i think next we need our hello world so here is a basic fine application probably the simplest one and for anybody who's familiar with uh go application it should look pretty straightforward we have our main package we're going to import the app and widget sub packages from fine and we'll see how they're used inside the main function first of all we're creating a new application using app.new now that's going to do all of the system discovery wiring up dependencies and making sure that everything's going to run and from that application we create a new window we've titled it hello on some systems this title will appear above your application on on others like mobile devices you may never see it but it's good to give it a name then we give it some content in this situation we're just creating a new label from the standard widget package with hello fine as the content and then we pass that into set contents as the main element of the window and we tell the window to show and run which is basically showing the window and then running the application it's just a little shortcut so when we run this code you'll see this window in front of you assuming that your current system is is running in a dark theme probably no surprises there hello fine prints to the window that wasn't really the most exciting thing so today i wanted to cover something a bit more let's make a markdown editor in fact the code isn't that much more complicated than our hello world you'll see that we've included the container sub package as well let's see how that updates things so our window this time is titled markdown editor and the widgets that we're creating are a multi-line entry widget which we're calling edit this is where we will edit our markdown code and we're also creating a rich text preview widget we're using new rich text from markdown of course the markdown here is empty because we're starting from from no content but you could pass it any markdown document in there the key part of the logic of this application is that we're telling the entry that when it's changed it should pass the data through to the pars markdown function of our preview widget and that's really it next we're setting the content of our window and we're using an adaptive grid from the container package and what this is doing is putting two items side by side but when we rotate into a vertical context on a mobile app for example the grid will adapt to the size available we'll see that and we pass it to the edit for the left or top and the preview for running next to it and so we run that code and you can see here we have a markdown editor running on our desktop we've got the content on the left that's currently being edited and a live preview on the right that's pretty good now we want to package it because a go binary isn't really a full desktop application we use the helpful fine tool here which you can get install with that command there and using find package we build a graphical application which can also be installed using find install onto the current computer we could also build for other systems like using the windows os as a parameter or in fact we could build for mobile devices like our android platform where we need to give just a little bit more metadata in fact you could build for lots of different platforms if you don't want to manage the tool chain there's a fine cross binary that can help you with all of that and here's a list of all of the platforms that we're currently supporting asterisk is a little bit of a work in progress but it's all in our main repository you can see here the different applications that have been packaged we've got on the bottom left android we could double click the app on our android mac or we could run the windows and this is it running in ios simulator or in landscape mode as you can see there if you've rotated your device and to release we're going to then use a similar command passing in the credentials from the operating system that apple might have issued us or perhaps that we've obtained and so i can't go into the details of all of the platform specifics but you can see there that that's how the helper command will get you through once these have been built you basically just take that and do it like you would a normal native app so we can drag the ios ipa onto apple's transporter app or we can upload it into the website for google play or the microsoft store and really that's it we've taken our first application all the way through to the app store there's so much more i wish i could talk about date binding file handling standard dialogues animations and so forth but there's just not time today if you would like to know more please join the fine channels on slack or discord and if you would like to know even more or you'd like support in for your business or your project do give us a shout at findlabs and we'd love to help you out with your applications i really enjoy the rest of the day it's been great to join you today thank you andrew i i have to say that that was a really fine presentation sorry i i mugged for the camera and i don't know why uh that was a terrible joke let's bring up our next speaker who hopefully have better things to say than me amit you there yes i am welcome welcome where are you calling us from i am calling you from sydney australia australia wow we have time zones all over the place today this is amazing excellent well you're going to talk about hugo and go in bed right yeah exciting stuff go for it all right thank you mark all right hello everyone my name is ahmed so today i'm going to talk about how i converted a static website um into a go executable using the go embed package so i want to start off with the problem statement um and i also want to start off with that there was no problem to solve i wanted to host uh you know the wanted is the keyword to host my new static website in the cloud on a virtual machine i did not want to do you know the usual way that i have been doing and most of you probably have been doing using you know things like github pages s3 there are various solutions that we can use but i wanted it to do it differently and hence my problem so i started thinking about the solution um so my initial plan was okay i'll just generate the html the css files i'll copy to the server put in probably nginx and be done but then i was thinking okay um you know i have been working on go recently a lot why not uh i try to write my server in goal and serve the files from the file system so that was a second thought and then you know the lightning well uh excuse the pun struck me and then i thought of the embed package which slowly uh uh and gradually when it was introduced in go 1.16 has probably become one of my favorite standard library packages and so this was what i thought would be the solution uh to the problem that i created for myself essentially so i'll write up my co um content in markdown i'll use hugo um generate html css and then i'll write my server you know with everything embedded and then deploy or copy the binary to the virtual machine in the cloud so this is this is how my first pass looked like so i created the new ego site i wrote my content i uh generated um the html and you know put in a nice theme got the html and the css files and then i went into so hugo by default generates the content into a directory called public so i go into my you know change my directory into public i initialize the go module and then i start coding up my server so this is uh the first key aspect of of this solution is writing the correct go embed directives so here you can see that i have got three uh blinds of going bit rectifiers i wanted it to be this way so that it's just uh you know one line is not uh like too long um of course i could have put it one line the key is of course the site data variable which is of type embed.fs which is essentially in simple terms it's creating a file system based on the the files that you have given it and the reason for doing it this way is so that we can then essentially create a handler for this file system that we have created using the http.file server and http.fs functions from the net http package and the rest of it is how you'd usually just normally create a http server we create a new mox we then call the listener insert function passing in the marks and that was it and then i built it so i built my server using the go build now uh so and then i sap my server which is the binary to my host one thing that tripped me off the first time i tried it was i just did exactly what i've written here but then i realized i'm going to deploy this on linux system but hey we know that go makes it very easy so use the go os and the go arc environment variables so i was hosting it on linux on an arm64 machine so i just did the go build and then copied over the binary once more and i copied it to this location which is which i'm calling local bin practical go website now the naming will become clear in a bit and so this is a linux red hat's flavored uh distribution that i'm running so i create a systemd file because i want this program to be running in in the background and it's a long live process essentially so this is my systemd file which is uh init process on linux systems and init manager sorry and i'm running it as a user nobody so that it has the least privileges um possible and i'm also setting up the listener address to be h08stro so my my server is going to be listening on port 8080 um i then create a dns record and pointed it to the public ip address of my virtual machine so you have the dns hooked up with the virtual machine now by default the dns https i want https so that means i need something to be listening on port 443 as well as we i need a tls certificate now remember my website or my server is listing on 8080. okay so what do i do i bring in paddy so i installed caddy via the repository that the project provided which meant it was automatically a systemd service i created this very simple cd file the practical gobook.net so that's the dns record and then the key part which is the reverse proxy so all i'm telling you to do is any requests coming into this uh virtual machine or or to academy on practical go book.net hostname just reverse proxy back to localhost 8080 which is where i started my go server with all the embedded um you know data in it and super easy started caddy and the thing that the certificate is obtained successfully okay and the result is and this is the big reveal i my new book is well coming out sometime this month and that's called practical go and that's the website i created for it so that is a problem i care for myself i do encourage you to check it out um i think a lot of you will find this book useful now okay going back to the topic of this presentation so and then when i do a call i can see that the server is scady so it's served like um you know caddy it's confirmed it's also http 2 and it's https so awesome now i wanted to improve things so i did a second pass and that was a lightning talk driven development so i created an automated way to create this initial server dot go and go dot mod i put it on on github it's the simplest of program you can think of using go templates and what i can now do is instead of having to create the go.modern server.go manually ah excellent thank you joshua uh and i agree the comment bed package thank god uh as somebody who has maintained a couple of those embed packages um is definitely uh i'm definitely very very happy with that ah so with that said uh let's bring up our next speaker joshua are you there yes hello how you doing yeah pretty good i'm good we're joining us from uh melbourne australia so okay so we got another australia wow these time zones we are all over the place today uh and you're gonna be talking about grpc which is something i know nothing about my business partner loves it i bow to his knowledge so i'm very excited to hear another person's take on it so please take it away um yeah so i'm going to be talking about how to automatically generate a cli for your jrpc apis this is a project that i started and i've been doing in my spare time and it seems to be really cool um the links and the problem the task that um we're going to be doing is we're going to be integrating with the billing ctl proto in google apis so this is just a um for people who have used protos before we have a service definition we have some methods um and we've got some types and so all the information that we need to interact with this api is right here um and the to begin with um we want to just quickly um use this this is just getting some authentication um tokens from the gcloud api because that's what we're going to be integrating with and the tool um building ctl is an example um using the library that i wrote so we can go cloud catalog list services we can see it's all got autocomplete we can go page size 20 for example we can add an authorization header in there so we can go authorization and then auth and [Music] there we go so we've got we've just called a grpc api um and so now i'm gonna go through how that happened um the basic problem um which my library is trying to solve is um when you start getting a lot of uh teams with proto uh grpc apis you start having like a whole repo that looks like this you have a lot of protofiles or maybe they're across um different repositories um but you need people to be able to call it um easily the first solution that comes up is using um grp curl but the problem with using grp curl for those people that don't know it's just like curl um and you can call a grpc api and you can specify the address but you need to know all this information so you need to have a piece of documentation side by side or run some commands on the side and you can add your authorization headers and then all of your information um so it's difficult to write because you need to know what you're actually integrating with um or what you're calling and it's a lot to remember so a better solution to this is using a cli um library like cobra and how this would work in a organization is that every team would have their own commands that they're responsible for managing so this makes it easier to interact because you can have things like auto complete and stuff but um it also means that it's difficult to maintain because and every team might have different implementations for their apis and so this is where grp ctl comes in what it does is it takes the file descriptors that you see in the generated code and if we look at this this is a piece of generated code that the google grpc code generator generates um and it generates this uh proto reflect.file descriptor file um and what this does is what this library does is it takes all the information within that to generate a cli tool that you can use on your um on your service um and so that's this uh these string of bytes here and it has all the information you need so if we see a service descriptor um these are things that are sort of contained within a file descriptor a service descriptor um that describes what the service is um a method and that gets generated to the first level command within the generated cli um and the met the method descriptor which in our proto file looks like this so get billing account for example um and um that gets generated to the second level um and then the message descriptor gets um generated to uh the flags so if we can see there i said page size 20 um that has um uh yeah so that's all generated and it's really easy to integrate with our cli tool now um and then what you get is a cli tool that was really easy to write we can see that it was only a couple of lines of code it's customizable and it can automatically pick up extra information from the profile and one of the things that you might be wondering is like how does this know where to send the actual request um on the google apis there's this annotation which is google api default hosts and that is set to cloudbilling.googleapis.com and so if we actually go to billing ctl and go help um the package has actually picked up that piece of information and put it into the uh defaulted the um the address string to cloudbilling.googleapis.com um so um you don't need to specify like where any of that config would be um yeah um and as a bonus um there is also a gr pc reflection um so if the if your grpc service has reflection enabled you can dynamically sort of call it so grp ctl i can go address equals localhost 8081 plain text equals true uh click tab when i click press tab um it does this the server reflection call gets all of the protofile descriptors like i was talking about before but it gets that directly from the api from the service um and then generates all of the things that we need to be able to call it and um you cannot know um what this you can only know the address and it will be really easy for you to integrate um as compared to something like uh proto grp curl um and so yeah that's the link um there's a go for slack no one's used it and if you want to see these slides that's the link there um cool thanks well thank you very much joshua it was a great talk i really uh you know grp one of my biggest problems with grpc is kind of exactly what you just said you know everybody's got different implementations so it's it's great to see people trying to corral that beast a little bit um excellent work so with that said let's bring up our next speaker liam you there hello yes i am here hey and where are you calling us from i'm currently in london so in the uk fantastic nice i'm fingers crossed i'm going to be there next week if my flight if my flights don't get canceled again yes they've already been canceled once so you're going to be talking to us about go uh in the edge like iot stuff right yes yes i am oh fantastic i love iot so i know nothing about it but i love it go for it let's hear what's okay hello everybody it's so great to be here my name is liam hampton and i'm a cloud advocate at microsoft a really quick uh agenda so what is tinygo why is it used what have i made with it personally and what am i doing next so let's start with saying what is tinies go it's a go compiler for small places as quite rightly stated on the readme on the github and tinygo.org so what does that really mean it's a subset of libraries from the standard library in the core language package and it's based on the llvm compiler infrastructure which is a collection of modular and reusable compiler and tool chain technologies now hopefully my slide can illustrate this quite well it is essentially cherry picks a number of really useful libraries from the main language which you have to have installed prior to having on your machine such as io bytes archive zip and then it it disregards things which are not necessarily seen as important like html templates or the net package and this is purely because um some of the 32-bit boards that you'll be using it on probably won't you probably won't be using it for that um so let's have a look at the cli it looks very very similar to the go cli very similar commands so you have build run test just with a few added extras like flash when you want to flash your board gdb and so on it's it's pretty similar same from same what is it used for well people often use it for microcontrollers and there's about 60 of them it supports 16 more than 60 microcontrollers arduinos bbc micro bits adafruit playgrounds and the x9 pro smartwatch is some of the ones which i've seen used the most um they're really popular boards and this is just a really fun way to get into iot if you're a hobbyist like myself it's also used with webassembly now if you're not sure what webassembly is then it is essentially lets you write your code so in this instance tiny go and pass it through a wasm compiler which turns it into machine readable code and then when you go to run the application through the web browser you can right it runs it because most web browsers support this and from that you get near native performance of your web app which is fantastic so where do i personally use tiny go i use it on arduinos now a little side track here uh i bought a bunch of arduinos thinking cool i'm going to run some home automation and boy was i stuck when it came to coding them because i didn't know c plus plus or micro python or anything like that i would i don't use it in my day-to-day development i use golang so i thought light bulb moment let's try and put golang on a microcontroller see what i can do so to prevent my lovely house plants looking like this which looks a bit scary it's dying in fact that one's actually dead now i created something like this i created a soil moisture sensor and it takes some calculations so it takes an input from the plant itself takes it to the arduino does some cool calculations uh and then spits out some human readable output for me that just essentially tells me i need to water my plant when it's getting thirsty and it helps my plant look like this which is over my shoulder it's growing quite nicely behind me secondly another really cool project i've made is a night light this is taking a photo resistor and and it start when there's no lights then obviously the lights come on and when there is light then the led lights are off for those that you want to see the boards um it's just a breadboard an arduino with a couple of resistors a few wires and leds nothing exciting but it's the principle behind what you stick in your garden at night with those solar panels on top the syntax looks really cool it's exactly the same as go so you have um you've got your bite arrays you've got your slices you've got um ranges it's all the same syntax it's really really fun to get into if you know go so what next what am i going to be creating next with tiny go so for my soil moisture sensor i'm going to be creating something which will edit water itself so i've asked santa claus for a water pump this year so i'll be able to implement this one and hopefully get that one going for my nightlight i am going to be implementing this in a letterbox and that may seem a bit crazy um however for context i live in a block of flats and in the block of flats it's a real arduous task to have to keep running downstairs to check the post every day so to prevent me from running down those stairs every single time i want to check the post i'm going to be putting something that looks like that in my letterbox and that will just tell me that when postman pat has been and delivered his lovely post then i will be able to get a notification and through that hopefully using mqtt iot hub in azure i will be able to do just that and there's a lovely little gif for you there secondly there is a tiny go playground so much like go you can just go online and it's an online emulator this will let you play with it sort of choose your hardware your specifications write a little bit of code give it a go and off you go i've actually written some pretty complex tasks on here where i have played with multiple boards creating multiple little loops and lovely led light shows just for a bit of fun like i said i'm an iot hobbyist i don't want to make the same mistakes i did previously where i buy a load of boards don't know how to code them and stick them on there so i thought i'd better give this a go but um yeah i'm an i.t hobbyist and thank you very much for listening to my lightning talk um if you'd like to get in contact if you'd like to talk to me about this um in fact if you're going to give this one a go please do let me know and you can contact me on twitter github or contact us and watch some meetups through the microsoft reactor for my work and that is a short sweet lining talk it certainly was thank you liam you're welcome thank you very much much appreciated uh it was excellent uh tinygo i still gotta play with it you know one of the things i miss about gophercon uh be not being in person is uh ron evans um and all of his hardware shenanigans uh where normally he would have a bunch of people in a room everybody hands-on playing with stuff time to go uh and every year of course he tries to kill me with a drone we all know that but you probably don't know is he often tries to get me arrested through tsa you routinely give me and this is the truth a brown bag full of just wires and capacitors of electronics and just random stuff to take on my plane ride home just a lunch bag and i don't know what any of it is and if i get stopped with a bunch of wires and stuff i can't explain what they do and they say well where did you get it i have to say well here's a picture of what happens this guy took me gave me these things to take on the plane so you'd probably never see mark again uh so with that said uh let's bring up our fifth speaker here today uh gabriel are you with us how you doing mike good i'm i'm doing great how are you doing good too thanks fantastic where are you joining us from so i am brazilian but today i'm speaking to you from istanbul turkey wow awesome i love istanbul i was there actually when i first got it to go i got to spend a weekend in istanbul with blake mizurani and andrew guerin who was on the go team at the time and that that turned me to go uh so have a great time i'm really excited about uh learning how to write some good unit tests go for it thank you so much so hello everyone uh i uh i am gabrielle busulu i'm working for macarthur live in brazil uh and today i want to pick to speak a bit a bit with you guys about mocking ingo uh i came i started to work on mecha delivery and working with goku by consequence uh six months ago and before it i was working with java and i when i did this transition i start to see how things works on golang and how people are making effort to do things simple so i didn't install too much frameworks around it people doing everything more pure stuff so even uh i knew it how to do some things in java but it was always surrounded by frameworks and i started to struggle to do simple things like mocking uh on goallength and uh reading some articles readings videos on videos always missing that enlightenment about the thing and today i'm trying to bring these things for you guys here so here i have uh i i have built this api to doing some course about uh gold and game testing and this api basically is connecting to uh an api that we have sorry for this we have one uh our company it's a that we're throwing the uh the country and showing some information about the country so uh basically just have one one url and we have a controller this controller and receive the id going through the service we have a provider and get the the [Music] vr here and you throw to our api good so what is the problem here the problem is that uh if i want to write when i run it running my my test here let's draw here test i have a test that i want to uh actually that's the behavior that is breaking it so country not found but when i'm running my test because uh on goal is too easy to just throw functions everywhere and uh basically we like on java we are obligated to put classes and uh always feeling with up so much things and chunking from works and here we go just throw functions and make things work so here when i'm doing my test it's working and the problem is that it's going straight to the api and returning let's see here um the result it's returning at 200 on my test that i want uh not found so mocking it's coming for solve this problem for us so um what we can do here to solve this problem uh because of time i didn't will write code with you guys i already have here on my master i will check out here so uh one of the things that we can do to solve this problem is that like if we check let me back for my other range uh as i said before it's easy to just throw functions and get the results that we want so before it i was going straight to a function and how can i mock this behavior uh and the other thing that is different on golank it's how to do their their deal we are dealing with interfaces here uh we can do composing with interfaces it's so easy you just throw interface here and you implement download it is so it's a bit different for me was working with java before so here i just wrote the function and the function is calling out the function function function so for mocking one of the strategies that we can have is that let me check out here it's start work with interfaces and on my service now i have an interface that now i have uh the behavior that i am expecting from the country from the this service and making this function now associated to the extract so now i can implement this interface it and because i have this interface here on my controller now i'm able to uh create another struct implementing the same interface and this is interesting on going like we can it's it's not a bad practice you implement some stuff like this uh so here we i implemented our interface i'll call it mock and i'm returning a function that i declared here the type of the function and when i'm going through my test now i can because of this function that i declared here and i'm using on the function that i associate with my mock i can say the return that i'm expecting so uh because of this now if i run my code yeah it start to pass because now i can give the behavior that now i can mock my function so uh the thing here is not presenting anything new but for me it was kind of enlightening enlightening you know like i've been working with chunk frameworks and they are doing all by yourself and you keep you keep other something sometimes not thinking how to do simple things like this just creating an interface and i can mock the behavior because now i i accomplish the contract of the interface so i can associate it here a function with return the behavior that i want and now when i'm using the server here through the country yeah it's working it's working fine it's giving the result that i'm expecting that i'm putting here so uh now i'm not connecting but here i'm mocking a simple function and we maybe we can we need to mock uh httpr request so i would like to uh give a tip for you guys i'm learning a lot from this repository here learning go with tests and this part of mocking is awesome enlightening a lot it's written from if i'm not one chris james and this article is excellent article how to create mock uh hp mocking in golang it's from sophie uh and this code that i i present to you guys it will be on my github here is not yet i will give posh and if you want to keep in touch my link is here thank you so much guys well thank you gabriel uh that was excellent uh yeah using interfaces and mocks uh and testing is you just have to it's it's definitely the best uh way to go about doing it especially when you start getting into some of the really really complicated uh types of things when we talk about you know http and talking with files and all sorts of stuff so it's an excellent talk thank you gabriel and hopefully others will uh get that aha moment too so with that said talk about aha moments let's bring up our next speaker tom who's going to talk to us about uh data science program at northwestern is that right tom that is right fantastic and are you in some sort of ionic chamber there whatever you have going on behind you i am in a vocal booth okay sound proof because i i live in glendale california as it turns out but the apartment uh building has many dog owners oh a lecture and the dogs start barking so this is where i come to do the zoom sessions fantastic i love it i love it well great i'm very excited to hear what northwestern's doing with go so please take it away thank you years ago when i was a a student at minnesota that's the home of the golden gophers i learned a lesson about programming in academia at the start of a course in robust statistics professor said i don't care what language you use for assignments as long as you do your own work i had a facility with fortran this is a few years ago but was teaching myself pascal trying to adopt a structured programming approach taking the professor at his word i programmed the first assignment in pascal while my classmates use fortran the first assignment comes to i walk my paper a program listing to the front of the class and hand it and it to the professor he looks at it quizzically and asks what's this i explain it's pascal you told us we could program in any language we like as long as we do our own work to which the professor says pascal i don't read pascal i only read fortran lesson learned academics are not especially open to new programming languages data science students today at most universities would have a similar experience if they handed in assignments in go or rust or any number of other contemporary languages rather than python with machine learning applications and ai python rules the day but watch out data oceans are choppy sharks are approaching recall the words of chief brody de quint in the movie jaws you're going to need a bigger boat i would suggest that that bigger faster boat be built with go now much what what i say in the next minute or so is not new to gophers remember that excellent presentation by carmen ando in 2017 the why of go go is machine efficient it beats languages that are interpreted as well as languages that depend on virtual machines go offers a rich set of tools for concurrent processing it is well suited to today's multi-core computers go is programmer efficient python may be touted as easy to learn but i would argue that go is easier than python go is simplicity by design a language with only 25 keywords it's easy to read easy to use and easy to maintain over time think of software development as a game of jenga we want to address the blocks at the bottom of the stack while ensuring that the entire stack does not collapse go lets us do this and i'm happy that the leaders in the go community are reluctant to add new features donald knuth had the right idea when he got to version 3.17 of tech he declared that there would be no new versions no new features only bug fixes and with each bug fix he would borrow another digit from pi changing slowly and carefully promotes programmer efficiency keep it simple keep it running and what about that software stack the infrastructure when python even bolstered by c or c plus plus is not up to the task data scientists turn to other languages and systems here is a popular so-called solution to python's performance problems data scientists turn to spark which is built on scala which depends on the java virtual machine and to provide easy access these well-meaning data scientists add pi spark to the mix is this the best way to address python's performance problems no consider a simpler stack it's go just go on the first day of this conference daniel whitenack demonstrated various ways of implementing data science solutions with go we can do this today companies value safety simplicity and the performance of go they recognize its strengths if go is good enough for google netflix uber dropbox paypal american express capital one salesforce silo and many others then go is good enough for the rest of us if go can provide an effective platform for docker kubernetes prometheus pachyderm grafana crowdstrike d graph cockroachdb aerospike and a diverse array of distributed solutions and cloud native microservices then go can be an effective platform for building data science applications using go for data science doesn't mean we have to forget good things the r and python languages provide we can be multilingual at northwestern we're doing just that we're using go along with r and python here's how the three languages for data science rank today according to i triple e python r and go in the top eight go is the newest of these languages trending up and with strong job opportunities take note gophers go plus data science plus data engineering is a winning convert combination these are languages courses that we're using go where we're using go as a primary language at northwestern and today if you go to a northwestern professor you may hear these words i don't care if you use go python or r as long as you do your own work organizations need data science and data science needs go we've been doing distance education for more than a decade most of our faculty consult with industry or work in industry they're practicing data scientists we offer five specializations analytics management analytics and modeling data engineering artificial intelligence and technology entrepreneurship most students hold full-time jobs in industry and or and study part-time if you're interested in data science or data engineering love to learn new things and want to work with go northwestern is the place to be well thanks tom that was excellent if i wasn't already doing a master's program i might consider it thank you so much it's really great to see go getting into uh universities because i kind of agree with tom i think go is a great first language i think it's better than python as a first language because you don't have to deal with the white spacing and uh the dynamicism went but again i might be a little biased i really like go as i'm sure you all do too i think go is an excellent first language so with that said we have time for one more talk so let's bring up our next speaker my hash you there you're muted currently sorry hey mike i'm here hey how you doing i am doing excellent great where are you joining us from i'm from mumbai india okay fantastic very very exciting we are all over the map in this session here i love it we've got people all around the world and you're going to talk about uh using go in the embedded world correct yes yes i'm going to talk about go in embedded world can you see my screen absolutely yes you're all set take it away thank you uh hello everyone i'm going to talk about an interesting topic go in embedded world and i think there are so many presenters already have talked about a lot about how you can use go in different embedded devices and products so i'm not going to discuss something very specific to any product what i'm going to discuss is just a single problem statement can we use go as a single technology for both cloud and device side development so this particular question came up when i was in a startup few years back and we were trying to find a very lean tech stack for both our devices and cloud site so we were pointing on different languages and we chose go over there let's talk about it i'm sure that there'll be so many other teams and companies who are also having the same dilemma whether they can have a single tech stack which they can use in both dev devices and the cloud side now first of all is go a solution if it is then why so as tom has already told a lot of things about go right i'm not going to tell you why go is really you know awesome language for the cloud side multi-core support easy for multi-threading faster development that the test framework in build test framework error handling um garbage collections so many stuffs right that helps go make sense for the cloud site development now the question is on the embedded cell on the device side right first of all this is a statically linked compiled binaries the points that i made here is very you know crucial for any embedded developer whenever it is finding which language he should use and how that can be helpful so first of all statically linked compiled binaries you don't have to worry about the dependencies easy to build and it is very only dependent on the kernel right the third point is the empty interface equal to void pointer i put this point you know after a lot of thought so any embedded you know developer who is working with c if you ask him void pointer is something which he always mostly use for transferring data from point a to point b right it's like really gold for him now what go provides is an empty interface which is you can use a void pointer not only that but you can use empty interface or concept of interface to design provide a pattern to your code so that you can you know use a change or support any new changes as quick as possible how good is that right now next what two key words go os and go work okay now when i was working with c couple of years back the main problem you know well that's one of my demands is i was not so good at make file and cmx files whenever i have to change the platform or whatever i add some new dependencies i always get stuck there and need a lot of help the go provides a beautiful way of procreating libraries creating uh creating your executables based on these two keywords google's go os and books are just tell which operating system and which architecture you want to build and it will build that binary for you as long as it's supported right and if you're talking about go in embedded world and if you're not talking about tiny go it doesn't make sense right tanygo library really makes your life much easier as i am better developer now as liam uh already told us about how easy to use tiny go and how easy to you know use this library to create a particular product as quick as possible now on top of that the thing uh i think liam told us that the 60 plus microcontroller available in tiny go i looked at tinygo today it was 70 plus 71 or microcontroller so the amount at which the acceleration is happening the amount of microcontrollers supported in tiny go is really huge not only this there are so many other libraries are now present embed go embedded to and it go there so many libraries are present which makes any embedded uh developer life much easier but enough positive let's talk about what is not there in go so if you're talking about embedded systems talking about microcontrollers you have to think about the advantage that go brings this that is a garbage collection and how this is a disadvantage for a microcontroller you may have to tune it down because you don't want your microcontroller to you know use this garbage collection feature and use it sleep cpu cycle for that this is something that we need to take care of in the go still there are some platform which are not supported so if you are working on a device which is not supported in go then you have to wait or you have to switch to c right now the third point i added it here to make sure that there are so many critics of go which says that go is having a lot of issues with the feature list that they have and it might not be good suitable for the you know embedded systems absolutely right even as for rob pike in gophercon 2014 go was actually not designed originally for the embedded usage but look at the timeline after that the amount of evolution that go has brought us it is completely you know awesome and amount of support the device support device side development support that go has right now as compared to any other language except see it's going in a much much you know higher speed just to give you how go looks like as a code as compared to c this is the same code this is the same logic that you will need to you know toggle your led in a adreno board with go the left side is a go and the right side is the c just to tell you how easy to write how is it to you know develop the code in go these days as compared to c of course uh the people who are in already see you will definitely have to you know pull yourself and look into the benefit of co but yes there is an easy way to develop and learn and do that's go provides so go makes sense now conclusion now before the conclusion is very you know can picky here what we need to do is we need to understand that go helps to get things done as quick as possible right it's easy to develop and easier to uh the timeline is very quick but we have to also remember which you know tom mentioned it properly that go is extremely simple and minimalistic they don't add features as they should be and which adds quick learning and helpful in reducing and reducing cognitive friction in any developer that you want to build it right with the rate at which go is evolving it can be a good and maybe only replacement option for c in coming future in embedded world that's about me and i think we made a right choice on choosing go as both cloud and website in our project hopefully it will be helpful for you thank you well thank you very much uh mahesh that was that was wonderful uh we've come to the end of yet another set of gopher khan lightning talks and it's been for me it's just been a wonderful week we've seen almost two dozen new speakers present some great new concepts some new ideas and definitely gave me a lot to think about and i hope you as well um i hope you've enjoyed all the programming at gophercon this week um but in particular i obviously hope you've enjoyed the lightning talks and i really hope uh to see you all in person next year uh well my gopher buddy and i hope to see you in person next year at the real gopher con and maybe i get to shake your hand and introduce you in person at the lightning talks with that said we still have a little bit more programming left at gophercon so enjoy it and i'll see you next year thanks a lot back to the studio thanks to all of the amazing presenters for the lightning talks this week you know it takes a lot of courage to kind of get up there and to try to you know reduce a complicated topic down to seven minutes and there's such a a wide spectrum of topics that we always see in these lightning talks so it becomes a really you know important part of the conference and one of my favorite things to see yeah you know we can't thank mark enough for putting all of that stuff to what i take that back we can thank mark this would be like six times he's gonna start to think we like him no no more for you so and also like geographically how spread out everybody is you know i mean we had at least uh north america europe i think we had africa i don't i don't know i'm i'm sure we had india yeah we just had india as well yeah so that's i mean that's impressive we are yeah definitely and then just the quality of the talks too it's just yeah we got seven minutes and then seven minutes and seven and every single one of these talks is jam-packed it's just really good content yeah speaking of thanking folks uh we should take a minute i want to thank uh microsoft and capital one for helping us take this thing virtual on relatively short notice i would say so thanks to them and thanks to sale for salesforce as well excuse me they're providing the captioning for the conference too so all those captions you see you can thank the salesforce engineering team for that and speaking of the sponsors as well let's bring it in i'm the vms guy the virtual sessions guy today uh bring it in real close um we've got open source security and action from praetorian uh they're going to talk about that go-kart uh package that they they we have them on yep um we've got more q a with the course hero talent team uh so check them out and then we've got more ama from the microsoft team so it's called gophers of microsoft go projects ask me anything session so do check those out reminder that you need a ticket the tickets are free just but you do need to go register on the gophercon.com website um to to go to those sessions yeah and if you're not if you've not found it in the last two days you can go under the gopherton gophercon.com agenda and that's where you'll you'll actually see the ones that are coming up pop up in the top if not you can scroll down to the schedule for the start times um again feel free to also interact with our sponsors in the discord many of them are hanging out most of them are all hiring so if you're in the market for a new job this is the perfect opportunity to reach out and meet some of the engineers and folks that work at these companies that you may be considering applying for and they already go surprise surprise some might write rust i don't know you're gonna have to talk to eric for that but you know they're here to steal the couple of rust people yeah they came to gophercon to find rust engineers and go engineers so check them out do check them out but where are the people writing pony in him you know you know we're we're down the rust rabbit hole and now we're into zig nib and pony i don't i don't know what to do with you man next time we're going to talk about brain it's like cutlass out there yeah yeah yeah so i have a question for you all how do you how do you debug are you old school like println print line you know or do you yeah sort of on ide it depends on what language or technology i'm using because in some cases the debug tools are are pretty good and you know i can use gdb or i'm not typically an ide person so that's not usually the way i step through stuff but you know things have good support in gdb you know i use something like that i just write perfect code if it's if it's quick i just print lines but yeah again so yeah oh yeah and and some some would say you're you're like a caveman you're putting print statements got here like like like a peasant but what do you do what do you do if it's a complex problem and you've got to delve into this oh my goodness yeah your likability just shot up despite the t-shirt here we go screenshot time again but uh yeah so we always always like to hear uh more talks about delve in in debugging and dap and all these cool innovations yeah just give a give a nice deep dive into some of that stuff so um up next uh we have uh mr sam kamenetz who's gonna dive a little even a little deeper right show us the how and the whys behind uh um using a dell how we can be a power user yeah power users and not just you know using print statements everywhere well i'd use brain statements i think it's okay right tool for the job you know right for the job sometimes you do need to got here you know yeah it's okay it's like javascript yeah that's it that's the only hopefully though we'll learn a little bit more how to get into delve so that you know we don't always have to do a print statement we can have another tool for our toolbox yeah let's hear it from sam kamenetz take it away hi i'm sam kamenetz i'm a software engineer at bread and i love using debuggers and i'll candidly admit it took me a long time to figure out how to use dell's goes debugger debugging a hello world application was easy enough but i got flustered by more complicated setups i encountered in my day-to-day job like debugging a test or debugging a dockerized application so i learned how to configure debugger for almost every scenario in this talk i'm going to demo all variety of debug configurations as well as how to reason about them so you can become a debugging expert i'll be talking about vs code's debugging features as well now this talk isn't just for debugger enthusiasts it's also for the debugger skeptical for all of you my pitch is this setting up a debugger requires understanding the nuances of how the debugger project and ide interact and it can feel like too much work to set it up while you're hunting down a bug but here's the thing it's a time investment but it's a one-time investment once you configure your tooling you can debug at will even at the press of a button and from there it's really just setting breakpoints before we get into the how with the setup i want to show you how awesome debuggers can be it's easier to make the pitch when you're actually excited about what the buggers can do for you first debugging without a debugger can get a little mind-numbing especially when you're in what i like to call the tweak build run loop so you ever find yourself writing code like this just a bunch of logging and prints adding a new one each time you compile obviously debuggers help avoid this but they can also do a lot more than that i think that one issue is that most people don't even know everything you can do with the debugger so here's just a few use cases say you have a function that's called in many places and you aren't sure what call chain represents the error case you can inspect the call stack with a debugger if a test is failing you can launch a debugger against the test itself uh say that's running fine locally but you're having issues in the docker container you can launch the debugger as part of the container uh let's say you're getting an unexpected no point b reference you can just uh set break points and watch the value mutated as execution proceeds alright so everyone's familiar with the basic debug operations of setting a breakpoint and running to hit it what you might not necessarily be familiar with though is that when you're using a gui debugger you get access to all of your local variables and you can see them updating in real time also you have access to the call stack where you can see each of our go routines and currently the one that we're paused in and see that we're currently in the playground function which was called by maine so suppose for a minute though that you didn't particularly care about seeing all your local variables but you wanted to focus on one this is where our watch window comes in let's go ahead and just focus on one expression let's go ahead and keep on watching i and now as we run through we can watch that update again and again as we go through a loop but suppose that we want to have things run a little faster let's go ahead and set a conditional breakpoint and we'll say the condition is o i mod 10 is equal to zero and let's also add a log point we'll just say hello world so let's see that in action so as you can see we're incrementing by 10 each time in our watch window and when we go ahead and look at our debug console you'll see that each time we run we're hitting hello world 10 times in incrementing the same amount so that's pretty cool suppose that you wanted to run a debugger but not by typing in all these commands at once or clicking around well you can use something called debug scripts and you can see here we have just a simple text file uh populated with basic uh commands that you would enter into the command line debugger and it's pretty analogous to our previous example so we're setting a breakpoint b1 trace point t1 and we'll go ahead and uh settings so we're only hitting b1 every 10th time and printing i each time we do that we continue a bunch then we set i back to zero and see what happens there so let's see this in action so that was kind of a linkedin you'll miss it situation so let's go ahead and see what we're doing so you can see right away that we're setting our break points and we're hitting our trace point t one and that's just printing out the name of the function that we're in each time we go through finally we pause when i is equal to nine and you can see our printout of not only the value of 9 of i as well as our source code so we go ahead and do that and you can see that we're incrementing by 10 each time until finally we get to 59 in which case we set i back zero and then the next time we run i is equal to 10. so suppose that you want to launch a docker container and then also debug it all in the same click well you can do that too observe see that we're starting our command right here flipping to orange indicates that we are good to go let's set a break point and run from here and of course in the words of dr facilier from princess and the frog there are things i haven't even tried namely rr debugger integration which lets you rewind execution of code there is a native debugger adapter protocol support built into the most recent version of delve i'll unpack what that means later on and then i showed you earlier in the demo that there are debug scripts one of the features of delve is that you can uh use a starlark which is a dialect of python to enrich your debug scripts full disclosure i never heard of starlark until i started digging into dell's documents safe to say debuggers are very very useful and make your life easier my goal is to show you what delft can do and how you can set it up for yourself for each scenario i'm going to show you how to launch dell as a command line application and then show you how the same configuration works when run through vs code's gui debugger so why use the cli well even if it's not the most ergonomic way to run a debugger it's a really good basis for understanding the commands before we introduce layers of our tooling my approach is to start simple and work up from there and you also might be wondering why we're using vs code well vs code is ideal because it's a common tool that's highly configurable and has robust debugging capabilities a lot of my co-workers like using jeff rain's golan id since debugging usually an emphasis on usually works out of the box i prefer vs code in general and for debugging because the debug settings are much more flexible there there's some really cool things you can do with it and it's also really easily portable uh for instance you can include the debug configurations as part of the version control uh we'll touch on that more later oh and one programming note if you'll pardon the pun if you have vs code the go extension delve you can pull my repo the one that i'm demoing right now and follow along on to our first demo let's start with the simple web server running locally okay couple things to point out here just for convenience i'm printing out the first part of the dell debug help command and then the actual payload is on line number two here where we're running the actual uh dell debug command and then we're separating the debugger commands from program arguments with this double dash right here side note you're going to see this a lot in all my examples where we're setting this environmental variable and then also including this program argument because my program requires that also i'm using make files because you should never remember something that you can make computer remember for you if you're not familiar with make files you can think of this as just a collection of very short bash scripts that each have an alias within all one file very convenient especially when you're setting up debugger stuff so let's go ahead and actually see this in action alright so we've launched our debugger and once it's ready we'll see the interactive dell prompt let's go ahead and set our break point on line 25 and then we'll continue and then you'll actually see where we're requiring that environmental variable and our program argument all right so now that we figured out how to run that in the cli let's talk about how we run the same example but in vs code so quick side note i'm not gonna invest time in showing you how to set up vs code or the vs code go extension or dell there are plenty of guides online that can teach you how to do that instead i'm going to focus on our launch.json here if you're new to vs code and you're not quite sure what a long shot json is you can think of this as basically every run configuration that we've specified here so every json blob is represented here as an item in this drop down menu and each of these is a different run configuration that launches our debugger with a different set of parameters now the best way to understand how do you use a long shot json is by referencing the official docs namely for vs code and for the vs go extension these pages were largely how i built this demo [Music] first attribute here is the name which corresponds directly to the name that we see in the drop down menu right here type refers to language and tools that we're using we also have a request for every configuration this can be launched or attached i'm going to touch on that later we have a mode which corresponds almost directly to the equivalent command line invocation program specifies the working directory where you can find the source codes and the executable here we're using a special vs code variable workspace folder which maps to the current folder that you're working in as part of your editor we specify stop-on entry meaning that we stop right at the top of program execution so we can then set up breakpoints and then resume execution finally we have somewhat self-explanatory args which are program args and then setting environmental variables for the duration of the run of the debugger let's go ahead and see this all in action we hit the plus button and then we go over here we can set our break point and there you go so let's go ahead and now run this so we're going to go ahead and set our breakpoint right here and also just to illustrate a future point i'm going to set a breakpoint right here so we hit our breakpoint we're running now and let's go ahead and actually activate our server post 8080 ping and you'll see as we hit this route uh the editor actually retakes focus as well as stopping at the breakpoint which is pretty cool now suppose you want to figure out why a test is broken well the elve can do that too actually going ahead and looking at this uh it's pretty simple the only real difference is that we're sending mode to test again analogous to our delve test command and then for program rather than just specifying the workplace folder we're then additionally specifying subdirectory sum package so let's go ahead and see that run looking at a launch.json configs for this there's actually not a whole ton happening here as you can see um everything's the same except for we're running mode test corresponding to delve test and then for program we're specifying rather than the workspace folder we're specifying its subdirectory sum package because that contains the test that we want to run so let's go ahead do some package.test set a breakpoint right there and we'll run and we'll hit it right then and there and you can see this testing object and all of its nested structs so earlier in the demo you saw me using logging points that's analogous to the functionality of dell's trace point which is a break point that doesn't stop execution but logs to standard out so you can activate trace mode and delve meaning every function matching or regex will have tracing but execution will not be stopped so again looking at this command pretty basic the two differences being that we're saying delve trace and then we're specifying a regex for all the functions that we want to call in this case we're really going to just be calling get message as part of the trace program so let's go ahead and see this run and remember that trace doesn't actually stop execution it just sets trace points at all the designated functions so when we go ahead and run curl local host 8080 we're gonna get a response and here in addition to our gen logs and our console.out we are going to also see these go routines get message this is part of our trace point so you'll notice that i actually do not have a debug configuration for number three because there is no direct equivalent in vs code's functionality for the delve trace command however you can mimic that with something i showed you earlier which are vs codes log points which more or less map to the functionality as a trace point so let's go ahead we'll run our original example here but rather than that we're going to separate point here and we're going to add a log message right here and we're going to say example three so we run and then when we run again we're going to see example three show up in the debug console and there you go before we go into more complicated examples let's talk about how debuggers work the correct mental model will help you reason about configuring your debugger now i'm stealing this slide from alessandro arzeli's 2018 gopher talk about the internal architecture of dell his talk goes into full detail of the architecture but the gist is that delve contains a client representing the ui and its service layer and then a server which maps the symbols of the program to memory addresses and manipulates execution of the program itself this separation is what allows us to support running a debugger on say a different machine be it a remote server or container as well as connecting to different uis when we use dell via vs code a debug adapter which ships with the vs code go extension using the dap or debug adapter protocol allows the ide to serve as the new front end instead of the cli interface it uses the ui actions that are standard to vs code and converts them into rpc requests to the server it also handles launching the server process for instance launching the debug server via the headless option the newest versions which must be opted into natively support the dap and require no separate adapter now that we understand the client server model let's try attaching dell to an already running program so you can see that i've defined this somewhat verbose command but this is really just building uh our executable one thing i want to point out here is our gc flags and what this is specifying is that we are building uh the n signifies we were building without compiler optimizations and then the dash l signifies that we're running without inlining both of which um are compiler-based optimizations that can interfere with how the debugger actually interacts with our running program so let's go ahead and see what else we need to do we have our make dry run which is pretty basic we're just running that build executable with our environmental variable and our program argument and then we're going to do make debug patch and simply what we're doing here is we're just printing out the eid based off the executable name and then we're running the delve attach with no arguments all right so let's go see this running in action which we're building then we run now let's go ahead over here and actually attach to our running program okay so we're in the debugger i'm going to go ahead and set breakpoint on line 8 and since we're broke at startup we're going to say continue so now we're for freely running and let's go ahead and curl our endpoint here you can see now that we've broken in the actual function and now when we resume we see that we get our message back okay so let's go ahead and talk about this launch.json configuration so you'll notice here that we're running a request hatch instead of launch now that you've seen a few examples i want you to think of launch as basically saying this debugger command is effectively uh starting the process that will also be debugged versus attach is much like running on a train and jumping onto it simply attaching to an already running process or already running debug configuration finally we also have to specify a process name that we're connecting to here instead of [Music] using a randomly generated process id we can just give it the name of the process itself so let's go ahead and run this so we're now running our debug executable and now in our vs code window we can go ahead and actually attach to it now so i believe we should be able to go into a package set a breakpoint and then we go ahead and curl and then we hit it now let's debug with a separate debug client and server you can think of this as debugging a program on another machine this isn't a scenario that i encounter much as a backend software engineer but when i was writing embedded software we'd load the application onto our custom hardware in a different building and launch the debugger against that even if the use case might be a bit special case it's a good stepping stone to our most complicated example debugging a dockerized application okay so there's a few flags here that i want to call your attention to first off is this accept multi-client basically this allows multiple clients to connect to the same debug server instance which makes it easier for us to actually disconnect and reconnect to the process then we have headless meaning that we are launching just the debug server waiting for a client to connect to it rather than launching both the client and the server in the same process as we were doing previously finally uh we're specifying a listen localhost 40 000 just to make it deterministic as opposed to uh listing on a random port and we're specifying api version two because my command line version of dell runs by default on version one but this allows it to use a vs code client which is running the api version too so let's go ahead and look at the command that we're actually using to connect to this debug server and as you can see we're doing dell connect localhost 40 000 specifying that we are attaching at the exact address that we specified for our listing so let's go ahead and see this in action okay so now we are listening and let's go ahead and say make debug connect now we're connected on port 40 000. so let's go ahead and set a breakpoint from package let's just make sure that we're continuing and then let's go ahead and curl the post and you can see that we've hit our favorite message all right so looking at this uh you'll notice the key difference is that in our launching against the running process we are in mode local but here we're in mode remote and basically that's just for when we're in the attach configuration that's toggle from looking from a process id to a post and port to connect to so and you can see right here we're also specifying the same host import that we're expecting to find our debug server so let's launch this by first running our debug server looks like we're up and running there and we're on example five launch against server okay so now that we're running we'll set this breakpoint here we're to curl localhost 8080 and we are able to hit our debugger pretty cool this is actually returning to the first example i showed debugging a docker container is probably the configuration that requires the most steps in knowledge luckily we have built up to this point i'm also showing how to set up pre-launch tasks in vs code which is critical to automating setup for many debug configurations okay so the first thing that we're going to want to do is actually take our docker file and modify it so it supports debugging there are two elements i really want to focus on here first you have to add this uh go get delve command so the actual golang image has access to the delve executable the next thing that you should focus on is that we are using cmd now in docker files remember that entry points not be overwritten when you're doing docker run bmd executes when you do docker run but can be optionally overwritten with a new run command when you're actually invoking docker run now if this solution feels kind of ugly to you uh you can always just make another uh docker file and call it dockerfile debug and run that instead so as you can see this is just a fairly bog standard docker build command just specifying the name of the image that we're building so let's go ahead and just execute that and since we're cached it builds right away so let's actually take a look at the command that we need to actually run and invoke the docker container so there's a lot going on here right now and it's kind of hard to read in the format so let's go to the slide for this the first line is how we are running the docker container and the second one is the command we are launching with remember we are overriding the cmd that runs by default for this container for those of you that aren't super familiar with docker we built our container and now we're running it we're exposing port 40000 meaning it's reachable outside the docker container and then via the publish flag mapping the container's port 40 000 to the host machine support 40 000. we could also expose it within the docker file as well okay so now that we've uh already built our docker image let's go ahead and run that as a container all right that's looking good to me let's flip over and then connect to that we'll go ahead and set a breakpoint right here we'll continue and then we'll just go ahead and ping check back in our debugger window and you'll see that we've hit our break point as expected if we're back in the prompt however the source code isn't loading now this originally caused me a lot of consternation but simply put after debugging a bunch i realized that the root of the issue is that because there was a mismatch between the paths on my local machine [Music] and on the paths on the remote docker file so the docker file has everything under the slash app and locally everything is under my go source so on and so forth this is where keeping the client in server model really comes in handy but since the paths don't match up we aren't able to find that file and hence why we're getting the no such file or directory right here all right so let's go ahead and open up our hidden dell configuration file in addition to a lot of other things that you can figure what we want to focus on here is the substitute path and remember that this is just mapping the source code that is configured within the actual docker container to what's on our local machine and this is from the perspective of the debug server so the from is actually in the docker image itself and the two is where a client is on our local machine so let's go ahead and uncomment this with our configuration change let's go ahead and rerun the debug server alright and let's go ahead and connect to that now we will go ahead and set a breakpoint and with the proper configuration we are now able to actually see our source code in the debugger let's break down this part of the run configuration this one is really based off of our connecting to a debug server because conceptually it's really similar just with some of the nuances of it actually running in a docker container the two things that i'm going to point out that are different are first off i'm using the substitute path which is essentially exactly the same as what we did in our delve configuration however the from is actually our local path and then the two is to the remote path because remember this is from the perspective of the client debugger instead of the server this time around the other thing that's happening here is that we have a pre-launch task which basically is running our docker image for us we're going to touch on this in just a little bit and break that down that lives in task.json then of course we have our stop on entry and then we have these two commands show log and trace basically both of these are what i use to get logs from the debugger itself to help troubleshoot this issue when i first encountered it this is a great step if your anything's acting kind of wonky with a debugger in general okay so let's go ahead and run and debug this docker container now you'll actually notice this is the example that i showed you at the beginning of this talk something i'd like to point out is that we've actually waited for the api server to start listening before the debugger actually launches so you saw in the last example that we were able to run the docker container and start the debugger all in one click that's linked to the magic of task so for context the s code allows you to define arbitrary tasks normally these are used for configuring file watchers or linters or other such tooling they live in your project's task.json if you've defined any task since we aren't setting up one of those tools the configuration can be a little hard to follow namely it expects a pattern of files and regexes to trigger so we have to fill in dummy values for this and remember we are invoking this as part of the debugger okay so this is our test.json file and we're looking at the task that gets launched as part of our uh debug docker launch configuration so the command to define is actually pretty simple this is just what you would type into the shell where it gets a little bit more complicated is that since this command doesn't technically halt on its own it has to be manually halted we have to specify if it is a background task further we have to specify at what point that vs code can stop waiting for this to finish and then go ahead and launch the debugger so here i've defined this as background object where we're saying that once we see the pattern appear in the terminal api we're listening at we are good to go ahead and launch the debugger mastering the task.json can be a little tricky but it's the key to becoming a vf code and dell power user i'm hoping you feel empowered to set up debuggers for your most common go repos you'll have to rely on the manuals to build that launch.json the manuals being vs code docs and the go extension docs specifically the bits about configuring the launch.json you'll also have to rely on a bit of trial and error the setup can be a hassle but once you make that launch.json file and lock it into your repo you are set for your whole team i've already touched on some extra mile things you can do there like pre-launch tasks and debug scripts so go out there give everyone on your team a new tool in their tool belt use delve and how to become a power user with delve and tools like it so what do we have uh i love sessions that kind of go into kind of day two things right you know because we watch and read so much content that like assumes happy path and like you know and it's cool you know we've seen we've seen dap now we've seen integrations with you know the vs code gold plug-in and suzy showed us all that she showed us integrations with possibly the best editor out there vim have we have we landed on that no i think it's a confirmed yeah it's confirmed there's not been enough videos on discord to say otherwise yeah so it's got to be different yeah it is so i got i got yelled at for doing a vim joke but i'm gonna do another one yes no i still this is how good i think vim is i still have it open but yeah you know we saw the last time the second time it was gonna be better i thought i'd maybe get some laughs that's okay um yeah but you know when we in this you know the same conference now we've got the deep dive into the underlying tools that power that those experiences which is really great to see you know if we need to if your vs code or your vim isn't working you can dive down and use the tool directly which is really nice yeah indeed so we've got an interview yeah and i mean speaking of the fact that everybody assumes happy path and things like that right yeah you know it we've we've talked a bit about you know cogen we've talked a bit about grpc and protobufs and you know the thing that a lot of people see in the day one as to use that kind of parlance is this stuff is not too hard to get started you follow some instructions there are quite a few steps but if you follow them you'll get it up and running you'll get your grpc service you know serving possibly even on the internet put it on a load balancer whatever but then yeah there you have it the day two is you know how do you evolve the schema and plenty plenty of other things there and there's lots of complications that come into it oh yeah oh yeah and so uh we've got uh alex mckinney from buff here to talk to us a little bit about those complications talk to us a little bit about this kind of stuff so i'm gonna put in the old earbud and we'll see if we can get him on the line so uh alex uh are you here yes can you hear me i can yeah great so yeah tell us a little bit about um you know these day two kind of things the difficulties what are some other considerations once we've kind of generated our uh our code for this like what are other considerations when we go to deploy this what are some other things we should think about and problems right yeah so i think you hit it on right on the head with the uh the simplicity of getting going like on the the day one you know there are a ton of tutorials out there is generating code is easy with protoc at the beginning but then you know when you get to day two and you start making changes those apis you start to understand that the the protoc compiler or the protocol buffer compiler itself which is the default encoding used by grpc is uh very complex there is a huge onslaught of different configuration options both tied to the protobuf schema itself as well as how you invoke the compiler um a lot of nuanced behaviors and unfortunately there's also a lot of undocumented compiler options uh but even once you start to get acquainted with some of those things complexity involved with like the go package file option or you know modifier flags if you're familiar with those things you you still fall susceptible to some of these issues with hey as soon as i need to use some of these protobuf files from common dependencies like google apis is a common one that is kind of baked into the grpc grpc go library for uh things like the rpc status type which is used for the grpc error codes and so on once you start to use these things you start to understand like okay now i'm depending on this these files from external sources how can i make sure that my copy doesn't drift between you know the one i that's published via google apis they do this by publishing sdks and generating them themselves in a github repository but a lot of these issues are not solved with regard to the general community and they still struggle from this problem of tying the protobuf schema itself to a certain representation of that generated code by kind of abusing the go package in that way so buff solves that firsthand just by saying hey we're going to provide dependency management for you so that you can just specify the dependencies very similar to what you're used to with a go mod like you can just require different dependencies and are included in your module so we've similarly created a protobuf module that's uh very analogous to what you're familiar with with a go module then beyond that we we have a ton of cool specifically tailored features for go including one that i'm particularly a fan of i refer to as managed mode so this problem i was talking about of you know tying a file option to the schema itself means that that schema is no longer portable across different environments it's no longer usable from one organization to the next this is actually something i went over and that buff helps you go faster talk but it goes into detail of saying you know hey this go package is tailored to my github repository and as soon as somebody else needs to use that api definition to generate their own grpc client they have to hand edit themselves and then that just again you're back to that api drift problem but what managed mode does it actually will set these options for you at generation time so you can remove the file options entirely and leave it to a compiler like buff to to set it at runtime which is uh not run time of your application but runtime of when the code generator is invoked and what this does is it makes your apis more pure it makes it not tied to consumer level concerns because these are consumer level concerns i want to generate with respect to what i need from the api and my annotations and my custom options not necessarily someone someone else's decision but beyond that uh this still has all the issues of okay fine you know i've installed proto-cgn go proto-chin go grpc locally i can invoke these plugins and get some generated files but you're still at at odds with you know the versions i have locally on my machine could be different than my peers so you build a bunch of custom tooling you use a build system to ensure this but those struggles still exist when you cross that boundary when you're referring to your customers or your clients that are generating code for a client that they don't own so what we do now is actually we let the community publish their own protobuf plugins which ie is you know protocingo and etc to the bsr which is our buff schema registry it holds all of the api definitions with regard to your protobuf modules the thing kind of analogous to go modules and once you have both your api definitions they're versioned logically together now i can invoke these plugins that are also versioned according to similar semantic versioning that we're all used to with you know github repositories and so on but now i can execute those plugins on my behalf but on the server side in a secure sandbox we actually have kind of homegrown our own technology around that using gvisor so we can ensure security for all of those things being executed on our servers and then so that alone is already so much better than where we were even just moments ago and then finally the last thing i want to really stitch on because i know it's super exciting to the go community is using all that technology we were able to build a go module proxy that you remove all your concerns about go code generation and all you do is go get a library that immediately gives you the go grpc clients and servers associated with the api that you've published to the bsr and again that's really only the beginning that's very slick it kind of sounds like a pretty a pretty big leap forward you know because i think a lot a lot of these i didn't even really kind of think about like i mean i haven't really thought about how to solve these problems but they are things we've ran in right into like how to manage versions between these things and how to share protocol buffers between yeah so we're in the stone age we're just copying that profile from the google or from someone else's so many people are doing this across the everywhere everyone's reinventing the wheel trying to come up with these solutions and kind of leaning on github repositories as okay let's use this as a source of truth then all the tooling you have to build to get that to be a good experience so many companies are just doing this repetitively and we're trying to solve it once and for all great i mean i wish we could talk about this for a couple more hours there's so much to meet here so yeah i mean thank you so much alex this has been although short it's been a really fulfilling and awesome and interesting conversation great yeah we look forward to see you in the uh in our slack channel or check out the documentation at docs.the buff.build and uh yeah give it a shot thanks so much alex thanks alex thanks cool thank you so we've got some vms's and we've got a talk coming up so what do we got yes so coming up next we're going to be talking to kevin dan gore from khan academy about migrating big python monolithic applications to services written in go so kevin take it away hi i'm kevin dangor and i'm going to be talking about khan academy's journey from a big python monolith to go services it's going to be a tale in three parts because i have a lot of stuff to cover uh and i'm going to be talking about why we did it how we did it in terms of the technology and how we did it in terms of the project and so that means i'm going to be talking about python go google cloud graphql code generators and burn down charts all of this in the 25 minute talk so before i get into that i do need to talk about what khan academy is because not everybody has seen our site if you haven't check it out we are an education nonprofit we have a mission to provide a free world-class education to anyone anywhere we have many thousands of videos articles and exercises about a bunch of topics and millions of learners come to us every month to learn on our site so when we first started talking about this rewrite publicly we did get some feedback with people wondering why we had so much code turns out there's a good reason we have a lot of features and people don't always see these features because they're not always target users of those features so keeping track of learner progress is something every every user sees because they'll see how far they've progressed in their skills and which skills they need to level up on but we also have login with clever and use with google classroom because working with classrooms is important to us because we are used as a supplemental resource in many classes already we also provide reporting for school districts we connect with the college board for sat prep we have a content management system to manage all that content that i was talking about and also to work with translators who around the world who have translated khan academy to dozens of languages so there is a whole bunch we have here and that's why we have so much code that's why our rewrite is not an easy project so why did we have to do it so to answer that question we need to go back in time to 2010 and in 2010 python 3 was still very new it was actually less than a year and a half old um as of the it was around a year and a half old i think in about the time of this this commit that you see here and that's um you know that's not very that's not very old most of the libraries hadn't converted yet uh there wasn't much available for python 3. plus google abbott google app engine was also pretty new on the scene at the time there wasn't much in the way of serverless architecture back then providing both scalable web layer and scalable data layer but app engine provided that and that stack actually served us really well it did scale quite well as we added new users as we added new features that worked great for us but the python software foundation said january 1 2020 was the end of life of python 2. and so we knew we had to we had to get off we had to move and we had to figure out what to do to make matters more interesting for us google in august 2018 introduced app engine second generation now this was actually i think a really big improvement to app engine because what it meant is that all of these bundled apis that app engine first generation offered have been unbundled in app engine second generation so an app engine second generation app looks like any old web app but you get to choose pick and choose which apis you use from the whole suite of google cloud so it was a big improvement but it also meant a lot of change for our code finally i will mention that because we are running an app engine monolith that meant that all of our instances were running the same code so if we wanted to be able to move to something else including python 3 that meant that we had to effectively adopt a services architecture at least for one more service which would be the new one and that adds a whole lot of complexity to our system and so we had to think about what kind of architecture we wanted to go forward to and as we thought about it we did some rough estimates and we decided we figured out that moving from python 2 to python 3 is a lot of work moving to a different language is probably going to be out 50 percent more work so it that's a significant amount but we thought we could get significant benefits from doing it things like performance um we like some of the tooling available memory usage all these things could be better and you know runtime performance does matter a lot to us so we looked at python 3 we also looked at kotlin which we had some experience with and we also looked at go and i probably don't need to convince you all that go was a great choice for us it has performed really well for us the tooling has been great and we're really happy with it so goliath is khan academy's project to move our entire back end to pyth to go plus services and we call the first 80 or so of the work our minimum viable experience we took the most important of that called it our minimum viable experience mve and you'll see this term later on so now we can get into the tech this is really cool we got lots of stuff to talk about here so the first part of our tech transition for this was finishing other technical transitions we had already started we were early users of react for example we wanted to replace all of our jinja templates in python with react so basically our go back end was never going to generate html pages for the web we wanted to get all of our service behind fastly which is a great cdn we wanted to move all of our rest apis as much as possible to graphql which we had already started doing so we had started these tech transitions and we wanted to complete them so we wouldn't have to port extra code over and the same goes for you know we had 10 year old product we had some features that were outdated and not hardly used so we we wanted to reevaluate some of those and so this is what our tech stack looked like our architecture as of january 2020 we had a lot more dragons to worry about back then but fundamentally our architecture was like this this is probably clearer and less fanciful picture of our architecture but the general flow of a request is it starts in a browser or mobile app it goes to the fastly cdn and from there it's going to go to the render gateway if you're getting your initial page view or if you're making a data request it's going to go to the graphql gateway that is colored yellow in this picture because it's javascript right now it is apollo server and basically the graphql gateway is going to look at the graphql request and fan it out across all the services we have about 20 services right now it's going to fan it out among those services gather the data and return it to the user and that includes also calling the python 2 monolith to get some of that data so our go services are all very consistent in structure they all have you know make file and a readme at the top they all have a command package or command directory under which you'll find the package main for any binaries we want to build it has some yaml files to configure some things it's got some graphql files it's got a resolver's directory which i'll talk about add a models directory for accessing our data store models so let's get into graphql i will give you a quick crash course in what graphql is like so that you can understand some of the architecture choices we've made so first of all we are looking at a piece of a graphql file here and graphql lets you provide basically a typed api so you define types in this case we're defining a type called user and your types can have typed values on them type fields on them and so basically kaid kade is our identifier for a user and it is a string and you can also define directives which provide metadata to the system about a given field or type next let's take a look at a query this is a query being defined on the server so it's defining a query that can be run by the client so a query something the client can query for is a user and they can run a user query query for they have to pass in the cade so they can identify which user they want and they will get a user object back okay and this is what a client-side query will look like so it's it says query you can name the query you can pass in parameters to the query and then you define what parts of the schema you want to pull out so in this case it's saying hey i want a user and here is the cade of that user and you can specify exactly which pieces of data you want so you're not going to over fetch in this case it's just saying just give me the username that's all i'm interested in so queries are not allowed to change data for that you use mutations they're defined just like queries except it's under the mutation type not under the query type but they work exactly the same way otherwise they can take in parameters and return values but the difference is they're going to mutate data on the server side and so that's what's in the graphql files on the server how do we get from these graphql files to the actual client to actually serving something to the client and that is where gql gen comes in so gql gen is an open source project it's great it is a library and code generator that takes your graphql schemas and gives you type safe apis that you can use to implement a server that will respond to those graphql queries let's dig into how that works so this is the package main for a service for the serv command rather and you can see that we're importing we have our own little web framework but our little web framework is basically just some conveniences around the standard net http stuff that go provides but conveniences for our services we're going to pull in the generated graphql code from dqlgen and we're going to pull in our resolver code and basically glue those together into an executable schema and we use that schema to handle the graphql requests that come in so that's what serve.main serve.handlegraphql passed into the schema that's what it's doing so if we jump into our resolvers and we look at what they're what they're doing the resolvers are basically gluing together gqlgen's generated graphql interface with our code that is going to be able to resolve the graphql data needed for responding to a query so we're going to focus on this user resolver down at the bottom here so again that is an interface graphql that user resolver is an interface that we are going to have to implement in our code in order to be able to respond to user queries so this is a piece of the user resolver our user objects have actually a lot of data on them and email is one of the fields that are on there so for every field you're going to see a a resolver function like this that basically is in this case going to take a context and a user object and it's going to return a string um it is a string pointer because it can be nil and an error so that basically every single field is going to look like this in the in the in the interface so this is our handwritten code now that is designed to implement that interface and this is what a lot of our resolvers are going to look something like what you see here so basically we are taking in the email we're taking in the context in the user just like we saw just following the interface we have our own dependency injection system where we basically upgrade a go context to match up the interfaces that we need so we have a nice typed interface that gives us access to all the services that we need within a service so for example when we need to access datastore we just have to include that interface in here and we will get the datastore interface back so the next thing that we need to do after we've got our context upgraded is we need to basically validate that the user making this request has access to this field and so yes this is actually happening at a field by field level um that we can determine what permission somebody has access to uh and so in this case not everybody has access to a user's email this is definitely not open and so we look at look at what's required for that access so the next thing is that we're going to actually pull from our data model what the what the user object looks like in the data and we get that by the kaid and every single field again is going to look like this which means that that get bycade function is going to is going to cache that for the request that's going to cache that user um [Music] this is how we make sure that we're not every single time for every single field pulling in that user object because we just we just cache it it's very simple but the pattern is going to be very similar each time we have this nice little ptr package that lets you do mappings from simple types to to basically pointer types because in this case uh as we noticed from the interface it's it can be uh nil so it's a as a string pointer and so we just have the convenience function to basically say hey if this is nil um just return an l for it um yeah and so one of the things also i'll note is that email is personally identifiable information pii um and so we make that very clear in our interfaces and we also encrypt that data on our back end so we we have to decrypt it um at the time um that we that we know that everything is authorized all right and so that is what an individual service looks like now some of the magic that made this whole transition work really well for us is graphql federation and basically the idea with federation is that the query comes into the graphql gateway and that is going to federate out the query to all the different services so so basically the graphql gateway has to take the schemas of all of those services including our old python monolith glue them together into one big schema that is what's what is served up to the users so let's take a look into how federation works okay so this query here is run by the assignment service okay so this is actually a service to service query but this query could just as easily be run on the client side but the idea with this query here is that we are we are going to query data and so we are calling that user query so that same user query that we saw previously we are calling for a user and we're passing in the user's k because we know that that's what's required but what we're requesting on user is student list now what's interesting about student list is that that is effectively a class in our system and the student list data is presented by the coaches service so the the previous service we were looking at was actually the user service this is something that comes from the coaches service so how does that work so this is how that works this is from the graphql from a graphql file in the coaches service and you can see that it is saying extend type user so the user type is defined over in the user service the coaching service is saying hey let's extend that type uh and add and and that extension is based on the key um of cade that's how it kind of glues users together between services as it uses that cade field to say okay give me this this user's data um and it's gonna say then okay i am providing on the user type student list and i take in a student list id which is basically a classroom id and i return a student list object and so the graphql date gateway is going to have to stitch that together when somebody asks for a student list so if you're asking for some fields of user they will come from the user service asking for other fields it's going to come from like the coach's service in this case now one really cool feature that we got out of this is that we were able to make the graphql gateway do side-by-side testing so what this means is that once we have an implementation in go of the same field that's in python what we can do is have the graphql gateway request the data from both services simultaneously and return the python data to the user but compare the result and if there's any difference in the result it's going to log that so we're able to look up in the logs hey are there differences that we're seeing and if so then we know oh we still have some work to do to make these match up but the great thing is this is working with real production data and so all the weird little edge cases that have built up over the years are all going to be covered by this testing works out really well and we can also prove our performance this way because we can make sure the go implementation is actually performing as well as we expect it to so how does side-by-side testing work in this case we have a field that is implemented or rather type the teacher campaign type is uh provided by the python monolith still but this field the live field comes from the campaign service the campaign service actually both campaigns and python are both defining the live field the difference is this um campaign service is saying uh we are migrating that's what this directive is for at migrate we're migrating from python to uh and the current state is side by side so basically what that means is it's going to tell the gateway hey when you look at this live field we know that it's coming from the python service but we're doing side-by-side testing right now and so that means that the gateway is going to run the side-by-side test that i just described once we're done with that and everything's hap everything's happy we can just change that state to migrated and that will mean that the gateway will only send the traffic to the new go service once we're completely done with it we just remove the migrate directive entirely and we delete the python code and that's all set so that gives us the ability to migrate even a single field which is as incremental migration as you can possibly imagine so um having talked to talked about side-by-side testing i wanted to come back and talk about this query that we looked at before that the assignment service is going to run to get data from the actually the coach's service ultimately um and i want to put this in more context so in context it looks like this because in context that query is living in a go file in a string with this with this special at gen client uh comment in there and so that lets me introduce gen client so gen client is really cool this is an open source project that we've created and what's great about this package is it's very much like gqlgen for clients it lets you create queries uh generate types type save go code that you can then use to run queries from go so basically that query was defined and that string is more or less thrown away because gen client turns it into the api you see here we call the gen client.getclassroom icon url for assignments and it takes in all those same parameters that you'd expect and that's generated code and the result that comes back you can see there's response.teacher.classroom.id so that is what the response is showing us and it's it's all nicely typed and it's also matched up when we when we take when gen client pulls this query out of our code it's going to match up this query with the schema so we actually get typesafe querying all the way through from our go code all the way through to the graphql schema on the service on the other side so it's that's pretty great basically it's like if student list changes on the coaches service we will know about it right away because our gen client code here will fail to build so i didn't have time to get into our test code our data store models library code custom linters we had a lot of really cool stuff that we've used to get us through this project and to put us in a good place with our go infrastructure but yeah so far the experience of writing go has been great we love the tooling that we're using and the awesome production performance it's been really great so i want to talk a little bit about delivering this project on time how did we do this project so goliath mve basically a little timeline it was march 2019 when we made the decision that we're going to go with go and services in this project we had spent the rest of 2019 basically building early infrastructure and the first service to kind of prove it out we had an overall idea of what the services would look like and then spring 2020 is when the team really ramped up when we really had like full staffing on the project and we completed it at the end of august 2021. what is mve mv ended up being about 85 percent of our graphql field which is about 5000 fields 95 of the requests going to the new go back end 750 000 plus lines of go code um so we wrote a lot of code um did a lot of porting and got it working so our earliest estimates showed us coming in at may 2021 this is our burn down chart from from goliath mv by july 2020 we had a really good sense of what our velocity was looking like where we thought it was going to go over time and so we basically just managed the project diligently to keep things on track people were very flexible the engineers were flexible and moving to other parts of the project if other parts of the project needed some additional help to keep it moving on track and you can see that we had a pretty smooth burn down or organizationally we value working sustainably and so this i see this as a sign of consistent delivery and not a death march like we were just working it working at work in it uh and that everybody worked really hard to get it through but we worked sustainably and consistently one of the things we did at the beginning of the project was we established project principles these project principles also helped to keep us on track um you know we clearly didn't want new python features because new python features were going would basically just add more to what we needed to port so that would be good a lot of the other principles are actually focused on keeping us from adding more to to goliath when we're reading rewriting the code we wanted to do as close to a straight port as we could and so that's that's what a lot of these principles were trying to keep us on track for so as of today the last 15 percent of the graphql fields remain so goliath continues but now it's just one project among several because we have a very comfortable move on our architecture everything's running and we're happy with where we are and i'll just close with i like go more the more i work with it which one of our khan academy engineers said and that's a sentiment shared by many thank you all to the community for making it so great uh and thank you for listening to my talk if you have any questions talk to me in the gophercon discord or hit me up on twitter i'm at dengor there and if you're interested in working on this kind of project go to khan academy.org careers all right thank you so much kevin that was a wonderful talk uh are you all ready for this i can't let's believe how good that you know because it's khan academy you guys get it right yeah yeah you get it all right all right peta gallery this academy is here at our conference oh oh just keep on rolling con academia learn from this peanut gallery i invite you to top this all three of us i invite you to i don't think it's going to take much i don't either because they have the they have the gift of images and being able to edit images and then put them into discord which so far has proven pretty pretty great they can't do it [Laughter] well seriously though three days three days here we're starting to get a little delirious yeah yeah we're on like three hours of sleep or something like that but seriously though i mean i i enjoyed that talk it was a really nice measured talk on how to move responsibly from one language to another of course khan academy is a huge service they have to do this well they have to have a measured way of doing it you know and the talk isn't saying you should get away from python python sucks it's not saying that it's saying but you are i am i am only not saying to write bots that's it yeah someone can someone can clip the video and just have me say but yeah you know i really enjoyed that i thought it had great measurement it told us kind of how we should go about doing a language switch like that if we found a reason to inside of our organization so when was the last time you did a language switch uh myself or at a company an organization another company um i'm moving from russ to go right now i actually did move a code base from python half of it from python over to go and we kind of did a similar thing like the services route about a year ago or so so it's kind of interesting to see you know that the paradigm shifts that even you inside your own mind has to make between you know in that case python and go i mean there's so much around the whole ecosystem of it right it's not just like the differences between like writing idiomatic code in this language there's dependency management and how are these things deployed and then you have certain things like statically compiled binary you don't really have to worry about things but like if you're going the other direction to a language that has you know a runtime and then you've got to figure out how you manage what version of that is on individual nodes certainly yeah and then you know and then you you got to get deeper into the language ecosystem sometimes too in python python two or three and if you go over to go you gotta start soon we're gonna have to start thinking about generics and things like that so it's it's you kind of hit the nail on the head there it's a lot more than just the language it's the tooling and the ecosystem and the support and the docs and the community i mean who wants to you know i mean where are you going to find people funnier than than this crew right here exactly i mean you know but on a serious note though like i think the community aspect of things for me it plays a huge role right it's it's it's being able to interact with others in the community being able to you know feel like okay you know asking questions and not somebody being you know a douche um to you um you know it's these kinds of things i think really make make it worth while being you know in a language community yeah i mean you got to feel comfortable you got to feel like you can enjoy this too you know absolutely absolutely so we've got some stuff coming up tell us what's going on in discord first i i saw some stuff going on out there especially with regards i'm sure i'm sure that some dad jokes uh galore there is uh i mean we haven't checked in on crowd strikes competition in a while i actually saw something they they've had some really interesting submissions people implementing try-catch unfortunately for monads someone already took the gonads repository right so you can't unfortunately this is sad news that i have to deliver to everyone but no going ads for a repository name no gonads for you another great one for was async await and the the struct to do it was pro mess like promise i enjoyed that yeah nice awesome i love it the puns too yeah but seriously crowdstrike uh there's a link in the peanut gallery for the gopher they're going to give away that that competition is ending now in about an hour and a half or so so get your submissions in and vote yes yes yes then speaking of things it's too late to get submissions in for we have the go for say game coming up yeah and so we're hoping that all of you trolled them hard hard please i'm so looking forward this is just how bad some of these answers are going to be me too yeah me too should be a hoot for sure i'm gonna be the most surprised by some of that like some of the things like were legitimate questions when when we played they were legitimate questions but you were not getting the responses you'd expect no i forget what one of them was other languages like your second favorite language or something along those lines so we were like we're like way off base we're like they're like cobalt and pascal like what is people oh man i hope someone put goal ball in there now let's just do the pure trolling aspect there so uh going along in parallel we also have a fun with fuzzing virtual meeting space coming up um if you want to attend that you don't already have a ticket it's free just go to gophercons website and register for that you'll see it on the agenda page but yeah i'm really happy with all the content that go time has been bringing to us throughout the event this this one in particular i've been really really excited for i'm i'm sure it's gonna be wildly entertaining yeah yeah and uh if you want to interact with the contestants and the hosts uh we created a gopher's say channel inside the speakers section of the discord and so without further ado let's do it it's go time one two three i stole it oh you're starting all right one two two three it's go time [Music] [Music] hello and welcome to this go time gopher con extravaganza yeah extravaganza yeah i'm matt ryan and i'm thrilled to announce that today again we're playing gophers say the excellent popular game show family game show based on family feud or family fortunes if you're in the uk which i am let's meet the teams we have team zero as well let's let's meet our contestants here we've got julie cues around hello julie hi welcome to go time slash gophercon we're also we've also got steve francia can't believe it hi steve hi everyone i can't believe you're here steve really can you mommy i'm here yeah well you won the competition so congrats john calhoun's also here john hey matt how are you hi john i'm good mate yeah i'm right i'm excited are you are you going to win i don't know okay that's good honestly i'll win but if my team doesn't carry me i probably won't okay good he's not going to be pulling his own weight there well natalie pistinovich is in town hello natalie hi matt welcome back are you looking forward to the game show i'm very excited to be the one person who is aware of the rules yes yeah because i already played it yes please remind me if i forget as well katie hochman also around hello katie welcome how are you doing i'm good i'm very excited to win despite john not believing that we can ah okay well you'll have to carry him that's it uh rob findlay's also here aren't you rob i am hi matt hey welcome to the go for time go go for time goton mashup yeah don't worry said it said it wrong on purpose okay i need to tell you about this this game that we're gonna play because it's lovely um essentially we've asked a load of gophers some questions and you'll have to try and guess the answers so you're trying to match the popular answers from the survey it's not about right or wrong it's just about what gophers say that's why it's called gophers say the top answers will go on the board any response that had five or more answers will appear on the board and in order to get control of the board each team has to do do a face-off basically where you'll just have a guess and see the highest score and that's how you'll take control of the board once you've got control you have to then guess all the other um answers and then you'll you'll win that's how you win you get points but be warned if you get three of them wrong you'll lose three lives and it'll give the other team chance to steal and they'll be able to take your points essentially so don't let that happen you cannot confer through the game but while you're stealing you can confer and i'll remind you of that later do we feel like we're ready to play and and i'm gonna announce the teams that we have here so team zero we've got it's natalie steve and julie you are our first team team zero there you go and you know don't take anything by the fact it's zero is it in the in the name of your team it's literally just a zero bound like a go slice in it and that's why i've done that john katie and rob team one you're our second team are you happy about that or slightly slightly different yeah okay okay good they're definitely happy we've been assured of that verbally um okay we're going to now look at the at the board we're going to share the screen can we see this we're going to jump into our first round here natalie and john you're going to go face to face and it's a we're going to have an interface off if you like puns um natalie what would you what would your guess be to this question describe the go community with just one word what did most people say when they were asked to describe the go community with just one word welcoming welcoming gophers say yes it's on the board in number two nine points there so john you're gonna have a guess if you beat natalie you have to get that number one spot you'll take control of the board natalie stole my answer and i don't know if my backup is uh uh i guess is friendly the same as welcoming i don't know how that would be categorized let's see if friendly is on there go for say yes friendly and where is it number three so natalie's team team zero takes control and we're over to steve francia steve how's it going steve are you going to have a guess what do you think the goat community said about this i was going to say welcoming and friendly okay but those have been taken so i'm going to say um good looking oh attractive okay go for say oh no i'm sorry uh attractive was not on the board you lose a life but don't worry she should have got two of them i agree yeah uh julie would you want to have a guess i'm gonna say um fabulous fabulous okay go for say i'm sorry nobody said it was fabulous well five or more people didn't say that we look back around natalie what do you think inclusive inclusive go for say no nobody said inclusive and that's your third and final life gone i'm afraid so now the other team team one have a chance to steal you can confer and just need one answer from you john the team captain after you've conferred with your team all right for my team does anybody think the word gopher might be on the board uh i think it's coming i would gopher that or like programmer or like nerdy i was just going to say i was going to say diverse i was that's i think that's a good one okay is that any answer you'd like to submit you're going with diverse you want to lock it in yeah we're locking diversion okay we're gonna lock diverse in go for say no i'm afraid not it did not steal let's reveal the board number six was opinionated five people said that in at number five we have fun with six people and in position number four we had helpful uh seven people said that in position three we have friendly with eight people welcoming's in position two with nine people and at number one it is awesome and that was said by 17 people so there we go a because rough start is fabulous yeah well there we go let's move on to our next round [Music] okay round two name a place outside your house where you like to code slash work name a place outside of your house where you like to code slash work this time it's steve versus katie going face to face to find out who controls the board steve uh i'll come to your i'll ask you after katie katie what's your answer wait that's me first okay um i have two guesses uh i will say coffee shop coffee shop good good good one i think gopher say [Music] yes of course and it's the top answer which that means that steve doesn't even get to have a guess you've just taken basically snapped his dreams out of the clutches of his and i i was going to say coffee shop so i'm a good one yeah good one okay um right so now it's rob's turn to have a guess you've got three lives rob what do you think we okay we used to work in offices so how about the office okay let's find out the office go for say yes indeed in at number three the office with 21 people by the way 38 people said coffee shop very cool all right john three lives still going strong where do you like to code slash work i'm gonna go with park in the park how beautiful go for say yes indeed number two park garden yard 26 people okay katie you still got your three lives i don't do this because i watch movies instead but i know a lot of people who do i'm gonna say like airplane oh airplane go for say no nobody does that outside they don't do that apparently they watch movies too i guess you lose a life but that's okay you've got two more lives rob what do you think okay how about um the beach some people were working do they well let's see go for say no have you met a programmer the beach well yeah yeah i know there's some cool ones aren't there not in this survey mate not me either okay john back back around to you talking to cool people what do you think i'm gonna guess campground campground okay it's go for say no can't believe it they can't believe no one's going camping to setting up a tent getting out of the electric camping and working so at the water sort out all the i mean look up the local amenities just to get get some cobra and they're not doing it i can't believe it katie what do you think um i thought we were dead so i wasn't even thinking um pressure's odd um i was gonna say they are dead they've had they've had three exes dad oh really i think i think we did i mean [Music] okay don't worry that's a mistake by me then okay that means something at zero yeah thank you thank you okay unfortunately that was your last life so that means the team of the team get to steal you can confer and i'll take natalie i'll take your answer what do you think so we get to talk to each other right now yes we get to decide together what about library oh i was thinking of a chord i thought train i was thinking trained too training live that's because you live you live in the us yeah maybe most of the people who took the survey live in the us how about a co-working space is that like the office um okay i think four five and six are our co-working space librarian trained by this order well i don't know do any of those sound good to me natalie you choose yeah that's lee i was about to ask julie what do you think but you're okay um yeah let's try train okay train to steal surveys i mean our gophers say no i'm afraid not let's well that means the points go to team one let's unload this board then number six was nowhere people that don't leave to do any work that got five answers in number five it's the pub or the bar six people said that you can believe that and at number four let's see if this is one of your steve number four was the library indeed 14 people oh that's okay don't worry though this is it's going well we've got team zero have 17 points team one are forging ahead with 85 points let's go on to round three [Music] okay round three the question we asked our gophers is the most useful go keyword is what what is the most useful go keyword but what did people say we're gonna find out which team takes control of the board when julie and rob go head to head i was expecting some music there it's all right um we've never done that piece before we'll do it for next time okay julie this time there you go julie oh boy um switch switch go for say oh sorry no that's okay you just lose one life no problem um rob you have to have a guess actually you don't lose a life we're just still finding out yeah who's going to take control so rob you have to get one of these to take the board well you i you can't switch without a funk so i'm going to say funk funk indeed gophers say [Music] yes indeed at number two 18 people said funk was the most useful go keyword and that means team one steals the control of the board takes it john calhoun what do you think the most useful go keyword is what do you think our gophers said this is one of those ones where i just expect people to not name keywords as answers uh why would that be john that's very unusual if you asked me to list all the go keywords i couldn't list them all i definitely get something wrong with select select go for say yes indeed down at number six seven people said select nice work okay katie what do you think i'm gonna say return return would be very useful let's see if the gophers said it go for say no and that honestly shocks me that that's not on there what i think you need to read yeah what are their programs okay well we can't figure that out now rob you're gonna have to you've got two lives still so what do you think okay i think people will have just heard the word go so they'll say go okay that that's how much respect you have for the audience of gophers it's useful yeah it is useful let's find out if they said it go for say yes and look at this top answer 29 people said that go was the most useful go keyword very cool indeed two lives it's johnny calhoun go on john do us do a guess var no you lose another life you have one more life katie that life sits delicately in your hand they're gonna set it free or crush it to death more um what'd you guess so i'm gonna go with i think people said um if if gophers say no these programmers are very certain there's no uncertainty yeah yeah they know what they're doing this is not like yeah if this if make your own mind up that was your last life i'm afraid this gives the other team team zero chance to steal please confer amongst yourselves and then natalie team captain i'll take an answer after you've conferred with your team all right this time we're doing this right what do you think defer i was thinking of that too interesting i was thinking panic or type but i'll defer to youtube this was meant to be let's go oh excellent i don't know when to do this now i feel like we need to wait for the end of the game show before i announce this one i'm just kidding defer go for say yes you've successfully stolen third space there with 11 people defer and you steal the points 65 delicious points coming your way so team 0 is now on 82 team 1 is on 85. let's reveal the rest of this board then what was the most useful go keyword number seven people said struct five people said that number six was select with seven people number five is interface with nine people said that at number four four including the number ten people said so and defer was number eleven funk at it it was a number sorry defer was in number three with eleven people funk in spot two eighteen people and go at number one there with 29 people well things are heating up here 82 and 85 is not much in it is there now we've built some tension it's time to go to round four [Music] bam bam okay so right what was what we did again what we're doing again okay the next question the best thing about the tech industry is what what is the best thing about the tech industry and natalie it's natalie and john to go head to head uh john what's your guess what's the best thing about the tech industry paycheck the paycheck go for say [Music] yes at number two the pay slash benefits 20 people there interesting okay natalie can you beat that can you find that number one spot twitter pardon twitter twitter twitter go for safe no i'm afraid no one gets that that means team one take control of the board katie we're over to you three lovely lives don't ruin them what's your guess i think people would have said the people the people that'd be nice if the people said the people let's see did the people say to people go for say [Music] yes indeed at number four the people or the community that they're in 15 people said that it's very nice isn't it uh rob you know can you find a nicer one than that or just any i i don't know about that but um how about the fact that we get to to program how about the work oh the work itself programming go for say [Music] yes indeed people did say that problem solving was we grouped into that 19 people at number three yes very cool three lives still johnny calhoun i'm gonna go with the impact the impact okay is the impact did they say that let's find out go for say [Music] they did indeed say impact they said opportunity or the impact to make change things like that not nice eight people there there's a number six and still three lives this is looking good katie can you keep up the running streak of success i don't know if i can um i i feel like the thing i want to say is probably within the category of benefits but i was going to say like flexibility um yeah flexibility okay let's see if it's up there go for say oh well done yeah work from home or flexibility 10 people said that i i tell you what i genuinely thought this was probably going to be the hardest question and you've really just nailed this very impressive rob i think we're back round to you aren't we i'm mystified i have no idea um i was going to say flexibility i i don't know the how about the opportunity to grow okay let's see i don't know yeah i'd doubtful let's see if more than 20 people said opportunity to grow gopher say no and you lose a life but it's okay you've got two other lives so john and katie are both going to get a guess here starting with you johnny calhoun down at the local saloon i do think equally mystified i think um let's i don't know i'm gonna go with go the language go go best thing about the best thing girl okay let's see maybe gophers did say that let's see go for say no i'm afraid not you lose your life katie last life to prevent the steel i don't know um very close with that answer apparently okay uh coding okay let's do it let's find out gophers say no i'm afraid not so that means the other team get a chance to steal can you get this right team please confer start conferring now my idea is conferences me too okay see you yeah i was gonna say travel all right so i think i think conferences is great yeah cool okay then that's our answer oh okay conferences conferences oh let's see if it's conferences is it conferences go for say no no afraid not that means team one steals all those delicious points and we'll look at the scores after the top answer for this one was in fact the tech the innovation tech or innovation is the best thing about the tech industry okay so let's check let's check in with it's calm down it's time to check in with the scores team zero in our zero in our slice of team here uh team zero has 82 points but team one out in the lead again it's 157 points you know celebrate yay yep there we go okay let's move on shall we to the next round round 5. [Music] someone else is having a party at their own house by the sounds of that okay um right here we go again this is round five this time the question is the worst thing about the tech industry this is this is what we're after now what's the worst thing about the tech industry and last time it was john and natalie went head to head so it's steve and katie and steve goes first steve what's the worst thing uh bugs bugs what a great answer burgs yeah we'll submit it in your accent then shall we you go for go for say yes bugs bad practices okay yeah there we go we'll give you that one bad practices um and bugs things like that okay in position four there 14. sorry 13 people said that okay katie can you beat that there are three other answers ahead of this one i feel like no matter what i say i'm going to get in trouble for it i want to say that's the most diplomatic way possible potentially like the room for growth as far as diversity goes okay so right yes diversity being the worst thing currently go for say no katy apparently that's all sorted all done yeah we've we're finished put that into the done column um okay so that means team zero you need it you've you've taken control of the board and julie it's your turn three lives have a guess what do you think the worst thing about the tech industry is um i'm gonna say the bad people the people but specifically the people who are bad the buddies bad people like villains and the like charlatans scallywags i don't know okay the bad people yeah okay let's see if scallywag i don't know if it'll be spelled out like that but let's find out go for say [Music] yes indeed people saying people people all the culture 49 that was our top answer merry merry festive holidays to everybody that was a little surprise and that surprise there means steve francia's going to donate 100 to a charity and that was what that lovely piece was so thank you so much steve that's lovely and uh and natalie we're back around to you now it's your turn to have a guess what do you think the worst thing about tech industry is three lives something along the lines of agile sprints jira something there okay so like management practices stuff like that okay let's see if that's on the board go for say no everyone loves all that not a problem at all death and diversity is all good yes steve what do you think you have another guest you got two lives left still uh i think it's uh the hours all the hours could be the worst thing about the tech industry it is nearly 9 p.m and i'm hosting a game show so let's see if what gophers say [Music] yes indeed steve number three there the pressure and long hours 16 people said oh yeah take a break everyone not you though julie because you're up to answer a question now you've got two lives still what do you think the worst thing about the tech industry is we're looking for that second answer um hmm this is very hard um i'm gonna say can i say like the bad impact yeah so the evil would you class it as evil yeah yeah negative impact of tech okay let's see if that's up there gophers say no julie uh not there down to one life yes indeed down to one life and natalie you've got your foot on it so what are you gonna do what's your guess it's the worst thing about the tech industry that code was already said right that was kind of the bad practices one yeah bad practice is an existing code was at number four there but number three we've got the pressure long hours and at number one is the people well that would probably fall under the existing code okay yeah oh man i have a phone isn't it i'm out of ideas um not the people not the long hours not the bad practice nothing about management then did we say that is that management practices that also doesn't work yeah that one wasn't wasn't up there either was it what are we gonna say or should we time you out does that mean we get to keep the life no it means the other chance the other team will get a chance to steal all right some document bad documentation okay let's go for that then bad documentation go for say no but what a great guess okay so now the other team have a chance to steal john i'll accept your answer after you confer with your lovely team so i'm assuming that if they voted the podcasts that they would have filtered that one out right [Laughter] twitter reddit hacker news like that kind of like the social media aspect what do you guys think i was i was actually thinking twitter but natalie used it for the best things about it but i think it's kind of like people as well like it can be very bad yeah i mean yeah i think that's good i was also like social media i i would hope that they would take either if that was actually the answer okay so let's go with social media okay social media let's find out go for say no they didn't say that so you don't steal the points that means that team zero keeps those lovely points but let's find out what was it number two on this one it was there's too much to learn there's too much to learn in this old tech industry of ours and it's making us angry that's what we've learned today okay i don't know probably wouldn't have guessed that one i thought there was some great answers there everyone how's it going so far having a fun quiz [Music] yay yeah i am i'm realizing that i do not resonate with all the people who took this survey we've missed like the top two answers several times now i feel like yeah that's it it's surprising you learn you learn things um this is a surprise let's check in with the scores eh shall we please team zero's got 160 points just slightly ahead of team one who have 157 but this is so tight and you won't believe it but the next round is double points round um round six let's go okay okay this is the final round for double points anyone could win the amazing prize any one of you three of you can win this and then you have to share it which i'm sure will be no problem the question for round six is we ask people to describe the go language with just one word describe the go language with just a single solitary word what did people say julie and rob you're going head to head starting with rob what do you think rob what do you think people said i think people said simple do you now let's see if people did say simple go for say [Music] yes and it's the top answer so it means you take control of the board straight away there and we're over to john you've got three lives there are three ones to guess the top answer's already taken simple what else do people use to describe the go language john one word i'm gonna go with verbose verbose okay and just for anyone that doesn't know verbose is when you uh like use lots of extra words that you really unnecessary and don't need to try and explain something to uh somebody or it could be in documentation and essentially you're using way too many words that um you know it's unnecessary essentially and the idea is tells you we have 11 minutes oh not 11 minutes to fill oh okay i won't carry that joke on them for 11 minutes yep okay good bit of comedy everyone we're all equally guilty katie what do you think is verbose on the board i guess oh yeah that was i said genuinely forgot what we're doing um and that's why i shouldn't know it's not on there unfortunately you know it should be now after that um so unfortunately yeah john you lose a life so thanks katie hopefully we can edit that so you know i don't look like an idiot no we can't apparently okay katie i'm gonna say fast fast the go language with just one word fast go for say yes indeed it was in spot number three there nine people said it so you get 18 points by the way for the simple answer because 25 people said it you've got 50 points there so this is there's a lot of points on this board and with two lives left rob's to guess robs rob um how about how about concrete concrete ah okay concrete let's see see if people said that it'd be interesting go for say no obviously not rob but good good answer but you lose the life i'm afraid john over to you now do you see this is the problem last life john fulfilling prophecy if i uh lose this for us um come on get in the minds of these surveyed gophers you you know them by now we've got to know them over a series of five rounds simple questions i feel like you know them pretty well you think readable readable i love that answer let's see go for say no they did not that is a surprise and unfortunately that was your last life which means team zero has a chance to steal and basically win the game if they can get either the second or the fourth answer from this board you're allowed to confer of course natalie i'll accept your answer after you've conferred with your team what do you think what do you say oh boy i i think i would say uh i'll give a few efficient that was fine that was what i have in mind i mean it is what it says on the go website which i put there uh we could also say fun fun is another one so fun efficient secure i like efficient efficient it's interesting that some of the other questions have like seven answers and this one has only four so it looks like everybody has the same same thing in mind or they just dropped out of the questionnaire okay i guess five answers gets dropped yeah oh so it's actually the other way around so many many answers many people answered the same things in this one yeah yeah 25 people said simple nine people said fast there's still two answers to get on the board positions two and four then how many people said efficient should we find out is that your answer yes would you like to lock it in i would like to lock it into number two please [Laughter] let's find out efficient go for say no we've been you've been hanging out in in germany too long natalie i've been reading the documentation it says there and it it yeah okay we're gonna award those points and let's find out at number four on this board people said with just one word the go language nine people at number two pragmatic 12 people said that it would have earned you 24. answers with like four or less then yeah we'll we'll do the data crunching and find out that sometimes happens there's lots of answers and it's not many people agreeing um only only if you had five or more will they make it onto the board of course but there we go let's have a quick look at the scores because this is the final scores now it's time for final scores now team zero got 160 points which is very respectable and i'm saying that to lessen the blow when we find out that team one thrashes you with 225 points please celebrate appropriately [Music] [Applause] so there we go right any surprises what do we think which which was the round john when you were saying that they missed out some answers what do you think what were you thinking of oh look i just felt like half the answers people gave on that last one were ones i would have expected to at least show up a couple times but it is hard when you know everybody chooses different words yeah i did call it that katie and rob were gonna have to carry me because i got nothing on that last round yeah it's a good thing that's why we're a team that i'm pretty brokenhearted i thought i thought we were going to win even at that last round i thought we had it i was like yeah we're going to steal it we're not efficient enough no mm-hmm well enough is really what it was that's true that is actually true or pragmatic i mean that just means you need to rebrand the go.dev website and just put fun real big i think we're going to have to just what were the it was fast simple pragmatic and fun i think that's what's going to go on the website now that's what people think is very complicated yeah but how do you feel do you feel like you got to know the go community a bit more as well in a way yeah and like do you understand them sometimes some of the answers are unusual aren't they i feel like i played this the last time you played this didn't you have questions that asked about editors and people are using that or i think one question was like what was the first language you learned so you kind of got to understand people better based on how old they were when they learned to program this one i don't feel like there was any questions that gave that away too much yeah no but you sort of just have to pick up the general vibes of people i think with this game but interesting fact in the in the first go user survey we had a bonus question which what was your favorite go keyword ah and uh and i i thought doing all these surveys would help me that's the only one it helped me with but um go and defer were both high on the list i remember somehow waiting for else to be on the list even though if and switch weren't i was just like if that's the case it's just somebody trolling us sometimes i think the answers some of the answers in in some of the questions because we've asked loads of questions so we have loads to choose from and some of them are like you wonder if people are trolling or joking and sometimes the joke answers enough people say it that it makes it onto the board as well because you know it's quite funny but i don't know yeah it's interesting to get into the minds of people i think a little bit speaking of that um we've got a couple of minutes um steve what we've been working on lately um i think uh i've been working on a lot of stuff lately um because that's what i do i work on a lot of different stuff but um i know uh i'll just kind of talk about some things that we're working on uh a lot of work getting ready for generics release which is coming up in uh in the release in february but the betas should be you know all the pre-releases we're working hard on uh i'm also meeting with a lot of our users that's a good part of how i spend my day is uh meeting with companies and individuals and projects who have adopted go and just hearing their challenges and their successes and i don't know it's one of my favorite parts of my job actually yeah generics of course coming in 1.18 in february katie also fuzzing is coming first class concern it is i'm so so excited that it's it's happening um i'm working very hard on documentation right now we're kind of in the bug fix and polish and document phase so um i think it's going to be important for it to land with docs or it's not going to land successfully if people can't use it so that's the goal that i have right now for this project yeah i know that makes sense it's i love the fact i love the way that it interrupts with the existing testing stuff like and our knowledge of writing unit tests can come in handy for writing fuzz tests i think that's that's a kind of really nice advantage to the fact that it's getting first-class support rather than being a sort of external tool absolutely yeah that was the main one of the main goals and i'm glad you like that yeah i do i really do um right well thank you so much it's been a pleasure um rob we had you if if our audience wants to learn more about rob and julie uh check out our go time back catalogue um and also listen to future shows because you know they'll definitely be on it hopefully thank you so much to everyone for joining this live feed this game it was a great time was had by all most great times had my most and we'll see you next time bye everyone hi hey there's close of music here we go i just smile don't let the smile go stale thank you to the uh go time team and the go team for i'd like to say a show that did not disappoint but there were some answers there that were kind of they were a little bit lack thereof the the answers were illuminating no for sure yeah we we have some work to do yeah indeed we do yeah well we don't have any work to do on the pawn front i'll share one from the penis oh we do wow the fact that we do means we don't right i feel like there's that juxtaposition in there well here's one from the peanut gallery to illustrate we were talking about khan academy before someone wrote in uh b green wrote in gopher khan you can guess what the spelling was just that's one word that's it that's great please tell me the accompanying picture of william shatner screaming go for comments on me how did none of us catch that one i know i know right we were going all over the board but we forgot the simplest one it was like literally right it was literally right there in front of us seriously oh man well anyway i we should just give a quick thanks all of our sponsors do go check them out on gophercon.com and in the discord go chat with them there's giveaways many of them are hiring um you know they're here they've made this conference possible uh so go check them out go go give them uh have some discussions with them and so forth i mean coming up we have another exciting speaker indeed indeed another return of an original gopher khan speaker an original an original original gophercon 2014. indeed indeed so for a lot of you in the go community this person needs no introduction but a lot of you are new and i hear over half of the community is new every year um you might have come on across his projects boltdb is actually one i i i use constantly in my projects he's here to talk to us about using go in sqlite he loves databases and you're going to see that come through and when he talks about that stuff so uh please uh help us in welcoming ben johnson to the stage ben are you cool hey thank you very much guys appreciate it um as you said yeah i'm ben johnson i work a lot in go and databases um and i'm going to be talking about writing production applications with go and using this great database called sqlite just a quick background on me i've been writing go for quite a while now about eight years now i write a lot of open source as well so bothdb is one example it's a key value store it's using projects like etcd or console i also am writing a project right now or have a project called lightstream uh we'll talk a little bit about later a lot of my focus right now is getting sql light really production ready and getting it to deploy out in all kinds of projects i do a lot of blogging as well so you can find blogs about go project structure on go beyond it's a blog i have and finally i just really love embedded databases i think they're awesome so this talk is going to be very biased but i hope you uh you follow along with me uh so whenever you start a project with go you're always kind of the biggest question is you know which database do you choose there's a myriad of options out there there's traditional options like postgres mysql sql server oracle those kind of things but you have a lot of new ones as well you have all kinds of options from amazon so rds aurora you have planet scale out now just tons of database options what do you choose now a lot of these databases they have you know benefits and they have trade-offs as well a lot of these are familiar so a lot of you have already used maybe postgres already a lot of these databases have been around for you know 40 years some of them so they're quite robust and they just have tons of features in them you know they just collect features over the years now you get some trade-offs with all this as well one thing is expense so if you have a separate server to run a database on that's an extra server to run if you're doing cloud databases they have they get charged for operation and a lot of these databases get bloated over time with all these features and then finally the last trade-off i think is one of the most important is high latency so connecting to another process or connecting to another server can really take up a lot of the time of your processing and your query speed now in addition to the client server and cloud options one i don't think people or most people consider would be sql lite um and it's you know for a long time was kind of considered a toy database or a test database um but really um it's so much more like you have it's a it's an in-process database it's embedded database which means it actually gets compiled into your source code and deployed with your application everywhere it goes uh it's super stable super safe this runs on all kinds of devices so your cell phone has it installed on it it's installed on airplanes it's installed in all kinds of places and it's been around for about 20 years now first released back in 2000 now sqlite provides a lot of great guarantees like any other database so acid compliance we're talking about atomicity consistency isolation and durability that means that whenever you write transactional data it gets saved safely and you can trust it in addition it has a lot of sql support you might not consider has things like windowing functions it has things like common table expressions recursive queries subqueries most of the things you expect from a sql database are in there and finally it has non-sql support that's really great you have full text search so things like elasticsearch you don't always need when you have the built-in full text search within sql lite it also has json support so if for some reason you are storing json in your database you can query it i don't recommend that generally though so as far as when you should really consider sqlite you know historically it was used in embedded software that was kind of its biggest use case it gets used on like edge databases things like that small resource constrained servers and that was kind of its purpose purpose for a long time now there's been a lot of improvements over those last 20 years specifically in concurrency so it does really make sense like a multi-process or multi-threaded database so because of that it's really useful for read heavy workloads you might consider it for something that like an ecommerce website for example where you have someone going to a lot of pages those are all read queries and then every once in a while maybe adding to cart or checking out those would be right queries it's good for you know small to moderate request load things like uh 10 you know tens of requests or hundreds of requests per second which probably covers most applications out there and you know really sqlite has kind of grown so much over the years that can kind of be considered a general purpose database for a lot of applications um you know obviously no tools used for everything and they all have their specialties but it's i know i consider it a good default to start with i think historically i always consider postgres as my option and then try to think about you know why would i use something else these days i really look at sql lite as kind of my default database and then consider why i would use something else so this stock's gonna be split up into three different parts we're gonna talk about kind of the development side the testing side and then finally we're gonna get into kind of the production database side and what that really looks like from a durability and a performance standpoint now when you connect to sql lite you have basically four options out there there's probably some extras but these are probably the most popular now the most common library out there is going to be the matt n library go sequel light 3 and honestly this should just be your default it's used in all kinds of places it's been around for years stable and great i have no complaints about it this is one i typically use there's also a project going on from modern c uh where they've actually transpiled like mechanically transpiled all the code from sqlite into go so you can actually compile it as a go library and then you avoid the sego overhead and any issues around cross-compiling next we have the low-level sqlite database if you want to avoid the database sql library you can use this one by david crosshaw and then finally tailscale is actually working on an implementation right now that has some improved benefits with concurrency over the matt and library it's a work in progress but definitely check it out they are doing some good work so when you connect to sql lite um this is actually going to be pretty brief um you use it like any other driver for database sequel libraries so you import it with a underscore before because you're not referencing the actual name of the library and then in the sql.open you are going to just pass in the driver name which is sqlite3 and then for the data source name instead of a url or a network path you're going to actually pass in the path to the the file on disk and then after that from then on that database connection really you can just treat it like any other database connection now i generally consider sqlite pretty configuration free there are a couple exceptions here that are worth noting and we'll kind of go over those but once you get these set there are some knobs you can tune if you really want to later on but generally it just pretty much works now the first one here is probably the most important i find it's called the journal mode and journaling and databases is how you safely write a transaction disk and what happens here is the default mode the original mode was called a rollback journal and you'd write the whenever you wrote dis or data to your database it would go into you take the old data put it into a rollback journal and then you write the new data to your database and then you delete the rollback journal there's a new mode in here since 2013 called the wall mode this stands for write ahead log and what this means is that you're going to write all your new data to the separate file instead of your database and you get a lot of concurrency benefits with this you actually get point time snapshots for every transaction as well as you're able to write and read at the same time there is a limitation on sqlite where you only have one writer at a time but you can have as many read transactions as you want next this is kind of an odd one if you're coming from other databases it's called the busy timeout uh the busy timeout sets how long uh a right transaction will wait to start now again sqlite only supports a single writer so that means if one's going if one's active and another right transaction starts by default it'll just fail immediately uh that's not usually what you want so instead we'll set this to the number of milliseconds we want to wait um i usually use five seconds sometimes you can use 30 as long as it's more than zero that's kind of the biggest thing uh we're going to use this format here with a pragma which is essentially a way to set uh settings on sql light so we're going to do pragma busy timeout equals 5000 for uh five seconds uh final option that's really important is foreign keys and a lot of these are for historical reasons as to why you have to set these things or why they weren't enabled originally foreign keys are not enforced by default which is pretty surprising for most people um you know sql lite started off in very resource constrained systems so you didn't want to necessarily use all that processing power for a foreign key constraint checking so what we can do here is we just set pragma foreign keys equals on and it will enable enforcement for your foreign keys so those are the main settings you want to set with sql lite i find another side another side of sql light you really need to kind of get used to is the type system uh it's a little weird but it's not as bad as a lot of people consider it uh the first thing to note is that the type system is pretty small uh you have integer you have real which is the floating point numbers you have text for readable text and blah for binary data but this actually pairs well with the go type system we have integers we have floating point numbers we have strings and we have white slices and finally we have null which is kind of no dated at all now the really weird part about the type system in sql lite that people kind of get thrown off by is that a a column isn't necessarily associated with the type it doesn't have like a strict type every value you insert into the system has its own type and type definitions on tables and their columns are basically ignored ignored by default so here down below you can see we have a create table uh with an accent integer and a y text but sqlite largely ignores those types and you can insert text into x or you can insert integers into y and it'll just let you do it which personally seems insane to me but that's how they set it up luckily actually just only two weeks ago they added support for strict mode uh which actually enforces the column constraints and a couple other options too so i don't have time to get into this presentation but it's something to look into the last thing you kind of caveat you got to worry about when you're developing is that sqlite kind of has some some types i really wish it did have it doesn't have these two are timestamp and decimal types uh for time stamps while there's no type actually available within the type system it does have some date time functions and these functions will work with three different types of dates uh the first one is iso 8601 and go this is known as rfc3339 this is a string format where it's nice because it's actually lexographically sortable naturally as you can just sort them and use them in indexes and it works as you expect the other option you can use unix epoc which is another second since 1970. this will give you an integer which you know shrinks down the amount of space you're using for each column and then finally there's something called the julian date i would generally advise against this one because most people have no idea what it is it's the number of days since about 4700 bc again it's it's pretty obscure but there are some applications for it but i would generally avoid that one now the development side once you kind of get it connected get up and running um it's really pretty good pleasure to use once you get around those caveats in the type system another piece of it that's super nice with sqlite is the testing is just super fast now the biggest thing with type system like if you come from an orm that allows you to run against multiple databases a lot of people will actually run their tests in sql lite because it is so fast again it's built into your application and compiled in there but the bigger thing is that you can actually has native support for memory back databases this means you know you're not going to have any slow disk access you're not going to have fsync calls everything actually lives on the heap we can do this by setting the data source name or the dsdsn to colon memory colon and then everything will just be in memory from that point on and again you just use a database like any other database except now it's screaming fast we can do some benchmarking here with sql lite again this is very much loaded in favor of sql lights but we're comparing this here go sql light in memory versus kind of a standard postgres setup that people use for testing um you know both are the default configurations and just kind of understanding how the performance changes uh so with postgres you know certain operations you need to do but you don't need to do on a memory database so if you need a drop of exists uh you don't need to do that for in-memory databases because you wipe it out every time you um you close it so for post crafts we're talking you know five and a half milliseconds for create table you know it's three times as long as the in-memory version and some of that is just the connection time uh and then when you get down to inserts and selects you know it's again screen fast where we have 25 microseconds for selection inserts versus you know around half a second for inserts and selects on postgres so again you know it's not quite apples to apples but it's a good idea to understand really the scale of difference between the two uh another thing you can do is parallelize your tests for sql light again because it is in memory there's really a shared nothing approach between the different databases for each of your tests you can use t.parallel to run these in parallel tests in parallel there's some options on go test where you can flag the level of parallelization the one caveat here is that sql light 3 sorry go sequel light 3 has a global lock on its connection so it'll actually slow down parallel tests if you want to get better performance for these uh parallel tests you need to use a tail scale implementation uh here's a quick benchmark here to kind of illustrate the point so with go sqlite 3 and tail scale they both run in just under a second and a half and here we're running about 10 000 small tests each test opens the database creates table uh does an insert select you know some very basic uh testing and then close us down uh we can run that about 10 000 times again in memory but as we do as you scale this out over more cores for ghost equal light it actually ramps up over time to three seconds and gets slower whereas with tail scale you can see that it actually speeds up quite significantly so on the development side we've really seen where you know it can be pretty easy to get up and running on the testing side we find that screaming fast and we can just constantly test and get feedback and what we need but then the thing that people generally worry about is production experience and that kind of factors into two different ways we have the the durability side and the performance side now before we get into too much on the durability side on sql specifically this is kind of a quick aside about durability in general i feel like over the last 10 to 20 years you've had a lot of conversations around availability where people had expectations of 100 uptime uh and that's not realistic really you know you're looking at 99 99.9 percent uh uptime and there's cost associated with trying to get higher and higher up time so it's really a trade-off of how much one you want to spend and the complexity around that system and what level of time you're expecting however on the data durability side we really don't have that same conversation and uh before we talk anymore it's just important to note that there really is no such thing as 100 durable data is you know you can make as many copies as you want of your data but you can always lose them all so sad but sure now there's kind of a spectrum of durability when we're talking about hard drives and systems so you know when people think about running sql light on an application uh sometimes they just think it's like a server in your closet that has a hard drive and that's not very durable i mean backblaze puts out an annual report where they find you know they check all their hard drives and see how often they fail and they find that about one in 100 fail annually so that's a 99 durability uh you know 100 chance isn't too bad for a lot of things but losing all your data you probably want something a little more safe now you can upgrade look at something like cloud servers where they tend to have a bit higher durability with their hard drives they can be rated they have other options around that too evs most of its volume so this is amazon's elastic block storage most of their drive types have three nines of durability this means that every year they lose about one in a thousand hard drives you know that's that's better than what we had with one hard drive but still it's not amazing when you're considering losing all your data um you can look at something higher like i o two on ebs this has five nines of data durability that means you have one and one hundred thousand drives fail annually uh you know this is getting some much something that's much more comfortable you know it's probably you probably have a bigger chance of dropping the database or deleting a table uh than one in a hundred thousand so you know something you can get more comfortable around in data durability terms uh finally we have s3 uh this is kind of you know on the far end of durability it has eleven nines which is crazy uh that means you lose like one and a gajillion objects every year so generally when you put something in s3 you kind of consider it safe to be there so we're talking about data durability beyond just a single disk we start talking about you know one or more servers so obviously single server is going to be your fastest cheapest office option but it's less durable you have options for doing a primary primary replica setup where we stream changes from a primary down to a replica this is obviously more complex you have double the number of servers so there's additional cost and finally it's you know it's more durable you have two servers instead of one there's also you know more complex situations you can get into distributed consensus for example you can have a cluster of servers that need to confirm rights before you let them continue this is really complex though if you've ever run say at cd you can kind of understand that there's a lot of complexity and performance you're trading off with using this but on the other hand it's really quite durable you can lose multiple nodes possibly and still have your data so uh that's kind of a side this little rant here with a quick story um you know the idea here is that no data loss is good but not all data loss is catastrophic um i'm not trying to throw anyone under the bus here i think gitlab did a great job documenting a an incident they had four years ago so in 2017 they lost about six hours of data which sounds you know that's rough no one's going to have a good day with that it's unfortunate but you know what happens sometimes they had a primary postgres replica setup so streaming data from one uh primary to replica however uh the replica just stopped getting updates for some reason they were trying to figure it out and during this uh while they're trying to figure it out the operator accidentally deleted the data on the primary when they were deleted on the replica and they lost six hours so uh terrible but uh you know get lab still around and they haven't fizzled out in fact they actually ipo this year for 16 billion dollars so again this is really a trade-off no data loss is good but it's not necessarily catastrophic so i'm not encouraging data loss by any means but i think it's something to consider how much your data costs and what you're willing to trade off for it in terms of durability and performance and cost so without that all said let's talk about data durability in sql light specifically so uh your your first option you should probably consider is just honestly regular backups um you know if you're talking about running something like io2 on ebs you're already getting you know tons of durability with that in the first place and if you're regularly backing up then you really have a fallback plan anyway after that if it fails so regular backups the pros here is it's it's super fast uh super cheap and it's really hard to mess up it's not a lot of configuration uh the main trade-offs here is that you start to get a bigger data loss window so a data loss window is when you just have data that hasn't been backed up and you lose your primary data set any time in between the last backup and your current data set is your data loss window now you can set up you know a regular hourly backup for daily backup to say s3 and again you know sqlite is just a file so it's really simple to work with we can back it up we can compress it show it up on s3 and don't worry about it which is nice we can use a time-based file naming schemes here so if you want to name it after the current hour you can just have it have replace the current or the previous snapshot from yesterday and just have a rolling 24-hour backup which honestly works great for a lot of applications the other thing too is b tree databases which is most sql databases uh they compress really well there's usually a lot of extra space in there so that they don't shift around records when they update and insert all the time and it means that if you have you know say a gigabyte database and compress down to you know a couple hundred meg and it's not as painful also something else to note with s3 that's uh interesting is that they make it so it's very cheap to get data into s3 but very expensive to pull it out of so it works great as a backup solution you don't do a lot of operations where you're restoring constantly but those operations pushing it up can be quite cheap so sqlite has a command line sqlite3 and here we can see at sqlite3 we'll pass in the name of the database that's mydb and then we pass in the full command here in the double quotes which is going to be dot backup and then the name of the file we want to backup to so again back it up to another file compress it push it up on s3 and uh yeah generally actually works pretty well for a lot of applications now if you can't handle that data loss window of an hour you know that it can be pretty long for a lot of people even if you have you know again say five nines of durability on i o two um you can look into other options so this is light stream this is a tool that i created um so obviously i think it's the best but you know every tool has it's it's a situation where it works well in so the idea with light stream is that because s3 is super cheap to back up to um is that we can actually continuously back up to light stream which is really nice uh sorry 2s3 so what lightstream will do is it'll snapshot your data at the current point in time and usually once a day and then whenever you make changes to your database it'll read those off that right ahead log that we talked about earlier and then it will compress those push them up to s3 and if you ever need to recover your database it can pull down that snapshot replay all the pages from the wall after that and you'll have the exact same bytes4by database afterward um you know this works too if you have catastrophic failure you can set it up to automatically restart and reload and go with that so again a bit more complex than your regular backups but you know it's a pretty nice option uh one user of this michael lynch he's written a blog post on this um where you know he's looked at moving from different uh cloud services over to lightstream and he found that he only spends about three cents a month on these uh the s3 backups so he finds it as a great super cheap option so something to keep in mind if you do download it lightstream actually runs as a separate process sqlite supports multiple processes connecting at the same time and your your actual application has no knowledge of light stream uh it's really like an ops consideration so here we have a light stream command uh there's a sub command called replicate and we'll pass in the path to the database which again is just a file and then we'll pass in the uh the s3 bucket uh here's just kind of a visualization of light stream so we have our database on our application server it whenever it has changes it attacks those onto the wall and then light stream is you know pushing up snapshots and continuously pushing up new wall pages um as soon as it happens so uh this is again a good option for reducing that that window of data loss but again it'll trade off of a little more complexity finally on the durability side you know the most extreme example once you get into data durability is clustered sql light which is a bit of an oxymoron it kind of seems but it works for some people so the pros to this is that it's super durable i mean you're going to run a cluster of servers so the cons here obviously that's going to be more expensive you know three servers or five servers is going to be more expensive than one and it's significantly more complex if you've ever run etcd before and try to manage that you know it can be sometimes a bit of a headache so keep that in mind and these kind of get split out into a few different types we have wrath-based so raft is a distributed consensus protocol etcd uses raft for example but arculite was written by philip o'toole and that's a great option he's done a great job with that canonical also wrote their own implementation of a distributed sql light called dqlite and then we have uh on a primary replica setup where we stream from a primary server to a separate replica server we have something called light replica this one i haven't used this personally it does have a gpl license so watch out for that if that's a consideration you need but they do have commercial license you can check out too and then finally there's kind of a crazy project from expensify called bedrock i haven't personally used this one it's blockchain based and i have no idea how they do it but it sounds very complex apparently works for them so kudos to them but i have not used it personally so you know on the performance side you know we've got durability if you're comfortable with the durability story then performance is kind of the next thing you really need to consider when you're talking about performance or sorry production now performance between sqlite databases is always hard to kind of quantify um you always see synthetic benchmarks out there where one database is x amount faster or x percentage slower but really kind of have to test it with your own workload to know what is actually going to happen in the real world because they're all again synthetic benchmarks now the caveat here is that most sql or sql databases in general are b3 based that means they have the same underlying data structure and because of that a lot of the performance profile is generally going to be fairly similar you know some are going to have a bit better performance some are going to have a bit less performance and it's really hard to say for each situation one might be better than another however the one consistent thing is that network latency can be a huge component of performance and this is really where sqlite shines because it basically skips over the network piece entirely because it lives right in your application so doing some benchmarks here against sqlite versus postgres um we're doing point queries in this instance here so it's running about 10 000 point queries on a fairly large database and it is we're calculating the time it takes on average for each query so for sql lite we see it's way down there at only about 18 microseconds which is screaming fast and then you know once we start going out of process to another process over some kind of network connection uh even on the same box so postgres running on the same box uh still takes 10 times as long just to get that one point query again this is you know connection overhead this is a serialization there's all kinds of you know overhead here that you need to consider you just don't have a sql light again it's a it's a lightweight database but you know again has some features too uh once we move out of the same server into the same availability zone here we see that it jumps up it basically doubles this is all running aws um but we see that it goes up to 300 microseconds you know again not a ton of time but if you're running hundreds of these queries or even tens of these queries that adds up to real numbers and then finally running it within the same region we see it gets three times as slow we're now up to almost a whole millisecond at 900 microseconds so going down from sql light at 18 microseconds to you know same region at 900 microseconds is a huge jump now you know that that's how most people set up their applications when they run their own database server if you start running against a cloud server that's a whole different story might not even be in the same region as you so you need to really consider latency at that point so once we add in cross region latency we really start seeing big numbers now this is adding in u.s east one so virginia to usc ii ohio which is not very far away but it still takes about 11 milliseconds you know you can't even see sql light at this point on the graph it's so tiny but you know it's something to really consider this network latency is going to be a big part of your performance now you might consider you know that queries can run in parallel and that works in some situations for sure but we see we when we do parallelize them uh we you know we get better performance generally especially as we have more network overhead but we really don't get anywhere close to that latency of sqlite because again it's in process and it's so close to your data now a lot of times you can scale up a single database on a single server quite large you know we have aws where it can scale out to you know 96 cores or some ungodly amount of ram which is great you know our machines keep getting faster every day if you ever do reach your maximum of what you can store on a single server and process on a single server it's time to start thinking about moving horizontally and horizontal scaling sqlite is not an often talked about topic because it's kind of that seems antithetical since it is an in-process database however uh the benefit here with sql lite is that it's so flexible you know it really is just a file that's that's all it is so we can do some pretty interesting things that you might have a harder time doing in other databases so for example if you wanted to isolate your tenants put a single tenant per database which can improve your security um you know they're all just individual files so we can move them around we could put them onto a cluster servers and then distribute those tenants based on like a consistent hashing scheme um you know this talk we don't have time to go into consistent hashing schemes but it's something to uh consider if you actually do want to fan out your uh your tenants and their data over a network or over a cluster so again options out there if you really do need to scale to those uh bigger data sets now as much as i love embedded databases and sql lite um there's always times when you just shouldn't use a tool so you know i want to you know get into that you know it's not all sunshine and roses out there there are considerations here as well i would say the the biggest time when you don't want to use sql lights by far is if you have a working system please please do not rub out your database and put in sql likes you know your colleagues will hate you you know you'll never find happiness in doing that so definitely avoid that with a working system now you know a good way to do it is to just start off small have a smaller application you know get going with that and if you really like it you know there's always some um some operational pieces you need to learn around any database so it's good to get comfortable with that at a smaller scale for sure another piece you want to kind of avoid if you have long-running transactions in your database usually you don't want those in any database really with sqlite is really a requirement though because it has that restriction of a single writer other databases get around it with some optimistic locking schemes which can work to some degree generally you want to avoid long-running transactions another time to really avoid it is if you have like a truly ephemeral serverless environment you know if you don't have a disk or if your disk suddenly disappears when your service is done running it's definitely time to you're going to lose all your data so that doesn't work great there are some options out there services like fly.io they kind of work similar to heroku so you have kind of those kind of dinos or kind of serverless environment that kind of feel we can push up uh to basically get repo and deploy you can do that but they actually provide persistent disks as well which really helps a lot so that's something to consider uh so you know just to kind of wrap up here a little bit um you know sqlite's great you know in a lot of different ways it's it's kind of a different paradigm from what you're probably used to with a client server database or a cloud database for example you know going back on the development side you know it's great because there's just no dependencies at all you know you import a library and you're done basically there's not really any configuration you just get up and running you don't have to worry about users or grants or privileges you know it just kind of works uh there's not much configuration you know we have kind of those three pragmas that we set the journal mode uh foreign keys you know that kind of thing so make sure you set those those are important and then again the type system it's kind of weird a little esoteric but it actually works pretty well against ghost type system and you know enforcing your types at the application layer and then as far as types go you know time stamps and decimals those are always going to be tricky but you know generally i would say that rfc 339 time stamps work pretty well uh on the testing side you know the in-memory support is fantastic you get your tests running super fast you can just constantly run them whenever you make code changes instead of waiting seconds or minutes you get your feedback in you know sub second even for hundreds of tests there is parallelization you can use but again there's a caveat with that the the go sqlite3 library doesn't work as well with parallelization and you really need to evaluate kind of the tail scale implementation even though it's you know it's still a work in progress uh on the production side once you actually do put it out there you know the there's a couple different options again you have your regular backups again you know some people may scoff at this but it's really a great option especially once you have um very high durability drives anyway you really you have a known data loss window but it still works you know great and you put yourself a pretty low risk of data loss in general uh if that doesn't work for you though options like light stream where you can stream your backups up to s3 also work pretty well again you're going to trade off a bit of complexity for that so just kind of go that i know that going in uh and then finally we have cluster sequel light which is kind of on the extreme um where we're using something like graph to replicate either through arc light or dq light we have light replica which is a primary replica setup or this bedrock which is the the blockchain based one although if you get into cluster sql light again there's not as much of a difference of just using a regular client server database at that point so that's something to consider too on the performance side you know single node performance is just screaming fast like you can't beat it once you remove that network latency and your serialization overhead you can really do so much um and bring your request times down significantly a lot of times you just don't have to worry about things like n plus one queries and this is just a whole class of performance problems you generally don't have to worry about sqlite and then you can scale it up you know there's pretty large nodes out there either on aws or whatever service you use uh and you can also scale it horizontally with some creative ways again sqlite is just a file so it's super flexible so really in conclusion i hope you give sql lite a try you know i love the database it's not great for everything maybe you'll hate it maybe you'll love it but i think it's worth a shot and uh yeah i really like it a lot so thank you for uh listening to my talk really appreciate it uh you can reach me either by email or you can find me on twitter as ben b johnson or pretty much anywhere on the internet has been b johnson and yeah thank you very much uh back to you all in the studio thanks so much ben oh that was that was a lot and the the main thing the main thing i took away from that was you know we have a see a simple option i was about to say sql option which would be awesome that is an option but you know we have a simple option we can start with right you don't necessarily need to start up a postgres or click the button on your cloud provider to get a postgres or my sql you know you can just get started single node write your app get it running you know when you hit the front page of techcrunch maybe then then you get your postgres in the cloud or whatever it's kind of sad because i feel really old thinking about the fact that that used to be the slashdot effect i know right the number of people watching this who are going to be like what what is that for for those who don't know slashdot used to be the popular geek uh hangout then it was replaced by digg and then oh it did and what came was yeah yeah that was the first plate yeah yeah that was for at least the first popular uh implementation right of the upvote yeah and the down vote yeah and then remember pounce pounce yeah it was uh it was created by the creator of dig and somebody else and i heard that [Laughter] master recognizes another pun master yes the pod masters have spoken yeah you know now i heard there was something blockchain in there did anyone catch what that was and i know there's replication and that that stuff is like very cool very cutting edge i never would have thought to replicate sql light or backup sql thing like that but i caught some blockchain strategies to do replication in there uh you know which which according to discord is really just merkle tree based replication but you know we got we got blockchain at the conference now yeah we have to somehow well do we have to somehow in blockchain does that mean we should mention nft for ben's talk no oh yeah we do that's the law that's the yeah well i know you had an unpopular opinion the last go time about blockchain i know right that might be not actual very first unpopular opinion on going but not your first unpopular opinion in general right and then i know right yeah i know right yeah i said blockchain developer was the new hotness of jobs and people are like nah man that did not we refuse that we refuse i mean just because you don't like it doesn't mean it's not popular i'm just saying you know you know there's always going to be somebody somewhere trying to make money on some poor souls so even if it's not uh here here's this will hit some discord people it'll rub them the wrong way even if it's not real money even peanut gallery what do you got for me let's hear it uh i'm i think we're this might be one of those times where we actually are very fortunate that we are not in person maybe they might meet you in the barbecue peanuts yeah that may be my unpopular opinion there yeah all right well um i heard we might be saving the best talk for last i heard something about a talk is it the best is i mean maybe we'll have to see i i think it's the best i mean look look at the speakers the caliber of speakers that are in this thing i mean you know one of them i mean both of them have been in the go community for a long long time you know they're passionate about this community you know they invest time in it they're funny they're charming handsome um you know like very eloquent you know of speech you know i mean it's it's like how can you not you know be looking forward to this talk you know and and obviously you know i'm talking about me as one of them as one of the speakers and uh and uh you know you know what's what's better than having one great speaker up there is having two who's joining me today this uh aaron schlesinger this guy aaron schlesinger and and he wore his best shirt for this talk he dressed up to the nines he needed to make sure that there was just lots of attention yeah yeah i wore a really loud shirt i went with blue instead of gray for for all you and for you two as well yeah yeah so aaron is gonna be the the moon to my son right he's gonna he's gonna be you know dimming down the light the brightness of my shirt you know with his own johnny turned it up to 11 and i'm here to turn it back down to like a seven six something like that we have a neighbor yeah come on let's not go too let's not go home all right so i'm thinking we we we walk on over you know to to the to the the area where we do our thing and then um you're gonna be okay buddy i i think i'll i'll be all right yeah it's gonna be a little lonely the end all right let's make it happen all right we're gonna walk over we should have some like music and tunes as we start running down you know run down to the podium to go do this yeah let's do it oh man all right all right all right here we are looking at our blank editor here so what are we going to talk about today so i was thinking right that every year we hear about how we get we get a lot of new folks in the uh go joining the go community for the very first time um and they're they're they're curious about go they're they're picking up you know some tutorials and picking up you know some of the idioms of the language and things like that um and everybody comes in like hey go routines and things concurrency that's one of the the draws of the language right and uh and a lot of times you know we end up also seeing um some misuses right of the concurrency um concurrency is great and go um but you can easily get yourself into trouble right when you have too much concurrency there is such a thing as too much concurrency right you know we get this toy we engineers love toys right and we get this go keyword oh man that's the toy that's all it takes you know you put go in front of a function and off you go right you know the puns follow us wherever we go you all thought it was over when we were over there no no no no no we're gonna be we're gonna be all up in it so i was thinking that um one one one thing we could do is about introducing sort of a a simple little program simple little concept you know um to sort of uh add some sophistication to it as we go uh and and to sort of introduce some some patterns and some ways of reasoning about you know a concurrent program so i thought we'd build like a quick little port scanner together what do you think sounds great sounds great so before we jump into building that let's actually quickly go over what a port scanner actually is so i have a quick little slide here um that's going to uh kind of sort of illustrate sort of the communication that happens right between a client and a server right and the client could be another server right you know whatever the case may be so a person initiates some network some network contact yes to another to another um server maybe you know uh on the other side of the world wherever the case may be right so usually there's a there's a there's a synchronization packet that sort of finds its way from the client over to that server if the server wishes right to engage in our communication right maybe there's no firewall or maybe there's no rule that says it it can't talk back um to that that the initial uh request it sends back an acknowledgement right of that initial synchronized packet right and the client now says okay i'm going to acknowledge that you sent me an acknowledgement to my original uh uh um synchronized packet there's a there's a joke there somewhere exactly exactly and then at last right the connection is established now we can actually talk back and forth with each other right so that but that process of actually connecting over to that client that needs to happen on a port right so the server must have a process right running locally that has basically is bound to that port to a port right whatever port you're trying to connect over there must be a process listening on that port right for the the host to actually receive your incoming request and says okay yeah i do have a process running on that board so i'm going to facilitate that communication all right so when we talk about port scanning really what we're doing is basically saying hey um can i get given a range of ports can i can i go through the list quickly right for a given host and sort of enumerate right the list of open ports enumerate the list of ports that actually i can actually initiate some sort of connection on onto not that you're actually going to exchange any data you just want to see it's just probing right is this part open so we're going to be doing some heavy duty kind of low-level networking we're going to probably expect a lot of times to fail to not be able to connect but we want to figure out what are those times and what are those ports that we can connect on successfully right exactly exactly and go right exceeds at this particular kind of application right it goes the center library provides a lot of the building blocks right to to facilitate right and creating the creation of these kinds of programs and a lot of our work is today is going to be based on the net package in the dial function because it abstracts away a lot of this sort of you know handshake process the back and forth you know handling you know drops and packets and all that stuff like all that stuff is sort of a low level concerns that we as the developers basically using the standard library package don't have to worry about and we're going to be doing almost everything with the standard library here absolutely one or two but you know ninety percent of our work is standard library library no yep no external dependency yep indeed indeed so let's flip back over right so the way i usually like to do these things is to basically you know our our task right what we've been asked to do is to create the sport scanner and i usually want to start with the simplest thing that could possibly work what is what is the minimal effort i could put in to just test out the idea right kind of similar in spirit to what ben was just talking about is you know we don't necessarily need to go and start up another database and all that let's just take what we got running on the same machine here we're kind of following the same spirit let's get the simplest thing working and then evolve it over time we're going to see nine evolutions of this yeah as we go as we go yup exactly exactly so what we have here is really is the the bare bones of what could possibly work so we know we're going to be using the net packages dial function to actually you know initiate that connection this is going to be a tcp based connection and obviously the the stem library the net package supports all kinds of connection types ccp is only one of a handful so we're going to be basically hard code the port that we need for now it's port 5432 i happen to have a postgres server running locally on that port right uh we're going to see if we can actually establish a connection on on our local host here uh and on that port and see and see what we get right and then we'll try it you know without on a port that we know is closed and they see what the response is that way we know what what to look for for an open board and a closed port right how does that sound that sounds great and look how i mean this is well this is 18 lines including the new line including imports but really we've got line 10 through 16. some of that's error checking yep the go standard library makes this pretty easy makes it absolutely very very easy to do yeah so let's actually do a quick go run here let's do this is the simplest thing right and then let's just run this and it says it's open right and then what i'm going to do is i'm going to change this i'm going to change my port here and say 5433 right and then i'm going to run this again and then now we're going to consider that for close all right so let's talk about the output here so when we call net.dal right so let's look at the the the the the results the the responses that come back so we're going to get back a value of type net con right that's going to be the connection value that we can use the object if you will um that we can use to to to do other things maybe we want to start exchanging data maybe we just want to send send some some some um text over and see what we get back but that we don't particularly care about that when our job is to find out is this board open or not right so we do have some methods on that struct that comes back we're not going to use them today right but we could you know in the future if we want to do some communication with the server once we connect to it or what have you exactly exactly what we're going to worry about more so today is the error value in other words for whatever reason regardless of why on arrow's return we're going to consider that port closed for inter purposes because whether or not it is closed because we couldn't reach the host or because you know there's nothing listening on or bound to that port on the host we don't particularly care an error is returned maybe at the time we timed out trying to get a response back it doesn't matter we're going to consider that board closed for on-time purposes that's why the different responses here 5432 again i have a postgres server running that we were able to successfully get a connection back but in the cases where it was a different port that we didn't we couldn't get a response back we're going to consider that close so that forms the the foundation the basis for the evolution right of our program in the next steps that's great this is not a lot of code i you know i've used go for a while so i happen to think this is very understandable i think someone from java or even python c c plus they can come in and get a pretty good grasp on this pretty quickly absolutely absolutely but the thing is we haven't done any sort of a scanning yet right so you know remember we typically trying to enumerate a list of ports right so let's let's basically you know augment our previous program by simply saying hey let's just range over a couple hundred ports right that's that's the that's i think that's more appropriate for a scanner right so let's actually see right when we run this version of the program what we get all right so let's go full screen on this so here we are we've just scanned a couple hundred ports right and we're seeing that the the same um sort of an error result that we can expect when the port is is is we consider that foreclosed right so if we scroll down we scroll down we're scanning ports we're scanning sports we should see 5432 here it is boom this is a port that that was opened before so now we've enumerated a list right uh now we have ourselves a list of all the ports that that we consider to be open but are you noticing a pattern here without scans yeah i mean we're doing these in order one after the other sequential in this case you know we're doing 5300 to 5500 it was pretty fast on your machines it's a pretty modern machine right but you know what if we're getting into the cases where we want to do tens of thousands of ports right i mean a lot of ports a lot of work hundreds of thousands might be a little out of the realm of reality yeah you know let's say five thousand ten thousand ports you know also you've got some number of cores greater than one on this machine i could probably use you know more of those things especially we're writing go so you know this might be a good opportunity to explore making this concurrent and probably parallel as well yeah exactly and the other thing is is too is that there is there is no real dependency between one port scan and the next right so these are these are there's a there's a term for for these i love this term you want it you want to take this one um how about we split it okay embarrassingly parallel oh we're not splitting the word the word we're supposed to put together okay get it together one two three embarrassingly parallel we didn't rehearse that yeah a couple off the cup right so so we we know that we we there's no dependency from one scan to the next so there's there's an opportunity here right to sort of uh make the work concurrent right enable that parallelism and because i have multiple cores in my machine we are going to hit some parallel workloads so let's see if we actually turn this sequential program this procedural approach to our scanner and do something a bit more concurrent and this is where we get to use that toy that keyword ah yes indeed this is where we actually get to use our toy so the i've been told right that the only thing we need to to make a go program go faster make it concurrent right just slap some go keywords in places and off we go let's see let's see what happens what happens so what we did here right we basically create ourselves a little sort of anonymous function that we are going to invoke all we did was we moved our our port scanning code right into that anonymous function so that we can actually you know sort of run that concurrently right outside of the the main goal routine right and then making sure not to not to have a variable shadowing um going on here making sure we're passing the actual copy of the port that we want to send in so that we don't get us into trouble with the with the loop variable and then basically it's doing the same thing right uh if the port if we got an error back market is closed if we if we actually got got to here you know consider it to be open and then we're going to say hey we're done after the uh we've uh we've done the work so let's do see what happens oh uh what happened there we got no output from any of the garages any of the growing that was a little too fast wasn't that a little too fast so what the heck is going on here can you tell me what i did wrong well when you have a go keyword that means that you have scheduled the go routine the goal run time is going to schedule the go routine doesn't mean it's going to start even start running it doesn't mean it's going to wait for it to be over and that's kind of one of our first rules of the day right we want to know we always want to try to know when the go routines are done not just schedule and not just start we don't necessarily need to know when they're started we need to know when they're scheduled and when they're done exactly exactly and in this case here we we just scheduled them and we just kept on going on our merry way for the best yeah i just hoped up for the best because a lot a lot of folks often don't realize that the the the moment you start running up a go program you are running girl routines right in in our in most cases right if you just have the your main if you have everything happening inside of your main a girl routine that is a girl routine right so now when you fire off other girl routines you need to somehow synchronize them you need to somehow allow one to wait for the other right or others or whatever whatever they're launching you need to synchronize them somehow so thankfully the go standard library makes that process a lot easier and there's a package i'm thinking that actually is going to help us add some synchronization to this to this problem let's go check out what that package is let's go check out what the package is and you know most of what we do from this point on is going to be all about synchronizing these go routines we kind of got this pattern we've got n plus one go routines where n are the workers and then that one is the mango routine that you were talking about and we're going to do a lot of synchronization between the one and the end and exactly exactly so the standard library has the sync package right basically the same package is going to provide us all of the building blocks that we're going to build on moving forward for this particular program so this is where we also ask for to have some sophistication right to our to our project basically i'm going to parameterize the host you know the from port the two port and other versions of this program are gonna have similar sort of a configurability as well as we go so i've got some defaults here so i'm basically saying my default front port it's gonna be 8080 my default 2 port is going to be 80 80 90 so if i don't override these things then i'm gonna add just a small range of ten ports to scan so and basically the rest i'm gonna i'm not gonna go through every every line here the rest i'm just doing some checks and making sure that okay you're not sending me a letter as opposed to a number football scanning and that kind of thing so let's get you're going to go for 80 90 to 80 70. exactly exactly basically yeah that would that wouldn't be so good yeah but so this is where starting line 39 this is where things start to get a little interesting right this is where we actually initialize well i should be careful here this is where we declare right a weight group so why why don't we have to actually initialize this weight group well if we look in the docs first the sync package and we scroll down to the weight group type it says that the zero value is initialized right right the zero value is useful right which is another idiom in the go community that basically you hear folks talk about make the zero value useful this is actually the same library living up to that to that mantra of making those zero values so we don't have to give it a value anything like that we can simply declare it and when we actually start using it like when we say wg.add for example um it's already there you're not going to get a a panic because you're referencing calling a methanol object or anything like that so you're good to go right so before we talk about the ad though uh we have to think about what are we adding what is it that we're going to be waiting on right so the weight group right the way that works is that you have to specify some sort of counter you have to specify how many things you're waiting on so how do we get that value how do we how do we unknow ahead of time what what what how many things to wait on yeah so we've got our list of ports so we're basically saying well we've we've got x number of ports so i should say n number of ports right we're going to want to start up for this exercise we're going to start up a worker per port in the list so we're going to say all right well how long is the list how many ports did you configure when you ran this code how many ports did you want to scan we're going to do our wg add and pass the length of that list exactly exactly and one of the empty patterns that we want to we generally want to avoid is one where we're actually calling this ad right inside of the the the the invocation of the go routine right so you know you can you're gonna see out there as you start to learn go and you're going to start to see some some tutorials and some code snippets that they don't always sort of provide they solve the problem yes but they don't always provide the best practices so this is one of the the ones where you actually should try if you know ahead of time especially if you're using a weight group that means that you have some knowledge of how many of these things you're about to launch right just basically call the ad method just once because you know ahead of time how many goals you're going to be waiting on in this case it's our range of ports right you do a quick little calculation but if for some reason you don't make sure to call it before you launch that go routine right if you ever call wg.ad like right before the defer wg dot done or inside the go routine somewhere else probably not correct that's a pretty good bet that there's something wrong something yeah something smells smelly with the coke there yes really is a good way to put that yeah exactly so we're doing we're doing we're not doing anything special here again we're going to iterate over our list of ports and for each one part we want to scan we're going to fire off a go routine and let the work happen inside of that girl routine and we have to ensure that when we actually the work is done within our routine that we decrement the counter that the weight group is is using right uh to help us wait on on the work to be done right and where is that work happen where is the waiting happening well that happens online 54 right you have to remember that you know if you can issue you can you can make use of the weight group you can add the number of counters but if you don't wait what's the point there's no point okay so that that's that's that's the kudu right there this is like 54 is really what you want this is where you're telling the main go routine that you know that you've just launched a whole bunch of different uh guru routines from from a lot line 43 on on to 52 right that you need to actually wait for these things and inside of those girl routines this is what the wg.done is doing for each one of those routine every time that work is done subtracts from the counter and then at some point weight stops blocking right and basically hence control back over to our main routine so blocking what is that actually what does that mean when you say you know weight is blocking and then it stops blocking when the when the weight group goes back down to zero right so our scheduler right is always scheduling okay who gets to go next who gets to actually do the next piece of work right and so our scheduler is the one controlling right who who is next in line who gets to do some work when do they get paused when is control handed over to to to another go routines and whatnot right when we talk about blocking here and especially in our main goal routine basically telling the schedule hey we want right to to to get some work finished right and but this this thread needs to be paused right okay needs to be stopped and then its tracks is to wait right here until we're ready to move forward so we are literally gonna not execute anything after the wait until the other girl routine those end goal routines are all done and they've called wg.done yeah n number of times that way the weight group knows okay now we can proceed the code scheduler hands control back over to the mango routine and off we go again all right all right so now let's let's now see right on the improvement right to this version of the program so by default we're going to have hey there we go that's our that's a list of the ten points before we just saw it done yeah right was it two years ago all right basically everything just you know we didn't know we didn't wait for the go reasons to be done now we we have we have uh um we actually waited on on this and this is going from 88 to 80 90. so yeah just a small range yeah small see i've got enough i've got enough cores on my machine this work was done pretty quickly right so this is the part where i want to actually see what happens again this is a very small range of ports which is just a just a handful right what if and now and now packages ship it right in and all of a sudden the customer right the the user of this program decides to throw you a curveball or rather they decide to actually use it an unexpected way all right otherwise known as testing production right exactly exactly so they decide okay hey you know what i'm gonna use this nifty you know port scanner and i'm gonna i'm gonna scan from say one to sixty five thousand okay like a very a very do i have the my number of zeros correctly there but that's better so yeah um i'm gonna have a very a very uh i'm gonna give your program a very bad day so if we try to do this now oh actually i need to specify the argument all right i need to say i think yeah i think you're right it's from port i might even have that in my hey there hey there we go oh yeah we're gonna use this one so we're gonna go from five thousand from five thousand to important to sixty five thousand let's actually see how our program behaves as written so here we go so oh here we go that did not go well for us that's called taking up too many resources that is called taking up too many resources right so this error is very different from what we're getting before so this is our host this is our operating system telling us ah you're using up too much right i haven't allocated that many file handles for you to be able to open right the operating system is protecting itself in this case and you know it's basically going to prevent your program from operating you know optimally right so we're handling this particular error um we're outputting you know what went wrong but you know obviously our program is not operating to spec and this isn't a great experience either right we would we would like to still scan all those ports we would like to still do it concurrently and in parallel right but obviously the operating system is not able to give us the resources we need to do it in this unbound unbounded exactly hence the title of this talk now this is this is exactly this is what unbounded concurrency looks like right but you don't have you you're not constraining your program to to the context in which it is going to run in this case yes we could launch 65 000 routines right to do the work but that's not the point right the point is for us to actually be good citizens within the environment in which we are actually operating and running right so we need to constrain this program in some way we need we need to somehow control the number of routines that we we have doing the work while enabling right for the concurrent work and for the speedy work to get done without running a foul of system resources that we've been allocated yeah so i'm thinking we need to do we need we need to introduce some some new concepts right for for this particular program so i'm thinking this is where we introduce the concept of a worker pool so what is a workable errand well before we had this unbounded thing where we were just doing go and we launched the go routine and the number of go routines we launched was dependent on what the user gave us right so we did whatever it was 60 almost 65 yeah 60 000 so yeah it was a very wide range yeah right so we obviously cannot do that as we saw so for with worker pools we're going to launch a static number of go routines and let's say five and we're going to feed ports to those go routines so we're never gonna have more than five but those five are going to work in parallel on all the ports that we need to do exactly exactly so we we're we're we're creating a set number right of of go routines that can go out and do the work and as they get done they come back and pick up some work right so for in our case here we've added a new flag right so we've actually modified our our our flags a little bit basically just to make sure i don't i don't have to type you know front port and too much all over the place i've just you know i've made some improvements to to reading of our ports here um and uh but the important thing here is that on line 24 you see i've introduced a new a new flag that's that's the workers flag the number of workers flag and i'm defaulting that to the number of cpu logical cpu cars my machine right so i think i think this is an eight core machine so that means that by default i'm gonna have uh eight workers uh in my pool which i think is a good place to start because there's no right or wrong number for this this is all going to be what is the sweet spot because you might be running in production you might running a on a 64 core you know multi like super beefed up machine or set of machines right that is going to be a very very different performance profile than the developer laptop yeah and you know folks will have to do trade-offs they'll have to measure how fast they want to run versus how much contention for go routines per chord would they want to run or per os thread and so forth right that's a whole nother talk that's a whole nother talk yeah that's that's a that's something that will basically leave to you to figure out for your particular situation or what particular workload what the a lot and the the correct number of workers is for you but the idea here is that basically what we're going to be doing and and i've also added some additional things here is that the the for in this particular main function i'm also capturing a signal for for when um i basically stop and send an interrupt to the program i can still capture the results so i can actually print them out so that's why that's happening here um and i've hidden the um sports scanning right behind a nice little helper function that's just gonna give me back a slice of integers right based on the the arguments that i pass in when i invoke the program so but the the real fun stuff starts on line 46. this is where i'm actually using make to actually create myself a buffered channel right so normally speaking right an unbuffered channel right facilitates some communication between the girl routines where only the the the where the information is exchanged when there is somebody on the other side waiting to receive a message how does the buffer channel behave differently so buffer channel has kind of a cue built into it where i can start pushing stuff onto the channel and it'll fill up the queue but once the queue gets full then i block to send right right but on the other side of course there could be someone taking off of the queue receiving from that channel so if someone is receiving from the channel at a similar pace as what i'm sending so let me say that better if someone's receiving at the same pace as i send then i'm not going to never block this right exactly so it's just a little bit of a queue or a buffer as the name goes in between the sender and the receiver and it can be used in this case of course to do our work to do our workers exactly exactly so what we're going to do is basically we're going to we're going to launch right on n number of grow routines and that's going to be sort of driven by the number of workers that we have in our pool and we're going to be using the porch chan right as the mechanism to feed ports into and for workers to pick uh pores to scan out of right so we need to feed some ports in there and others are going to be picking and picking up from there so the way we do that is by basically we've we've abstracted our way we've tucked away all of that behavior right into this worker function which in the basically we're calling we're calling a new worker we're basically we're we're preparing a new worker to start accepting work right based on the capacity right based on the number of workers that we have this is what we have what we're doing online 50 here right so let's actually go take a look right and then we'll come back to line 53. let's go take a look on for the definition of worker right so what we look like if we take a look at the signature of this function here we're going to take in the host yes that's that's it we we know we know we need the host but the the directionality right of our porch chain and the directionality of our resource chain yeah that looks that looks interesting right yeah what's up with these little arrows we can change explain those things to us aaron well you said it directionality right so we can enlist the compiler to ensure that for the first one for porch chat that we only read from it we wanted to read only yeah and that that directionality that arrow is on the left side of the chain keyword so if inside of worker if we try to write to porch chan our code won't compile which is really nice you get the compiler to write a whole class of tests for us right and then similarly for results chan well the arrow is going in to the chan instead of out so that's the one where we can only write to we can only sin on it yeah and i love making the compiler do work for me so this is a huge boon for me as it should be for all of us exactly exactly and what's happening here so basically uh um we're arranging over a porch chan right and unlike other four um ranges the when you read from a channel only one value comes back right there's no index and value it's just the value that you're picking up from from the channel and now we're just ranging on the port and as long as on a channel and as long as this value is coming in right we're gonna we're gonna be doing some work right basically we're doing the same thing we're doing before calling that dial inspecting a response and if we notice that okay there is that we consider the port closed we're gonna send a zero right onto the resource chain indicating that basically that that whatever port that was we don't care about it anymore because you know the port is closed so we're gonna send a zero on that because we need to send back right an integer we need to send an integer into that port to indicate that we're done with that work right so we send a zero in there and we continue on to the next one right so we're going to keep iterating until that that port is is until we receive a signal right that this port is closed and we're going to see where that signal was actually sent back where this worker is actually is actually uh um is actually invoked right but what we're going to do here is basically if we get to the point on line 112 where we manage to get past the point where we do the error check and and we we notice a closure if we get packed to the to the 112 we can get past in down to 112 we're going to consider that part to be open so we send that p whatever p happens to be port 5432 for example that's going to go into the results chain right and and this is where basically we're feeding feeding back right the the the result of our operation or our scanning operation we're sending back the results into that channel so that wherever the worker is being invoked we can actually do something and read some values from that so we've got a consumer here worker is the consumer there's going to be n number of cpus workers running they're all going to consume and you know if you send a port to the porch chan one of the n workers is going to pick it up do its thing send back the result yep so let's look at the producer now yeah so what's going to happen so in order for us right to steadily feed in ports into this particular um porch chain that the worker is actually waiting on we actually need to range over right our parts to scan uh list right remember porsche scan is is basically the what came back when we actually you know identified okay you give us this from range this two range like i get back a list of integer uh ports to actually scan so i need to actually i'm going to arrange over this feeding in the port numbers right but why am i doing this in a separate routine here well that's a good question right so we're feeding stuff into porch chan but in order for the port to get to get processed we need to get something back on results check right so if we were to in the main go routine if we were to feed into porch chat in the mango routine we would eventually block right so we need to do this in a background go routine n plus two now with the background routine because every time we feed something into port's channel we're gonna block and if we go further down under line 59 60 61 62 63 that's where we're going to receive from the is it called yeah results chan right once we receive from results chan then the passing part on line 55 that's gonna unblock and we'll be continued exactly exactly remember we have a buffer right that we need to keep you know loose right so as as things are sending in there the moment the buffer is full right there can be no more sense right so but we need to be popping things off right of that result shane in order for other things to actually be able to go in right so the the we basically it's like a conveyor belt we can't have you know a bottleneck anywhere so we need to be able to read it from the other end in order for new things to go in so that's exactly that's exactly why we have a sort of uh this said you know we have the feeding of the pores happening a separate go routine and so that we can actually have the the reading off of the result chain happening here as well and again the important thing is we start up we schedule in the background the person that puts stuff on to the conveyor belt because the person who's taking stuff off is in the main go routine exactly exactly and then at the end here we're just you know cleaning up after ourselves in this case and then we actually print the results so let's actually see this this version of our program in action this is the uh number four so let's actually do go run and this is going to be four make that go and let's actually give this one the same number of ports that sort of tripped us up right uh before right remember we just ran a follow-up the operating system said you had too many file handlers open we just have too many gurus and trying to do too many things so if we write the same range again here so this time right we're back to having the same kind of errors we're getting before that hey the connections you know is being refused which we expect that that's our indication that uh basically the the port is closed but you notice that we haven't run a file of the operating system yet we haven't had any errors that says hey you've got too many file handles open you know you you've exhausted your your resources and things like that so we have we have constrained our concurrency you know using the workable pattern we have a set number of workers doing work they're fitting in we're fitting in ports to scan we're picking things up on the other end and actually our program now behaves a lot more um stably and it also looks like we're actually doing this stuff concurrently yeah not just you know like like the first one where we're just going sequential one two three four yeah exactly and you can look at the port numbers here you know it's completely out of sequence right which is exactly what we want all right so let me stop this and oh actually the results and the signal handling we got we got our results back so we got 45 54 32 and 64 63 um open i think 6463 is uh because of uh discord i think i have this yeah this card opens this weird you know channel gallery this story's got some some stuff going on i don't know if it's shady or not but probably yeah probably all right cool so now that we've got ourselves um the uh the the workout pattern i wanted to introduce a pattern that we don't often see right well so worker pool is like a legit pattern is it legit banner you can use this in production it's solid absolutely absolutely but there's another one that i i find to be quite elegant right that i don't often see use in the go community and that's a semaphore can can you try and explain succinctly what except what the semaphore is i could try i haven't i haven't used semaphores in a long time so i kind of see semaphore as similar to a worker pool except it works in batches right it's like you you batch up let's say num cpu units of work or ports and once they're all batched up and ready to go then you kind of unlock and you start doing them you start executing all eight in your case eight you execute all eight of them once they're all done you start batching up eight more execute them all ones so it's kind of like the opposite you don't stream work and the workers don't just pick it up as soon as they can you kind of put everything on the conveyor belt the conveyor belt you stop the conveyor belt you get everyone to pull stuff off the conveyor belt do their work and then you put all the new stuff the new eight ports on the conveyor belt and just rinse and repeat until it's all done mm-hmm right right another another um analogy that i found to be quite useful um for for beginners or at least for people who sort of uh those who go to restaurants or maybe right so and a lot of us have started doing that again you know since you know things start to get a little easier these days oh we finally get to finally finally get to see other people in the real life um so imagine imagine you you and your your three other friends right go to a restaurant and and you you need a table that seats in four people right um and currently the the restaurant is at capacity every table maybe it's a friday night everybody's out having fun and and you're basically out in the the lobby the waiting area because there is there is no table there's a table with two people right um you know but two open seats right but you need a table for four friends you can't all four of you must be able to go in there and sit down at that table right you can't have sparse you know like you know one one spot here you know two spots there you need a table for four people right so to me that's what the semaphore is i need all four of uh of us right to go and sit at a table right basically that's our allocation we need all four of us to be able to sit in to actually do the work of eating right and you know talking to each other and having fun with each other right um or or we simply wait until that becomes available right so that's the whole point of the sample that's how that's how that sort of uh that's what the facilitation enables but semaphore is not yet part of the standard library it's part of the experimental uh a set of packages under golang x right but that's still you know don't don't let that fool you that that very much you know that i use this in production it's it's very much part of the er ecosystem and the part of the standard library or at times these things sort of uh graduate into the standard library proper but you know you have no fear of actually using this library so let's actually see how we get to actually um use this so i'm just scrolling past a lot of this because we've already seen much of this already and let's focus on the uh sort of how we invoke december 4 here so this semaphore i call on the stem four package and i called new weighted so this particular function will give me so much to basically the the maximum weight right of the sample four so how many things so what's the capacity right of that restaurant how many how many things can how many spots do i actually have and basically now from from then on we're able to say well what is the acquisition weight meaning remember when i have my me and my three other friends we needed to be able to sort of you know we need four spots to basically go into this whole um restaurant and actually sit down and eat that's the acquisition weight right so there's the maximum weight because should be the capacity of the of the restaurant right and there's the acquisition rate i need four spots at a time to be able to go in and sit down and actually enjoy my meal right that's what these two are and these numbers are completely arbitrary but they work in relation to each other right so from there we basically say hey i'm gonna uh uh iterate through the list of porsche scan right same as we've done before but this time i'm going to be using the semaphore to acquire right on the uh using that particular acquisition weight right and this call is a blocking call so that means this is us sitting in the lobby right waiting waiting to be able to go in to to to sit down and eat right because there's not enough spots for us yet so we're waiting right here so as at the moment we've enough uh folks have acquired the acquisition weight right the moment that we can no longer extract right that acquisition weight then the query becomes a blocking call so we saw blocking calls with the weight group if you've used mutexes and go or other languages they're blocking calls in there in those languages or with go mutexes as well it can be really useful here yeah absolutely absolutely and when when as folks get done in the restaurant and they start to leave right they actually release right which is what we're doing here we have a basic a separate girl routine and that go routine the actual scanning operation gets done where we scan the port and basically we're deferring the release right of the weight so basically imagine there was another party of four that just got done eating now they get up they pay their bill hopefully and they work and they walk out right so that's that's the release they've just released right the acquisition weight now you with the same acquisition weight right can now go in and sit down at that table where previously it was occupied right so that's that whole mechanism so in the interest of the time i'm gonna skip running this one and i'm gonna jump to the next concept the next pattern i want to take a look at and that's the pipeline so when i say pipeline what comes to mind for you um kind of like an assembly line to me you got one person doing something they hand it off another person doing the next thing they hand it off but each individual person they all just do one thing at a time they know how to get something from the previous do their thing pass it down to the next right right right so the way i like to to sort of visualize this is basically you have all these steps right and and this sort of this conveyor belt as as you illustrate and each one of these steps obviously is is is handled by a go routine right so we have you know the the first one it gets done with the work you know you know through a channel right communicates to the next step and in the process and then again through another channel communicates with the next step in the process all right so let's actually see this pattern right at play in in in our code so let's flip back oh we need to there we go all right so let's actually see how we actually put one of these together so we've already seen much of this before we've seen reports to scan up we actually need to open up the right the right program all right here we are back in action all right so you know we we've we've uh um we've got our ports to scan and we're gonna this time we're gonna be writing our are the results to a file and by the way there's going to be a repository with all of this code sample so don't you know you're not missing out on anything you're going to have to you're going to get to see all this and play around with it at your leisure right and basically what we're doing here is this is where we we have a pipeline invocation right so the the way with this invocations happen it's kind of odd looking right so you're starting from the innermost function we have a generator right and typically in when you have a pipeline you need to actually generate there's a there's a starter there's a starting point right that generates basically the the work that needs to be done in the pipeline and that's and that's where this is notice the the return type that's a that's a that's a directional channel right for scan up what is let's take a quick look at scan up scan obviously says hey it's an encapsulation that says hey given a port and and we're going to find out whether it was open or not if there was an error on us we're going to capture that we're going to capture the duration right pretty simple we're just capturing all the details about the scan operation that's what that's about right so that's going to generate basically a channel for us of skin operations that we can read from right to do work okay but the results of that right is passed into our scan now our scan is the next step in the process right next step in the pipeline right so we're feeding in that that channel into the scan the scan then is going to read from that channel as well and in whatever work needs to do right it's going to be doing but its result is also right a channel of scan operations which gets fed into filter right so that that conveyor belt mechanism right and then the last step in the process is the store which does the exact same thing receives that that channel so you can see how the channel becomes the conveyor belt that same channel is what basically every step in the process is using to pass information to each other so i noticed something here we've got you know the standard people who do the work we've got someone to to not do work but just pass stuff onto the beginning of the conveyor belt we've got someone to take it off and put it into a file but they're all waiting for the previous person we don't have any do we have any more concurrency in this that's the thing by doing this though we've actually we've reintroduced the sequential processing right of step by step by step by step so the way i'm glad you brought this up because the way we can avoid this is actually by using a different pattern called a fan out and fan end so again in the interest of time i'm going to let you play with the with with all these code samples on your own but i'm going to introduce the fin out and feminine pattern instead and see how that looks a little different right from from the previous uh version right so on the let's actually let's put this on the left side let's put the fan out on the right side okay okay so this was our original right implementation this is the pipeline right and on the right side this is where we're actually fanning out right we have we still have a generation step that's going to give us back our our channel of operations that we're going to read from but we're going to fan out by basically the scan call is basically going to receive basically that channel of scan operations that we can read from but we're going to invoke it multiple times in this case we're doing so manually but you can actually you know you can program around this right you can have a loop that's basically driven by maybe let's say the number of workers that you have working on those different steps in the pipeline but we basically we split that work into three different uh um channels right to to actually to actually perform the work thereby bringing back the concurrency that we lost by making things sort of linear yeah right with that with the first implementation of the pipeline we got this model now where we started with linear but it looks like it's super easy now to make it concurrent instead of calling scan once now we're just calling it three times or end times or whatever now we've just easily pretty easily we have three go routines exactly exactly so i'm going to close this one right but i do i do want to do a quick walk through what the uh with our scan what some of our pipeline uh steps look like so if i go to the scan all right so oh i didn't mention the the generator and the scan and the filter so these are these are you'll you'll notice a pattern here which each step of our uh of our pipeline so the generator this is the starting point this is when i said i received the income the ports that we need to scan and returns that channel of skin operations that we're going to be reading from so basically we create this this uh sort of buffer channel right and then we feed in right the ports right to to the the that outbound channel that we're going to feed into other steps that are going to read from that right and we return these things and the reason why we are closing here right so we always talk about basically the channel right closing the channel being a signaling mechanism why do you think that's important here that we actually we need to return this this uh this uh this channel and we need to close it here but in other cases we we're doing this sort of um because this is the first step in the process the generator right the the the point i'm seeking to make is more more readily illustrated with this perhaps there's the scan operation where we we need we need to send back an outbound channel as quickly as possible because we are in this pipeline right but we're actually uh the the we need to actually close out this this channel to signal right to the further steps down the road that there's no more data coming yeah right so this is a great way to broadcast something this is the exactly when you close a channel you're signaling to anybody who's reading on that channel that there is no more data coming right that's why we actually perform this work here inside of that it would defer the close of this uh of this outbound channel here after having fed in after having done the work whatever work we need to do right sending in the scan into the outbound channel right this happens concurrently and we return back to outbound channels so that any other step in the process down the road can actually pick up and keep reading and know when there's no more work to be done and this is kind of got a pattern here create a channel yep go use our go keyword to schedule some work yep immediately return the channel after it's scheduled and then and then we'll kind of move on to the next step and move on to the next step exactly exactly so let me actually quickly show this version of the program which is our fanout nope and you find the appropriate this is hard to read i think it's been a long it's been a long conference for us [Music] all right i think this is uh nine yep it is nine and uh we have that and we have uh the ports here and then we run pretty pretty right range reports and then in this case here we found the the the one port that is open actually the two persons are open 54 that are doing the suspicious port from front discord discord and we can see how this sort of again we use using that that the fan and fan pattern we're able to actually split the work across multiple go routines and actually merge all the steps back in so that we can get ourselves a result and our program successfully ends and this is cool because we were really easily able to put filter on the end of our pipeline so we don't have to park so we don't have anything exactly all these things yeah our filter function which is exactly what we have here we basically said hey i want to filter out only the open channels right which is another step in our process right for uh um in our pipeline right basically you want to give me back the things that i care about right so one last thing we're going to look at before before we before we wrap up here is uh the are we looking at number is it number 10 we're looking at uh context i think we're looking at number yeah i think we're looking at number 10 because one of the things that that that is useful right and and being able to sort of launch all this all this work and being able to sort of uh know when when things are uh when things are done being able to signal to to to go routines that hey you're done you can you can go away now because girls are not garbage collected right they're gonna hang around until you return from them when you say you know hey i'm not using them anymore right so we're basically augmenting the the uh the um our pipeline by having a mechanism of signaling right to to the every step of the pipeline when when we no longer need the pipeline to to be running right and this goes back to our you know always know when your chain is going to shut down always know when the go team is going to shut down know when to signal for it to to to go away because again these things sort of hang around so this is where we're actually using not a contact but we're using an actual uh a done channel which is a very common you know idiom within the go community basically having that done channel passed around just like you were the contacts okay and this guy has given me since we've been talking about this has given me an idea how we should actually have another example to this it actually uses the context right in addition to to the example here that uses the done channel but this done channel here you can think of it as the the signaling mechanism that we hand over to every step of the pipeline so that they can be made aware of when they should stop doing the work and so like you said this is a really powerful and popular mechanism to broadcast something is done if you use context for doing for signaling things or timed out or manually finishing something this is used under the scene it's right behind the scenes behind the scenes and absolutely absolutely so you know you might see some some um some change some sophistication added here before we're sort of manually doing our fan out here this is a more sophisticated way of actually based on the number of workers in creating the number of basically doing that same thing out work we're doing before and then basically we're doing some filtering here on pass in this time every step of our pipeline actually we receive as part of the invocation and as part of the arguments for for that function invocation is going to receive the done channel as a signaling mechanism meaning that if i were to go to say the uh let's go to the filter open as one of the steps in our in our as an exemplar right you see that it received the done channel as you know as a as a read-only channel of empty structs right the empty struct is the common commonly used for for for for signaling uh on channels and again we have the same thing we're doing before we have the inbound channel from which we are actually going to read uh um things to things things to do uh you see i'm telling you it's it's it's confidence and then basically we're returning into the the uh channel of skin operations again again for whatever comes that comes at the uh next in in the pipeline and we're following the same exact uh procedure here but the only thing we've introduced here that is a worthy of note is the fact that we're actually now using a select block right so basically not not only sort of doing a work we're doing before but listening in on this done channel so that if we receive a signal right that says hey you should you should stop now you should there's nothing more that's going to come through on this pipeline because sometimes you you don't need the pipeline to keep operating anymore you know you maybe maybe you received an error and and you really want to sort of bail out of there there's nothing to be done anymore you want to shut things down so receiving a message off of that done channel is going to be our cue to return from this go routine right so that now the school routine goes away basically you know you can get garbage collected now we can actually you know sort of shut down things cleanly and this is a really popular pattern and you know production concurrent code exactly you are going to do some operation inside of a routine send receive on a channel you probably want to put it in select and you probably want to have a done channel as well exactly one of the cases in the select exactly so the final final run of our example here what i'm going to do is i'm going to use this filter error function right you know and i'm going to uncomment some things above so you can actually see this in action i'm going to use this filter function here as that is part of our pipeline to basically say hey if the the the i want to extract out of that pipeline as things are going through that conveyor belt i want to plug out right the ones the scan operations that actually aired out okay obviously i'm gonna find the ones that basically when you do the scan and uh um for whatever reason right there was an error but the errors i'm gonna be looking for is one where there are too many open files remember that when we're in a foul of of of the the os and basically saying hey you're using up too many resources when we had too many things happening running at the same time i want to plug those out right and the way i'm going to do that is by actually invoking this program you know basically asking for way too many groceries to be running at at any given time lots of workers right so and but that filter function is actually going to give me back right the one where i've actually run a foul of of of my constraints here and i'm going to return that so let me go back onto the top here and then we're gonna we're gonna end up some with the with the with the summary of uh the best practices for this so i'm gonna yep you got something going on oh no okay i thought i thought i heard you're about to say something maybe like a pun you know when you want to drop [Music] all right cool i do think it's cool because you know we decided we you know imagine we got different requirements from our pm or from the cto or something we've got our pipeline and we're essentially going to comment out filter open and uncomment filter error and now we've got different behavior right so we're basically you know the person at the end of the assembly line we're going to fire one of them and we're going to bring in someone new who's going to do slightly different work and now we've got a different program so this is the power to me of of pipelines is it's very composable but it's very orthogonal i suppose right indeed indeed so the first time i invoke this program i just want to make sure it works nothing is broken right so i'm going to invoke this program as is right this is the one with our workers so it actually did something let me make sure i've got my my uh um okay i've got the appropriate number of ports okay okay good good because remember we're trying actually the reason why we didn't get an output is because i'm filtering out right i'm only saying give me back the errors that happen all right so in this case i actually i scanned right basically this default number of ports and there was no error so my program actually ran successfully even though there was no output right so let's actually change that a little bit let's actually do this but this time around i'm going to provide a like ridiculously wide range of reports file descriptors and then i'm going to ask for a ridiculous number of workers right let's actually see right if this goes according to how i think it's going to go right the operating system is going to tell hey at some point this process has got to way too many file handles open right and our filter error function in our pipeline should be able to pluck that out of that stream of errors you know happening as the scanning is happening our filter errors should be able to pick that up let's see if that's friday but we're asking our trusty machine and trusty programming language all right ready ready ready here it goes here it goes here it goes up it's running it's running oh here we go all right two files there we go it's amazing we actually plugged out right the too many open files and what else did we do when we actually uh caught this right remember when we talk about the whole signaling mechanism so what we did here we were ranging over right plucking values out of the filter error step in our pipeline right and the moment we were able to read a value right we decided you know what i don't want this program running anymore we got an error we didn't like it we didn't like it we we're banning out of here we're done we send a signaling mechanism to any other steps that might be in the pipeline you know that map perhaps could be wrapping filter whatever it is you know we're not doing that here but you can actually see hey if i had other things wrapping filter whatever it is i could send this done signal right to everybody say hey we're out of here we're bailing right and then we move on and then basically our program can now basically shut down cleanly right so these are a lot of of of cool tricks yeah and tips right let's see if we can actually summarize this into a handful of of of of uh key takeaways let's do it all right so the first thing i would say is start this one start simple i love it you saw us we just used the net.dial we found the one thing in the center library that we needed to use for this we had a you know a port that we just basically we just hard coded into the program we said hey what's the sympathetic work and we basically we could validate the idea and then busy build from there we evolved from a v1 that you know we're putting in production on day one we evolved it maybe we fast forwarded through three years of production experience here you know we evolved it we got more complex as we needed to as production bugs came out or as requirements grew indeed should we do it one two one two three yeah identify the one two three embarrassingly parallel exactly i'm way too much fun with this oh yeah yeah so yeah not every problem you encounter right is gonna be suitable right for concurrency right and when you do you need to identify the parts of your code that actually can run right in parallel that that can't as a good candidate for concurrency right out of that is when they don't have dependencies of each other exactly if there's no dependency from yep from one to the next that's usually a good spot right for for for uh contemplating concurrency what's the next one avoid run away go routines that goes back to like always make sure you know when you're scheduling them at least when you're scheduling them and when they're going to be done exactly exactly and and as you saw us basically saying throwing a bunch of routines you know at the task and the operating system basically saying ah you're using up way too many file handles in that case that was a constraint that we're violating right with our program so you need to understand and what is the context in which my program is going to run what are the resources that i'm going to have available right what's the network throughput i'm working with what's the database constraints what's the latency whatever the case may be identify the constraints that and within which your program is actually going to to execute so that when you have grow routines doing the work you actually know what what boundaries that are going to bump up against and another thing to remember if you're coming from c c plus or java or other other operating systems that give you access to os threads go routines are not os threads right so the operating system is not going to automatically schedule your go routines the operating system is not going to automatically limit your go routines the go runtime lets you have lots and lots of go routines indeed indeed so leverage the standard library right we didn't need to go far yeah that was that was pretty much it the only thing that was that was foreign if you want to call it that was the december 4 that came from the golden x yeah but pretty much everything is is standard library yeah use the more advanced pattern as needed and that's key the last two words there as is needed we evolved from the simple thing and it kind of goes back to the first bullet point yep we evolved from the simplest solution as we needed to as we found out that well a user might come and test in production and give us 10 000 ports to scan well that's not going to work on our server or on our laptop so we need to add more patterns to to manage that requirement exactly exactly so the repository with all the source code and even the things we didn't have time to go over is all available at this url and this is my twitter handle and this is uh aaron's you know feel free to to get in touch with us especially those that are new to the community we welcome you this is this is a safe place and definitely reach out to us um don't hesitate don't don't be don't be shy right you know we're hopefully more than more than happy if you couldn't tell we love this stuff your currency is our jam distributed systems is our jam so please reach out if you want to talk about this stuff if you have questions if you you know want to work through a problem anything absolutely absolutely and i think that's all we got so i think i think you wait did we forget somebody crazy i think we forgot something oh we left aaron we forgot about it no seriously i couldn't be happier to spend the week here with these two likewise yeah it's been a fantastic conference i wish we could have done it in person hopefully next year we can i know you've got to see a lot of our faces but you know first off i want to thank the uh production crew that's been working on this conference year-round you know heather jose julie from convention designs that have been working on this year-round re-planned the whole thing with us in two months truly here and we went to virtual yeah yeah two years in a row re-planning yeah the comments i think we said uh at the outset of this conference they may have planned five conferences for two years or something long yeah exactly yeah so thank you to them thank you you see the three of us standing here and we were at the desk but there's a lot of people here that you don't see we've got ross here behind the camera we've got matt and camera running the audio and video boards out there i don't think they have a camera on themselves no no we've got matt over there furiously editing videos so that hopefully we can get all of these to you on our youtube channel within the next week or so and also jose and julia are sitting out here outside this window too i mean they're the ones that bring this content to you they're the ones who make us look really good they're the ones who make our job easy yep which is you know despite the shirt despite the shirt yeah i mean despite the shirt how do you invite this shirt to come up here with you we don't have a wardrobe in fact you are the wardrobe person by you wearing that we don't even have to think about ourselves i make you look good anything on anything really a bathrobe anything please don't come in the bathroom that's been done [Music] yeah and you know thank you to our sponsors we couldn't do this without you uh you know this first year like being able to bring this to you completely for free yeah um so we couldn't do this without our sponsors if you haven't gone and shown them love please do go to the sponsors page on the gophercon website go check out their sites see what they've got going on i think some of the giveaways might be going on i don't know whether they've they've done the drawings yet if you're looking for a job please please go talk to them that's the overwhelming majority of the reason why our sponsors come here yeah yeah check them out in discord as well i think the the crowdstrike uh contest might be over now but you know hopefully yeah hopefully hopefully there's not any more horrible generics code out there but i'll be studying it i'll be i'll be uh watching what not to do authors out there what not to do yeah thank you to all of our speakers those that were able to fly into the studio and have some fun with us for the last couple of days uh thank you to the speakers that weren't able to make it and were able to get us recordings and things like that like on very short notice like the production quality of these things thank you to the editors that have been like furiously working on getting these things ready yeah lightning talk speakers thank you so much for contributing there should we do one more thank you to mark i think we maybe all right he gets one more one that's it yeah all right should we do it one two three and then uh thank you mark one two three thank you i hope he sees that and cringes yeah i know he puts a ton of work into it he's he's really really passionate about doing the lightning talks both in person and continuing to do it virtually you know if you run into him on discord or on slack you know throw him a thank you talk to him about lightning talks next year yeah yeah well i mean everyone here in the gophercon family is always looking for new lightning talks for new content again it's a great way to get started in speaking if you're new it's a great way even if you've spoken a lot before it's a great way to test out a new idea as johnny said earlier it's a great way to challenge yourself to cut down your content or your idea to seven minutes good luck with that that is tough so then it's another one there for you yeah yeah and the the go time team yeah indeed indeed yeah of which they are missing one i know but hey couldn't be in two places at once i haven't figured out that trick yet you have to choose who you love more and you know we came out hey hey i didn't say that i didn't say that matt you know matt's going to be like you chose that yeah you chose you chose blue and gray man oh man but absolutely yeah the that team was basically its own sort of operation going on as well and and behind the scenes and making sure that this conference was and went off uh uh as well as it did yeah so big thank you to them as well indeed and thank you to everybody everybody thank all of you for coming and hanging out i know that the last two years have been tough on everybody i know there's probably a lot of virtual conference fatigue at this point but you know we appreciate you watching we appreciate you hanging out and interacting with us and discord and heckling us a little bit in the peanut gallery i i don't know why i amazingly like being heckled it's weird maybe it's it tells it tells us that you love us yeah he who heckles loves he or she anyone he or she who hackles or it validates our imposter syndrome like no you you really are you really are impossible you really are an imposter don't worry you don't have to think about it anymore yeah everybody knows now everybody knows now you know we all know we're good we're good you know we accept you as you are so yeah that's a wrap thank you so much everybody thanks again to the learn videos we're going to have records oh yes recordings will be up probably in about a week okay wonderful yeah but thanks again to the learn tv team for putting us up in the studio thank you to microsoft thank you to capital one thank you to salesforce for the the captioning uh you know we couldn't we certainly couldn't do this on short notice without the help of those folks and the rest of the sponsors too yeah all right thanks everybody we'll see you next year hopefully in person [Music] [Music] [Music] [Applause] [Music] [Music] so [Applause] [Music] [Music] you

Channel: Gopher Academy

Views: 120

Keywords:

Id: VHn7aaiO9sM

Length: 461min 46sec (27706 seconds)

Published: Sat Dec 18 2021