ElixirConf 2021 - Yiming Chen - Promox: Roles, Protocols, and Mocks

[Music] [Applause] [Music] [Applause] [Music] hello everyone i'm mimi i'm an elixir developer based in china today i'm gonna share a library with you promox as you can tell from its name promox is a library for defining works in our tests if you are already familiar with mox which is another popular library in elixir to defining mocks then you would find promox public api to be very similar in the so the usage of promox is very simple and its implementation is also very simple as well so today's talk is not about how to use promox or how to implement a library like promotes but today's talk is about why would you choose promox at the first place why would you use promox basically so we would touch on these three concepts here in the next 30 minutes rows protocols and the marks so to answer our question of why would you use promox we need to answer this question first when would you define your own protocols because if you are not using protocol at all then it doesn't make it make sense to mock protocol functions right so i guess for most of us the answer to this question is that when we have a bunch of data structures and i have a bunch of structs already then we would define our own protocols to have some polymorphic behavior for these data structures but in this kind of scenario then it might not make sense to use promox or mocking protocol functions but what if we define our protocols first then define our implementations so that's basically today's talk so in today's talk i would introduce introduce a kind of design that's called that's called role-based design and we would see how to design based on those uh to make our code more flexible and how to gain elixir use protocol to define these rows and how to use drugs to implement these rows later and finally we will see how to use promox to test these interactions between rows before we dive into today's talk i want to thank these people for their inspirations and helps i want to thank this great book growing object oriented software guided by tests and the workloads not objects this great paper this talk and the promox library are both inspired by these two great resources and it's they are highly recommended and i also want to thank queen wooten for her great talk flexible software composition using protocols and you would find the the idea very similar and i also highly recommend menu to watch this talk finally i want to thank my company 2b because promox is a library extracted from a project i built to s2b and without the support from my team i wouldn't be able to try this new idea and contributed back to open source of course to be is now hiring in both the us and in china so if you want to solve large-scale problems with elixir feel free to [Music] reach out to me later and join us yeah now let's dive into today's talk so the first question is what is role-based design in the what is a role basically so to me a role is a responsibility plus a bunch of interfaces so the responsibility is a job to do is what mission we need this role to accomplish and the interfaces are the messages we can send to this role and uh what this what messages this role can send out to other laws basically the interfaces is how this role communicates with other roles in our code or in this world basically now maybe this idea is still a little bit too abstract let's see some examples first i i think this kind of of draw based design is very common in our daily lives let's consider pc here i guess you are all watching it watching this talk uh through a computer so in front of you is i guess it's a display and for each computer we have a keyboard motherboard and on the motherboard we have cpu memory and the disks so each of these hardware components take a load in our computer and for the display its job is to display all these pixels so we can see them and the interfaces it talk to the computer is maybe the display board or the the usbc port by splitting our computer into components like this and each component take a unique role in this system we make our computer very maintainable because if we want to have a better display we can just plug in a 4k display here if we want to have a better cpu or more memory we can just throw more hardware to the motherboard if our display is having if our disk is having some trebles we can replace the disk with a new one in the this is basically an example from our real life so i also think that the low base design should be very familiar to our elixir and airline developers because bin processes also have growth here um let me use open as an example here because uh obama is my favorite library to process background jobs in elixir and to accomplish this job to achieve this goal open has this supervision tree here and whenever we start an open supervision tree it would start a notifier process of middle life and for each queue it would start a queue supervisor and it also starts a bunch of processes for every plugins we use in the under each queue supervisor it would start a format a producer and a watchman so all these processes have its own role in this kind of beam virtual machine and they focus on their own job focus on their own responsibility here and all these processes at the same time they talk to each other via their interfaces by sending each other messages so they can collaborate together to achieve the same goal which is to process the background jobs so basically without these isolated processes and there are collaborations open wouldn't be able to do its job so well by uh i think by splitting our runtime system into rows like this and notify our midwife and so we make our runtime system very related because whenever a process dies we can just start a new process that takes the same role for example if the producer dies we can start a new process that takes the same producer role and because it's taking the same same role which means it's doing the same job of the producer and it also can receive the same messages the producer can receive in the send out messages the producer did so the system can work as before because it has a new component that plays the same at the same row in the now what if we can put this kind of row-based design into our code and i think if we we can do that we can make our code more maintainable and more resilient now let's see how to do that in elixir with protocol now let's face a real problem here let's imagine we need to move a bunch of large files from aws s3 to google storage and uh this job might seem seems simple at first glance but if we have millions of files and each file file can be gigabytes of large then things are going to be different because we need to make sure all these files are transferred correctly from s3 to google storage so to do it well we need to follow these three steps or even more so first we need to copy these files from s3 to google storage of course we need to do that and then after that we need to compare the file hashes between all the copies and the new copies to make sure the files are matching and finally we need to copy the database records let's check these files so we have because we have too many files we need a database to manage them so we have database records for the s3 files and when we move these files to google storage we need to copy these database records and update them so they can check files for the files on google storage so how to implement this logic in elixir so with low based design we can define two rows with protocol so one is called mover and another is called comparator and uh let's see how to implement how to define them in elixir so first mover is this kind of role that its job is to move the data from source to destination so the data might be some kind of files or it can put some kind of objects or it can also be some kind of database records in the the interface series it's very simple it's just one function called move and this move function can receive a mover struct and the source map and the destination map so each mover knows how to find the data it needs to move based on this source map maybe this source map contains the s3 bucket in the past info in the destination the nation map contains the google storage package info and google storage path info so if the data move succeeds this move function can return ok otherwise it would return an error tuple containing the region and then we can have a comparator row here and the comparator job is to compare the data between source and the destination so let's say we have different data on source and the destination the competitor would find the difference in the return to pole saying that the data are different and and the reason why this data are different and if they are the same the comparator would return the atom same and with these two rows we can define different implementations we can have a file mover that moves the file from s3 to google storage and we can have the db mover that copies the database records from all the records to new records and we can have a concat mover that takes two movers in sequence in the code stem in sequence and now we can have a compare after mover that takes a mover and a comparator and also calls the mover first in the the comparator to compare the data gets that gets moved so for the comparator role we can implement it with different hash algorithms for example we can have a md5 comparator here so with all these implementations we can solve our problems but let's see first the how to implement this compare after mover first so a compare after mover is a struct that takes two fields one is the underlying mover which would do the actual data moving job and another is the comparator it would use to compare the data afterwards and it provides a new function that just return a constructed struct now let's implement the mover protocol for this compare after mover struct so when we call a move function or compare after mover it would first dedicate this over the move function call to the underlying mover in the if the underlying mover does its job it would call the comparator to compare the data between source and the destination and only when the mover successfully moves the data and the comparator says that the data are the same this compare after mover return okay otherwise if the comparator says that the data are different it would return an error saying that the files are different now and the reason is the reason returned from the comparator so now you you see an example of how to implement this mover role for compare of the mover and we can do the same for our other implementations like file mover and the db mover so i'm not gonna show all these details here in the now we can control uh so let's suppose we have all these movers already and now we can compose them together to solve our problem so first we can build a new file mover and uh pass it to uh compare after mover with the md file comparator and then we pass this compare after mover to the concat mover with db mover so basically here is the body of our solution and you will see that it's basically matches what we planned before so first we use the file mover to copy all the file files we need to move and then we use the compare after mover to call the md5 comparator to compare the file hashes between all the file and the new file and finally we copy the database records with db mover so i think this solution is very elegant because it's basically translating our plan of our solution to elixir code here and we also have a natural layering of abstractions here so at the top level we have this concat mover and this com cat mover is composed from compare of the remover and the db mover and then the compare of the mover is composed of file mover and the md5 comparator so with this body of our solution we can be go fancy here like we can replace the db mover for example with another so we can connect to different database or something in the we can maybe replace the file mover with a cheaper or faster solution because moving these large files can take some time and take many network bandwidths and maybe we can replace the md5 comparator with a different hash algorithm implementation and the body can stay the same we just plug in different components or different implementations here so to move around our logic and to change our logic here but since all these components play the similar roles so they can work with each other well and can be composed in this body of our solutions now we've seen an example of how to use row-based design and how to implement load-based designing elixir now what if we want to test them especially the concave mover and the compare after mover now we should use some box so now let's see how to mark these rows with promox so we can test our compare after removal before that let's circle back to our real world example here so imagine we need to test a keyboard i think the open obvious solution is to connect this keyboard to a pc right but this solution might be a little too expensive because when when you are building many of the keyboards you might not have a pc or a pc is too complicated you may build a dedicated simulator to test the keyboard you want to you want to build so this simulator can receive the signals sent from this keyboard and the same in the testing that when we press a key for example a here and the key code a can be sent from this keyboard and our simulator can receive this key code and verifies that the logic is correct so this simulator is the work here in our real life in the marking our test place the similar rows as well so we can see how to test our compare after mover with pro mods so now we need to test the case that the compare after mover should return an error when the the underlying mover succeeded but the comparator failed so we built two marks here the first mark is for marking the mover row so the mover uh so we would expect the mover the move function to be called with this mover and then it would always return okay adam and then we would build another mark for the comparator row so we would expect that the comparator the compare function to be called on this compare return mode and it would always return a tuple saying that the files are different even though we don't have any files to compare and the the reason why they are different is for tests then we call the compare after mover with these two collaborators and then we can observe that the compare of the mover should return an error tuple and the the reason should be that the files are different and the reason is the same here it's for test so by building marks like this we can test our combinators like compare after mover very easily because we don't need to set up a complex scenario to make our comparator fail or to make our mover pass or to make our mover fail uh and so on and so we can just simply saying that hey in this test the mover is always going to succeed and the comparator is always going to fail so and then we can observe against the behavior for our compare of the mover so now you see how to use promox to test our compare after mover and but if you are an experienced developer you may think that you've already been very familiar with raw based design and the protocol is not the only way to implement the robust design in elixir you may use behavior or even anonymous function to implement this kind of design and composing scenes together so why chose pro procore and the prologues so for me i would chose a protocol for its explicitness because when you define a protocol with depth protocol we are defining a explicit slow in our system if we are thinking in that way and uh this might not be the case for anonymous function we might not have a way to define those explicitness explicitly and now when we use definitely implement to define implementations we are also explicitly defining these implementations and all these uh definitions are very searchable because we can just grab these keywords in our code base and we would see how many rows we have and how many implementations for a specific role in our code base in the for depth struct we can store dynamic config configurations in these structs so for example we can compose multiple implementations together in the dynamically in runtime so for example i'll compare after mover we receive a file mover and the md file comparator and this can all be changed dynamically during runtime and uh it's also very explicit in our code in the on the promox side we can define multiple mods because each mark is just a struct and holding all the stops and the expense so we can define maybe two marks or even more for a single row like mover we can have a mock mover one and a mover two or even more and what's better is we can pass these marks to other processes because these marks are just struck in the they are just data structures can be passed around in the beam virtual machine we don't need to allow another process to use our mark or to have private mode or global world or things like that finally when we call a protocol function like mover don't move we are specifying log dependencies explicitly we can just grab or search for mover don't move here in our code base and we can basically other times we can get all the places that depend on this function or depend on the mover role in in our code base so i would highly recommend you to give row-based design a try and when you do please consider using protocol to implement it and also use promox to test it and you can find promox on github and i'm on twitter at dsdshcym and you can find my blog it's iming.dev and feel free to reach out to me if you want to discuss more on these topics and that's basically my talk for today

Channel: ElixirConf

Views: 632

Keywords: elixir

Id: Df81LbdRd0A

Length: 29min 20sec (1760 seconds)

Published: Mon Oct 25 2021