Talk (Software - Day 2) - Rules Rule (Creating and Using a Rules Engine)

[Applause] good morning this is spike on sweden 2021 day two of our conference and we greet you all from the software stack uh today our talk number two is called rules rule creating and using a rules engine and i'm really happy to introduce you all to our guest leonard frieden from stockholm sweden thank you very much good morning good morning let me introduce you quickly leonard is an experienced consultant from agical he is currently working as a developer and team coach at funell in addition to being a polyglot developer partial to dynamic programming languages such as elixir ruby or python he is also an avid mob programmer excessive pe drinker myself as well perpetual journeyman moderately mad owner of three horses if i'm not wrong and tabletop roleplaying game master and designer so dear leonard now the stage is yours please begin thank you very much it all starts deceptively simple and easy our code is straightforward and to the point but then we add a conditional to it and shortly after another one time passes someone perhaps a colleague or your future self comes along finds the existing f else clause and adds a few more ellipse one even happens to be a nested ternary bit by bit our codebase devolves into a bewildering pyramid of doom more akin to the starting point of a convoluted code carter rather than an actively maintained system worthy of being in production but do not rule out hope i'm leonardian i'm a polyglot mob programmer and consultant working for educal here in stockholm for the past two years i have been on assignment with funnel as a team coach that role involves being a teacher a mentor a code artisan and a purveyor of face pawn inducing puns and other team building and team growing activities one day when i regaled my team with some story we came across one of these pyramids of doom it certainly looked like someone had ripped off the gilded rose carter and stuck it into one of our api integrations but no it was actual working code moreover it had grown over time each and every addition and change making sense in isolation a regression fixed here a more robust handling of flaky responses from the third party api there all sensible and useful on their own but the whole suffered the legibility suffered we the team suffered taking a step back it is so easy to end up in this kind of situation especially when the code in question had passed through many hands and with few if any tests to be seen thus our first order of business was to get the code under test and in doing so we created a safe vantage point to refactor it from but into what at this point i suggested that we could build our own rules engine before going any further i think it is prudent to clearly demarcate what i had in mind when i suggested building a rules engine if your first step for information is wikipedia you will find this article called business rules engine this is great because it contains the words rules engine but at the same time it is terrifying as it contains the word business i might add that many developers do not perceive themselves of being in the business of well business surely that's the job of another department nevertheless reading the very first paragraph of the wikipedia article is even more frightening a business rules engine is a software system that executes one or more business rules in a runtime production environment apart from the word business so far so good the rules might come from legal regulation company policy or other sources legal policy that does not sound very developery a business rule system enables these company policies and other operational decisions to be defined tested executed and maintained separately from application code hang on there separate from application code oh there oh dear right here well i'm sorry this is terribly embarrassing but my sincerest apologies but it seems as if i've submitted a talk to the wrong conference except this was not at all what i had in mind when i proposed that the team should build one so let's go to another article namely martin fowler's 2009 blog post on the matter a rules engine is all about providing an alternative computational model this definitely sounds a lot more developery don't you agree a good way of thinking of it is that the system runs through all the rules picks the ones for which the condition is true and then evaluates the corresponding actions the nice thing about it about this is that many problems naturally fit this model these are excellent points we will touch on them shortly but first how come this sounds nothing like the wikipedia article at one time there was the notion of building an entire system using a tool like this but now people wisely tend to use rule engines just for the sections of a system the production rule computational model is best suited for only a subset of computational problems so rules engines are better embedded into larger systems well there we have it this can be a useful tool for us developers after all judicially used and applied a rules engine can help us when tackling a great many problems there are situations where one might need a complete full-featured rules engine using shaming inference rules or fussy logic however in our case making heads and tails out of our api response handling code was the chief concern we did not need anything other than the simplest of rule sentence after all even deeply nested if else statement is nothing more than the practice of looking through a set of conditions finding the first one that matches and then executing its associated code or action if you will each individual condition and its associated action is what we refer to as a rule and it is the first building block of our rules engine just like its abstract concept a rule object has two properties a condition and an action the condition is a predicate function that takes a single parameter a state and returns either true or false the action is another function also with an arity of one it is called with the same state as its condition and returns whatever makes sense for that action it will only ever be called if the condition returns true next up is the rule zambian at its barest minimum it is capable of interpreting a set of rules applying them to some state some context our rules engine accepts any number of rules and can be told to apply them to an object when the engine runs just like its abstract concept it goes through the set of rules calls the condition of every rule in turn until one matches at that point it calls the action of the matching rule and returns its result that's actually it this is the bare minimum we need to make a simple yet useful rules engine we have the rule object encapsulating a condition and an action we have the rules engine object with its run method the third remaining step is the vessel in which the devil himself dwells in other words the details the details of our rules the actual conditions and actions and the state they are applied to in order for us to talk about the details we need some let us take a piece of synthetic code and refactor it using a rules engine for presentation purposes it's not possible to show you the actual code that the team refactored but this piece of code will suffice it is considerably simpler and much more straightforward than where we began this journey but it illustrates a number of the affordances the rules and abstraction provides this example function is handling the response to a call made using the requests library it evidently handles a number of special cases if we have received no data as in the case of a 204 we will not be able to call the json method of the response without it raising an error also since there is no data to parse in the first place we might as well return some same value in this case an empty list we can easily picture how a good response containing data might be a json encoded list of some sort representing the lack of such data with an empty list is in a general a much much better idea than returning none next we have an elephant elsif someone has seen that we want to treat both unauthorized and forbidden requests the same way and there seems to be a domain specific error to race in this case a no access error in third place we look for 404s not found when accessing a specific endpoint perhaps we're using some using the same response validation and handling function in conjunction with multiple different requests at any rate data from this endpoint might be completely lacking when which we also handle in a similar fashion to the 204 no data response above lastly we use the generic race for status method on the response object to handle any other http errors we have not handled ourselves followed by parsing the json content and returning it with this as a starting point let us see how it can be expressed in terms of a rules engine we will assume that we have managed to get the function fully under test as it's the very first step to take when refactoring we begin our refactoring yes we began a refactoring by flattening the nested if depending on how many levels of indentation you have in your pyramid of doom this will likely add a number of lines stay in course though and have faith in the refactoring process this will become much nicer in the end we also notice that the function contains an implicit else clause let us make it explicit so that we better can discern our rules our conditions and our actions now extract the conditions and the actions name them identify them and identify their parameters are they all the same or do they um do some of them require additional input in this case we have the good fortune that they all depend on the response objects you may wonder why we bother to return the value of the access error function to which we have extracted the erasing of a no access error there is no real need for it in this case but it serves as a reminder that the rules engine will return the result of the executed action if that action raises an error like here it will make absolutely no difference but we strive to mimic the behavior and form and the form of the rules engine as this is just a step of the process extracting the special case uh of requests against a certain endpoint returning 404 not found which we ignore that gives us our first taste of reusability it has namely the same action as the 204 no data case the previously implicit else case now made explicit gives us force though apart from the somewhat clunky name for a function that both handles other http errors and passes the json content if the response is fine we realize that it does not have a conditional to be to be extracted except of course else is nothing more than an e-lift true we could name this always true but i find it reads better as otherwise it's now finally time to introduce our rules as all of the tiny functions we extracted all are called with the same argument namely the response object we have already determined the state the context for our rules to operate on thus we only have to convert the conditions and actions into rules put them in a rules engine and run it on the response object and for comparison we can see the the our reflected if on the left and our rules engine on the right now notice how the definition of the rules engine is separate from the act of running it this means that we can define it somewhere else for example in becoming a constructor of a client and using it over and over again with different inputs with different contexts or once again since we only rely on a single object here the response object why not replace the handle function altogether by just pointing directly to the rules engines run method still this probably feels a bit trivial to some or even most of you let us pause for a minute and reflect on what benefits the abstraction of using this kind of rules engine actually grants us firstly it forces us to think about what context we are applying the rules to we cannot simply hide another dependency somewhere in the fourth level of our necessity if else statement as using rules engines requires us to have a single state that is passed to each condition and possibly onto an action secondly we have to extract our conditions and actions into named functions tiny ones could be implemented as lambdas but that has its limitations apart from the often reduced readability as we've seen in the prior example extracting named functions not only gave us reusability but it also allowed for operating on a higher level of abstraction abstractions rise to the top while details sink to the bottom this vastly aids in reading understanding and maintaining the code if you do need to see further details it's merely a matter of using your editor's go to definition shortcut thirdly separating the rules and the application of them opens up new possibilities imagine the following scenario you have a user that wants to be notified when a certain event occurs however the the means of being notified they're they're not exclusive as per the user's preferences we may have any combination of the user wishing to be sent text message an email smoke signals or a carrier pigeon this is very much not the case of going through the list of possible ways of notifying the user and picking the first one that matches the user's preferences the user may very well wish to be informed in multiple ways perhaps the fomo is just that strong with this one thus we cannot apply the rules engine as is however we only need to make a small addition to it by adding a run all method to the rules engine we retain the use of all our abstractions semantically if the run method behaves much like a flattened if statement the run all method behaves like a map over a filter every rule with a matching condition will it will execute its action and the result of all actions will be collected and returned as a list this allows us to implement the top level of our notification sending like this the details of how we determine whether to send a certain type or particular type of notification and how we do it that belong in their own functions at a lower level of abstraction but wait there's more if the actions we trigger are not interdependent if they have no impact on one another and the sequence in which they are taken does not matter why would we want to execute them in a sequential fashion let us add another method to our rules engine let us add run all in parallel implementing this requires a sliver more of code using python's excellent thread pool executor we can map over the rules and execute the actions of the ones with matching conditions due to the nature of the executor's map method we wrap the running of a rule into a function that closes over the state notice that we have slightly altered the process of applying a rule to a state it still holds true that we will call the action of a rule if the rule's condition matches but it does but if it doesn't we will now return a no match object the reason we do this is to be able to discern between a triggered rules action returning none and a rule whose condition does not match in the first place the latter would implicitly return on and we would not be able to tell the difference lastly we remove the no match objects from the result and return them as a list with this in place switching from sequential to parallel execution or our notification sending process is a matter of changing the method we call on the rules engine from run all to run all in parallel well that and writing thread safe python code but that is outside the scope of this talk something else that is outside the scope of this talk is fully fledged business rules engines i may have in in yes said that they are not meant for developers but this is of course not the case and there are in the right circumstances and given the right problem domains they can be exceedingly useful tools if and when you do end up in such a situation it is well worth reading up on traditional rules engines and some of the underlying algorithms that underpin them processing thousands and thousands of rules with the simple rules engine that we built ourselves would be very inefficient the commonly used ret algorithm is a significantly more advanced way of finding matching rules and only applying them more advanced much more complex and utterly overkill for most of the things we do what is not overkill is to consciously and constantly looking for better abstractions regarding the things we do at the end of the day this talk is not really about convincing you of the splendor of rules engines but it is all about inspiring you to consider the level of abstraction your code exists on instead of adding yet another elif to an existing if consider using a simple rules engine instead of conflating your models and the way they are persisted to some database have a look at the repository pattern instead of checking for undecided non-values all over the place perhaps introduce a maybe monad or at the very least diligently return sane and safe if not meaningful values from your extracted and named functions if you want to see more of the rules engine that we built at funnel it is open source and it is actively being developed further currently in addition to what we have seen here today there are a few convenience rules such as no action and otherwise no action only accepts a condition with its implicit action always returning numb useful as a replacement for a break for example otherwise only accepts an action with its implicit condition always returning true this covers the case when we refactored the else in our example before the methods run all and run all in parallel are optionally lazy so they can return generators instead of lists if so decide and some forthcoming additions include convenience functions for creating simple conditions and actions and as well as support for selecting a subset of rules to run by including or excluding rules given certain tags and with that i thank you for your time and attention thank you very much this was really dense short useful and inspiring thank you let's see let's see if we have any questions here uh there is something from helio i think it's i'm not really sure if this is a question just a comment [Laughter] okay uh so helio also he's also praising you i'm fully agree there is another one am i right that this is not something if i understand it correctly this is not something really specific to python as long as the syntax of some other language allows you to do this you can implement these rules in any other language you want so this is something fully abstract and functional which just goes naturally into any other system absolutely exactly i mean the the one thing uh that that um that is kind of required uh for for this for this type of implementation is that you need to be able to treat functions as first class citizens you need to be able to pass function pointers somehow um but but yes it's it's it's not python specific in any way or any shape this is a thing thing about it is that that the the entire idea of using using this uh in in our case back at funnel was that the python simply lacked some of the mechanics that uh languages such as elixir with with a much better pattern matching uh provide and this was sort of a way of getting the same same concept applying the same concept so to speak but in a pythonic way oh a complementary question um from kenny in what context would a rule engine not apply what are the limitations well i mean as with with all abstractions uh they're all wrong some are useful uh just like models um it's probably overkill if you have only a few very simple cases and by that i would basically mean if you have a simple simple if else it's overkill to you so rules rules engine if you have some process that you know is always going to be a conditional that is always going to be binary or either is or you're either that but and there's there's absolutely no way uh or shape that you will ever add other conditions to the to to this to this statement then a rules engine would be absolute overkill um but the thing is that as soon as you start adding these elsifs these eli iliots they generally hint that there's um that you probably will be able to add more of them and you probably will need to add more of them and over time that is going to become much clunkier than if you had been using rules and then in general you can consider that there are three magical numbers when it comes to to computer science and development zero one and arbitrarily many uh and if you and every end up integration of arbitrarily uh many the rules rules engineers can can possibly be a good fit let's see if there are any more so for nothing one more thing is it something that uh you need to implement explicitly every time you need it or can you define for example a set of abstract classes in a package which you just get and extend i mean for sure i mean since since uh the rules engine takes a number of rules and the rules are just you know a couple of of all objects so if you want to reuse things you could conceivably have um have defined uh rules for let's say that you you're always um always checking the status code of an api response uh you could you could just have the sort of this this basic set of rules like unauthorized forbidden not found et cetera et cetera you can just have those as functions then they always take just a response in and they you they know that we will be passed the response object check its status code and return true or false depending on whether it's 404 or 401 or whatever and that can be definitely reused um and and that's for and since you basically note how the creation the instantiation of the roots and you need separate from running it that means that you can create the rules and you can just add then all the the number of rules you want want wanted to contain and then you run it on different states so for for presentation purposes it was simple to sort of have uh the creation of the rules engine and running it in the same function but i would imagine and the usual way we use it is that we we created once when we sort of instantiate maybe a a client an api client and and then for every request that is handled we must call the run or run all depending on on uh our use case a method of the rules engine with whatever state we wanted to to to look at okay clear thank you so people keep thanking you but we have no more questions and we have to move to the following talk so first i would like to thank our audience for your questions and comments and i would like to thank to leonard and we really hope to see you next year at the same conference it was a pleasure to meet you here it's all mine okay now bye bye to everyone bye

Channel: PyCon Sweden

Views: 6,044

Keywords:

Id: Lsi1ZhmbNDc

Length: 30min 12sec (1812 seconds)

Published: Fri Oct 22 2021