When Python Practices Go Wrong - Brandon Rhodes - code::dive 2019

our first speaker is Brandon Rhodes and you most probably read on our code of code I've page his technical bio privately Brandon likes hiking and most probably his next trip will be the Grand Canyon in Arizona but I must say that we have quite nice hiking trails two hours drive from roads off so I have a question for you have you ever considered Poland as a hiking destination please welcome onstage Brandon Rhodes with his speech when the peyten practices go wrong testing good morning thank you for the welcome and thank you to the conference organizers for having me I love visiting Brad suave and I like talking to your community this is my talk when Python practices go wrong I'm not going to talk about Python the language and its features that might have been good ideas or might have been bad ones this morning that's a different topic and it's also one that we can't do very much about once features have been added to the language they tend to stay but it can be very impactful to study our practices around how we use the language because those are flexible and continue to evolve so I'm going to discuss four different areas of Python practice today I'll start with something relatively uncontroversial to get us started and look how the community has learned the good and bad aspects of using that part of the language and then I'll finish out with three topics where my ideas are a little more controversial because I think the community still has some things to learn and still has some practices to perfect in some other areas so let's get started my first topic is compiling new code at runtime Python has they're slightly different an exact statement and in an eval function as do many dynamic languages which let you get a string at runtime that contains some code that was not available to the compiler when your modules were loading you get that fresh string you send it to a vowel it is Lex tin two tokens it is parsed into an ast it is compiled to bytecode and it is run the entire string to compilation process runs a fresh for the string that you pass so that the code can run and produce a result that this isn't available in languages where the compiler is gone by the time your application is up and running it is a feature of dynamic languages that compiling novel code in the middle of your application is possible but is it a good idea some beginners to a language like Python start using eval for any dynamic operation they can think of they think I need to get an attribute whose name I don't know until run time maybe the user is selecting from a menu and that is when you learn what attribute you will need to display such programmers when they are beginning think that they need to go through that entire compilation process in order to say my object dot and the attribute name because many early languages were called dynamic merely because they offered eval they were dynamic purely textually in the sense that you could construct new phrases in the language at runtime but python is more powerful than that python is dynamic because it offers introspection and data-driven object operations without making you build strings and recompile them using eval instead for example in this case there is a get adder built-in that lets an attribute whose name you have just learned is a string be looked up at runtime without going through an entire compilation process so we steer new programmers in Python away from eval because for whatever dynamic operation they want to do whatever introspection they want to do better mechanism than eval is almost always available so how does a Val wind up getting used if thou wound up having a great future it was widely used in the language today it's just not where we first expected it for example you can go look in the standard library you will find that we have in the language a class fact it's a function that you call and it gives you a new class every time you call it you tell the language that you want a mutable struct a named tuple it's called you tell it the class name you tell it the list of attributes that you want it to have and it generates for you a convenient class that takes all of those attributes at instantiation time and then can let you access them later Wow Python builds classes at runtime how does it do that if you look under the covers it's using exec on a string it is not painstakingly going through by little bitty steps and building up a type object manually no it just builds a big string which is the class statement that it wishes that it had at a load time and it runs it with exact and out comes the class that it wants to return to you in fact this bothered some people a contributor once got on the Python bug tracker and offered an alternative that instead builds the named tuple class without using exec without recompiling a whole string from scratch the core developers turned it down they had tried that before themselves it turned out and running the string was faster I thought is pretty efficient at running Python and they found it was easier to get correct the class statement built a class that was perfect or is it they tried to use little introspection operations to build it it was easy to get something wrong or incomplete so templating types was judged in the standard library to be a legitimate use of eval another use is called doc tests documentation often in many languages include sample code maybe your readme shows what looks like a little Python prompt with an expression that produces a result very often inside of the doc strings that we try to remember to put at the top of classes and functions and methods we might also illustrate because very often it's easier to understand an illustration of what something does than a textual explanation show instead of tell someone seeing these examples once thought well what what if we could execute what if we could run the code inside of the documentation then we could see whether the output still matches what we claimed the code does in the documentation and thanks to eval it's something that you can write yourself I mean don't try there are some edge cases that that are a bit sharp but the basic idea is just go through the text and any lines to start with what looks like a Python prompt can be a valid turned into a wrapper and compared with the next line in the file and what it claims should come out when that expression is entered this became in the standard library the doc tests module a standard way that Python programmers keep their documentation in sync with their code now we did have some learning to do there were some downsides some people started trying to write all of their tests as dot tests and there were some problems it meant then as they wrote a huge text file to have all of their tests that the code snippets were all coupled a mistake at the top would break everything else even if those later features in your class were really working it was it was just awkward documentation it would start explaining how the function works and then would start talking about all of these edge cases you didn't really care about you realize they were talking about them because they needed them tested not because they were really interesting to you and so they didn't wind up being a good model for real tests but a great great tool if you keep it limited to testing documentation but we had to learn where that mechanism based on a vowel was useful where it was not but what was the greatest success of all of eval it has to be the ipython notebook the Jupiter notebook as it's now called this was easy I believe the project was started by a scientist Python makes programming that easy they wrote a loop that takes input initially at a prompt but now also in cells in the browser it runs eval on the code that you've sent in and can send back graphics images charts data tables and raw Python output in an interactive display that lets you instead of running your statistics analysis program in one shot it lets you load up the data and then work with it live an ability powered by a vowel that is making Python the tool of choice for statisticians scientists and data people all around the world today possible not just possible but easy for someone to get started implementing because Python is dynamic if Val turns out wound up having only limited use for application code I write thousands of lines of Python without ever running eval but it gets very wide use usually not in our applications except in named tuple but in all of the tools we build around the code the tests the development environments and the tools I'm now going to go on having used that as an example topic of how we learn is we work with a language over the years about what its features work well and work badly for now going to go on to a second topic let's talk for a minute about pythons object model the way people use it what is the object model well python has a number of operators that are supported by it's built-in types in sand floats know how to do addition subtraction comparison and other operations but it's not only built-in types that get to be part of that ecosystem any class that you write in the Python language can opt into the same operators if it adds special methods that the language recognizes a special because they have double underscores at the beginning and end like certain symbols in the C and C++ language that are special to the pre compiler the we call these double underscore methods underscore with our operations and operators here are some ways that people have tried to use those let's look at 3 dunder methods first dunder bool dunder bool let's your object participate in if statements while loops and other built-in constructs that have to make decisions true or false based on a value in Python a value really means an object because everything in Python is an object and your objects by implementing the method under bool can also be eligible to be the condition of an if statement or a while loop many built-in types have already opted into this system to make Python comfortable for C programmers C++ programmers and Java programmers integers declare themselves false if they are 0 floats declare themselves false if they are 0 and all kinds of containers consider themselves false based on their length they take their integer length as determining whether they are true or false programmers love brevity I don't know if you've ever noticed this or tried to read the code of someone that tries to be very brief but in no less an authority than pep 8 the official standard for how we should write Python code it says this for sequences strings lists tuples when doing if tests and while loops use the fact the empty sequences are false what habit is this trying to encourage it's trying to encourage this a beginning programmer writes this if the length of the sequence is greater than 0 but then they learn more about Python and they say wait 0 is already false I don't have to compare myself to 0 if I want to skip the body of the if statement for a 0 value all I have to say is if Len of sequence and it means the same thing then they learn a little more Python and they realize well wait a minute sequences already judge their own truth or falsehood by their length I can just say if sequence it's officially recommended in pep 8 and it just makes developers feel good because it's fewer characters problem and this is something I only really realized in the last year or two when working on very big code bases Python code is always in danger of becoming a type desert I mean empty and barren wasteland and the Sun is beating down on you as you walk through the heat thirsty for any knowledge of what types the variables are this isn't so much a problem in smaller programs but in a huge system hundred thousand lines of code you can run across a routine like this and ask what is the type of users the only hit you are given is that it's used in an if statement how much does that tell you almost nothing anything can have a dunder bool method because the object in an if statement can be anything it could be a user collection that has defined a dunder bool method it could be a normal Python list but of B user objects it could be a list of usernames that could legitimately be called users in fact I've been in code where I suspected one of these things added a debugging print statement and found that users was 12 some people don't always call that user count or number of users they think users clearly a number might not be clear to the reader six months later or even to the author in the Zen of Python which I hope you have displayed on your screen as you sit with your tea or coffee in the morning and get ready to code Python when you type import lists this and the Zen of Python prints on your screen you see that statement explicit is better than implicit in Python code where a systems become larger and larger it's often a difficult question what the type of a variable is might not be a good hint I have found myself stepping back from that pet bait recommendation and keeping things explicit rather than implicit making it clear to myself giving myself a reminder whether the if statement is attempting to make a judgment on the size of a container or a judgement on the value of an integer those of you who have used the language will will guess where I got this last year I had to do some work on a code base that was in Go Go does not believe an implicit trueness or falseness you have to do the comparison you have to show your work when I came back to Python the if statement almost looked unfinished when it just said if users so I didn't tell anyone I've just been using go but I wrote the if statement explicitly where I could see what it was testing and I suddenly felt that more knowledge was being left in the code then when I omitted it don't tell anyone try it and see if it does the same thing for you maybe it will work maybe it won't but that's an area where I'm finding a common Python practice deprives my code of knowledge that maybe I'd prefer it retained all right the second part of the object model is called get adder this is a fallback for if you try to look at an attribute of an object that doesn't exist if C dot foo isn't there but get out or is defined then you can have dynamic code that decides on the fly whether there is a foo attribute or not either raises at attribute error if it thinks it shouldn't exist or returns a value for it and a client can't easily tell the difference between whether that that was a real attribute that was there already or whether you made it up on the fly this powers some very important patterns for instance the proxy pattern in the Gang of Four a standard design pattern if you are wrapping a class with 30 methods traditionally requires you to implement 30 methods in your proxy it's much much simpler in Python because all the attributes of person tries to look up on your proxy can dynamically be dispatched through to lookups on the wrapped object and then you can add an if statement if you want to do something special which is the point of a proxy object so some patterns become vastly simpler simple vastly more simple in Python because of this dynamic ability what people got another idea instead of wrapping another object what if we made something that appears to have every method you can think of if you look up a name it will say oh yes I have a method of that name and return to you a function but it could instrument that method to record every time it's called and what arguments it's called with you get an object that knows you know whatever methods you try to call will suddenly seem to exist and will be recorded and this is now in the standard library and available to power your tests a test instead of actually letting you connect to the database or allocate a window in the operating system can instead create a mock object call your code and see if the correct set of operations was done ah that's so great so powerful what could be the problem mock the widely-used encourages tests that in one sense are not really tests the ideal is to have unit tests that yet small collections of functions or classes make calls to their facade their outer layer of calls and see how they behave and then integration tests that really hook code together to see what it does together I have seen code bases where the code is never actually hooked together like it would really work where every single test is simply testing a piece of code against a mock so the only guarantee you have is that it works against a fake object but you never actually know until your customers try the code whether it really works my complaint then is that in fact tests start to lose signal when mock becomes routine instead of a reluctant workaround and I think our Docs and blog posts too often focus on how to use mock I feel like I've seen a dozen blog posts with that title rather than also telling you when you should use mock and that it has downsides and dangers a final built in bit of dynamic nough suis like to take advantage of is called dunder call the Python syntax F followed by some parens is an operator named a call F in that syntax is often a function or method but it could be any other object instead any object can define that dunder method and opt in and so if you have a template class that only really does one thing useful render you give it some keyword arguments and the text gets filled in well programmers love brevity and they think well I only have one real method I don't have lots and lots of things the template can do to just that need to be distinguished from each other does the one method really need a real name they look down at the bottom of the screen look at that they're wasting seven characters dot render well that's the only thing a template in its right mind would do anyway and so they make this pivot they switch the verb that the template performs to dunder call so that now the template simply acts like a function it takes some parameters and returns a string and they are now happy and at peace with their code having saved seven characters every time the template is used what's the problem readability I realized this when I was debugging inside of a framework one day and saw a call like this yet template args close paren open paren close paren open paren close paren what are those parentheses doing I had no idea it turns out does anyone remember XHTML crazy crazy times this was back in the era of XHTML the first pair of parens built an object graph that represented xml objects the second pair of parens then flattened them to flame to plain text could I tell that by reading the code no I'm sure it felt clever to give each of those objects a default verb with an under call method that didn't even have to have a name but I am going to come out against that practice I prefer to just burn some extra characters have the line be a little longer and know what the code is doing know what operation is being performed I at all costs avoid under call in favor of real method names but wait I hear you call you've thought of an exception what if I'm working with an API that expects a plain callable then my class definite he needs add under coal method I have an excuse if I am going to pass tea to a framework and the framework says pass me a callable I have no choice I'm going to have to be brief and add dunder call to my class so the class needs a dunder call right my answer is no no it doesn't because of a neat feature of Python called a bound method the syntax T dot method name does not have to be followed by a pair of parenthesis triggering the method it by itself performs an operation that JavaScript another languages call bind and that can be done I believe in C++ with a wrapper function dot by itself in Python goes ahead and does that operation that requires more work in other languages T dot render is a plain callable acts just like a function but when called is a method on the particular object you use to the left of the dot so even if your framework requires a callable that doesn't mean that the method can't have a name alright those are a few observations of where I think people have gone a little too far in enjoying pythons object model if maybe hurt their codes readability in the process my third topic getting serious now pythons mutability mutability the ability of things you've defined in your code to then start changing at runtime that sounds like a bad idea it generally is let's look for example modules and classes in python are mutable you know my you might have written them one way but they can be changed later if someone is feeling unwise or adventuresome if you import the default module string and you don't like how limited the set of ASCII letters is you can go change that constant to a new list of letters that you prefer the language won't stop you you'll be entered disinterested to see them how the rest of your application runs with this constant having been changed in a class you might set a class attribute value to one but someone can always come in later and change it to two because classes are mutable in fact not even methods are safe you can come in later override a method with a new definition that you prefer and the rest of the application will run with the new method in place instead this as you might guess has over pythons history resulted in some predictable mayhem did I mention that programmers dislike repetition here's the sort of things they try they love saying I use the dry approach don't repeat yourself and someone I won't name them was writing a web framework and they noticed that if you pass the an object representing the incoming web requests to your views your code has so much repetition request request request all the way from the top to the bottom of your application clearly this needs to be fixed they said hey Python global objects are mutable global object doesn't have to be a constant it can be a live object whose attributes change let's eliminate the repetition they decided to do this oh let's not pass in the request as an argument ah let's eliminate all that repetition and have it be a global object that you import and then behind the scenes when a web request comes in after importing your module it can for the new web request is received overwrite the attributes of the request object with the properties of the new web request and then call your code that will see in that imported class instance the information that it needs this it turns out I think I will argue is a bad idea data when you come back to debug something in the middle of the night later because a customer has run into a problem data is now entering that function from two different directions you can no longer read the argument list and having your head all of the things that could change the functions behavior instead live data is coming in from some import statements that could also affect the outcome of your view function tests are now going to have to mutate that global object before calling of you in order to completely control the decisions that it's making threads are going to have to overwrite that one request object you're gonna have to put a thread local in place so that different threads all looking at the same global can see a different set of attributes and then a sync framework will now break your web framework unless you teach the async frame work that each Co routine is going to need its own request object swapped into that global right before the code is called and that's difficult if you've imported it and it needs to be injected into the views module so there are a number of downsides to this I just prefer the repetitions that I can see the data that my view is acting on it's bad but guess what the situation can get even worse there's another Python web framework that I won't mention that asked why import requests at all if you have 10 different modules with views oh you're having to repeat import requests at the top of every one of them it's maddening don't repeat yourself and so that web framework loads your module full of views that reference a request that's never mentioned and it doesn't matter because it sets it gives your module a new global at runtime before calling any of your views so request is magically there without boilerplate developers hate boilerplate and this gets rid of it the code is shorter it also means that you now unless you know the special rule can't see where request comes from it appears magically and your IDE has no idea where it comes from say goodbye the tab-completion say goodbye to right-clicking every time you mention requests little wavy red line underneath because vs code can't see where it came from it was done in the name of making code more brief I'm not sure that it made it more readable or more useful a global state it's useful in emergencies but shouldn't we discovered I would argue be used routinely what would an emergency be that might justify global State I'll just briefly mention that gating is a big problem out in Silicon Valley right deep inside of some code you talk to a database and now someone comes to you and says oh we're starting a beta it's private to just three of our customers and we need them to connect to a different database well good grief by the time you're down that low in the code you don't know who you can't get at the current quest you don't know who the current user is and the team hasn't been passed down there for you to make the decision you could go through all of your code and start passing team everywhere but the experiment ends in four weeks or it's supposed to if someone remembers to back it out and you don't feel like modifying lots of code that shouldn't have to know about the team just because we're doing an experiment for a customer at a low level that is a situation where I've seen Global's grudgingly used to good effect to get knowledge deep in a routine shouldn't normally need to know it because we're we're doing an experiment changing some behavior based on something the code wouldn't normally know but except for situations like that I try to avoid passing knowledge to routines through Global's I prefer instead to speak to them with arguments another use of code mutability is in code that's about other code like tests unit test dot mock has another feature that I haven't mentioned yet it is called patch it manages mutability in a much more robust way than if you are running around trying to set values or install new methods on your own patch is used with a width statement and while the indented code block beneath the width is running C dot method is going to be replaced by another collar ball that you provide and patch is very very careful whether the block of code gets killed with an exception or whether the block of code E in successfully patch makes the guarantee that the original C dot method is put back by the time the block ends so you don't break all of the rest of your tests with a temporary maneuver patch lets you run with a class a function or a method briefly substituted for something else patch is therefore become the official mechanism for testing code that's coupled to side-effects let's consider some simple code you loop over some books for each book you want to fetch it so you build a URL for each URL at the very low level you go and do an HTTP request to a server to fetch some information simple enough question how much of that code on the previous slide can you test without triggering real HTTP start on the right all right we can't call download without real HTTP trying to run because it does a URL open but notice that that's also true of the other routines if you call fetch it will call download and a real HTTP request tries to go out on the network if you call get books it inevitably calls fetch which inevitably calls download which does rely oh it turns out that not a single level of this hierarchy is callable without real I oh trying to take place are you going to need to run a small web server on localhost in order to test get books over there on the left none of that code in fact can be tested without a web server up and running the code succeeds in abstracting away the i/o if you read the top level you don't see any raw HTTP requests but it fails to actually decouple it even though you can't see the web request being built if you're up at the top level it still isn't inevitable consequence of running that code it's not decoupled from it what is the solution to that if you want to be able to test get books ok first the real solution is to turn your whole application upside down look at my talks or other people's talks on the clean architecture or in other communities it's called the hexagonal architecture where you turn software upside down io should happen up at the top level the business logic that are in the subroutines shouldn't know where data is coming from or where it's going that makes the lower levels easy to test they're often pure functions and then it's only when the top level is called that you're in danger of real IO taking place okay but what's the second-best solution many of us are stuck in situations where we can't completely re architecture right now and things like the clean architecture are still emerging in the java community the c++ community and python you might not yet know the clean architecture well enough to architect everything you normally code in that way so if we want to keep the IO on bottom coupled to everything else but be able to test that's when we use patch you go patch that IO call URL open and now can call that top level function get books knowing that sample data from your patched version will be passed back to it in the test rather than a real web request trying to take place unfortunately not all programmers stop there there is a worst case you can move to where instead of patching the IO you think oh well if patching is a good idea let's not even have get books talk to fetch let's make a fake fetch routine so that all I'm doing is testing get books some people think that unit testing means each little thing needs its own tests whereas in fact unit tests means create a unit by choosing some edge of a smaller system and only test things from that edge but it doesn't have to be as small as one class or function the unit could be this whole little cascade of three functions when people create units that are too small and are always using patch instead of letting get books really call fetch then the same problem happens is with mock you often get tests that don't actually test whether your application will work for your customers when the real functions are operating together like testing objects with only mocks testing functions with all their subroutines patched falls short of being a real test and you lose half the benefit of testing why because there's two reasons we should write tests one an automated alert when code is broken because it's better than your customers telling you but two noticing that oh good grief I can't I can't call get books without patching IO is a hint that your architecture could improve writing tests and noticing do you need patch and mock all the time is pressure that helps you understand the or architecture needs to pivot towards more pure functions if you're always patching and mocking the problem away you lose that signal to improve architecture so python is dynamic and allows us to change things at runtime for tests that's good heavily coupled code if you've written it you can move forward it's still testable in Python thanks to patch but there's a downside the bad is that overuse of patch not only makes tests less meaningful but ruins what the tests would teach us about our codes architecture one last danger of mutability is import time side-effects Python modules are executed top to bottom at import time each module starts as a blank slate and runs its code sequentially this is a surprise and this is different than many compiled languages not only can a for loop for example run inside of a function if you just unand nth you can have a for loop that runs and does i/o at import time you know why would you ever do dangerous or bad things at import time it happens like this one you start with your configuration name and port the database living in the python code number two on a dark night you discover that that's an awful thing to do because a customer calls with an error you roll back to yesterday's version and because your configuration is in your code you're also trying to use yesterday's name for the database server which you had migrated last night and you're now talking to the old data so you switch you now understand that config shouldn't live in your code you switch to loading config from a file but it's the middle of the night and it's easiest just to do it from that top executable level of the module because I mean opening a file it's what Python does as it loads all your py files and compiles them right it's in the same spirit as what import is doing anyway at import time opening the little JSON file where you've now put the config feels so much like importing a Python module that we just go ahead and do it at the top level so then later it's another programmer that is at the helm when the config moves inside a database and that code at the outside level of the module now tries to talk to Postgres to get the config where you're keeping it centralized for everyone to read and then the database port moves to zookeeper you can't even talk to Postgres until you've connected with zookeeper the result a module that can't be imported or tested until zookeeper and Postgres are both up or you will get an error at import time and it will fail my advice is to never start down that road admit no import time side-effects at the moment you're tempted to add some executable code to the outside of the module instead create a setup function that you call over is the first thing in your main routine you want to keep your code importable without trying to talk to the database trust me one last thought about this topic you could do all kinds of things in a module and I see people trying to do all kinds of things specifically in the dunder init dot pi if you have a package or directory called my package import my package in fact in fact imports that dunder init PI if you say import my package dot models first the dunder init dot PI is run then models dot pi if you say import my package dot views dunder init PI is imported first and views dunder init Bob pi is thus an obstacle and a cost in the way of all of the other modules that you might want to import programmers keep putting code they're adding expense to any little thing you might want to import from the modules beneath it and worse yet some dunder init not PI files for convenience import all of the modules beneath them thus ruining your ability to ability to cherry-pick one thing you want you now can't touch the package without waiting to load everything inside of it my advice which is not what everyone would say is to keep dunder init dot P empty of code look at my sky filled project sky fields dunder Annette dot PI has its doc string so your IDE can tell you what sky field is but it has no code I do understand that my users don't want to have to import things manually from all dozen of my modules so I have an API module that imports all the important classes so you can say from Sky field API M court and the list of everything you want I don't make people go ransack each of my module separately for the classes they need but it is possible to provide that centralized location work all your classes can be accessed without making dunder in net PI expensive so consider pivoting to done during it dot pies that are empty all right that ends my thoughts on some practices around immutability I have one last topic to discuss object orientation indeed the difficult and fraught topic of the subclass I will give an example there is an old module called threading it has an object called a thread as you might expect you can get a function the you it defines a sequence of things to do you can create a thread and start it running that function but there's a second way to use thread you also as an alternative can create a specialized subclass of it that has a run method that has all of the lines of code that you want to execute it provides two different ways that you can provide thread with the code you want to run sub-classing is a slightly awkward choice you now have two new objects of class and a method instead of just one extra four spaces of indentation you can't now test the tasks code in isolation you have to instantiate a thread first it's a bit awkward but it is an option it is an illustration of a difference very crucial between composition and specialization which I define as follows composition means putting simple things together specialization means making one thing more complicated crucial topics in software a problem I had is that Python provides an extra temptation some languages don't have towards using specialization as a way you construct software it happens in something called the M in problem this is described among other places in the Gang of Four book design patterns you discover some of your programs programmers sub classing window because different operating systems work differently oh but some of your other developers have been sub classing it to adapt it to different layout algorithms normally this now means you're going to need as many subclasses as there are operating systems times different layout algorithms the Gang of Four say that you can get out of this m times n problem through something they call the bridge pattern have two separate class hierarchies windows that are the low-level operations and layouts which use those low-level operations you then at runtime can get to class instances and at runtime compose them together where one holds a reference to the other where the layout algorithm has a reference to the window it should be using for drawing thus and M times in problem where you have a geometric number of classes you need to define is reduced to an M plus end problem you only need one OS window one version of each layout and you can mix and match them at runtime freely as needed I will mention in passing that I think there is another pattern that in generality is even out beyond the bridge pattern you can use data to couple code instead of behavior the Gang of Four don't spend much time on this idea you can have a layout routine that given the size the metrics of the view doesn't talk to any other code at all but simply produces some data a list of graphics objects that need to be drawn that you can then pass to window dot render without the two pieces of code knowing each other's interface at all but instead only knowing the data classes it's a little more work because you have to define the data classes it takes more code it completely decouples the two pieces of code and even lets you jump in and do things like post-processing the graphics objects before the window sees them is there any advantage to coupling with nouns instead of with verbs with methods I like the inside of Fred Brooks writing in 1975 he famously said show me your flowchart your code and conceal your tables your data structures and given just your code I shall continue to be mystified but show me your data structures I won't usually need to see your code it will be obvious if Redbook Brooks was right coupling classes with data structures will be easier to understand than coupling them with behavior and so this is called usually the pipeline pattern where each element of software is able to run to completion producing data and it's the top level of your code which hooks them together which provides the relationship between them instead of the classes needing to know that each other exists so the m-by-n problem bad pattern bridge is the Gang of Four solution pipeline is if you move all the way to the code not knowing each other about not knowing about each other at all and now Python comes into the picture and Python has a solution which doesn't even make you go all the way to the bridge pattern Python provides a temptation to not even go all the way to that solution by doing something else which are called mix-ins let's look at an early experiment that still lingers than the Python standard library now I'm not criticizing the programmer of this because it was the 1990s which were crazy times we were just beginning to learn how a powerful object-oriented language should work they wrote something called a socket server a base class for writing services that receive incoming requests from the internet these are the methods of base server there's a lot of them you might think well that's a pretty big interface for users of this class to be calling it turns out most of these methods are not the public interface they're visible in the public interface but you should never call them the documentation specifies this there are only three of these which are in fact the external API that today we would move out into a facade class but here the three API calls that you use to start and stop the server are mixed in with all of these other methods that you're never supposed to touch because it's when you call those public though those three main methods that they then call activate server buying server service actions and then as a new request comes in out of the asterisk by the way means that you're supposed to specialize those you're invited when you subclass this to adjust those if you need to to be particular to your service you are not supposed to specialize handle requests because it has a lot of complicated logic if you want to customize it there's three more methods that it calls that you can make more specific if your service needs to do special things to get the requests to verify that IP should be talking to you or not and then to process it process requests if you don't override it calls by default finish requests which is actually what handles the requests we finally actually do respond to the customer and and send some data back because finish requests could raise an exception or hit a timeout process request runs it inside of an exception handler and you can specialized subclass handle error or handle timeout if there are special cleanup actions that your server needs to do if a request dies in either of those particular ways this single class is in fact several layers of behavior you can see them right here it take it took me 10 or 20 minutes of reading the docs to put this together and figure out where the layers were but there are clear layers within this class that are interacting why are there so many behaviors on a single class well largely because it was 1990 and back then the idea and object orientation was that the point of having classes was modeling if you have managers and employees in your database you want a manager class and an employee class that was the idea they had back then if we only have two nouns when we're doing low-level saket operations you can have a listening socket or a connected socket you necessarily will only have two classes a base server and a handler and you get everything having to do with a server and stick it on one class and everything having to do with a request and put it on the other if your training is to think of classes as structs for modeling the world you'll create one class for each noun instead of for each interface or behavior with the result that you'll generally have too few classes you'll model patterns of behavior with dense graphs of method calls on the same object instead of clean separate interfaces you'll wind up with one big multi-story class instead of several sleek single-story classes that then cooperate together the bridge pattern tries to get you to split things into layers abstraction and implementation in separate class hierarchies the s in the solid principle if you follow that set of object-oriented guidelines the single responsibility principle says classes should have only one reason to change if use two different reasons to specialize a class then maybe it's two classes hiding inside of one alas the base server has many reasons to change so many different directions that this single class needs to be specialized it really wants to be broken into separate layers each of which has a single purpose with the again a facade that provides only the public methods and back behind it clean interfaces you can specialize to specialize the network behavior and to specialize what the service does with the data it receives yes three different classes at least are needed here to wrap a single actual resource in the end there really is only one listening socket but to properly model its behavior we need three classes not one so given the situation it's got itself into how does the base server class survive needing to be customized in different directions at once mix-ins the fact that in Python if you declare a class it doesn't need to have any relationship to the class that's being combined with threading Mixon doesn't inherit from or adhere to or or have an interface that relates to anything else it's just a bear a little class with one method if when I create my service class I list threading mixin burst in the list it's method will be preferred over based servers version of the method for processing the request that is how the socket server survives its problem you select the base class you could have done it the other way but you select the base class in this case according to the protocol you want on the network then you mix it with a mixin that provides different concurrency models that's how it lets you specialize in two directions at once without needing M times in different classes mix-ins let you extend a class in several directions at the same time without solving the actual problem which is that the class is too complicated so that I think is in one way a strength but in one way a weakness of Python is that you can get around actually solve a problem that requires a real design pattern by stopping short of it and just using mix-ins with your overcome overly complicated class instead it's nearly 2020 and the reason I bring this up is that new Python books are still recommending mix-ins as the best of breed design pattern I saw one the other day there is a major Python web framework that thinks that mix-ins are the way to build views this is still an active design pattern in the Python community that I think has better alternatives let me pour on architect texture is going to happen despite our best efforts so it's wonderful that Python does have powerful mechanisms for surviving it when it happens but mix-ins too often I think become routine instead of an emergency survival mechanism so in conclusion looking back at all of these examples of the way that Python is very powerful but that lets you work around things I'd say that Python is a flexible and powerful language it's so powerful that we often work around poor architecture without noticing it a powerful language then by itself is not enough the ease and power of the Python language need to be combined with the experience of its community for healthy long term software projects thank you all for your attention I'm Brandon Rhodes you

Channel: code::dive conference

Views: 41,003

Rating: 4.9172416 out of 5

Keywords:

Id: S0No2zSJmks

Length: 60min 51sec (3651 seconds)

Published: Tue Dec 31 2019