#PyConEstonia 2020: 'Object Oriented Programming from scratch (four times)' by Raymond Hettinger

okay so uh hoping that you can all see my uh screen right now so here's my name for those of you don't know if you don't know the year here it is you don't know where you are there's where you are and my topic today is talking about object oriented programming what i'd like to do is build it up from scratch several times in a row with the idea if we work in python every day everything around us is objects but like a fish swimming in water you're not conscious of the water so i'd like to take us back to first principles and give you several fresh perspectives if you've seen one of them before i'm hoping that one of the other ones will be completely fresh for you so i have a whole series of mini adventures for you about uh a total of about five of them uh and the fifth one is uh time allowing so let's uh start on the uh first so the tools that we're about to need is uh name spaces and namespaces can be implemented in several ways in python and one of them is with a dictionary and so we make an empty dictionary and i can store in there raymond is red and then do a lookup in that uh dictionary and this thing is taking my keyboard away from me you will see how well that continues to work as i type all right so we've got in uh uh i'll look up that is one way uh to do a namespace but there's another as well and that is whenever you save a variable it's being saved somewhere and the question is where is that somewhere the answer is it's also in a dictionary called globals and so it turns out whether you think you're using a namespace or not whenever you uh save a variable you're always using some namespace so it's part of our world in python to always have namespaces with us in fact if i update the globals directly i can see a change in the very one thing that we don't really like about how dictionaries and uh work is that we have to use the square brackets and we have to use the uh quotes in order to do a lookup and that can be a little uncomfortable for globals our speed up is we don't need the square brackets and uh at all we just type the x and the value terms right it comes right out but there is another way as well and it's in types okay and it's called simple namespace and so a a simple namespace is a wonderful thing it is just a data holder like a dictionary where i said a x is equal to 99 and a y equal to 100 and if i take a look at this namespace um how is that possible it is possible because i didn't type yes there we go never talk and type at the same time just saying all right so the namespace looks just like the object that i created although it's funny that when it prints out it says namespace instead of simplenamespace and unlike a dictionary where you have to use square brackets to access all of the fields you can use a dot so dot x and dot uh y in fact the reason that people like simple name space instead of dictionaries is so that they don't have to use the square brackets and the quotes so i've shown you three different name spaces ones that you can make with a dictionary ones that uh is just globals and one that uh is a and types which is a simple namespace and they all have different ways of accessing them the way of accessing a dictionary was d square brackets x but in globals you don't need any of that you just type x and it looks it up in globals and with the simple namespace it is namespace.x so three different ways to do a name space and those are the tools that we're going to need so i have a small project uh to work on which is i'd like to emulate how dictionaries work and a way to do that is to make a small hash table of size 8 and we can store all of the keys in an array of our list for i in range 8 arrange n and we can create create several buckets likewise i can do the same with a value array when i do that i have three things in memory i have a size i have a group of eight buckets and eight very uh eight places for uh values so suppose i have a key value pair where the key is raymond and the value is red the way you can insert into this dictionary is you need to choose one of the buckets according to a hash function which is quite easy to do we compute a hash of a key modulo the size of the dictionary that gives us a bucket number somewhere between 0 and 8. so you go into the key array look up the uh bucket and append to it the key and do the same thing for the value array to store the value this is an emulation of how dictionaries really work the actual one nc python uses very close to this technique except instead of chaining it uses uh open addressing if you're interested in that i have a separate talk on that subject that has been very popular so i'll give you a send a link to you for those of you who are interested and this is how we say things in a dictionary so here's the way i've been working the project is emulate how dictionaries work the variables are in the key array and the uh value array and the place name space where they're stored is globals and my uh my execution technique is manually typing commands so we'd like to move up and improve this uh a little bit and make our project better the first improvement is to uh move the operations into functions and for that i've already put one together for you where i just moved it inside a function and this is a dictionary emulation using just functions we've got a lot of screen space here so i could probably make my type size a little larger for you do that momentarily and you can take a look at the code on the uh on the uh right so let me see if we can make this a little bit bigger now this talk is going to have a lot of code in it and a lot of live demos okay all right so the first thing we've got is setup and what setup does is create our variables and you can see where they're stored they're stored as global variables now whenever we uh i do a store operation a store does exactly the operations i just showed you so after three stores down here the contents of the array are we have in the key array and the value array and the positions of these are determined by the hash function i can continue to star uh other people bob uh like screen and we can see where that goes in the array so this is a nice improvement the other tool is lookup and lookup will look up one of these variables and we'll look up raw raymond and that goes directly to the raymond cell and pulls out the red without looking at the others and this is what gives dictionaries their super fast performance but if i look up uh omna we will get a uh akira so our improvement here is that we were working in functions so our project is emulate how dictionaries work are these are variables and the place where those variables are being stored is in globals and our execution technique is now a little bit better we have moved things into functions but we have a limitation and the limitation is this in globals are in one namespace a variable name can be used only once okay so we've already used in k array and v array so our problem is in one name space we can use these variables only once which means even though we have these three functions we can only have uh three dictionaries uh one dictionary and all of its variables are stored in globals which is actually a terrible place for them so here's the engineering problem the engineering problem is uh quite simple one name space is not good enough if one name space is not good enough what do we need can tell me in the chat window talk to me uh i am feeling all alone here if you all chat me up i like to have two weird conversations with you while we're going along and that's uh uh the correct answer is we need more name spaces so we've only got one global's namespace we'll need another and a way to make those is using dictionaries so our next improvement is to rewrite the uh code this time passing in a namespace and the namespace will be a dictionary and so to store in instead of just saying n equal 8 we store it in a namespace instead of globals and the same thing for the k array and the very right otherwise these functions are exactly the same as before and our new approach is to emulate how dictionaries work with the same variables but the namespace will be user dictionaries our execution technique is functions and the benefit of doing it this way is we we can have multiple name spaces which means multiple dictionaries so in namespace one is a dictionary and my functions have been modified the first argument is we pass in the namespace which is just a dictionary and if i need to look up a variable i find it in the namespace and if i store it put in the namespace we say that namespaces are how the functions communicate so when i execute this namespace one is the first dictionary okay and it has the k array evaluate and it has colors raymond red rachel blue and matthew is yellow but we also have a second namespace that has the same variable names but this time we're storing a second dictionary there's a different v array i have a mac rachel has a pc and matthew has a chromebook interestingly we don't have to pass in our own dictionaries we can also pass in globals so namespace 3 happens to also be globals a lot of things will come out here the other global variables and what that means is everything in namespace 3 can be looked up directly like the n and the key array or we could say namespace three in and key array and it would work so this technique is very flexible you can pass in your own dictionaries or you can use the built-in global's dictionaries our big advantage here is we can now have three uh different emulations if one name space is good three is better and our technique is we make the namespace manually and then we pass it in as the first argument to each of these functions and this is a generally useful technique and i hope you all learned something new however there's a problem writing ns d array every time i want to get to v array well that looks ugly and are we happy with that no because the zen of python tells us differently the zen of python tells us that beautiful is better than ugly so we'd like to have something prettier than this and so the problem is namespace the array is ugly the solution is somewhat obvious the solution is simple name spaces which i showed you a little bit earlier so let's improve this one more time this is exactly the same code as before but this time so we say simple namespace and with a simple namespace instead of writing something like nsvar which is unattractive we can change it to something that looks pretty like nsv array so this looks really nice it says go to a particular name space and find the n in that namespace and the v array and remember the reason we have in multiple namespaces is so that we can have several instances this code looks prettier than before and more natural and the way we use it is make a simple namespace and pass it in as the first argument all the code here at the bottom is the same so i can do a lookup in namespace one for matthew and find that his color is yellow and look up namespace 2 and find that he has a chromebook so two different instances this time when we look at the namespaces they're going to look a little prettier than before it says namespace and it has the key value pairs otherwise it looks amazingly like the dictionary however there's a problem something we don't like the problem is every new case we have to manually create a new simple namespace that is this part right here this line and this line that's no fun but worse we have another problem every single call to the function has to manually pass in the name space for us there's a solution to that problem and the solution is pythons object-oriented programming python classes and so we'll uh move up to uh dictionary version four this one is class-based it's exactly the same code as we just had there's a different uh except all of this on the right has been indented on the left and also i've renamed namespace namespace is too obvious of a name i need something much more mysterious and what's more nothing is more mysterious than calling itself self is the name space what does the class do for us when we call dick it builds the name space for us every instance has its own dictionary the other thing it does for us is instead of having to pass in the namespace as the first argument manually the dot does that for us so when we say d1 store what it does is it passes the d1 in as the first argument the namespace viewed in this way you can see that all a class is is a beautification of what we have already made here otherwise the logic here is exactly the same so this is fantastic and we now have a class that works the same way as our other functions and multiple same specs it probably doesn't surprise you though that there's a problem and the problem is this i have to type d dot look up d1.lookup for matthew better if i spell it correctly and that is unpleasant to have to type this method name likewise if i want to do a store i would have to say d1 dot store matthew and give him a a new color orange it is no fun to have to type this our problem is the method names are too hard to type the solution is operators is change the method names so we can use operators and it's a very simple thing this will be exactly the same code predict class magic brother that's number five and this one looks exactly like the code on the right except setup now has a name vendor init store is now called under set item and lookup is now called thunder get item the advantage of those name changes is instead of having to call setup separately it gets called automatically and instead of having to say store we can use square brackets instead of saying look up we can also use square brackets and what i've showed you on this journey is that object-oriented programming is not particularly magical it's the idea that uh one namespace is not good enough therefore we want multiple ones we need to pass in the namespaces to the functions and the most beautiful way to do that is to put it in a class and to have the operators call the methods for you but there's nothing magical here these are things that we would have done anyway without object-oriented programming in fact before object-oriented programming it was very common to pass name spaces or environments or context into objects i hope you saw that i got a fresh perspective on object-oriented programming and that is adventure number one with that in place let's go do adventure number two i'm going to tell the object-oriented programming story again in a completely different way a long time ago in a galaxy far far away it was 1980 and in 1980 there was a software crisis lots of people knew how to program computers we had a lot of computers we had some big projects and in the 80s almost every major software uh project in the entire world was a failure not all of them but almost every single one failed to some extent and this was called the software crisis of the 80s and the early 90s i'd like to take you through that crisis and give you a solution to the problem once upon a time we had a big pile of stuff called data that we gave variable names if these variable names seem terrible to you it's actually how we used to name things a a 1 a 2 b c d short variable names were very common back then so a was an integer b was a list which we called a vector there is a two-dimensional array another two-dimensional array that's square a string a file a unicode an open url and so lots of different kinds of things one big steaming pile of data but we didn't just have data we had something else we had functions that we could call on that data and these functions include closing a file computing a vector cross product a dot product eigenvalues of a square matrix matrix inverse matrix multiplication the size of a matrix multiply two numbers read a file get the length of a string write to a file get the length of a unicode string uh uppercase unicode string uppercase a regular string uh read a url get the size of a vector and uh write to a file a big pile of functions and all of these things existed in one namespace and if you detected the theme of the talk already one namespace is bad how bad our problem was this whenever you wanted to call a function with the data you had to know which one there's only uh about uh what is that uh a dozen pieces of data and a dozen functions but 12 times 12 is the n squared problem there are 144 possible connections between these things on the left and the right and the way we used to call uh write our code is in the form verb now noun which the verb is multiply and the nouns were multiply a and c together and uh matrix multiply the matrix e times the matrix f and this was life during the 1980s and it was terrible did i mention there was a crisis yes everything was falling apart what was the problem with what we were trying to do here there were actually four major uh problems one problem is knowing which function went with which piece of data and if you ever called the wrong function with the wrong data you got the wrong answer for this amount of data and this number of functions that's not really a problem but when you get up into thousands of functions and thousands of pieces of data a very common source of failure is a person would call the wrong function with the wrong data more than once we lost a probe on the way to mars for exactly that uh problem uh the aetnt telephone network uh went offline for eight hours for uh one day for exactly this problem it doesn't seem hard until you've got a lot of functions and a lot of data and it's an n squared relationship between the two another problem is these functions are not well organized there is no organizing principle one thing that we can do is some type checking to know which option to go the problem is let's say you have a function power how 2 to the fifth and 5 squared are not equal to each other but to a type checker they are both integer to integer maps to integer and that tells you that a type checker can help you a little bit but it can't make sure that you get all of the pieces of data correct your program can still be wrong and still be type correct there's no way to introspect this object and ask it questions what kind of object are you and we'll know what to how to do and the learnability problem was terrible if you started work at uh bell labs they gave you a stack of books three feet high and gave you six months to read them and you learned all five thousand of their functions before you could do anything useful it was a terrible learning process six months before you wrote your first line of code a lot of risk called the wrong function with the wrong data not the network offline type checking didn't help you no introspection this was a terrible time so let's go fix it the big idea was objects and what is an object the data is directly associated with the functions that operate on that data so in addition we store a is 5 but in addition we say the functions that are allowed to work on that are the five can be multiplied or get the absolute value r uh this vector the three things that can be done on it is a dot product a vector length and a cross product now every piece of data has a list of all the functions that work on that data this is a huge win but it changed the way we wrote code instead of verb now noun it is now noun dot verb now we would say object a integer a dot mole c dot uh matrix e dot mimo is times e transpose and matrix f is m size or hello.upper which probably seems very familiar to you so what is the notion of an object every piece of data knows the functions that are allowed to work on it what are the benefits here one of the big benefits is it's no longer possible to call the wrong function if you have a and you call dot product on it it will know right away that's the wrong function and give you an error whereas previously it might have seg-faulted the entire system so we have solved the problem of calling from wrong function with the wrong data also we have introspection if you get a job and you're told go do something on f you can do a dirt listing on f and it lists out all the things it knows how to do you don't have to learn all 5000 functions before you do anything useful that's where places like dropbox and facebook that's how it's possible that people are able to contribute a line of code in their first week at work and that's solely because of the object-oriented programming so this is a major improvement however there are problems as there so often are in my uh talk and the uh one problem and i'm losing my screen here is we have repeated this list of functions small and abs multiple times we've repeated this one multiple times as well and this one is multiple times and so this eats a memory like crazy if every function it takes every piece of data has a very long list of uh functions so it eats memory like crazy because of redundancy also we don't have any way to describe this group of uh functions it doesn't have a name that problem is of course easily solved after the invention of the object the next important thing was a class and yes it's true you can have objects without classes but what are classes classes are just those groups of functions that you create exactly one time and you refer to them by name so for a we store exactly two things the number five and then it's uh class is a scalar we store two things for b the list and that it is a vector 3. one cool thing about this is we're now giving these names and i can have standard names for this these are a stir this is a file and this is unica but what are these names they are references to the list of functions so when you say a.mall what it does is it says a is a scalar what's up scalar and finds small and says that's an allowed operation this is great uh it takes a lot less memory than what it took before to repeat these list over and over again we have the list exactly once and our other benefit is we've given the concepts names and we can now say go use all of the square matrix operations and here's the values allowed for square matrix you probably suspected there's going to be problems and there is a problem here and that is this vector supports dot and vlan but a vector of length 3 also supports cross product you'll notice that this group is contained in this one the subgroups are in this so there is repetition that is unnecessary over here in addition we haven't expressed any relationship between any of these the solution to this is very straightforward we call it inheritance what is inheritance inheritance says what does vector 3 do it says it does everything that vector does plus it adds a cross product and you might say vector 3 is a kind of vector that has a cross product to find what about a square matrix a square matrix is can do everything a matrix can do but it can also have a uh inverse and uh we can compute eigenvalues for the matrix one win here is that we have many fewer functions on the right far less redundancy this improves code we use another thing is that we've given names to these relationships and we can express structures of ideas that previously people kept in their head but they didn't have a good vocabulary for and our calls for uh this are the same as uh uh before we've got g the string if it's string length and d the uh vector length and then f the matrix size but there's something that we don't like about this all three of these things have the same concept the size of the object this one is the size of a string this is the size of a vector and this is the size of a matrix they are three different uh functions but they have the same concept and it overloads the mind to have to remember all of these names but it's not just the mind that's a problem sure you have to remember all the names but there's an even bigger problem a horrific problem that was characteristic of almost all of the languages of the 1980s what we used to have is big case statements you're looping over a bunch of objects and you say if it's a matrix print the matrix size if it's a vector call the vector len and if it's a string get the string lit and that way we can get the size of all of these objects it was very common in the 80s and 90s to have codes where 90 of the code was a big pile of case statements if it's a long-term life insurance company policy if it's a short term if it's a whole life insurance policy there's a term life with an annuity and each case had its own function and this writing code like this would destroy your soul it would eat you alive and make you want to become a flight attendant rather than a programmer this was terrible so we needed a new invention and the new invention was just a naming convention it's called polymorphism and polymorphism just means we use the same name for different uh functions in different classes so instead of calling it vector lin we call it len and instead of calling streamlin we call it lin and instead of calling it unicode lin it gets called lin now when you're learning an object-oriented language you don't need to learn a lot of concepts you can learn very few concepts the learning curve went way down and look at the calls now jalen h lynn and eflin these are three different functions and these two multiplications this one's a matrix multiplication and this one is a regular multiplication and they all look the same not only is it less of a word on your mind that really nasty chain of ifs and elifs we saw before is now gone and our foreloop looks like this loop over all these different types and call lin on it polymorphism isn't something cute it will save your soul from destruction if you ever find yourself writing big either if you lift chains or case statements in a language you'll say hey we solved this problem a long time ago with uh polymorphism and that takes us to modern times and that shows you another way to look at object-oriented programming what is object-oriented programming it is a solution a direct solution to a long series of problems that we had over and over again we really did use a code like this uh it was common for a scientific company to have a library of 5000 functions i remember as a kid seeing giant tables full of bugs where the functions are indexed alphabetically and a person would have to look up the function if you updated it you gave it to a secretary to type it and then what they would do is pull out the old page and stick it in the book and all of the documentation was in the uh computer this was a terrible way to program if you're ever given a language that looks like this you should run and say that's so 1980s and object-oriented programming solves these problems for you it's not a perfect solution for everything but it's a direct solution to real problems and that is our second adventure in inventing object-oriented programming from scratch i'd like to do uh one more uh adventure which is to show you something in the collections module called a chain map and that is something that uh i created so let me show you how the uh chain map all works it's actually a very simple thing my screen is rather busy so let me clean it off and we'll say from collections import star and a chain map is a very simple thing it takes multiple dictionaries and it connects them together i have a dictionary a is one e is two and i chain it together with b is three and c is four what it does is it takes multiple dictionaries and it gives you a single view of the dictionary chain so d square brackets a searches the chain and finds the a is one db it finds the very first one in the chain scanning left to right and it returns the b is 2. we see that this one gets shadowed but dc doesn't find it in the first dictionary and it goes out to the next one and then dd doesn't find it at all and that view places a key error that it's missing it's called the chain map because it changed several dictionaries together so let's look at python's object-oriented programming we have a class p and np is a namespace p dot x p dot y and p dot z are these three class variables we have another class see that y and z are these two numbers i have instances of that class and in the instance a dot z has been stored in the instance name space and b z has also been stored in the instance name space which raises the question how do all of these things weight together and it looks like this for uh a if i type in z these z's get shadowed by the instance if i put in a dot y it works its way up from the instance to the class and finds the 444 and a dot z a dot x rather works its way from the instance all the way back up to the parent b has a different name space so b dot c is different but otherwise its parent classes still give the same answer for y and x and that is basic object-oriented programming in python what i'd like to show you is that it's actually a very natural concept python objectory object-oriented programming is just a tame map i'll take the uh variables of a so here's the instance dictionary for a and the uh variables for uh c that's the things that we stored in c and we chain them together so now i'd like to show you that the chain map is isomorphic meaning identically structured to these data so the instance a we have a dot z but in the chain map if i look up z i get the same thing if i look up a dot y the chain map gives the same thing and if i look up a dot x the chain map gives the same thing the second one has a different element at the beginning of the chain b and so b dot z gives the same thing as mb z the delta y is the same as the chain map for y and x and this concludes your third adventure in object-oriented programming what i've shown is if you have a language with dictionaries and you can chain them together object-oriented programming happens naturally it says look in this namespace look in this namespace this in this namespace we do this in the real world all the time you go to the branch library looking for a book and if you find it you check out the book but if you don't you put in a request and it goes to the main library and they give you the book but if they don't have it they put in a request to another university's library and they pull the book from there there are several different name spaces your local library the main library and then the chain of other libraries that's a chain of namespace and if you've ever borrowed a book you've used object-oriented programming that is your third adventure of building up object-oriented programming from scratch and i'm hoping each one of these gives you a completely uh different perspective another thing i'd like to show you is that it arrives naturally from uh into the modeling so when you're describing data object-oriented programming just happens so let me see if i can organize myself here and find that uh there is a programming language called bloop those of you who've written uh seen the book good election bach have seen this language before this defines a procedure factorial and it's a block structured language it defines an initial output a cell value fixed length loops multiplies an output times a cell and adds one and this is a procedure in that language to compute a uh a factorial now what i'd like to do is parse that language so i wrote a uh a compiler for it uh called the bloop compiler and the bloop compiler what it uh does is let me give you the uh i'd like to show you that one parsed one bloop short right now here we go the ast for the group there you go what uh abstract gentrix tree is in a compiler is an analysis of this code and it uh creates an object tree and it says we uh the whole thing is a program that consists of procedures the first procedure its name is factorial it has a parameter in and it's uh not a test it has a block of code and the first thing it does is an assignment to uh this output name which is not a boolean and the value of it is one this is a step in compilation if you can transform something like this into something like this you have written half of your uh compiler so this is an abstract syntax tree and these things are descriptions of the entities that occur along the way which raises the question where do these names come from how do you describe a program when you start describing a program concepts come up and they have a relationship to each other and it turns out the most natural way to uh model that is through object-oriented programming uh you know you can write regular classes for it but it's also easy to use either data classes or name tuples which is what i did so the abstract uh set tax tree is described in a language called asdl and it says every program consists of statement procedures and some calls statements are either a procedure a block assignment a conditional a loop an abort or quit and expressions include binary operators numbers booleans id identifiers and calls and l values and then we have a definition of l value this is the structure of the knowledge it emerges from studying the programming language when you study a language its inner bones our skeleton of structure is described something like that but now we want to represent that in my thought and it turns out this translates very naturally to classes and so i was able to describe it quite easily a class program is a name tuple which has procedures which is a list of statements and calls which is a list of expressions what is the procedure well every procedure has an identifier a list of parameters an indication whether it's a boolean or whether it's a test or whether it returns a number and then it has a body with a bunch of statements so what we just saw that output on the previous screen was really a nest of all of these things the point here being whenever you're studying something complex like an entire programming language there is some inner structure and when you start to describe that structure you end up inventing new words in relationships between these these are called entity relations or er diagrams and it turns out the most natural way to express them is in terms of classes that contain other classes here we're using modern type annotations uh along with the names and because i used name tuples my building all of these classes took about five minutes in fact the only hard part was analyzing the language once i had analyzed the language implementing it was uh just a matter of minutes once you have these classes it becomes very easy to loop over all of the children and display all of the nodes the other uh bit of modeling that i want to show you is i've invented my own version of twitter okay so you've got real twitter right here and what's in it you have people's post you have a picture you have a summary and uh user uh user names clear is obviously a very complex uh website which raises the question how difficult is it to model in python well i'm going to uh build my own version of a twitter and let's see if i can find which screen it's on and i've got it up running on a server right now let's turn back to our this really sees control of my computer so leave that length though so this is currently running on my computer it's my version of uh twitter which i called messenger you could search for python on it without logging in and it'll give you uh people's tweets or uh messages all right if you log in it'll give you more and you could go in and see my friend gavin who has posted three times he's following two people me and uh barry uh of warsaw and he has uh one follower my favorite data scientist and friend michael sellick we can also uh search for particular topics or we can find out what devin has to say and devin's the person who taught me that coriander and cilantro come from the same plant so how difficult would that be to model in python and the answer is not difficult at all i'd like to show you with through the power of object-oriented programming how all of that is modeled if you all want a link i can give you all of the source code for uh for that entire program uh the back end is about 100 lines the front end is about 100 lines and the html is about 100 lines so very short program a user is represented by a string a timestamp is a float a user's hash is a couple of bytes and some more bytes for assault a hashtag is a string a post something you call a tweet is a name tuple and every tweet has three parts a timestamp when it was tweeted the user who tweeted it in the text of their tweet uh 140 or 280 characters in addition we have info on users every user has a display name email we've stored their hash password and they have an optional string biography and an optional uh this is a string for the photo so uh it's a file name and the data structures we store them in is every post is stored in a deck arranged newest to oldest we also store it in a default dig so that we can look up let's say all of donald trump's post and find them newest to uh oldest and it looks like i'm coming up on my q a time here in just a moment so let me wrap it up really quickly uh hashtag index is also a dick so that you can look up a hashtag and find every post for that hashtag following is a default dictive set where you look up one user and you find the set of everyone they're following likewise followers is a default dick where you look up a user and you find everyone they're following and then there's a regular dictionary where you look up a user and you find their info this one page describes all of twitter you should be dazzled and amazed by that this is the power of object-oriented programming it naturally expresses entities and relationships and that completes our fourth adventure of reinventing object oriented programming from scratch from solving the original function problem to applying name spaces the chain map point of view or it falls out naturally from end of the uh relationship modeling it turns out there are many many problems in computer science and many of those paths end up at object-oriented programming much as we've implemented in python i hope you've enjoyed uh this talk i'd like to uh open it up for uh questions and uh i'm hoping that someday you invite me to visit estonia in person i would love to see your country a lot to learn and we can actually start off with the questions and answers so um you can either send in your q a or i guess all of you are unmuted if anyone just wants to ask a question okay is there something i should be looking at so i can see their questions i've got a chat window up but uh i don't see anything nobody has written yet anything in the chat but they can if they want them okay good enough well anyone ask me anything i'm not is um a chance that you could grab a question for me and repeat it for me so i can answer because i'm not making out anything yes uh there's one question about what are your thoughts on functional programming i love functional programming i think it's awesome i wrote the iter tools module because i love functional programming so much that said functional programming doesn't work nearly as well for these problems functional programming is not good in modeling all of our twitter like a pile of objects are functional programming is not particularly good at managing a series of namespaces like we did in the dictionary option to begin with it can do it but it's not its forte so i use functional programming for things functional programming is good at i use object-oriented programming the things objects are good at and the two actually mix and match pretty well for instance in uh the code that you can see on the screen when we want to find all of the posts for a user everyone that davin should see it's only two lines of code because that code is functional programming so we look up in our data structure the user daven and find everybody that he's following and we loop over those and we uh find in user post the post for every person that davin's following then we run a merge on that and that builds a iterator that merges the post newest to oldest then the functional tool i slice will pull off let's say the first 10 post and this turns it into a list this is fast it's efficient and it's clean because of functional programming so uh more about functional programming should be in another talk but i think the two techniques blend together very well when i'm processing data i use functional programming while i'm laying out and managing state i use object and the two fit together perfectly great question next there's another one about what you think is the best way to apply this for a small sme um uh i would let the person who has asked the question uh i think it's carl want to also small and medium enterprise oh i see when i was very young i learned the programming language basic and the first example i gave you in the software crisis uh came from the kind of code i wrote in basic when i said that we had these function and data this is what my basic code looked like and basic was great because it was easy but it was a problem because there was only one name space it didn't scale so i had no problem writing 50 line programs that played lunar lander or a star trek game but if i ever tried to do something serious the language would fall apart very quickly now keep in mind modern versions of basic do have objects and they do have name spaces but the version basic i had didn't have any of these things and so my problem was even though i was a decent programmer the tooling i had it couldn't scale and i couldn't solve any interesting problems so i would say to you object-oriented programming is there to help you manage the scale that you have at a small to medium enterprise that in fact programming without objects with just a big pile of functions and data reinvents all the problems we used to have and it was a disaster for small and medium businesses so this actually is a solution to your problem it is a technique for managing complexity which is something that you want one other aspect of object-oriented programming we didn't talk about was encapsulation and the idea is people are giving you objects fully formed so as a small and medium enterprise you don't have to write all the code yourself you download a package from the python package index it has some objects in it you load the objects and run the methods on it and you don't have to know anything about what's inside and that's why it's possible for a small group of people like 20 developers was all it took to make instagram five developers built the original version of a dropbox the original version of a youtube was written by a team of size under 30. object-oriented programming and pre-made packages made that possible in other words uh object-oriented programming is a gift to you in this time from your uh parents and grandparents from the uh 1980s and 1990s they saved this up for you so that your success would be possible great question next there's a question is method overloading a good thing the way you ask it it sounds like it could be bad i don't think so uh so uh the word overload is actually overloaded itself and it's used in a couple different ways sometimes people mean it in the sense of c plus plus if i'm out overloading an operator and python doesn't do that our operators are hardwired plus is hardwired to dunder add so instead of overloading the operator we make a class that knows what to do with that operator when it's called and that's a very good thing it's why the core language has a plus that adds integers and strings but extension pass packages like numpy are possible that add arrays together so the reason that python is still significant in areas like data science is because of the equivalent of operator overload but perhaps you admit where you have a parent class and a child class overrides one of the methods of its parent and you're asking if that's good and the answer is sometimes you can't get away from it the reason that we have inheritance is so that the subclass can reuse the parent methods but sometimes the parents methods don't make sense and you have to override them so the good part of inheritance is you get code reused the bad part is sometimes you inherit methods you don't want a solution to that problem or a band-aid is overloading it however when you use that band-aid you get the good effect of a new and better method but you get the bad effect of the child class isn't substitutable for the parent so it's a trade-off like many things in engineering you're solving a problem as easily as you can but almost all solutions to problems beget their own problems uh i had a friend who was a uh went to pharmacy school and when she graduated she told me what she learned and basically this she said every single drug we have is a drug because we know it solves some problem however every single prescription drug is a prescription drug for a reason it's also bad for you in some way every single drug you can take has some negative effect otherwise it wouldn't be a prescription drug so she said the science of pharmacology basically boils down to look at what problems you have that are really important take a drug and trade uh solve a major problem in exchange for getting a more minor problem overloading methods is exactly like that you're solving a big problem inherited a method you didn't want you're repairing it and to do something useful but you'd lose substitution another excellent question thanks for asking uh yes there's another one what is your personally favorite part of it tools model the combinatoric tools that would be product combinations and permutations when i wrote them i had a mathematical interest in them but i had no idea how powerful they would be so when i'm writing tests i take uh event user events the user can do step a b and c but maybe they could do c first and then dna so i run all permutations of a thing a user can do what if a user sends you money just before they close their account permanently two days before you were about to ban them from a user but their trial membership still had a few days left in it what should the system do those tests are hard to think of but permutations can generate all of those tests likewise a product every category of user with every category of action i found an amazing number of ideas uh particularly in testing are very easily expressed with the combinatoric tools they are my personal favorite thanks for asking um other questions are interesting um there's one what is the best thing written in python that you have seen and why i've seen a lot of good things in python so it's uh difficult to narrow down but i think the first time i saw inside a microweb framework i was dazzled and amazed how clean the code was so my personal favorite in terms of readability is bottle i know that flask has become more popular but flask is rather large big enough to where most people never read the source code if you read the code for bottle and you can read it in about a day you know it's about 3 000 lines of code you will know completely how a microweb framework works how templating works and you'll realize it's astonishing what can be done with a smaller monika so that is uh one of my favorite pieces but i bet if i thought about it more i'd find others so if you want some weekend reading the kind of weekend reading i do download bottle b-o-t-t-l-e and read the source code for it it won't take very long and it will make you smart the best way to pass an object from one endpoint to somewhere else what's the best way to pass an object from one end point to somewhere else as in between two different machines i guess yes if you were passing it between processes or machines you either have to destructure the object into components manually and then load it into a format like json for transport our pi uh pi protocol so uh manually unpack it into a uh a binary format those are pretty decent ways but the easiest way is to use pickle it's fashionable now for people to hate pickle for lots of different reasons but it's really astonishingly good every ah let me show you an example i wrote this version of twitter and it is the entire backend for twitter the problem is it's all in memory and i can't share it between machines which raises the question how much effort is it going to take to take all of the tweets and all the relationships between them and transport that between machines the answer is down at the bottom is i save it by opening a pickle file and do pickle dumps and it takes me one line of code to save the entire state of all of twitter all at once and it takes one line of code to restore all of that uh okay pickle is really really good at what it does now you should never unpack a uh unpickle something that you don't trust so if you're moving pickles around you should probably sign them or only accept them from yourself don't accept them from outside parties they do have some disadvantages but what they do is astonishing and that's how i move objects around it's also how the multi-processing module does it as well and i believe pi spark uh uh uses a similar technique twisted has a variant on it i think it's called a twisted jelly and there are a number of other tools but pickle is my favorite it's supported directly by the language and it's well tested and even though a lot of people hate it it's very very good uh so next question is what is your opinion on better classes i love them uh if they are taught to you properly you know and i teach python for a living so i can in a few hours take people from kind of intermediate programmer to being able to handle descriptors and medical classes and properly taught you realize they're remarkably simple meta classes in many ways are no different from regular classes they just have a different application uh so i do like them uh in one thing that never fails to astonish me whenever i am working with people in medical classes helping them create them is many classes we create are like the ones that i showed you today they can be quite large so classes can have uh dozens or hundreds of methods or be several hundred lines long but many useful meta classes are three lines long or five lines long or eight lines long it doesn't take much to make a very useful uh meta class and the kind of things that uh medical classes are often hard to get in other ways that said if people aren't thoughtful about how they write meta classes they can easily create chaos if they're rearranging the internals of python so it doesn't behave as people expect so they are dangerous in that it's very easy to create atrocities that are difficult to debug so if you use them for good things they're great but if you use them haphazardly they can really create some misunder serious misunderstandings and that's given them a bad reputation another problem is that if you don't if you start to use meta classes everywhere you'll find that you override exactly the same methods and it's fine until you try and use two meta classes together we do have techniques for solving that it's called our cooperative classes but it takes some skill to do that and most people would rather just say i hate meta classes and avoid them entirely now the thing is if you don't use meta classes you end up with a lot of substitutes like other various folks like there's a new one done uh dunder and knit and that saves you from writing a med meta class class decorators are an alternative but all of these uh alternatives were created simply because people either didn't learn meta classes they were scared by them or they got burned by them once so do i like them yes does lots of great code relied on them sure the django orem is not possible without a uh a meta class uh python's abstract based classes created by quito is not possible without meta classes uh it is often the simplest way to get something done so uh i'll summarize then with the the spider-man rule with great power comes great responsibility use meta classes for good not for evil they can tie you in knots but they can also solve problems elegantly so there you have it great uh i have one more question but uh i would just take this as the last question if there's someone who wants to ask a question uh by just unmuting themselves we can go ahead with that question instead or i would just read this out why don't you just read yours uh you're reading them so clearly and perfectly i welcome your next one okay so um someone's asked for recommendations for types of projects to work on that help with gaining a better understanding uh object-oriented programming [Music] i'm going to do a little evasive maneuver on that question in that i'm i can't think of categories of projects that teach you more object-oriented programming uh you know if i had to grab one right off hand it would be what was done with uh the twitter this kind of project of modeling a website so go out and model within a few lines of code try and model wikipedia or craigslist or facebook or some major service and it should be that the core functionality should be something that you can model in one page that's a reasonably good exercise uh also if you build a compiler and that is not an easy project but if you go to build a compiler uh that other example i gave you with the abstract centix tree that'll teach you a lot about uh object-oriented programming but the invasive part of this is i don't want to recommend projects to go to just just solve real problems don't go do made-up projects to learn it uh a enhanced version of your question is what is a good way to improve my skills in general and i believe the best answer is read good code so open up the collections module and read the code inside you'll find some code written by quito you'll find some code written by me open up the func tools module and you'll find some code written by uh wukash for example or myself or nick coughlin and so learn from uh the people who are the best in the business the standard library source code is easy to find it's not necessarily the most beautiful but there's a lot of good techniques in it and that's the way i improve my skills so every now and then i find a project and i download it and i just go read all of the code when it's written by somebody i admire so i spent some time reading some of the source code for number and i learned things that i didn't know uh and i had recommended a bottle uh bottle is an a one day read and at the end of that one day you know a lot about objects so grow your skills by reading excellent code there's a couple books that will really help one of them is called refactoring i think that is by martin fowler and there's a couple of principles that you should look up one of them is called the grasp principles okay pick up martin fowler's book on uh refactoring that will make you good at uh objects uh really quickly so this will give you some of the theory and you should learn some things like list golf substitution principle so learn some theory but mostly uh uh i'll go read code and in great code you'll hear people talking about single responsibility uh uh principle for example i'm refactoring and list golf substitution uh principle uh there's another fairly important principle called the law of demeter and these will give you some rules of the game uh the refactoring book like i said is just excellent it's examples are in i think java or c plus plus uh but it's very easy to write those examples in python and learn from them i spent a weekend with that book and it improved my skills and so uh read uh read good books you know read the theoretical principles read good code and uh a little bit of heresy read something other than python python people don't have uh all of the uh answers c plus plus people know some things too java people knows things too small talk solved a lot of problems a long time ago in fact pick a language wildly different from what you know and just spend a uh a day with it uh if you spend one day with fourth it will change your life forever in terms of how you uh program you spend one day with apl you'll finish that day with a headache but you'll walk away coding differently for the rest of your life haskell you can't spend one day with i wrestled with it for about a month before i was able to get my first programs up and running but it was a minded prologue will grow your brain so i would say learn different patterns of fog but another way if you want to grow just don't get stuck no well otherwise you'll find that you've uh done the same thing the same way your whole life and will finish the long software career and not have learned a lot of things so get out of your shell go see new things and go to conferences like this one and listen to people that's my advice hope it's useful for you amna thank you so much for inviting me when uh this coronavirus is done do i get to come see you all in person yes please we would love that if you visit us who knows when maybe when it's all over i would love an invitation to come see you in person thank you all it's been an honor thank you so much thank you for your time thank you everyone who was here um i'm getting in a lot of comments about how awesome the lecture was everyone really enjoyed

Channel: Thorgate

Views: 18,478

Keywords:

Id: 8moWQ1561FY

Length: 76min 18sec (4578 seconds)

Published: Tue Nov 10 2020