Functional Programming in Python: Lessons from Haskell and Clojure - Anthony Khong

can everybody hear me okay thank you very much for coming to the salt this is social program in Python lessons from Haskell enclosure so this is me just sharing my journey you know doing a little bit of functional programming in other languages and then coming back to Python you know some of the things that I learned I just want to share I hope you learned something new today yep let's get it going someone dijah home Ben Cohen in Doha that put I died last year I I lived in Bangkok I was in a coda for a year and a bit that's how I learn Thai I actually started as an economist in Cambridge but then I decided you know statistics a bit more interesting so I got into computational stats in Oxford then I went to a good as the the data scientists for experiment platform and now I'm a co-founder at a red mark so with my partner there did it we do a little bit of growth hacking and CT as a service so enough of that this is the my plan of attack for today I want to chat a little bit about defining what a functional programming language is I just want to lay that groundwork and then I want to cover three maybe little lessons that I learned from Haskell and closure and how to put it back in Python and the idea is that how do you still write Python as it is pythonic but still gain the benefit of these functional programming practices then it'll just be a brief concluding remarks so the first thing is that it's Python even a functional programming language hands up if you think that it is so few and No yeah I feel ok so I'm gonna answer that but if you just do a quick google or youtube search on python functional programming like you get so many hits and you think that python is you know there there's actually something that functional programming and Python but you know if you really look at what they talk about you know a lot of them goes into iterator saying that it's an important foundation of functional style they get into these functional constructs of lambdas higher-order functions and this comprehensions it's only in the third blog post that you get you know what I think is the real meat of functional programming immutable values and pure functions so for me functional programming is all about programming without mutable state right you'd hear perhaps intuitively that functional programming is about programming with functions but it's not strictly correct right it's programming with you know the way we define functions and python is not exactly the same as functions in a mathematical sense because within a function and Python you can actually grab like a global state and then modify something to it and you return something else and such as in a way that you know your your your output doesn't only depend on the inputs so it's not a pure function so it's not programming with functions in that way and if that's the line that we're gonna take with functional programming what is it that makes a programming language functional and here I would argue that programming with a mutable state must be one organ ohmic you're not fighting the language in order for you to do that and secondly it has to be idiomatic so this is the style that is encouraged when the community that you should write your coded and ironically if you get you know one of those blog posts that you found there's something called learn functional Python in ten minutes but inside there it's actually arguing that hey functional programming doesn't fit the Python ecosystem very well and here I think he correctly invokes that you know one of the lines of the design of Python you know if you do import this and Python one of the lines is that there should be one and preferably only one obvious way to do this and if you look at the functional programming constructs that are useful for programming without States you know sometimes it's not idiomatic and Python you know I'm talking about higher-order functions people don't really use maps filters at some point Guido wants to get rid of reduce partial so partial and compose you need to import and Pepe doesn't let you do you know lambda you know assigning lambda to a name you want to do the DEF you know and recursion is a mess in Python so in some sense you know maybe Python is not really a functional programming language that's my argument but why why even talk about functional programming in Python right so I feel like I need to really preface this I I believe that you don't need to do all of those like hundred percent immutable state I think we can be very pragmatic here take what we can and I think there are so bigger lessons and concepts that we can pour back you know sometimes side effects are ok you you don't need to be like castle you know so I want to get the benefit of functional programming languages while staying with idiomatic Python so not necessarily writing these obscure higher-order functions with that in mind I want to get to the very first thing about functional programming and that's immutability so immutability is this idea that once you assign a value a variable shouldn't change its value right and just looking at it like that might you know it's not maybe entirely clear what I'm saying so I like to have an example so this is an example and Python so I have a function G it could be somewhere else in the code base you're not exactly focusing on it so you don't know what it actually does and then you have a function f that you know presumably a pense 999 to or less so you declare your this XS 1 2 3 and then somewhere in there you you apply G to get YS presumably when you apply F to access you should get one two three nine nine nine do you think we can make that guarantee in Python no right because you can be really doing something nasty with G for instance I can make a side effect here and then instead you get three two one nine nine nine and if you think about that there's another way to do this right you you think it'd be equivalent to do negative indexing but it's not so if you do reverse it actually mutates that list whereas if you do negative indexing you actually make another copy of the list so there's performance consequences here but then you do get a vastly different results so here the list is mutable I want to contrast this with what happens in Haskell same program in Haskell right you get some G you get some F appending 999 you've got some excess and then here's why is applying G on XS and actually in Haskell you can always always guarantee that this is one two three nine nine nine it doesn't matter what G is this is because mutations are side-effects is controlled by the type system in Haskell we'll get into that in the next section but this comes in very very handy why why is it handy right like that there are a few benefits for you to writing this in an immutable way or immutable style firstly its predictability you know people in the functional programming community like saying your code is easy to reason about and really this is what they mean when they say it's easy to reason about it's predictable when you're looking at a function your output depends only on the inputs so that you don't necessarily have to look at other places in the code base to find out what exactly is happening there and composability so the name of the game and functional programming when you want to build complex B years is that you compose small functions together and you get complex behavior by building on these small functions and it turns out that side-effects is very very hard to compose and I you know if you put a side-effect there you don't you don't get this benefit and finally testability I'm a huge fan of TDD like I hate it when I have to write mocks like it's just not fun to set up so what does that mean in Python right how do we get this benefit in Python and I would argue that avoid non-local mutations if you can help it and what do I mean by that right like local stateful variables are okay say you declare a stateful variable in a function and then you make a mutation then and there and then you return it that function is still pure this idea is captured with even in haskell write something called this trick thread more not but it's not good there the problem is when you mutate something and then the effect can be seen elsewhere in the program so that whenever that thing is mutated someone has to keep track of what that mutation is and that's hidden so just to say like temporary local mutations are okay so this is okay right like I'm making a side-effect but it doesn't matter right here this so this is what I'm alluding to like you know you don't need to have this haskell strictness every time but you can still get the benefit one of my favorite quotes in the Python community is that we're all consenting adults Raymond had in here actually in the context of this is that you know when you're dealing with us like a third party classes you don't go inside the internals and then change things that because you don't know what you're going to be changing and so to the same spirit right like whatever you're changing then you're changing global state like you don't want to do that like for me consenting adults here mean okay let's let's try to avoid that as much as possible so don't make a non-local mutation that's that's number one so I want to go through the the second point is there is anything not clear is there any questions about immutability okay I think I hope that's clear so the second thing I want to talk about is this idea of functional core an imperative shell I'll get to it or what that means but I want to take a detour to Haskell so I I kind of left you hanging with some of Haskell right so I told you okay everything's pure you can't make mutation and there are no side effects but how do you even do anything you saw and hassle because if you need to print to standard out that's a side effect if you need to save to a file that's a side effect you need to send and receive request that's a side effect like how do you achieve this and has some right so there's something called the IO type the input-output type and really the side effects are controlled and it on by Haskell's type system and let me elaborate what I what I mean by this an IO type is it takes a concrete type to make another concrete type which bear with me so there's something called an IO of int just like there is a list of int what is a list of int a list of int is a list whose elements are integers that's what a list of int is a list itself is not a type a list of int is a concrete type an IO is not really a type it's not a concrete type but there's an IO of int so what does it mean to have an IO of int an IO of 1 means that you arrive at this value of 1 using some sort of a side effect or stateful computation so I can have an IO of 2 same thing two is just a number but I arrive at that number two based on a side effect so I can change this thing so I can +2 201 to get a three but if I have an IO of one you cannot get out for the type system doesn't let you to get out so you can get an IO of three from I of one right because well if you arrive to one we're using a side effect you cannot arrive to a three we're using something pure like once you're in you cannot get out and you can you can combine these functions together using what's called the IO by some examples just to crystallize this clearly put certain is a is a print function in tassel it takes on a string and it gives you an IO of nothing so think about it that this is like a print function and and and python right takes in a string doesn't it gives you none it gives you nothing but here you have to mark that side of fact that you've written to stand it out with an IO so it's an IO of nothing gate line is a bit like input and Python it gives you an IO string because it's a side effect people write something you cannot control that as a pure computation that's an IO of string the the last thing is what's called the IO bind but that's how say you've got an eye of one you want to change it to IO 3 that's how you do it or I go off like a string of one or whatever so that's going a bit deep right but this is a program that you can write that basically prints enter attacks you give it back something and then it'll just print it back oh excuse me so these two things are equivalent and the point that I want to make here is that forget the funny symbols they don't matter they really do not matter but Haskell makes it hard for you to compose IO actions once you have a side effect they're like you're making your life very very hard to do the next thing it's not as natural so what do people do in Haskell so they end up having this idea of pure inside but then the imperative like the stuff that you have really have to make a side effect as a thin layer of your application so you push out these impure code to the fringes of the code base so that what's inside is pure what's inside is just immutable stuff and we can do this in Python right there isn't a compiler to say hey you did it wrong but you can rely on your own discipline to say hey you know this is a kind of critique or logic so maybe I shouldn't be doing the side effects there so you can always delay the necessary side effects to push it off the core you do it at the end and sometimes you know we use Python partly because it has a really good third-party ecosystem right and sometimes they're very muted mutating so what you can do is a build a functional abstraction such that you only deal with your own thing which is immutable nice and immutable and then do the mutation right at the end I'll give you an example so recently I had to deal with this library called xlsx writer it basically just writes spreadsheets for you so this small little code here basically is just it makes a workbook you graduate and then you you write this label and value to the row and column that you like you write you can even write a string or a formula then you close it and you're done what's like a very simple spreadsheet problems all of these software side effects and as you can see like if you think of this as the entire application you don't have a functional core these things are inside the core so you have like an imperative core and it looks like it's not a problem it's it's it's it's easy enough to understand but the problem is that what if I want to shift everything ten columns to the right presumably you can do this again at simple enough he just +10 everything but this seems wrong right this seems wrong because well you're dealing with the business logic there you know you're you're putting your your labels your values there why does shifting it 10 places there has anything to do with it and why does it have to be conflated inside the business core that makes sense it should just be a simple modification of the code just you know I made this thing stops working there's another one here that's that's okay so it stops working if sorry yeah excuse me so you did all the logic there you just want to shift it to the right and that should be easy where's this you kind of you kinda have to get into the forest a little bit and then like find where you have to input it and this replication as well this is what I mean by saying come side effects side effects are hard to compose because essentially you have the state argument that's implicit it's not there in the arguments list but the global state is actually part of your it's part of your argument and this is like a just a very simple fix that you could do is like it's just a functional wrapper really the important thing is that you want to put you wanna write where at a coordinate and then there's a cell which just takes in a value and a coordinate and this is immutable right like this it's just numbers and you can also have a function that just takes in all the cells and then write it in one go if we do this I can rewrite that thing to look like this so I accumulate all the cells in one go I've got the items cost totals or whatever and then I put everything together and then I make the right there so that's an edge of my application only at the very end why does this make sense well if I want to shift everything to the right I can reuse this all cells like I don't change anything from the logic like it's one York one composition away and then everything else stays the same so that's kind of the idea it's it's it's nicer to test as well again no marks because you know you're just looking at the cells and the coordinates they're just there integration testing like this is easy enough but what if you're you have like 10 sheets and then 100 books like how you know you want to make sure everything goes together where I said there are just values I guess there and I think after this top there's property based testing that's something that you can write it goes very well of functional programming that's it's awesome yeah so this is my case for a functional core but imperative shell that's the entire series on immutability and how to structure a program I want to change to something a bit different now so I'm newish to closure this will please correct me if I make any mistake so this would be a lesson from closure so just an example code it doesn't matter what it says I took this from encounter which is data science and stats library for closure it's a list right so that means you work directly with the abstract syntax tree it has all the really nice parentheses you don't see any types so it's dynamically typed so what's so special about this language right what kind of lesson can we draw from it for our Python code so one remarkable thing about closure is that even though it runs on the JVM and you have all the powers and all the object-oriented glory of Java idiomatic closure encourages you to use this you know fundamental data structures so people typically use you know as a default list factors and maps like they don't create class on top of a class on top of class integral class so a Python analogy we did this before it's as if like okay like forget forget this like we're just going to do this like yeah but why all right so it's very hard to read a closure tutorial without coming across this quote it's Alan Perlis who is the one of the pioneers of computers right computer science it is better to have 100 functions operating on one data structure rather than having 10 functions operating on 10 data structures and this is a statement about composability right like if you have a hundred functions operating on one data structure you can arbitrarily compose them together so that you can make complex behaviors using small pieces that are tractable for you and closure has this wonderful functions that are out there already you can't really see this but these are the functions for lists vectors Maps so this is a closure cheat sheet and they have what's something called the sequence protocol so these are functions that work on all these fundamental data structures arbitrarily and agnostically so it doesn't really care what that thing is you can always get the first second last rest next and first on all these stuff and yeah it goes back to composition right if you have all these things that's enough for you to compose them together and you don't need anything new and I argue that you know this should sound somewhat familiar for for us Python users right because in some ways it's like kind of a duck typing right it's if it walks like a duck it quacks like a duck of treated as a duck and you know if you've given me a dictionary it has a row and column on it I'll treat it like a coordinate and if you have you give me a value and a port in and I'll treat it like a cell so as long as it satisfies these conditions you know you know I'm gonna treat it in a particular way so that's doable and Python but it seems a bit sloppy right like you've got like this nested dictionaries and all that like it seems sloppy it seems like you're you're missing something and yeah I mean we do do something right we first thing we lose encapsulation you know you can say that this is a coordinate and maybe you can have like helper methods that that you know maybe it shifts itself or whatever and you don't have to really worry about that internals of what iWhat actually is there everything just works whereas with the dictionary it's like you're exposing yourself to everything and discoverability you know this is the point of like static methods right you know people put related methods together in a class and if it's just a dictionary like you don't have these methods it's nice to know like you know these are sometimes bundled together but they can't be now so closures answer for like these two problems is like having well-defined well structured namespaces so it really goes back to the programmers discipline to put stuff in the right namespaces and then you're you're you know you know where to find stuff and I want to deep down a little bit on the this idea of validation though I think this is the interesting part something that we you know can actually change the way we write Python you lose the very little validation you have left in a dynamically typed Python right so maybe you can have a coordinate you can put 0 & 1 there but you can't you know you can't instantiate something without a column let's say but you can always instantiate an empty dictionary so if you represent a coordinate as a dictionary you don't get this check so there's a very nice technique that I feel is underused in Python but it's really gaining some ground some closure and that's designed by contract designed by contract as basically a bunch of asserts of preconditions and postconditions but structured in a nice way I'll give you an example so this is a function declaration that just increments something by one right and it's presumably it takes a number so that if you give you give it anything else you get these unuseful errors but you can spec it so there's a library called closure spec that allows to you that allows you to give it some contracts so these are pre preconditions and postconditions and as an example so this is a spec for for my function look the arguments have to be a number and the return has to be a number that's all it's saying so that when you call it with a string you get this nicer exception it says look hey you told me that it's gonna be a number but it's not so it did not confirm suspect it's nothing else then it's then an insert that has a nice error message to it and the idea is that look hey you get things for free if you do this in closure specifically if you spec something you get this generative testing for free so this idea of property based testing so you can generate a test example or test cases automatically and then you can check whether or not it conforms to the the function that you actually wrote it can also act as enforceable documentation informative error reporting and there's nothing stopping you from using the contracts to just define types but then you can do it better like say you want to have a list of just numbers but the list has to the lengths have to be bigger than six try doing that in Haskell you can't so sometimes it's more powerful right than static types but there's a catch all of these things run during runtime not compile time really what that means is that you the programmer is responsible for testing once you've laid down all the SPAC infrastructure so you know but Python doesn't have a static type checker anyway even if you do you know you you know even if you use my PI or PI err you can you can also combine it with this to get something powerful and luckily there's a there's this library called contracts and Python unfortunately it doesn't get many stars but do check it out and and do give the author some stars because I think it's amazing it's very simple it requires your preconditions so it checks what your argument should look like and ensure as a post condition so it checks how your result should look like with or without in or you know in relation to the arguments so this is a function that just you know we intentionally break it I took it off the the readme directly and the idea is that you know if you do this it works okay and if you do that like it gives you a nicer error and you can combine this with property based testing which is really really nice anyway that's contracts I want to quickly conclude I think I'm running out of time I've got 13 seconds always favorite mutability you don't have to go full haskell but you know if you don't do it unnecessarily structure your code so that you know as much as possible you want the core to be functional you know it's nice and you can compose nicely there and the the shell can be as messy as it is marine yeah within reason so finally try using this techniques I can't say that I'm using it myself fully but I you know it's something cool that I learned I want to share this idea of AI designing by contract you know when when I you know I pitched this talk there's so much to talk about right and really this functional programming is this it's a goldmine of insights that you know you can port back to to to Python and you know you know people say you know you you learn a new programming language and makes you think slightly differently I think it's true on you know with the condition that you learn a different family right if you learn Python like Ruby is just yeah it's not gonna do anything for you if you know Java and Haskell you know Scala's you know just maybe just the implicit learn Haskell learn closure Aida prologue address all these funky languages you know I I you know I I really believe that you do this even though you're not going to use it in production you're gonna extract something and it's gonna make you change the way you write your production language at work I put some references there and that's it thank you for listening [Applause] I also use okay so I think there's a functional API for scikit-learn I think that's out there but I not necessarily recommending you to do that might not be production stable look there's a time and place for everything right sometimes you know when you really need to do the heavy lifting you want to be standing on the shoulders of giants you want to have you know be using all the work that as people put on numpy scikit-learn and spark and tons of flow and pi torch right you want to do this and being a contrarian there even doesn't matter how beautiful category theory is how beautiful Haskell is you know I I would still recommend staying within that ecosystem for machine learning but you know for some stuff stupid stuff like writing to an excel like I think it's a bit of fun to do this thank you yeah Michael yeah so you mentioned ability right but generally bringing that were not explicitly made print as certainly trade offs that isn't we and a Kent so we're here at because they which is a cat-scan enclosure with other than language level to reduce as much energy as possible there's pretty strong make coffee and right semantics the data structure that Jade because I mean it's a change and everything else references in memory because when it's a beautiful venue folks you know you don't have attention so make tricks on clear price unworkable yeah right the traded constant memory for the mere memory for not really that much difference in expressiveness so we can leave those trade-offs always worth it in languages related the language core itself doesn't actually really because he said it's a right time and place step so asking you to believe he'd ever place in a language yeah of course like I can only say yes to that right there isn't a good persistent data structure library in Python and maybe someone will write it someday and we can all use it but I've always stressed that you know in your own Python you try to make it as idiomatic as possible as soon as that doesn't make sense you saw you don't do it in the case of trading these some performance considerations yeah I mean again that's pragmatism right like if you're gonna ride like a billion cells don't do that if you're gonna write a thousand whatever you know so yeah there's this you know I preface this with all the pragmatism that there is there's a time and place for these I find these things really nice to to Express yeah use it in the right time and I would disagree with you that it's not as expressed you know you don't get get as much expressive it see does that make sense but I think maybe we can we can check a little bit yeah I I took that example directly from the thing right so that's meant to be idiomatic way of you to use that library which is yeah not not fantastic in my opinion I think that's one more right since job the basis that they still face that I read Kassadin world thank you still day all exposed but I think it mentions yeah chocolate some way I'm getting my funding yeah it's a good question like I tried it myself I hate it the way it looks so you know when you're composing all these functions together it's you know especially higher-order functions one you need to do a lot of imports like partial is not built-in right reduce it's not built in anymore it's a hassle there is a few things out there there is a library called tools with a Z or Sai tools which is like a rewrite of tools with the Python some constructs that are interesting you get the pipe and all that which is quite nice and there is also a Python transpiler called coconut that's also very interesting though that seems to be a bit bloated nowadays but people are using it in production so that's that's quite interesting so I guess have a lot these but again like I wouldn't force the issue if it looks terrible

Channel: PyCon Thailand

Views: 3,466

Rating: 4.9545455 out of 5

Keywords: pycon, clojure, PyCon thailand, haskell, Anthony Khong, pyconth2019, anthony khong, python language, pycon thailand

Id: UgnCnEDrMM8

Length: 38min 38sec (2318 seconds)

Published: Fri Aug 02 2019