Using Cython to speed up Python

hi there Serdar Yegulalp here for info world at IDG today I'm going to demonstrate Cython, a system for compiling Python code to see for the sake of boosting performance as many Python users now Python is convenient but not always fast see on the other hand is fast but rarely convenient scythe on lets you get the best of both worlds scythe on provides a custom set of type annotations and syntax elements that extend Python and allow it to be compiled to high performance see existing Python code can compile as is with scythe on but probably won't gain much of a performance boost in that form however if you add scythe ons type decorations the resulting C code can be optimized enormous ly sometimes by many orders of magnitude scythe on is most effective when optimizing Python code that's numerically intensive like algorithms you can write a slow prototype of an algorithm and regular Python then converted to scythe on for production speed what's best about scythe on is that you can perform this conversion incremental e you can start with a code you already have and optimize its performance gradually instead of rewriting the whole thing from complete scratch to demonstrate how site on can speed up Python code dramatically I'm going to use a visual example a Python implementation of Conway's Game of Life for those who don't know it Conway's Game of Life is a mathematical game a simulation of a colony of cells as they grow reproduce and die off because Conway's life is mathematically intensive it makes for a good stress test of a languages raw performance here I've written two versions of a Conway's life program the first is in pure Python and as you can see it runs very slowly the numbers you see popping up there are a rolling average of the amount of time it takes to compute a new generation of cells around 3/10 of a second for a 400 by 300 average and that's after we did a lot of tricks to speed it up as much as we could like using a pre computed lookup table since that computation is almost pure math it's a great candidate for being siphon eyes now let's look at the second version of the program some of the key functions have been moved out into a scythe on file which uses a dot pyx extension when we use those scythe on versions of the function in our main program we just import them straight in as if there were regular Python objects that's another reason scythe on is handy it works with your existing Python apps naturally as you can see the scythe on version of one of the key functions is almost identical to its Python original except for the presence of type declaration syntax many existing Python constructions like the range object translate naturally into C provided we're using C types for the variables in question incidentally this function also makes use of another powerful scythe on feature many python objects like strings or arrays use what's known as the buffer protocol a way for a C extension to access the raw binary data used by the object this way you don't have to go through the Python runtime which is slow to get to it you can just access the data as a raw byte array we're making use of this feature here accessing the raw data of the self-life arrays so we can read from and write to them directly a bear in mind this can be dangerous you may need to make sure that no other Python objects attempt to access the same data in the meantime and you can always use safe object access for things like areas at the cost of some performance but this kind of power is available throughout scythe on whenever you want to reach for it before we can use the scythe on version of our life program we have to compile the scythe on components to a library module I've written a helper script here to do that if you're on Linux scythe on will use the C compiler available on your Linux system typically GCC or clang Windows users should install the Visual Studio C compiler available for free which is what I'm using here now when we run the cytha Knives version of Conway's life you can see a dramatic speed-up and the amount of time spent on calculating each new generation of cells is at least two orders of magnitude smaller sigh that also provides you with visual tools to get an idea of how optimized your code is when you compile a scythe on module it generates an HTML report file alongside it lines and yellow indicate that the code isn't fully optimized because it still references Python objects in some way if you click on a yellow line you can see the underlying C code the functions where we do most of our hard work generation and render are almost completely optimized the only unoptimized code is in the setup parts of the functions which we can't do much about and they don't run in tight loops anyway so they're not a major source of any bottleneck but up here in the init function which sets up some of the fast look-up tables we use to compute each generation of the game field we see a lot of loops and a lot of yellow that's because this function is more or less a straight port of the Python version with no type information added to it if you don't add any type information scythe on assumes the variables in a function are just generic Python objects resulting in slow code to speed this up all we have to do is add explicit type information by way of these two lines here and as you can see after recompiling the function is now pure see there's a great deal more to scythe on than what I've demonstrated here like interfacing with existing C or C++ code or using multi-threading but this demo should give you an idea of how scythe on gives you the power of C and the convenience of Python together if you have questions about this video please add a comment below don't forget to subscribe to The IDG Tech Talk channel on YouTube and for more Python tips be sure to follow us on Facebook YouTube and info WorldCom

Channel: IDG TECHtalk

Views: 86,328

Rating: 4.9415112 out of 5

Keywords: Python tutorial, Cython, Python performance, Python speed

Id: 8DuyATDaIdM

Length: 5min 44sec (344 seconds)

Published: Fri Apr 03 2020