Cython: Speed up Python and NumPy, Pythonize C, C++, and Fortran, SciPy2013 Tutorial, Part 1 of 4

on tutorial given by Kurt Smith so occur it is a soft pick software developer Alan thought and also a very regular and experienced trainer and he has a PhD in plasma physics from the University of Wisconsin and he is also somebody who knows scythe on extremely well for contributing to it and so he's going to show you some of the all things that have been in there and that are really powerful and some of the new things as well so I'll even the floor Thank You Jonathan thank everyone for being here and I really appreciate everyone going through the possible pain of verifying that siphon will work on your system before this tutorial really appreciated I'm encouraged by the relative lack of support emails that I got that I think everyone's in pretty much good shape and I was able to get almost everyone in working shape except for maybe one or two that I was unable to get to so um at the exercises I can certainly sit down with you and hopefully won't be too many whoever system that needs need some TLC too to get in shape so really make things go smoothly today so I'm Jonathan sided I've contributed to scythe on and I did it because it was a project that was just fantastic really liked it the idea of it is great just taking Python code allowing you to optionally add type information and speed it up I used it for extensively for my graduate work I did high performance massively parallel turbulent simulations of Astrophysical plasmas and the core code was all in MPI and Fortran and everything before that was Python and scythe on and everything after that was Python at scythe on so on the front end was all Python really and in the back end all the data processing all the everything else was was Python and scythe on for speeding things up so I found it to be a fantastic combination for doing science and I'm extremely encouraged by the activity that's going on improving the toolset for doing scientific computing in Python so I thought is as you'll see here it's becoming more and more part of the infrastructure of doing scientific computing and Python and we'll see why that is hopefully by the end of the day today all right so let's get started here so I should mention I'm fully aware that I can run in presentation mode and PowerPoint except when I do you and I flip back and forth between PowerPoint and anything else about the fifth time it crashes on me so I just leave it in presentation mode and I hope it's large enough for you in here so that's what I'm gonna stick with so just get to the core of what we're doing today on the Left we have a simple Python function in computing the Fibonacci sequence okay and it's using some nice Python goodies like multiple assignment a comma B equals 1 comma 1 for I in range n we do a comma B equals a plus B comma a and if you haven't seen this before it's one of great pythons great ways of doing many things on one line in a very expressive way so I'm going to take this function and I'm going to put it into an eye Python notebook oh geez all right so here's my fib function and fib of 10 gives me let's just do fib of 3 just to keep it simple we get 5 back now the point of Saipan is to take something that starts out as Python so we're very expressive with it and add optional type information to it okay and eventually get down to something that looks like this C or C++ or at least conceptually is equivalent to the C or C++ on the right here okay so on the right we have the equivalent C C++ and then down here is how you would send conceptually merge the two you take the Python on the left the seriously plus on the right and you add the type information from the C to the Python syntax on the left and you get this able to stick an INT in the argument signature so it looks like a c integer now and even though this isn't valid Python anymore it is valid scythe on and you and I know exactly what is intended here even though the Python interpreter doesn't write this line here is essentially a C declaration where we say C def int I a and B so we're declaring I and B is I a and B as ents the seed F is just a scythe on new scythe on keyword but it's you can essentially ignore it it's just helping the scythe on parser figure out what you're declaring but that's all you have to do and so I thumbs able to take that code and generate essentially this C and C++ code on the right here okay so that's a core of what's going on now you may be wondering about run time and if the Python version runs at one unit of time then the scene C of us plus version is going to be 70 times faster the scythe on version is also going to be 70 times faster if you run this okay so we're getting the exact same performance out of the scythe on code that we would expect have the seen C++ version as well now you may be wondering what about what if I were to do this all in hand what if I were to take a C extension create my own see extension module in C and use all the headache and pain of pythons dis to tills and get all the right compiler flags right and have to worry about all the details of the Python C API to get everything working right this is what you'd have to do for that simple Fibonacci function now granted this isn't that much code this is a complete C extension module and it will work if you take this and stick it into a C file and compile it properly but it's still a lot of stuff there and you have to know all the details about how to use pi r parse tuples correctly how to use pi build value right how to create your own PI method def table how to call pi PI nib module right those sorts of things so scythe ons taking care of all this for you and the really surprising thing is that this handwritten C extension module is only 40 times faster not 70 times faster like we saw with the the scythe on version here you may be wondering why in the world is scythe on faster than a handwritten C extension module the answer is because scythe on is generating some heavily optimized C code for instance scythe on doesn't doesn't even call pyro parse tuple it creates its own it does essentially the same work as pyro participant arts to blow up in a much faster way that's just one example so when you're using scythe on you may be thinking well I make I'm having making a compromise between Python and C but in in fact you often get code that's faster than if you were to write it by hand okay and so that's one thing it's not just you don't you don't have to feel like you're compromising and performance you're actually getting better code then you may be able to write yourself okay so that gives you just a taste of what's going to happen here today so as the ability to take Python speed it up by adding time different type information let's talk about what exactly scythe on is and more formal sense so I like to think of scythe on is a Python like language that is designed to improve pythons performance by adding type information and it's not uncommon to get three orders of magnitude or more speed up especially for numeric programming and it's not necessarily that scythe on is that good it's just that Python is that slow at numeric programming okay and so all you're doing is you're just tapping into the the performance of native types for that you know I read able to fit inside of a single register rather than having box types that pythons provides like box integers where our boxed floats or boxed complex values for example so pythons got a lot of stuff around all the core data that's representing that integer that float or that complex value scythe on does away with all that and it you get much better performance as a result you can wrap external code with scythe on that's the other main usage and you can wrap C C++ Fortran anything else where you know the ABI for that language the application binary interface you know the name of the symbol that's in that object file you can wrap that with scythe on okay and these are the two ways that scythe on is most commonly used for wrapping external libraries and uh giving you a lot of control a lot of power to be able to give a very custom very nice Python interface and speeding up existing Python code okay we'll be going over both of those today there's a scythe on command that comes with the scythe on source package all lowercase scythe on and that command it's job is to generate an optimized C or C++ source file from a scythe on pyx file that's it so all scythe on does is the source to source translation tool it takes scythe on source and it generates C or C++ source for you and it's up to you to roll that generated extension module into your infrastructure thankfully um there's several other tools that make this even easier so we don't always have to worry about getting compiler arguments just right that sort of thing and we'll we'll see plenty of those later on today and as I mentioned here the C++ source is then compiled into a Python extension module and so scythe on it just does one thing but does it really well and it's up to you as far as how you want to use the results of what site on does for you so some other big features that are good to point out about scythe on it is built-in support for numpy there's a lot of numeric users of scythe on and so they paid a lot of tension to it and suicide on has first-class support for working with numpy arrays it integrates with ipython we'll see that just in just a few slides here and the whole idea of scythe on the whole name of scythe on is combining C with Python so and it doesn't a very fluid very flexible way alright so how are people using scythe on well question yes um we'll see in a little bit that there's nothing automatic about siphons rapping you have to do everything by hand although there are separate packages coming out I'll mention at the end today that takes care of some of that automation for you okay but yeah there's nothing automatic about it there's another question yeah here good question so um again it all depends on what you're rapping but it's psyphon so swig by design will generate a compiled extension module and a dot py file and when you call the wrapper and swig you're calling through that Python file and then that calls into your extension module so there's an extra layer of Python between the compiled code and swig and the your code that's that's calling it so by design there's there's an extra layer there that may or may not make a big difference for what you're doing but it's generally scythe on if you just look at just the wrapping layer scythe on is generally faster than swig so for simple stuff they're going to be almost identical in performance for more complicated things scythe on will usually went out by you know a measurable margin okay all right so just spent half an hour this morning looking at some of the fundamental packages for scientific computing in and I thought I don't have to explain any of these packages here you all know what they are and just looking at how much syphon they use and so we have numpy sai pi pandas sympa psychics learns like it's image sage and NPI for pi we just tallied up the number of scythe on files and that includes py x files py d files and PX i files and then the total number of lines of code and thousands okay so numpy i was surprised to see the number i had any psyphon code since its to my mind it's kind of a bootstrapping problem but um you have siphon that uses numpy heavily but numbers using scythe on to some degree so um but anyway it's it's there I was really surprised to see a little bit so 5,000 lines or so Syfy uses it more about 18,000 lines pandas uses it heavily for a lot of its optimization pandas uses underneath the covers something called I think it's called bottleneck or fast bottleneck something like that and it uses scythe on Eveleigh as well to speed up numpy simple use --is it tremendously the one that I should really mention here is sage sage is driver of tons of Python development sage well Robert Bradshaw was at University of Washington he worked with dr. Stein on sage project and a lot of Robert's work was thinking that the lead on the syphon project he's still the lead there and so sage drives a ton of South on development and you can see here it uses a tremendous amount of scythe on for what it does so definitely so I thought would not exist today without the support in the background the support in the development time devoted to it from sage Leslie MPI for PI if anyone's doing parallel programming it wraps the MPI library using scythe on so if you're not using scythe on directly you're using it indirectly given that it's becoming part of the infrastructure doing scientific computing in Python before I continue what time is our break so I can make sure a time things 1033 thing three okay thanks B four three okay okay that's that's a strong incentive all right so if I don't stop by three you guys are just gonna get up and go anyway so okay before there it's gonna three thirty yeah okay thanks all right so again the two modes of using scythe on one speeding up Python so we have our fib function we add our type information there with scythe on that's your job you add the int information there the CF compilation line you call the scythe on tool either indirectly or directly and it spits out this generated C code for you I'm not going to show you the generate C code it's quite ugly a lot of mangled stuff in there but you can make sense of it if you need to and it actually I learned a bit about optimizing C code from looking at how they optimized the C code generated by scythe on the other direction you have existing compiled code C C++ code here we have a little function called fact and just recursively calls fact again to compute the factorial very simple little function it's nice it can fit it on one slide here and then we have scythe on code that's designed to wrap this simple little function fact okay and we'll go through all this in more detail later and here's just four lines of scythe on code we just tell scythe on what the interface for this C function is and then we create a Python function that will wrap it and then scythe on again spits out a generated extension module and we compile that and it compiled along with our C code and we have now our extension module that's wrapping our factorial function okay so those are the two main modes of working with scythe on the two objectives and there's this new mode that's come along fairly recently last couple of years that you can use scythe on very interactively VI Python okay and there's a scythe on magic percent percent site on and we'll see how to use that here so if you don't already have an eye path on notebook up and running I to do so now sighs here so in order to use scythe on ni Python dynamically first thing you need to do is tell ipython that you need to pull in the scythe on magic so what you do that is by percent load X underscore ext scythe on magic like so if your system is able to run that line you're halfway there if it isn't means you have an old version of either I Python or scythe on and and things aren't working so you need to update update either or both of those and then now what that does is it provides several scythe on magic commands when we're going to focus on today is the percent percent sight on magic and if you stick percent percent sight on inside of a ipython notebook cell the code now ipython will treat this code is scythe on code you're no longer in Python code you're in sight line code so we can now say def fib int and I can do CDF into a B I it gamma B equals 1 comma 1 and for in range and a comma B equals a plus B a a ok now the moment of truth is when you evaluate this cell it'll take a little bit but if it works you will now have a compile but what's going on is ipython will take this code will stick it into a source file somewhere compile it into an extension module load that extension module and load in the essentially do from extension module import fib for you so now we can save fib let me get this function this is the name of that module that it just created dot fib okay so psyphon is doing a lot sorry I pythons doing a lot here for us and the really great thing is that we just care about the scythe on code we don't have to care about the compilation side of this whole thing so it has this very dynamic feel to it which is just fantastic for interactive use of sight line and trying things out of course we can call it fib of 3 for example so I realized after I wrote my slides out that I use intz for everything here and fibs Fibonacci sequence grows really quickly so you quickly run out of room so don't expect this to be a very good implementation Fibonacci sequence but it's there as far as the the idea that all it works right we could make these lungs and set events to have a little bit more room to play with here okay ah good question so let's try it out let's do time it fib of 3 so turn a little bit and it gives us 67 in a second 68 nanoseconds / / / call okay let's create a Python version of this PI fib and a comma B equals 1 comma 1 it come B equals one oops B okay so there's my PI fib Oh P grade C identical coded the scythe on version - the typing information so we've run that and that will do time it - of three okay it'll turn a bit and eight ten nanoseconds so more than an order magnitude speed-up okay now this is very much a micro micro performance measure but uh as you saw before this code sight line generates is heavily optimized and this makes a big difference especially for the wrapper layer when just getting your Python code to just get to the underlying C code there's a lot of Python between there there's a lot of calls to PI R parse tuple there's a lot of checking of argument types that sort of thing Seton's and a lot of work to make that as fast as possible while still being correct and so that's often where the payoff comes especially when you're doing calling functions out of our mostly just function call overhead and not too much work in the body so you can just C++ as well we'll see that yep you can do template we'll see that later on yep that's another tutorial I don't I don't know enough to be able to tell you so yeah I wish I could tell you I don't I don't know enough to be able to be able to say I'm a big guy so until I Python has been inside of the cells I'm not going to use it that much so all right um let's continue here so um I should mention that if you for whatever reason you aren't able to use the ipython notebook or you just prefer the QT console for example or just the terminal version of ipython this will work in the terminal version as well you can do percent low text siphon magic and then percent percent site on gives you access to the same siphon magic there okay so it's this is a lower-level ipython thing and we're just I'm just showing it to you in the ipython notebook just one way of accessing it okay okay so that's ipython so again I'm just showing you different ways of working with psyphon and and the different ways it's intended to be used the performance um it it will be to some extent I mean you are you're getting down to the sea level of code so but for that matter I mean your Python is going to be somewhat however dependent as well oh so they're almost almost the same I see yeah for that I wouldn't be able to provide much guidance I'm not sure I'd have to look going to more detailed if to figure out why why that is the case okay so just a flow diagram to just help understand the the process for working with psiphon source files so the extension for silent source files is is conventionally pyx and that's a holdover from whence iPhone was something started out as a project called Pyrex sort of by Greg Ewing and so I thought fort Pyrex but it's a friendliest fork in the world where Greg helps out the scythe on guys and scythe on guys help Greg out as well so the the reason for the fork is that scythe I wanted to be more general and be more aggressive and optimization whereas Greg was more focused on very just one of very specific things in PI racks and wasn't willing to accept some of the more edgy features at scythe on hats so but scythe on kept the pyx extension and so you put your code typically in a py X file your scythe on code and py X file and you call the scythe on command either directly or indirectly and it spits out a C extension file okay here if it's the scythe on source code is fib WI X it'll spit out fib see and then again it's up to you or some other tool for you to compile that see extension module into a Python extension module and on Mac or Linux it'll be called fib that Esso and on Windows it will be called fib PYD okay and if you happen to be wrapping code like we saw earlier you would compile along with that fib dot c NEC dot C files that you want to wrap okay so it's all going to be wrap contained inside of that's extension module and then you can just load that into Python is if we're regular Python module go on your merry way okay so let's go into that a little bit more in a little bit more detail so at the top we have fib that pyx we just stick our Fibonacci function in there we've seen that enough times now and then at the bottom here we have set up fib py and for those of you who haven't seen Python dist utils setup files this is a very common pattern that you'll see you import multiple things I never remember what these are I just find out pull someone else's from the web somewhere and modify it and the core of this is just pulling what you need and then you create an extension object with capital e extension and you give it the extensions name here fib and then you give it a list of sources and you include in there any pyx files that you need to use and also any dot c files or dot CPP files for C++ if you need to and dis you tools will take all that those source files and compile them into an extension module for you okay and the way you fire that office by calling the setup function using this syntax you give it the extension modules you can have more than one extension module here which is why you give a list of extension modules and this command class is just tying in scythe ons ability to run the scythe on command automatically for you so that's all that's going on there so this is kind of boilerplate that you use for using pythons distritos to compile scythe on source code for you so let's see that in action here so I should mention that if your you should have unzipped that directory and you'll have believed its name Sai Sai PI 2013 scythe on tutorial you should be seeing about the same thing that I have here perhaps not the untitled IP and B that the one I just created recently so let's go into demo and fib there's a lot of stuff here here's our set up top PI file and here's the same code that we have saw on the slide okay so we have our extension extension object and then we're calling setup with that extension module okay yep yeah that's a newer feature I've not used it much this way will work this is the older way of doing it that's that's got a little bit more sugar for automating things so you don't have to pull in as much boilerplate to make that work but what I'm showing you here is just kind of a older way which is which will certainly be around but you can look in the online documentation for the newer way of of doing this sort of thing okay so that's our extension module we want to now what we're going to do is going to tell Python to build this extension module for us from the command line okay the way you do that is by make sure the right python loaded okay so I'm going to say Python the name of that extent that this utils driver scripts setup that PI and then you say build ext and then lastly - - in place okay so that's my line on Mac or Linux on Windows you're going to give one more argument - - compiler equals min GW 32 okay that last option there is only for Windows you do not put that you do not use that on Mac or Linux so here's Mac and Linux and again for those of you haven't used as utah's before just briefly what's going on here is we're calling python with this script the build X is an argument to the setup that PI script or telling dist utils what to do we want you to build an extension okay the - - in place is saying this utils stick the result of this operation in this current directory don't stick it in some deeply nested build directory somewhere I want to use it right away so give it back to me when I'm done so you run that what good yeah good ah I don't know enough about the siphon magic to be able to say I think I looked into that briefly and the I don't think you can although I'd have to play around with it to make sure does anyone here know - the answer can you set the compiler directly on this site on magic command I apologize for not knowing this I just don't use Windows enough to need it um I presume you got an error when you try to use a scythe on Magic on Windows is that right okay I apologize um I'll look into that at the break actually in let you and get back to you so from the command line when we run this you'll see a bunch of output the one that's of interest oh let me get rid of some things here first see so I need to get rid of the build products first to force it to actually do what I'm expecting it to do there we go so here it tells us that it's siphoning fib pyx to fib dot C so there that's calling scythe on underneath the covers for us to generate our extension module and then pythons just utils kicks in and takes that fib dot C compiles it that's what this line is here this GCC call right here into a dot o file and then it takes that object file and creates an extension module we'll fit that s o for me to use yep question yeah sure so it's the same as this Python setup that PI build X built underscore X dash dash in place and then dash dash compiler equals min GW 32 okay pushing ah well so let me leave this up here just a little bit the this filled that I so that's the whole goal of this is to give us this extension module the in place tells digital's put that extension module in this directory don't stick it inside of your build directory where you would otherwise that's correct yeah and that was those part of the setup earlier so if you have anaconda that should work I've tested it if you have and thought canopy it should work as well if you've installed the lib Python and the mingw packages question that's correct that's that's all very specific as far as how it's tying together what what commands you're allowed to give to to district Hills that's right okay so if that's successful on Mac and Mac and Linux you should see a fib dot SL as a result of this command and on Windows you should see a fib PYD as a result okay so fire poops fire by python and import fib safe fib now it tells us it's module fib from fib that soo so it is indeed loading this extension module force to use and I can say fib dot fib this is of course our function I didn't put a doc string in there but it tells us as a built-in function or method and you'll see that if you do lend question mark for example or I don't knows if you do string dot stir dot blanking right now our strip or something all these built-in functions are methods that Python compiles into the language itself you'll see that same type for those so you see siphon is now giving us the ability to compile something basically into a lower-level way accessing the I don't States compiling into Python itself but it's it's creating built-in function or methods for us rather than just regular Python functions or methods okay yeah that's correct yeah thanks for mentioning that so I should mention that this line that we had to do here when we're doing it manually this is essentially what ipython is doing for us when we use the % % site on magic ok so that's the that's a big benefit there question it'll it'll show up here yes mm-hmm that's right question hmm interesting I've not seen that before what does it do are you are using regular Python or i-5 that's correct yeah the the it will be able to introspect docstrings but won't be able to introspect the function signature you'd have to put that in the docs training manual yourself if you want that to show up that's right okay so far by Python let's just test it out here import fib fib fib call with no arguments to see what you expect fib of fib of string tells us an integer is required now this exception is slightly different than the exception you would get if you had the equivalent Python version of this function defined when you let's just look at fib here quickly again so here fib it takes an integer as an argument and when you call this function fib is going to restrict that argument type to something that can be converted into an integer right there okay so you're statically typing N and you're going to get an exception at that point in your in your process so it's if this were pure Python function you obviously couldn't type the argument as integer n here and if you passed into a Python version of fib a string it would take that string and keep on going with it without any problem and the only point you would get in errors when you try to multiply that string by a float or do something invalid with that string later on so you're going to get much further in your Python code than in the scythe on code because scythe on is now restricting the types of the of the objects you're working with okay so there's this other way of working with ipython that's also very nice and called pix import and what it does is it allows you to import a scythe on source file as if it were a pure Python module and it looks and it detects changes in the scythe on file and it recompiles it if necessary and if it's if it's already compiled this same version of an extension module it'll just load that cached version for you so it's great for very simple cases you can't use it for wrapping stuff unfortunately but it's again it's it's great for just doing very simple stuff so let's give that a try here quick actually it's just a time I'm going to show you that in a little exercises coming up here in just a little bit but I should mention at least the the syntax you use for it is down here you import picks import this comes with scythe on so you have it ready to go and you would say picks import dot install so that basically allows picks import to hook into pythons import mechanism and now when you import something picks import comes in and says are you trying to import something that's a py X file and if it is then picks import will intercept that import call compile it for you into extension module and then load it in as if it were just a regular py file so it's totally transparent now whether you're working with a scythe on source file or Python py file question um it's a good question I believe the dot py file reference I've never done it before because we just confused me so I've never tried but uh so again this is really nice just a couple lines you can stick in there and now you can treat a fib that pyx file as if it were a fib that py file and scythe on will be called for you automatically okay there's a bit more work to be done if you're on Windows to basically tell it which compiler to use but we'll see that in the exercise here actually the next exercise the sync exercise we'll see we'll see that example okay so the very simple hello world example sounds like number of you have been following along alright but those of you haven't you're welcome to try this one out so again inside of the zip file there is a exercises directory so go in there and then there's a hello world directory and a few files here okay now there's a ipython notebook so if you're hypothenuse working you can just say ipython notebook scythe on hello world i py and be fire that up and you should see this come up and again all this is doing is just reiterating what we saw earlier today so just some description here we're loading in our scythe on magic create a very simple add function using scythe on just takes two mints and it returns some polynomial those ends here's a python version of the same pi add and then lastly we are timing it hope they didn't go too fast the going to show the so you know inside of my exercises directory hello world and I just ran ipython notebook scythe on hello world i py and be ok I'd fired up this ipython notebook and I'm just now executing these cells one by one so now we're timing to compare the two different versions the first ones Python this sorry the first ones scythe on add second one is Python PI add you see it's about factor two faster in this case so not huge speed up but they're actually for this case most of the work is done just at the calling Python to get to the C so there's just a lot of wrapper work being done here just for kicks let's just create an empty function so psyphon def no op takes nothing and it does nothing just to see what the wrapping overhead is for cyclin so the fastest possible function you could ever wrap a scythe on and we're just going to see what the overhead is for calling a do-nothing function so we do time it now no op run and so more than 15 of those nanoseconds up here is just calling a is just a wrapper stuff okay and if we give it two arguments and B just to make it a little bit closer to totally high going on earlier one to try that out see if there's much difference so they're 120 nanoseconds is just for taking two python integers converting extracting the underlying c integer and calling it do-nothing function okay rushon mm-hmm so let's write an O op in pure Python no op PI a bee pass and time it's coop PI one to get that one a try mmm just just in that cell so just in the cell is cycling code out once you exit this cell now your index cell down that's now Python code again okay excuse me um and yeah oh yeah sure sorry so the first question was um the % % site on magic it's limited only to the cell that you're in the question was does that turn on site on for the rest of your ipython notebook the answer is no to that it's only in that limited that one cell and then your question was how do you how do you limit scythe on just an empty line oh right right yeah in the command line that's right so if you're if you're interactively using ipython in the command line let me show you briefly back here um you have to give it a complete block of code and once you exit that block of code now you're back into the regular Python mode okay um see so here let's look up here at our original timing okay so here's our original timing where the functions are actually doing something add the scythe on version of add takes 124 nanoseconds the python version of add takes double that okay now our no op versions are the same you see a no op version of the scythe on version takes 120 nanoseconds so basically all the time here is being spent just calling just python to to uh to see conversion that's just all in the wrapper layer the actual body of this function is almost for free so of course these are very trivial functions if you're doing something anything serious the body of the function will be doing more work so this wrapping overhead won't won't be as much of a hit all right so that's the ipython part of this exercise and it's a very simple exercise just to make sure people on the same page so let's quickly go through the ups where am I so that let's look at the pics import part of this so let's look at use pics import py again I'm side of exercises hello world so here's use pics import and here's our pics import boilerplate it's very simple import pics import and then pics import install that's all you have to do should say that's all you have to do on Mac and Linux I apologize I didn't get this the Windows version of this and this exercise but it is in the sink exercise a little bit later on so just dis bear with me here for to see how to do that so now we can say import scythe on hello world and that's a py X file and now we can you just say prints iPhone hello world add one too as if it were again it just feels now just like a regular Python module okay so here's Python hello world that fix it's just our add function inside of a py X file okay let's try that out Python use oops story use fix import it runs and again it just runs as if it were a regular Python source module now what I'm going to do I'm going to make sure I get rid of any cached versions of this that I may have compiled before so I'm going to remove this directory seven by default we'll stick the pics import build products inside of this dot pyx build BLD inside of your own directory so I'm going to get rid of that whole directory to force pix import to recompile for me ok so take my word for it it just recompiled and and rent imported the extension module and ran it okay all right so that's the pics import part of this lastly let's let's do it all by hand so we can we have a setup that py file so here's our boilerplate importing what we need from dis to tools and siphoned out dis to tools and we create an extension object here we give it the name of the extension scythe on hello world and we give it a list of sources here it's just one pyx file later on we'll see pulling in more more stuff into that and then we have our boilerplate setup call ok alright so let's run this again the magic incantation is python setup that py build ext - - in place like so that's on Mac and Linux and Windows is - - compiler equals mingw 32 ok so get rid that line so my Mac so run this and first thing again you'll notice is it's siphoning size and other world to scythe on HelloWorld dot C and calling GCC for us to create our object file and then compiling everything together into our extension module ok pick simpler um there's several yeah you have much more control over using it this way if you have a larger project we have several scythe on source files you might have control over let's say you want to give someone an extension module but you don't want them to have to have scythe on installed to use it then you would use something like this to generate the source source code for you compile it yourself and then ship it off to them with pix import you have your users have to have scythe on installed and just another dependency that sort of thing there you can be more certain that this version will work because it's using lower level Python stuff and the pix import version might not work based on your user not having a proper setup so but if it's just for your day to day work for your own research for just speaking up your own code definitely go with pix import that's going to be a lot easier to use if your interactive work go with the percent percent site on inside of ipython notebook if if that's working for you as well another question yeah it does this two tools comes with Python so that's that's built in that's ready to go um you know that's a question I think it I'm pretty sure it does no no you don't need to write this at all it's essentially generating that file for you and doing a lot for you the only thing you need is this import picks import and picks import install that's it alright so that was our first exercise again just very very basic just to make sure we flush out some of the basic ways of working with scythe on okay we'll get we'll see more interesting exercises here in just a little bit okay so let's go a little bit more formally now as far as what's going on how do we how do we declare C stuff using scythe on well typically what you do is you just tell scythe on I'm in declaring something as C so I use C deaf okay and CF is a new keyword that's a new scythe on keyword and the way you use it again is if you want to declare local variables you say seed F and then you declare after C to F as if it were a sea variable okay leaving off the semicolon it's going to be just as if you declared this as an int and C correct that's right C Python and int C Python edges as a sea long whereas this is a true C integer here if you wanted to long here you could say seed of long and that would be a C long okay so yeah this is not a Python int at all this is definitely a see it okay whenever you see C tough anywhere you are making a C declaration okay so you're working with C level things okay if you want you can create I should mention that this function fib is a Python function generated by scythe on okay so it can be called from Python it acts just like any regular Python function that's because it's a regular def function if you want you can create a c function in scythe on using CDF instead of Def for the function declaration so here we're saying CDF distance and we give it a return type float here and all this is doing is taking to see arrays pointers to see arrays with a length n and it's just inside of the body here it's just giving the distance between these two to see vectors okay and the think the way you should think about this is when you create a CDF function like this you're declaring you're writing C with a Python like syntax okay you're no longer writing Python code anymore this thing that you're creating this distance function is a C function it's not a Python function anymore and it's not available to call from Python it's declared static inside of your C source file that's generated so it's only visible to other things inside of that source file okay if any way you know anything about the static keyword and C and there are reasons for that but the important thing is that you're writing C AC function using Python like syntax so the name of this function is distance it returns a float and it takes a pointer to a float as X pointer to float is Y and then an integer n okay and of course there would be no way there's there's no Python analog to a C pointer right so you wouldn't even if there's some magical way to make this call from Python it would not be obvious at all how to make this work out of the box with with Python with a function signature yeah you can absolutely say seed avoid and have a void function that's perfectly perfectly fine but yeah whatever you can do and see you can do here it's just got a little fun this kind of like a Python like syntax for declaring a c function okay we'll see in a little bit how to do structs and unions and that kind of thing question correct yeah we'll see in a little bit a way around that okay using a new another keyword I just want to get through C def here in a little bit so the function arguments are float a flow pointer float pointer and an integer those are also see declarations you don't stick see def in front of those though okay so it's it's one place you don't use CF for declaring a sea level thing now here's another way of using CF inside of this function we do see def : and an inside a an indented block we have all of our declarations as if they're in C that's equivalent to doing two separate lines CF into I and then C to float D equals zero point zero okay so those two options are equivalent I when I have a big block of C declarations that's I kind of prefer to do that just because it groups things a little bit more nicely okay but they're equivalent lastly down here just want to put it in here for completeness we'll talk about in more detail later we're declaring an extension type here with a CF class particle okay and we want all details of extension types in a little bit but inside of these extension types you can declare things at the class level as CDF and that has particular meaning as well okay and again we'll we'll go into that later I just want to have a one slide all the places you would use the CDF keyword ah you know scythe on can generate c99 I mean it'll generate C code and I think it supports c99 features as well I have not tried it yet so I'd have to look into that but I would see no reason why it doesn't I know it supports like c99 complex variables and that sort of thing so it'd be worth looking into okay so here are three versions of a distance function using deff CDF and this new keyword now see PDF we'll see what that is in a bit all so the first version here is just pure Python and this is also valid psyphon okay so scythe on is a superset of Python you can anything that's valid Python is valid scythe on as well okay so this is Python or scythe on if it's if it's in a scythe on extension module you can just call it from an external Python module somewhere okay so this is available to the Python level this is our C F load distance again this is not available to external Python code this is only available to scythe on code inside of this source file or inside of the the ipython cell that you happen to be defining this in down here is a see PDF function and see PDF is a new scythe on keyword and it's designed to essentially get around this limitation that C def functions are only visible inside of the source file that you're you defined it in so cpf what it does is it actually creates two functions one is a C function only and if other psiphon code calls that function it will call the C function and then it creates a very thin Python wrapper around that C function and that makes that C function available to the Python level okay so it's creating two functions for you a c version of this function and a python version of this function but C PDFs are a little bit are somewhat limited in that you have to give arguments and return types that can be converted to and from Python types okay so our distance function here that took two C array pointers you couldn't take that and just declare to see PDF that wouldn't be allowed because in cyclin doesn't know how to take something and convert it to a C array out of the box okay uh it's been around for a while now much as long as I've been using it which is five years or so for five years at least I'd say um the question was how new was CPF it's been around for for a number of versions of scythe on um so what I had to do is I had to take my C array pointers here and convert them into typed memory views okay so this is a little teaser for the later part of this course here so what are type memory view is well we'll see how to use those and what they allow you to do later on but this now essentially gives us the ability to work with numpy rays or buffers of memory that sort of thing and it's compatible with Python so it'll work okay all right so def functions available within siphon and Python but they have Python overhead so they're a little slower CDF functions are just pure sea things so there's almost no overhead they're going to be very fast but the restriction is that they're only available within that source code and lastly cpf functions create two functions for you a Python layer around a C core okay scythe on will call the core the seek or external Python code will call the Python wrapper for you so you get the best of both alright so here are some examples of using these deaf and sida functions I'm not going to go through this in great detail and the interested time but just suffice it to say we have a function here that's calling a smaller function Inc and here Inc is a def so you can call Inc outside of this size own source file instead of our pry thon interpreter here and it works over here we've created fast Inc and it's a CDF function so it's now available only within this pyx file so own fasting seq can call it but you cannot call fasting outside of outside of the pyx file okay then here's a version using CPF and just illustrating that you can use fasting outside of this source file as well as inside ok ah there will be no slowdown when calling it from scythe on because it's going to call the C version of that function it will be slower from the Python there because it if you call from the Python interpreter it's going to have to go through the Python layer to do to do that right still is going to be slowly a Europe whenever you're calling from but so the question is that as speed implications of of calling this to me fast inside a scythe on and the the only way to get to it from Python is to go through the Python layer and that's going to have to do all those type conversions and it's going to be there's going to be Python overhead no matter what you do there yes we'll see that a little bit good question yes all three allowed in classes we'll see that as well no no CPF doesn't to use totally transparent you just define one function you just you just pretend it's just one function and siphon takes care of whether it's being called from Python or from other siphon code and it calls the right version for you but you don't worry about that at all okay so question is why not just always you see PDF and the answer is sometimes if you have a assay of a function like this that takes pointers as arguments right pointers are everywhere and see but there are no pointers in Python like true see pointers in Python right so you could not declare this as a CPF that would not be allowed okay CCDF functions you have access to all C but you can't call them from Python CPF you have to make sure that the arguments signature is compatible with being called from Python so they're they're more limited in that route in that respect question yes yes that's the capsule exercise absolutely um so the question is declaring CPD functions do you lose optimization of of just c-level stuff rather than yeah well in the body of this function you're still able to use as much C as you like so the bodies of the function is perfectly you know you can unpack stuff and use seat pointers inside of the body that's perfectly fine it's just the the argument the function signature that has to be compatible with Python that's the only restriction to CBF functions okay okay so just very briefly going talking about c import another psyphon keyword there aren't too many we're through almost all of them I think and what C import allows us to do at least one one thing it allows us to do is pull in stuff from other C header files that are wrapped for us with with scythe on ok we'll we'll see it in more detail later on but I just want to talk about this one part of c import for the for the next exercise so here i'm importing stuff from python in these first two lines here so from math import sign is PI sign so again I'm this is you know familiar to all of you here you're using just the sine function inside of the Python math file and this is a Python function that's got a Python layer on top of it and it's wrapping the math that H sign underneath for you okay there's also a numpy universal function that lives inside of a numpy namespace so you can say from numpy import sign is and p sign for example and this is a universal function that's calling sign underneath it covers as well and it works with python scalars and with numpy arrays rights you know all about that so now here let's say we want to access the sine function from the c standard libraries math dot h we can do that in scythe on because scythe on ships with wrappers for all these standard c and c++ headers we can say from Lib seed math C import sign okay and if you want to import malloc and free from standard lips eat at standard Lib and say from zlib see does Center Lib C import malloc and free so again this only can go inside of siphon code this is not valid Python obviously but this sign here that we imported C imported from Lib seed math gives us access to this c function sign so there's no Python overhead when calling this sine function here you can of course do this inside thing you can say from math import sign as PI sign and call it that way but when you call PI sign at the top here inside a scythe on code you're going through the Python layer so you have to convert let's say you're calling PI sign with the C integer as an argument or C float let's say it has to convert that C float into a python float give it two pup

Channel: Enthought

Views: 36,853

Rating: 4.9725084 out of 5

Keywords: scipy, scipy2013, python (software), cython

Id: JKCjsRDffXo

Length: 71min 31sec (4291 seconds)

Published: Thu Jun 27 2013