"Extending MicroPython: Using C for good!" - Matt Trentini (PyCon AU 2019)

good afternoon everybody and welcome to this session of PyCon au our next speaker is Matt Tran teeny who will be speaking to us at other wonders of extending micro Python please make him feel welcome nothing like getting stuff done why the last minute is there okay extinct market so you'll see on a bunch of my slides today there's a whole raft of QR codes just sitting in the bottom right corner as we go through you'll be able to scan them on your phones if you really want to but they're there for reference for later this first one will take you to the the slides themselves so that's good if the levels are okay back there yeah so Who am I I am the guy who's not making those slides to over there we go I'm a software engineer I've had about 20 years experience which makes me feel a bit old as a cinema buyer I've been fortunate enough to sort of play with the entire software stack so all the way down from very small devices all the way up to large websites and desktop applications in between the whole shebang but a common theme of mine are supposes to work with embedded devices as part of a larger system so I got into micro Python a few years ago just because I think it's wonderful and so I just want to acknowledge pretty much the front row here if you could just put your hands up Damien Oliver Sean Andrew and Jim have all really helped with this this talk it's quite a detailed talk and some of these features of micro paithan weren't even working before yesterday so we've pulled it together through if you really heroic efforts so thanks guys if it works out great it's because of them if it's not so good today the heads almost certainly my fault saying why this talk I think it's necessary there's some good reasons up here on the on the screen I think but sort of speaking from the heart I guess is the sea has been entrenched in the embedded space for for decades since the seventies right and nothing has really changed if you look at the way we're doing development today not much has changed at all like the tools have gotten marginally better the composite got marginally better but you know if you look at other parts of the software systems we've gone up through high level languages we've got better thrilling for unit testing I've got much better idea support in other languages and it really improves the developer experience so for me I see my cap on the micro Python all high-level languages like it running on an embedded device as a way of disrupting that and breaking that cycle and allowing us to move faster in the embedded space so my vision is that micro Python can be they go to or hey go through at least of being able to develop on that embedded space now there's a couple of reasons for that has said all the Python goodness that we all know and love allows us to develop faster but specific to the embedded space being able to develop interactively with a ripple is a complete game-changer as far as I'm concerned and if you've got sort of these stubborn sea guys in your office that sort of latched on to it and don't really want to change or evolve or look at new things go show them the ripple show them that they can change stuff at runtime show them that they can do in I Squared's to scan and if they're not converted you're going to struggle to cover them and I've tried anyway I think it's I think it's an important thing I really want to sort of push the micro Python angle but there are genuine performance concerns so it's an interpreted language it's not as fast as a compiled language like C and we need to tackle that there's some some good options that we have and the guts of today's talk will cover one of those options okay so specifically the main dish the I fill it if you will is that we're going to create a module in pure Python and a spoiler alert it's going to run pretty slow we're getting again we're going to then re-implement that in C and it's going to run a lot faster and I'm going to show you how you can wrap that so that you can access that from the micro Python space now that's that gives us the best of both worlds because we can do most of our development in micro Python and when performance dictates we can drop a drop back down into C and handcraft some stuff to make it work a lot faster it's really powerful concept I think the only problem with this concepts I suppose is that it does require you to build the micro Python the blob so you'll see module that you create gets built in alongside the whole of the micro place and firmware which you then deploy to your device so it's an important distinction like any good meal we've got a couple of entrees today we're going to start with some background I just wanna get everyone up to sort of the same level about talking about what micro Python is and some of the fundamental concepts around what a port is you here is talking about ports and you need to understand what that that means and how they're built so we'll cover that we also have a look really quick look at micro optimizations I like to call this is these are things you can do to thank you these are things you can do to improve performance without having to get down and dirty with with C so they're a little bit easy to apply in the imagery of the case and you can get some genuine performance improvements from them as a dessert we've got native modules now this is the feature of who's it's been definitely a bleeding edge but I mentioned before the regular micro Python C module requires you to build a whole family binary and they of module lets you not do that so you can download and use your regular micro Python blob and then you can build your own module and just load that at runtime it's a natural evolution of going down this route but it's it's a really wonderful way of developing we'll also talk about some ramifications about what that feature can unlock so let's get into it what is micro Python actually clever show of hands who's used micro by then so 70% that's good keep your hand up if you've used C in the embedded space probably the same people like yeah it's that's good to know so micro buttons written in C it's an implementation of Python designed to run on microcontrollers so the main difference there is I guess it's really concerned with minimizing resource usage in particular ram now memory use is probably our first constraint when we hit these micro microcontrollers but it's not just the language like Python we inherit the idea of batteries included so we have a bunch of libraries as a bunch of the standard libraries that are implanted we also have hardware library we call it machine and if of good I won't play with it yet but if we've got a microcontroller much like this one it has a bunch of however peripherals that we want access to machines one of the libraries that we supply that provides those those features you also get a file system and as I said before you get a repple as well and so all this gives you a pretty wonderful starting point for darlin okay so the structure of a micro Python system I heard and the other talks the hand-drawn pictures of the way to go especially the stick figures apparently with the smiley faces so you're allowed to umm attach ourselves to that somehow anyway so over here we have a comparison we we have a compiler and in much like C Python you have a compiler which turns your Python files into byte code okay and then it's what that's what the virtual machine can run you have a runtime which is responsible for doing stuff kind of cost over some of those details but it has access to the modules that are packaged as part of micro Python and those modules can be in Python or they can be in C it doesn't matter it's the runtime so you'll see that in the bottom little blob here we've got a whole bunch of things that I wrote down which my handwriting is almost illegible so I'm surprised if you can read them anyway and then we've got some off to the side now the distinction there is what we're going talk about in a second which is a port so a port is a variant of micro Python or a blob of micro Python which includes the compiler of the virtual machine the runtime which is all cross-platform it's also the collection of libraries and they can change depending on the on the port that we're talking about depending on the hardware we're targeting and we've also got the machine stuff so we jump back to here these libraries on the right I'm machine specific they're definitely they have different implementations depending on which microcontrollers you're using right so if you're using a Sam D chip you're using an sp30 through using the stm32 these manufacturers will provide you with ways of accessing their hardware at a low level but of course that's different from port to port so at this point each of the ports of micro PI's are actually built slightly differently because they've got different dependencies it also means that all the stuff for the left of this block these boxes on on the right all that word gets gets reused for the most parts or cross-platform code and all the stuff to the right needs to be redone for each of the ports right because we need to figure out how we make our connection to the I squared C features of that hardware although the pins that are part of that hardware so fundamentally that's kind of the gist of a port it's probably worth saying that microphone works on a lot of different bits of hardware so this is a photo I took the other night I just bought a whole bunch of devices that I have I had a quick count there's like 80 different microcontrollers in that page and there's about five different families across you know 20 or 30 different boards it's quite a lot of large number of things and so each of these things has a different port so I get back to something here at the bottom here there's a line of thoughts about board support because you're familiar with the embedded space you'll be aware that there's usually a family of microcontrollers but when the pedal hits the metal the boards are important so the way that that that microcontroller is constructed in a board there's certain pins that can be broken out and they can differ even within families of micro microcontrollers so the concept is that you have a port which describes like a family and you would have a board which describes a physical board of a microcontroller and so today today we'll be playing with a couple devices here one of them be SP 32 based which is this one on the left it's a tiny Pico and one the rights of Pi board D which is stm32 based if those names don't mean anything that's perfectly fine you can come talk to me afterwards and give you more detail it is worth covering which ports are available in Micah Python so on the Left we've got all the official ports the QR code will take you to the microphone page which has the list of or the folders of all those different families and on the right is unofficial ports but are still gaining in popularity I suppose and there's a whole bunch of other weird and wonderful ports out there for various custom bits of hardware the top group the top group up here is sort of officially supported and ready for primetime I guess you could say they're widely used and very active in development the bottom fewer somewhat interesting the Sam D is an interesting one because it's quite new we've got very little hardware support for it at the moment but it's notable because a lot of the Adafruit boards use that that microcontroller family so it'll be great to unlock more of those features and that's something that's in active development right now the CC things your honor and the teensy board are just listed because we do have support from it's it's there but it's less active I suppose they're not being as actively used as f is another one that's interesting because F is a a real-time operating system and it obstructs away a lot of the hardware so there's an extra layer seeing between micro Python and the hardware itself but that means if we can target the way Zephyr operates we can use any of the boards that Zephyr supports so there's 150 boards as if a user's of the Pacific and work on Micah Python can now accessing his O's by sitting on top of Zephyr so it's quite an interesting area of development and finally there's a JavaScript version as well so we use them scripting to get down to wesen and the rabbit-hole continues down yeah worth mentioning I think a really interesting area for those are in the hardware space is around risk five both in the soft core space and in the hardware so we've got a really good support and microbiome for that now propelled through I've got listed up there because it's a weird and wonderful amazing bit of hardware and they have the first iteration of the propeller one board you can go look this up later it's got some really unique hardware characteristics very good at multi processing execution but they had quite an esoteric sort of language that they developed themselves for the first generation the hardware for the second generation the hardware they've picked up microfiber and run with it and it seems to be unlocking a lot of features really quickly for them so I think it's really exciting area diamond right straight into the micro optimizations we've got avoiding Global's which is generally a good thing to do anyway in my cup I think it's quite important we'll see here on the left is what you shouldn't do because if you look at what's happening is that pin on and pin off for actually in the global namespace so the lookups for those are a little bit slow so just simply keeping a label to those methods if you're inside a tight loop that can give you some performance benefit we did some measurements and it's like 10 to 20 percent faster in some cases now that's a fairly contrived example but if you're looking at getting some performance it's a really easy one to do constant good this is red because I hadn't quite finished the example but I can walk you through it it's just a simple extension so that when you store a variable you can store it with labels of const and that means that micro pathum won't do it as a lookup it'll just replace the number wherever it's used in the code so it saves you a dereference optimisation there's pre coverage so I mentioned before if we can get back to that diagram its way back just like cpython when you have a compiler and we compile it down to virtual to by code so the virtual machine can ingest it you can skip that step by doing precompilation that means that you can create that by code and store it as an empire file just like the PI C files in the Python world that means you use a little bit less RAM because you say that you avoid the compilation step which takes some RAM and it's also a bit faster to get going because we don't have that compilation step at the point where import occurs probably worth mentioning too normally the conversion to bytecode is transparent like and see Python but it's a little bit different than in the we know don't normally store the bytecode on the device normally occurs in memory as soon as you do an import okay so you import your code you get your bytecode that's stored in RAM and then from then on it's executed from there but we do have the concept of my file so you can do that manually and skip this step it does slightly complicated appointment because normally you would just get your PI files onto the device now you've got to have some sort of make file to create your Empire files okay first in modules so first modules takes that a little bit further and takes your bike code that you've created from your pre compilation step and stores it in the micro Python binary blob so this requires you to build micro Python but it also allows you to store the pre compiled by code in in amongst that binary blob now this is useful because it saves further memory because we can execute from flash so we saw this in the binary blob and we just execute this stuff as we need it right sorry Damien I hope that the okay picture native Viper and assembly are a bit more advanced things and Damien cover these really well last year so I'm not going to go into too much detail but native in vibra in particular changed the way that code is emitted from the compiler to the virtual machine so instead of creating byte code we actually create native code there's some complications with this I guess in the sense that it can use more memory but you can get some real performance benefits so I think it's it's quite interesting the assembly option is a bit different it lets you create Python functions in amongst your Python which translate directly into assembly for that for the platform you're targeting so if you're hand crafting a really small sort of tight loop assembly can be a really good option and I'll direct you to Damien's talk because he'll show you how to make it like 400 times faster in assembler so it was really cool right so we're at the main meal for 25 minutes right we're going to be talking about QR codes it's probably no surprise right I got QR codes all over these slides generating a QR code on device like this is actually fairly computationally difficult it's all built around this concept of the reed-solomon algorithm which if you've used the CD or a DVD or bluray device you'll be familiar with even if not explicitly it allows you to encode data with redundancy so the reason these pictures can work in QR codes is because there's redundant data in there and the pictures you know throwing away some of that data because you're just drawing it over the top of it but the reed-solomon algorithm allows you to get that data back anyway it's a wonderful algorithm but also a little bit slow to use on an embedded device so yeah QR codes are being QR codes are the the task that I'm going to use today to look at this stuff but it could've been anything could have been free transforms which are computationally expensive could be in the camera module could be anything that takes a lot of time it's also did help that there's an issue raised in the micro Python ticketing system that says we don't have a queue our library so now he's got one so is the dives a long way first thing I did like any good Python developers to go under the web and find out what was out there and there's a whole bunch of QR libraries project Naoki was really interesting to me because it implemented QR generation in seven different languages it's got really clear documentation there's lots of code comments it's all about being really clear sort of hold it up as a good example of how to do QR codes in any of those languages they're all about a thousand lines of code and yeah it's it's excellent now I should say that micro Pisan is built around Python three five more or less and there are some differences between micro Python and C Python so the pipe when if you're going to go through this effort of taking a library that's designed for Python making it run on micro Python there's always some work to be done and so I thought we'd go in a little distraction looking at those kinds of things that that I went through for this particular particular task so my goal here was to use an ankle code on the C Python n micro Python I wanted to run on Python 3 X so the first thing I mean III was that there was a reference to future because they supported a Python 2x now a Wi-Fi password here is vital means Python 3 so I just got rid of that and so we've we've gotten rid of Python 2x back has compatibility this stage it would actually really easy to add that it's just too regionally to stub for the future to ignore that in micro Python but it wasn't that big a deal the regular expression library mycoplasma is designed to use a lot less memory or a low amounts of memory and I was missing a really odd little quirk that they decided to use in project Naoki which was a /z argument has anyone used the slash that argument in regular expressions I had neither it was it's designed a match to the end line off for multi-line matches there wasn't any even any multi-line matches in this project so I couldn't figure out why they use that so got rid of that that was easy this version info it's slightly different in micropolitan varies a ticket about this not sure will make any changes but in micro price and it's a couple whereas in regular cpython it's a class and the class provides access to major minor and patch versions by label so with the tupple and so see PI's actually supports both so you can do index so zero matches major one matches minor but yeah markup iron only supports the the indexed access so it changed to index access and that was fine but we're considering adding the label support as well in libraries libraries are usually where you'll spend most of your time porting your micro Python code which is worth mentioning that there's micro Python Lib I'm often surprised that people aren't aware that there's a separate repo with these libraries and we do a terrible job of communicating it so it's here right and this is community maintained although it's curated so that it always works with microfiber main line and it's you keep installable which we should cover as well just quickly that's the github for the that page goes on for a while because there's over 130 140 libraries now you pip is just pip from micro Python effectively but this is kind of a bit of a revolution when you're first doing development so on an embedded device like this one which actually has Wi-Fi support I can install one of the libraries if it's got pip support directly on the device so I needed either tools that one has been published it's a pip so I can just you pip install microbiota tools yes and there's no no other tools needed for that it's just pretty cool and the embedded see developer me is like oh that just that shouldn't be that easy that's just not right which is great if you're using a device smaller device for one without Wi-Fi you Pips also supported it just downloads the files to your PC so you can transfer them across later it's not much more difficult it's just it's just so cool being out again you keep install something it's great yeah getting back to Michael Python Lib 130 to 130 libraries it is fools went in that just worked straight out of the box Nikki also required deck now we have a deck implementation in the core micro Python binary in the in the main main line but it's actually been it's quite a cut-down version designed so it consumes no extra RAM it also didn't have the functions that I needed so that was a bit of a problem or functions that Naoki needed but there was an alternative implementation in microfiber lib are you people installed that got almost all the way there but it didn't have sub scripting support I'm gonna add that but in the meantime I can just convert my deck to a list right before it gets used then we can use a subscripting there so I've actually introduced a performance penalty to this but for the purposes of this presentation it doesn't matter I will fix that it's okay so all that being said this is an example and it's it's fairly typical I've found importing a true Python code down a micro Python and still supporting both environments there was 12 lines that I had to convert and so I think that's pretty amazing to be honest like to be able to take from the vast amount of Python libraries that are out there with a little bit of work we can then get access to them if they don't work straight out of the box so it's a 900 line library twelve lines that needed changing it's pretty cool right QR codes in Python if this works hey it's pretty close hey there's really big I'm just summarizing this because I don't have a ton of space the project nehe Python class kind of boils down to the use of three functions it's got a bunch of others as well depending what you want to do with your QR code but these are the only three that I needed for now that's twin coda texts that takes actually it takes a string let's try to summarize it so it's nice and easy to see depending on the length of that text it kind of dictates how big it needs to make the QR code to store enough of that data so once you've encoded it it that's a static function so you actually it actually returns an instance of the QR code class that QR code class can then be queried for the size of the QR code and then you can get the module and the module is you it's confusing I know because python community but module is actually what they call the pixel in a in a QR code so by iterating through the the QR code you can build up that picture of that is that a QR code all right oh look that that is actually I don't even need to do anything because that's the QR code that comes out of it so it's nice and I've just printed this at the command line with a Unicode character which prints a white space and space for the other one and don't know if it's you know let's see if we can we can't squeeze it down so if you've got your phone's that should scan hello world they don't want to test that please tell me it works I think was hello world yep yep confirmation does it work from the back I tried in the other room and I had to zoom in zoom in digitally but it did work I'm gonna go with yes that's fine right so that was great the problem is this that QR code takes about fifteen hundred milliseconds to run to generate and that's a really short QR code I found that if you made them long so if you've got a URL that's 30 characters that takes like 10 15 sometimes 20 seconds to generate which is a long time from microcontroller be busy especially if you got things like watchdogs right so that's a problem so what about see see progeny he has a scene fermentation which is great it is worth noting and being a little bit unfair here because the C variant has been optimized to use less memory but when I ran it I won't go through the process of building C here but when I ran it that was the amount of time it took to run almost any QR code regardless of the length so that's a big difference as I said I'm being really unfair to micro Python here I'm stacking the odds heavily against it because this is a computational problem which C is very good at solving and Micah pythons is really hard for it to solve well and I'm comparing and optimized to optimized but for the purpose of this talk that's great I can tell you that yeah we can we can use this right because what we want is we really want some of that performance all right that's pretty good but first some of the feedback I had was that the deep dives were kind of long and this one's a little bit down the down the weeds so just take my own guys okay everyone just like stand up I know this sounds contrived but if nothing else it means that I can stop and just have a bit of a drink isn't that better I especially feel it's difficult for us because we've just seen lunch right and then you come into a dark in the room and you're listening to the micro Python deep dives yeah we all good good good okay how to represent my jewel okay so what we need to do here is we're not committing to building a c module we're now committing to building a micro Pisan c module which means we need to wrap that c module okay so the details here are important there's a whole bunch of methods whole bunch of macros and what we need to learn is about marshalling because we've got values that are coming in that we need to call so on a micro patent side we want things to look just like they would as if it was a Python object we were calling and what we didn't need to do there is marshal values across that boundary okay the interesting thing here to be aware is that the the structures are we going to create and see to specify how that marshaling occurs are the exact same structures that the virtual machine creates when you are creating your Python code in the micro Python environment so what we're doing is we're sort of matching all of those structures in the C world to what the virtual machine would do that can make sense okay so this is this first example is the nice simple one that we is actually the I mention before the our documentation not great for all of this stuff the only documentation we really have on it is this when you find it [Music] okay right this actually really reasonably well-written and walks you through a basic example now that example is this one what we're gonna do is we're gonna take is that readable if I make it a little bit smaller okay this whole green box we're going to talk through and you're gonna understand that won't get sick so the goal here is to take a function which takes two integers and we're going to return the addition of those two integers right we're gonna start baby steps right so I just get back to here I think I've got the useful right hand-drawn stuff good on the Left we've got what represents what represents a module in micro Python it's effectively is obstructing the see world that struct has a number of pointers one of which is a table of function pointers which were indexed by a string okay now you can sort of blur your eyes and you can see that if you were import a module this is how the micro processing environment learns what it can do with that so we're going to register the module that's the thing that we had in the left right we're going to say that we're going to give this module a name which is example we're going to link it to this which is actually with the module definition okay that module definition is linked to another thing which is linked to another thing this is the module name this is the name of the function we want to index this is the pointers of that function which takes those two ends so we've built up a bunch of pointers that eventually point to this function through a series of structures this function is intended to take two objects and these objects MP herbs which is effectively a type of but it has a type built in but in the C world it's it's void star with a whole bunch of infrastructure around it to get the things out of it that you need so when we register this function we tell it that it's a two parameter function and here we've got two parameters and so the compile will check a lot of these things if they're not right if you have another right number of arguments here it'll tell you internally so when we execute this function we're going to get an integer in the C space out of the object that gets passed in in both cases and we can return them as a new integer and that's it I say that's it there's a fair bit going on here but yeah it makes sense any questions with that just what we dangling is it too much to grasp in a forum like this hello yes so are there differences between things are the difference between the micro Python API and the C the PI the C Python API no is the short answer yeah so there are differences in the way they're done but it's supported microphones and supports all the same things so you can pass classes you can pass tuples dix the whole shebang where it gets complicated when you're doing those marshalling is trying to figure out like what types you actually want to pull in and out the other the other half of this is so we've got we've now got this is C code then we need to add to our build system right so we've got to choose a port to build right because we've got dependencies for say stm32 or ESB 32 or not these macros are required to tell to tell a build system what to do there's a there's a folder which by convention is modules in the root of the root of the micro Python check out if you're clearly repository so you'll throw there'll be a modules QR it's just hard to show you because I haven't got many pixels and modules QR and then this example and then we just need to enable that module and build system we'll take it and run it actually here's a good example so it's it's literally two files now I just said before that we need to choose which port to build it in this is actually cross platform or cross port code it works on any of the ports but the only difference is that you've got a you can't build across all the ports because they have different dependencies different compile tool chains that kind of stuff so the same code will work on all of the ports if you've not used any port specific code cool I think that's all I had to say for that and the example is here [Music] right so I mentioned that there's a whole bunch of macros and yes macros and methods that help you bridge that marshaling gap they're quite challenging to keep that all in your head if you're using it a lot then it becomes quite second nature but I'm sure as you know your response when you saw it it's like oh that's a lot of boilerplate to to try to wade through now we've got poor documentation on this as I said but during during researching this this talk we stumble across two things the first one was a guide put together by a guy named Mike Teachman Mike's put together a git commit which has a whole bunch of marshaling examples it there's like 900 lines of examples if you want to pass in a single single parameter you're going to pass in a list if you want to return this if you want to return that you use keyword arguments all that kind of stuff so it's a really helpful guide I've got a link to there but also we had this idea that wouldn't it be cool if you had like a website to just describe what parameters you had and then the website could generate the boilerplate so if I just pull this back a sec all of you and put your hand up just to embarrass you really all it was kind enough to to put this together I find it lost track of each workspace was on it's dead space say I would love to you I should have LinkedIn that little vein smart thing today it's gone what's your sorry now just roughly you like your feet areas yeah good man all right good that's hard ok so this is a stub generator that all of us put together and so the nice thing about this is that you can specify what parameters you want what output types you want and it would generate all the the code that you need so you don't need to remember everything we've shown you today it doesn't yet do everything it certainly handles all of the simple cases where you've got floats strings lists and tuples that was a neat addition I haven't used yet and let you specify the argument names all that kind of stuff there's also a checkbox here which will add extra boilerplate to show you how you can do extra things but right ok well if anyone out here is used JavaScript raise your hands again fantastic because that means you can all generate pull requests for this right we need to extend this straight so that we can take dictionaries so we can take classes and so on so I think it's it's a really useful tool so thank you all for that that's cool right if we got 10 minutes is that 10 minutes to 20 great no problem easy okay right so we're gonna wrap this cube our library I want to get to the maybe this thing so we're going to take a string return a couple of tuples that's a little bit different to that class example before the class examples a little bit more difficult so I thought we'd go through this one first and steal my thunder but okay sorry folks I'm struggling with the let me get to the right place there's a few different examples all mixed in here I've caught it encode text - for no good reason probably cuz the second time we try to get this working and this goes for that definition of saying what arguments are available I'm intentionally glossing over this because I think we're gonna be struggling for time but there's a way to specify that this function takes keyword arguments and so some things are optional which of course in C is quite a difficult thing to do so you can see here you get arguments and you get a pointer to a list of arguments and there's a system built up so that you can specify what you expect those arguments to be and then there's a parser to get those things out so all that cooks in to say that at some point here this is the nayuki function which is in C to say pass it in I want you to get in this case the string argument the string argument which is our text in my world it also takes in the ACC variable ACC is the amount of error correction that you want that's one of the parameters in the system I've made that an optional argument in this case you can also raise exceptions there's no reason why you can't in this case it's a value error but it could be anything I answer that's a useful thing to like again I can't really stress enough that anything you could do in micro PI's and with sufficient persistence you can figure out how to do it in the C world so we're getting the size which is necessary and then we're building up a tuple there's a bit of magic here to create a new tuple and massage that into shape in the C side but effectively we return a series of rows and columns within that table let's see if we can get this working it's a great song okay naturally the demigods started playing with me before because it appears that one of these cables that I've got is not working that great again problem is I don't know which one it is sorry possibly just speak amongst yourself for just a moment people the best laid plans and all that right good sign so I'm using a tool called a shell which allows us to have a serial connection to to a micro Python device this one here the SP 32 base one it's a tiny Pico we're gonna import my QR library that we've just built before now thank her texts let's create this right we get a couple of tuples of stuff now that is rang through see that's pretty cool the nice thing about this is that my Thunder that was stolen before is that that takes 11 milliseconds to run so I think that's that's really amazing I'm like I can't stress to you how cool that is right yeah I really could have delivered that better but it's yeah right so sorry the next thing I wanted to show you really quickly is that you can we talk before the you can construct any sort of object including a class because that the way that's returned is different to the way that the project naoki python implementation worked right before we had the three functions that return to class the class had access to functions to get of the data you can do that with micro Python but buckle up so this is a expanded view of the one we saw before Damien I need to get you to sign this because it'll be worth something someday I'm sure at the top we've got modules the module links to a look up to find the functions that are in that modules those functions could be an instance of class or a definition of a class their class has a bunch of functions you can associate with it one of those function pointers points to a table which is the methods that are associated with that class one of those methods on the class needs to create an instance of the class itself that instance then winds back to say what its definition is don't get that close yeah cool so you need to build up that structure in the sea environment to present that kind of interface in the micro Python world okay it's gonna be really hard to show that much code here but I'll give it a good crack okay starting at the bottom that's not even the bottom that's looking like the bottom right so we register our module no don't register our module we set up the lookup forward just like we did before we define a dictionary which is going to be our things that are inside the module now these kustra is the got service before but it's worth mentioning they're literally a string lookup there's part of the build process it goes through and identifies all these things and turns that into a string which is everything that's after MP underscore kustra under school okay so this is defining a class type we've got a kind of bridging with two things here this is there's a couple of things going on the only important one here is the QR code and we'll dig further into that so that's defined here this was the the class definition excuse me we have the methods that are defined in that class and then for each of those functions we put the instance of them and you'll see the first parameter or any of these functions is gonna be an in value and like we retrieve that just like we would in Python that that's referring to the instance of the class itself so you do all these bits and pieces and again I want to get the the web tool to help generate some of this stuff but ultimately you come down to something which can create one of these things which is the in code in code the function that was the same as just before this is calling the project Nikki stuff I feel like I'm losing you so I'm gonna go straight to the demo so I'd be just like before we had this is a static method I can prove that to you later but static method the returns an instance of a QR code okay so write that class that we've defined in C now has these functions but we've already generated the reviews encode text to generate one which is populated we can then get module look and could iterate through all those to get true or false from everything so disconnected memory let's get a print function eventually we'll get a print function so if you look at what this function is doing its iterating through this yeah it's iterating through the QR code to to get the module and spit out by those Unicode characters and that should be HelloWorld again that 11 millisecond penalty that we're paying occurred when the encode text function was called but now we've got if you think about we've got we got the best of all of our worlds because it's fast we've got a class that we can do we can generate that in this comprehension we could generate it however we want to it's just as flexible as if it was in Python so I thought that was kind of worth the details they were heavy and they're hard but come talk to me if you want more information the final thing that I'm going to struggle to show I think is native modules they mean if you want to help out native modules separates the deployment of your micro Python blob from the actual sea module itself this is this really new stuff and if you think about it you especially if you got embedded backgrounds you might understand why you've got a whole bunch of pointers that when you load them they need to be reorganized in memory because that could be placed anywhere in the memory space of that micro the nice thing about this is when it's all working when we've got all the infrastructure to do so it's just going to look like a regular Empire file so we can compile it and it can have byte code in it or it could have native code in it it doesn't matter just from the usage in micro Python it's just gonna be purely transparent so we can get that that that benefits in in performance and we can also separate it from the micro plot micro Python system so I won't demonstrate that we mentioned all that it is worth thinking about how that can affect micro partners a system because it could actually mean that we can trim a lot of the micro path and system down to a really tight sort of core and then if we get better support by you pip and by our infrastructure we can just install the things that we need for our device and that could really save some flash and some RAM but we need to do a lot of work here to make sure all of that works right you can think that each of those native modules that we build is port specific so if you want to do this right you would register for each library the owner deploys separately you need to register that for each of the ports that you require them so I think it's a really cool thing to think about we've got a lot of potential here I think yeah but a lot of work - all right just to wrap it up just think we're getting close to the time I want to spend the micro pathing like performance issues like they're real like we saw there to take a second and a half on a small QR code but we can overcome that by dropping in to see selectively and really in a targeted sort of fashion and that way we can get the benefit of developing at a high level and still having great performance so yeah I think the and just to wrap it up the native modules is a really exciting part in the future that's all I wanted to talk about technically I did a couple of things couple things I'll just mention briefly great I just want take another couple minutes because and diversity is something that I think PyCon does a great job at promoting but I really feel that like the embedded space suffers worse than many others so if you're an underrepresented groups you're a woman or whatever you happen to be and you're interested in this stuff don't be afraid come talk to me and I'll I'll spend time getting you guys involved so I really think it'll help the ecosystem so I'm sure a few of us feel the same way so please reach out I'd love to get more diverse interesting involved in market pathan and another favor like if you don't care about today is to look it for other talks you've seen there's a lot of effort that goes into this so please just let the speaker know that you really like their talk so if I could just ask you guys like one of the talks are you saying over this weekend go tweet or go saying I mean I think generally people who are attending PyCon do a good job of this anyway but it's really gratifying to get that kind of feedback so please do that and the final thing I wanted to say is like just a call to arms like microphone is exciting it's really it's really interesting sort of project and we could really use some more help so if you've got a interests in you know there's a whole bunch of aspects we can we can use people in documentation we can use people with JavaScript now to help our our tooling if you've got C background Python background we can we could use help across the board and I think it's really rewarding and it's a really welcoming community through so yeah thanks for this thing I do have one little demo before you do your spiel and Oliver I think I should get you to do this um first of all you guys are all software engineers scope creep is a real thing and yesterday we were like we really should get this working on a device as a micro Python talk right and so I think we got that working relatively quick like you can render a QR code to a little display that happened relatively quickly but of course you send like you saw the photo of this last night that was like at 11 o'clock at night or something and was like you know we could put like this is a Wi-Fi it can be a web server and you know then people could get on there and generate their own QR codes so this thing's acting is a web server it's it's advertising an AP and all of us connected to it I think yeah so when he browses when you scan the first QR code that comes up on power up it will give you the IP address of the web server that you can scan so on your phone you can then see the web page that this thing's serving up to you and over if you could do the magic stuff you can type in whatever you want and we'll generate the QR code and display it that's a little self contained device the only thing here we've got as a battery the this guy has the wi-fi's all hosting it what are you doing what they're not swearwords there give me on the video as long are those so that I think you've pushed that it's open-source now so if you're gonna go check out that code it's available it uses this what's that how long did you make that string if you want to go see how the sea library's integrated and have a real example of the web server working you can go check that out so I think they are done thank you a small token of our appreciation thank you are there any questions we have a time for probably two or three questions yes we'll start here great talk thanks man so but before you got into porting the whole library across does micro Python have any support for profiling and working out if there's any other so if if there was a smaller chunk that would have been profitably optimized it gives you sort of tools around the fringes so you can certainly measure the duration of function calls that kind of stuff but in terms of like applying those micro optimizations and see if they're gonna have a benefit or getting an insight not really no I mean I'll ask the brains trust here on the front row but no there's no easy way to profile like individual func like like you would do with the sea you know you'd run a pro few profit or something you could use decorators to wrap functions to enter and exit I guess to time them that would be like there's not there's not time anything that's easily available but there is there is some work in progress and there's a proquest for adding set trace support to microphone so you can trace all byte codes and function entries and exits a really sophisticated cpython level sophistication for tracing that's a little bit more time before that gets in but that would then be able to be used for profiling although that's something you'd need to optionally add to your build because that adds overhead to everything so you'd only have that for profiling purposes you can't so a song with some embedded experience you can sort of get a feel for where you're gonna run into problems and so profiling manually is it's painful but it's not any other questions I might thank you so much to you and all my team it helped me a lot I was a PC developer for ten years and then I started writing embedded so within a year I'm writing like products in court embedded area so my questions are that people like me who are very new to embedded space where we can help in micro Python where we can contribute the first place at saying it's it's something that so frenching is high but documentation like we could really use some help there we we try to get from the google season of season of docs and we had we have quite a rich plan now of where we could improve that documentation so we could hand tasks out to people if they're prepared to do that it's also a really good way to learn about microfiber beyond that walking through the quick ref we've got a quick refuge described as a lot of the basic functions for each of the ports it's a really good way just to walk through and sort of become aware of what you can do with the system yeah any other questions yes I really find the unique sport useful for testing stuff out is there any way to fake pins in it yeah it's it's something I was I wanted to cover with lack of times so that you need to prefer one that doesn't know is it's a port with microprocessor a genuine port like all the others but it's just built on Linux so you don't have a machine module which would normally give you the pin support and all of the hardware access but this is Python right you can mock that out so it's quite easy to generate stubs to to pretend that those exists and in fact there's a project online we should remember the name but it's just it's like micro Python Stuber look that up but it actually will stub out a whole bunch of those things and you can direct that into your unique support so that it generates a whole bunch of things which do nothing I give you all the function entry points that your code can be tested on it that's really helpful and really should have mentioned that like the UNIX port is exceptionally good for testing and recover that a little bit yesterday but it's wonderful for having at a real fast iteration loop building testing on UNIX before you even deploy and play with your hardware it's it's it's a great great tool for that are there any other questions cool no thank you Matt thanks [Applause]

Channel: PyCon AU

Views: 7,011

Keywords: pyconau, pyconau2019, Python, PyCon, PyConAU, MattTrentini

Id: 437CZBnK8vI

Length: 63min 32sec (3812 seconds)

Published: Mon Aug 05 2019