"Open-Source Communities and Ten Years of MicroPython" - Damien George (PyCon AU 2023)

foreign welcome everyone thanks for coming to my talk my name is Damien I'm the creator of micro python which I started about 10 years ago some of you may have seen me give a talk before and some maybe not this talk is supposed to be non-technical there's a few graphs and plots of some data but otherwise it's mostly just about my experience over the past 10 years so what is macro python so for some people this is new and some people it's it's not um this is just an introduction now micropython is two things micro and python micro is sort of small and also means microcontroller and python is as you know the language that we're here to talk about and micropython is a complete re-implementation of python from the ground up no lines of code are shared with python it's it's a it's a complete uh a rewrite uh and The Way It Was Written was so that it was small and efficient um with the resources that you have at hand so when you're running on a really small computer a microcontroller you don't have much memory you don't have much hard disk space you don't even have a hard disk and you don't have much CPU so there's not many resources at your disposal to do things but you still want to run a high level language in this case Python and so micropython was implemented with the aim to be able to do this to run python you do list comprehensions for example and async IO to run that in the context of a tiny little embedded microcontrol that may power for example a coffee machine um and this started uh during my time as a theoretical physicist which was my previous life and I got sidetracked when I did a Kickstarter in 2013 uh to start to bring micro python sort of true fruition uh it took about six months to develop from April 2013 is when it started and then the kickstarter was at the end of that year and that was quite successful um and then there was another Kickstarter a few years later after that and then many collaborations um since then with with various companies and people and institutions and people such as yourself contributors hackers uh Educators who've contributed and used micro python so it's been been quite a journey in 10 years and I didn't really expect it to go where it's where it is today but I guess I didn't expect anything if you're asked what's going to happen in 10 years you don't really have such expectations but uh here we are and I'm just trying to sort of convey this things that have happened in those 10 years with a sort of a name towards describing what it's like to actually maintain a big open source project um the uses of micropython today are pretty varied so we did quite a bit of work with the European Space Agency to make micropython robust enough to run in space and it's actually used as like a high level application control language in satellites uh it's the reason there is so that it's easy to update things whilst it's flying because traditional space-based programming you have you know in C or C plus plus or more ancient languages and it's very inflexible but these days they want things to be much more flexible so they can update code when it's out there and you know it takes 10 minutes or something to actually send data to it the other things have been in LEGO Mindstorms so Lego have used micropython to implement their new Mindstorms kits for teaching in schools also various commercial products like smart vending machines various medical devices um vehicle counting and weighing devices on the autobarns in Germany it's what it's used for a lung elastomer to measure lung elasticity came out recently there's many many applications also in teaching in research Labs people use it to you know heat up DNA samples and and things like this so it's it's become quite widely used because it's piggybacking on the success of python itself as a really you know easy to use language and then people who want to build something for physical with the hands like a robot or you know a toaster or something um they don't necessarily have a software engineering background but python allows them to actually program something to make it work and if you could mix you know this ease of programming with actual Hands-On you know make an LED blank and a motor turn people can make their eyes make their ideas real in a very a very straightforward way so that's kind of the key selling point of micropython is it's mixing microcontrollers and the Python language together um you can try it out yourself there are many many different Hardware platforms that you can get started with some of them are very cheap only a few dollars uh some of them you know more expensive you can run it on lots of different things with lots of different uh plugins Motors lights whatever you like you can actually try it online and that bottom link there so if you haven't played with it it's very easy to get started now um this talk is trying to give a bit of an insight into what it's like to maintain a big open source project like micropython I mean it's not as big as python itself for example or the Linux kernel but it's still you know reasonably um scienceable project and there are lots of challenges that come with maintaining the source code uh developing the code and then maintaining the community and building the community around it because obviously you don't really have a project if you don't have people using it or you can have your side hobby projects but you know a project that's out there in the world the users are really the big part of the project and we all know this with python and conferences like this so the social aspects of software engineering is I guess what I'm trying to convey here I'm a very technical person and I love working on micropython because it's very technical but as part of that I've been thrown into this you know thing of building a community and doing social sides of of um of computers and technology and I think it's it's really interesting um and really eye-opening so also at the end I'll give a bit of advice if you were ever going to set up your own project um what sort of things you can do at the start to make it easier later on so the main I guess the place where I spend most of my days on GitHub at this webpage or at my terminal and GitHub is is the sort of the Nexus of where everything happens uh it's micropython is free and open source so free means you can use it without having to pay and there's no licensing it's MIT license and open source means the source code is there and available and anyone can access it those two things are not always the same and you can have one without the other but micropython is both of those and you can access the code if you go to this GitHub website now um the open source development that we use uh is uh well it's based on git Source control uh if you're not familiar with this git is sort of it changes your way of thinking about code or at least it has for me and that code is not about the code well it's about the code you have but it's also very much about the stepping stones that you make every day to revolve your code from one spot to the next so a commit is a small stepping stone that you make your code has this form you make a commit and it changes its form and to me the commits the Delta the change the stepping stones are really the important part um because you can look at them in one go and say this is what I'm changing and it comes with a test and it comes with documentation and this is how the project evolves and we're very very strict with commits and how they're formatted and and The Styling and the commit messages and I think this this is really sort of part of keeping the community on track when it's developing a project um the having a very strict git history and a strict history of Stepping Stones allows you to easily track down bugs in the future when they do pop up so you can go back and use git bisects which is when you like we have a bug we know that this code is good this code is bad so somewhere in here the stepping stone went wrong and you can easily track it down and then fix it and it's also good just for you know people who are new to the project and they want to see well why was this change introduced they can go back in time and have a look and usually hopefully the commit description explains why this change was made so I think having a having this really strict git workflow is really important for us we have a lot of testing we've done a lot of work in testing part of the work with European space agency was to add a lot of tests to get close to 100 coverage which is really important so every line of code has a test associated with it now the coverage is never perfect because there are still many more scenarios than just making sure every line of code is run but we do have very good coverage and we are very strict to try and keep that up and the continuous integration whenever there's a new command made there's a lot of tests that are run automatically and you have to pass those tests to get that commit in to get that next stepping stone done um it's so it's a really good safety harness you can be really confident that if you make a change to the code even if you don't know what's going on in the whole code base if you don't break any tests then probably your code's okay and by the same token like you can't you can't just go randomly deleting lines or anything because of the test will break so it's the test harness is really there it's like a safety barrier to make sure we don't do things wrong Hardware in the loop is something we don't have yet but we do have offline Hardware testing which is important as well and the releases uh the release cycle is not very strict but we try and aim for releases maybe every two three four months and the releases do take a lot of effort because you've got to get to a point where you're confident with the code and you know you're going to release it and people are going to upgrade and maybe they're gonna you know the coffee machines now are going to run the new version of macro Python and you don't want it to break down because you know embedded system is really hard to debug so we're very strict to make sure that the releases are equality and you know known to work and hopefully not have any bugs or at least no bugs that we know about um or if we do we can document them so the continuous integration testing we have helps us there but then also there's a lot of hard testing on real Hardware before we actually make a release it's also fun to write the release notes although that takes me a couple of days uh of full-time work to actually make a full set of release notes and it's good because it looks back on the past few months of what you've been doing and it's like this is a summary of my life these release notes but it's yeah it's good and it's nice to make a you know a summary and then put it out to the world and everyone reads it and they're like okay I like this new version because it has this and this feature um so this is this is sort of the day-to-day grind of of working on a piece of code um one thing that is really important here is funding so it's it's been 10 years since I've been working on this uh and you need a source of funding if you have a open source project that is you know going to live for so long because you know you need to live um and you can see here I've got we've got uh GitHub sponsors uh which is a program on GitHub where people can sponsor for like a dollar a month or a hundred dollars one off or whatever you like and that's a continual source of revenue for us now since 2020 and you can see there how much that's in Australian dollars how much we're receiving from GitHub sponsors so 2023 is small because it hasn't finished yet um but we get you know a sizable amount of money there from GitHub sponsors and that helps to pay sort of two part-time maintainers um not myself technically um so although yeah there is a bit bit left over there but it is good the other sources of funding we have are Hardware Sales because it's a hardware software project we can have Hardware that we sell and some of the money that from that can be used to support the development which is great but there's also a lot of others contract work that I do and that other parts of the members of the team do as well so people come to us and say we want macro python in our product or we want this feature added to micro python or we want this thing improved and uh you know they pay us to do that work for them and usually that work that they've paid for goes back into the open source community in the sense that it's not hidden behind anything it's contributions made to micropython that everyone can benefit from so that's really great but you know that someone can pay for something to be to be done um so I yeah I can't really stress enough how important it is to have funding and I think there are a lot of pure software projects out there that find it really hard to get funding um and and to continue on um there's been lots of stories you know about about one-man projects that die because that person doesn't want to do it anymore and then people are relying on this project so where does it go so we're very fortunate to be able to continue and to sort of Thrive and have enough money to live and continue doing what we love which is develop um code the other point to make is about licensing which is a really deep topic I have a single line we use the MIT license the GPO license is kind of the opposite open source license to MIT if you want to say that we try we steer clear of GPL because in an embedded environment it's really dangerous but difficult for customers who want to use micropython to accept the GPL because it means that any code they write like if they're making a coffee machine again um then they have to open source all the code for the coffee machine which doesn't work they want to sell a coffee machine they don't want to have to open source the code for it so MIT is sort of yeah the choice that we've made and I think that was a good choice because it's allowed us to be successful people use micro python because they know they don't have to release their code to the public yeah as I said it's a big topic licensing um so the community the community is very important as I mentioned before and in micropython there are sort of these four portals of the community mostly can interact with the project um apart from we also have in-person things events uh once a month at the hacker space in Melbourne but online there's GitHub which I told about before which is where the code is there's also GitHub reviews also for discussions which is like a forum we used to have a phpb forum but it was getting a bit old so we switched to GitHub discussions we also use GitHub for the wiki we had a separate Wiki at one point but we again tried to consolidate there and we use Discord now for chat we used to use slack but we're using Discord so we have moved around uh you know it's um sort of this is another part like it's it's not easy to choose a technology to use to interact like we had to make choices here and some people were not happy with those choices um they were like we want to stick on slack or we want to stay with phpb but as you know the maintainer and open source project you have to make decisions to keep things moving and there are reasons we went um one way or the other to keep up with you know the pace of technology and change but these sorts of things you don't really think about when you're thinking about coding and software engineering and I don't think you know any University course would teach you you know how to choose a Wikipedia page or a you know a chat interface for all your users but it's it's really taxing and you know the the team was we spent quite a lot of time discussing this and thinking what should we do and that takes away from our time you know we need to make a living but do we spend you know 10 hours discussing what platform to use so you have to make decisions about what you spend your time on and these things these things you have to be um conscious to take up your time this is a graph here of the monthly contributions to micropython over 10 years um which is the taken from the the git repository so all the stepping stones that we've made each the blue and the orange uh the commits made per month so the scale goes up to there to a few hundred so there's a few hundred commits per month the blue is the ones made by myself and the few other core maintainers and you can see that sort of trending downwards the Orange is by a sort of contributors who are not the core maintainers and that is trending slightly upwards there towards the end which is really great um the green line is actually on the right hand scale there that's the number of unique contributors per month so who've made a commit to the repository so that's going up as well so this is a good Trend in that the maintainers are doing less commits Excel people doing more commits and there are more unique people contributing to the project so that's sort of you know that is not without a lot of hard work to Foster an environment where that can happen where external contributors can come in and make contributions to a project especially one where it's um you know you have to make sure that the code is still space grade in the sense that you can actually go up in a satellite and run and not crash the satellite so we have to be very careful about introducing commits loot repository introducing code changes and I think it's uh something to be proud of that we can get to a stage where we have a lot of external contributors so you know we have in in a year we have over maybe 150 to 200 people contributing and we have around an average of around about three commits a day which is a lot like if you sustain three commits a day for a year over 10 years that's a lot of work um this here so this is a plot of the number of people who have downloaded some micropython firmware to put on a piece of Hardware I like um an esp32 board or something the different colors are the different boards that your different ports um but different boards that you can use different targets architectures so the orange line is the Raspberry Pi rp2040 which is a really popular board that came out at the start of 2021 and that's sort of taken off a lot that got a lot of popularity the blue line just below it is esp32 which is espresso's new Wi-Fi board well not that new but you can see that started around 2017. the green was their previous Wi-Fi board the esp866 which started off earlier than that then the red is then stm32 which is the pi boards architecture so this is showing obviously that we're having more users or at least more people downloading but hopefully it's a good indication of of users so downloads per month so we've had over three million total downloads in so this is eight and a half years because we didn't start downwards you know immediately um so this is showing that the community is definitely people using is growing and the previous um that graph was showing you know people contributing to the code uh the way that's trending so building a community uh it's it's really hard and people have to do you know marketing branding Website Maintenance um I didn't really like doing those things but you'd have to do them anyway um some people are very understanding when you're part they're part of a community and they're like yes I understand that this is your time that you're spending on this um and that's really great um some people are less understanding most people have good intentions but sometimes people just like don't understand and get stuff wrong or get confused and are rude and you know they'll say oh this doesn't work and it should work and this is a terrible project or you know they could they get the bad end of the stick they come to some part some corner of micro python that's just not well tested and doesn't work and it gives them a bad impression of the entire thing um so it there is a lot of a varied sort of interactions that you have and people have different experiences um for example to make a bug report people are supposed to you know tell you what the bug is and make it so you help help you to fix it we haven't we have an issue template where if you want to submit a report there's a template that you fill out that was a we added that so that people would actually give good bug reports um but it's been almost completely ineffectual ineffective people basically just delete the template and write whatever they want and usually don't even say you know what version they're using what Hardware they're using this so like my Wi-Fi doesn't work um and you're like the complete loss to help them because there could be so many reasons why the Wi-Fi doesn't work um so some people you know most people have good intentions but um usually the good people are the people who don't say anything and I just quietly understanding the people who are who don't understand are the people who make you know take your time and you need to help them but you don't need to but you choose to um it's important to exit expectations but people rarely read documentation or anything so even if you have a Manifesto people are not going to read it and they're just going to behave the way they do anyway um and some people well a lot of people with micro python come to it to try and learn like how do I turn an LED on and how do I blink an LED and then they get stuck and then they ask for help and complete I mean this is great we're teaching people here but who has time to help them like do you take it out of your time to do that when you have other pressing things to do to write code or to maintain a website that's you know crashed or something so you really have to decide where your time goes and you know what you want to do and with a big Community hopefully there are other people that can step in when the questions are simple and as a maintainer you know you're needed for when the questions are really hard or there really is a deep problem to solve um so I think the main one of the main points is that as a maintainer uh it's important to make sure that there's Direction and that people don't get lost and be like oh there's no decisions here uh everyone has different opinions and we don't know what to do so to maintain his role really to set the direction to say out of all these choices we have we're going to choose this one because of this and this some people are going to be upset some people are going to be happy but at least this is a decision made and we can move forward and then people can deal with the decision and say okay well this is how it works and I'll I will work around that um but if you just let it sort of flounder and there's no Direction then I don't that's really not a good way to run a project but it is very rewarding I mean I you know there are times when it's tough and you don't want to open your email because there might be emails that are people you know not abusing you but you know it's not really nice to read responses sometimes um but then there are other times when you get really nice fruit replies from people or you know thank you it's a great project and things work really well and I've used it for this and that and it's really helped me a lot so it is very very rewarding when things work um a lot of people are thankful and grateful which is nice and there are definitely some standout community members that help and go out of their way without any without asking them and they help to you know Foster the community and answer questions and be there all the time which is really great um and as a maintainer you have to recognize those people and the people who have expertise and are willing to help and really allow them to do their thing and Foster them and Foster their activities and say yeah well just step back and let them do what they do well and also you know I ask people when I don't know something um I ask them and say you know you know more about me than this topic so can you please help um so that's that's good uh this is a an obligatory uh AI generated image it's such a fantastic technology but anyway this is just uh to show you the next bit is about bug reports and contributions uh so when as I said before bug reports some people do not give very good bug reports because it's not reproducible if you're going to give a bug report please make it reproducible and then at least someone can try and fix it uh but then on the other side there are contributions a pull request someone who has an idea about how to change the code and they're like I would like to change the code this way and when someone makes a pull request um you're like okay here's another 10 hours of my life gone because Apple request it I mean that was joking in a way but a progress is a lot of work to go through uh sometimes it's you know it's a few hours reviewing at the code giving some feedback and then they go back and forth for maybe a few weeks where you say you have the assessor changed has to be tested you review the code again and eventually it's good enough to merge and then you go through the last little details of making sure the commit message is right and all the tests passed properly and maybe the CI broke down a little bit so you have to fix that and it takes it takes a lot of time to get a contribution from someone in and that's not to discourage contributions because they're obviously what make the project work but as a maintainer and if you maintain or a one-day maintain a project you have to be aware that you need to do a lot of work to you know Foster People Foster the community and get the code in and get it working sometimes it's easier to do the work yourself and just say okay I don't know how to fix this I'll just do it but you can't do that forever so you really have to learn how to get people uh efficiently contributing to the project and one thing we do down the bottom there is we we try and automate as much as we can in terms of contributions so people will have uh a pull request and it will go through the you know code formatter and spell checker and code styling and even commit formatting and making sure they've signed it off and everything and those things are all automated so we don't have to worry about that and people can check their code formatting make sure it is correct fix it if it's not and once all those things pass what's left is to review the actual code which is you know the the meat of the of the problem um one thing to mention is that sometimes you know you get sort of lost up in a lost in a conversation or a heated argument or a discussion or something about code and you know I think a lot of us have probably been there and one good thing to try and do is to distance yourself from the actual personal interaction and talk about the code rather than the people and instead of saying why does your code do this you say why does that code do this so don't talk about the person's code you just talk about the code and that removes the sort of human there is human element there and that's important but when you say code doesn't work you can almost feel like an attack on someone who wrote that code so and say instead of saying why doesn't your code work why doesn't that code work so some maybe take home points about what I've learned it's hard to maintain open as a project it takes a lot of effort but I'm still here after 10 years and it's I still love the technical side and the community side is really great it's really rewarding you can't please everyone there are always people who will think that you've made a bad decision but you just have to go with it um if you're passionate and just follow your instincts and have some nice sort of like a Manifesto or a goal or a real overarching principles in this case it's micro python so microcontrollers and small and python so we've got to try and Implement python but we've got to do it in a way that's small so we're not trying to re-implement python so when people say oh it's missing this and this and this and we need that to match I'm like we can't because it won't work then like it won't work on a microcontroller so we have these two pillars micro and Python and we have to make sure we follow those for all the decisions that we make um so I think we're very fortunate to have a project that has strong ties to hardware and software so it's a real physical thing you can make something and it's also a software technical project and that really gives a good balance and funding as I said before if you want to do a project long term you really need to think about how you'll find the project maybe you can do it in spare time but in this case the project is large and we have multiple maintainers and some people working on it and that's the only thing they're doing so we need to be able to pay people to do that um if you're going to set up a project I suggest you know you choose your license from the beginning write a code of conduct pick a git workflow that you're comfortable that you like and stick with it uh make sure you have a nice directory structure so you don't have to move files around at the end uh version well from the beginning hopefully using semv if you don't uh read about it if you don't know um have some really good code formatting and linters and things set up from the beginning uh Choose You Know pick black as your python code formatter and clang format or something else um and stick with that because that will really help you from the beginning and then I think it's just some final words that I think sum up a lot of the you know a lot of the time during the day when I'm working um I feel like this is that when you put yourself out there online when you've made some code or when you make a change when you make a comment or a post in a community you're really putting yourself out there um and you know that's hard because you put your you know your person out into the world and when you're maintaining an open source Community I think you have to be aware of that and make Foster an environment where people are comfortable to put themselves out so when they put up a piece of code to review you realize that maybe put a lot of time into this or they've made a bug report which is really detailed and they put their self out there so if we have a community where people are comfortable to put themselves out there you know you can build that Community up and make it a really nice place to live well to live online and to work and I hope that's what we've done with macro python over the past 10 years has fostered a good Community where we have members who enjoy being part of it because yeah I do and I think it's it's it's been yeah a very rewarding 10 year so far and I hope that there is more more to come there definitely is so thanks thanks for listening [Applause] thank you very much Damian and we have time for one quick question anyone would okay oh look yeah I think I'll go for the close one sorry thanks Damian um how have you uh engage people to you know assist you with testing on I guess the variety of hardware and applications yes uh that's a good question we we have automated testing which is obviously people don't have to to work with but for Hardware testing uh people there are community members or people who do that when there's a like a when there's a pull request or a piece of a commit for a particular piece of Hardware that I may not have or that may be difficult to test other people will test it um you can just you know at them on GitHub and say at you know X can you please test on your hardware and they do that and that's great yeah so there are people who will test on various pieces of hardware for us and applications sorry application applications you mean um so people have got micro python applications and how do they test them yeah is there any pre-release testing that we have I have a few uh private applications that I do tests on before we make a release to make sure that they're still functioning and they yeah they do good integration testing of the uh well the whole system yeah um usually people will test the non-release code like before we make a release like they'll be like we need to up we want to use the latest micropy because it has a feature that we need and I'll test it on their project and then there'll be a bug report the next day about how something broke and we fix it so it is a really continuous process there's always generally a lot of testing from other people which is very helpful thank you thank you so much and here are your speaker gifts and thank you very much presenting today [Music] thank you

Channel: PyCon AU

Views: 1,569

Keywords: DamienGeorge, pyconau, pyconau_2023

Id: iEQQWpjgFd8

Length: 34min 0sec (2040 seconds)

Published: Mon Aug 21 2023