ElixirConf 2021 - Jason Axelson - My Scenic Companion

[Music] [Music] uh welcome to my talk i'm uh i'm talking about my project uh my senior companion about how i use um scenic with nerves to create some fun fun automation that i use um every day so a little bit about me first i started developing an elixir professionally around 2016. we were i was a ruby on rails shop and we had to add like a chat survey so we picked elixir and i just fell in love with the language i'm part of the elixir ls core team i've been just doing a bunch of different looks or things since then i maintained various projects here and there and i'm also a software engineer at felt and we are a collaborative mapping tool that is written in elixir and that's interesting to you we are currently hiring so come talk to me after and let's rewind the clock back to elixir comp 2018 uh that was my first elixir conf i was super excited to be able to go meet everybody at the community listening to launch a lot of great talks there's a really good nerds training where i actually received a raspberry pi 3 with a touch screen and i was able to witness the release of scenic which is a graphical framework for elixir that i'm using um as part of this talk and so when i got home from the conference i started thinking oh what should i do with my new raspberry pi my new hardware and so i started thinking i came up with a few problems but before we get into those i have to give you a confession because this is all all hobbyist stuff it's not for production use so if you look at some of the codes some of it is like very much like um you could say like over-engineered or i spent more time on that than i need to and then other parts are just completely completely messy so just a warning okay so the first problem is that it's i use i listen to pandora which is a streaming music service and i interface it through a command line client called pianobar and so in order to control music i have to switch to that terminal window and additionally i use multiple computers using the software called barrier and sometimes my desktop looks something like this and so it can be just really hard to switch to the correct terminal window and then in tmux switch to the correct session and then switch to the pane that has piano bar just deposit music so i decided to solve this with software problem number two my day can tend to be a little unstructured there's lots of notifications and emails and pings and everything so i need some way to kind of enforce some structure on it and i really like the pomodoro technique you've probably heard of it you spend usually like 25 minutes on where you're working like focus and you spend five minutes rest in between and they repeat that throughout the day but the problem for me is that i tried a bunch of pomodoro timers but i just didn't really like any of them they either had just a bunch of different things and i like building software and another the third problem is that my coworker which was my wife so we share i work from home like many people do now and we share an office space and if she's like right there when i'm starting a munich i can just like let her know okay i'm going to meeting i'll be aware of that but if she's like in the other room then she won't know so i had to come up with a way to kind of let her know about that because if she just walks into the room my webcam does have a blue light which is currently on here but you can barely see it even when you're this close and like when you're across the room you just you can't see that so the solution to all these problems for me is to build my scenic companion so it's made up of three different parts there's the piano bar interface in the upper left there is a pomodoro timer on the right that lets me do my pomodoros and on the third is a meeting indicator light that i can turn on and off so why build this so number one i think it's just really empowering to be able to build software for yourself it's nice you're the only user you don't have to do any user studies you don't have to you don't have to worry about as bugs as much in a sense because like you know what all the bugs are so you when you when you're about to encounter them you just don't do that behavior and you do something else until until eventually you just get so sick and tired of that you just fix it and another thing is that as programmers we really want to try and get into like a state of flow or we're loading like our whole code base into our head and and one thing that will break that immediately is context switching so for me one of those things is just fine switching windows because then i may end up on some random website or something so by having a separate interface that's completely separated from everything else i know exactly where all these controls are at all times and i can just reach up and press the button also elixir is really good for home automation and i just want that to be like used more and with nerves and so i helped release a few libraries relating to that and also i just i had the hardware so i figured maybe i'll just do something fun with it so here's a hardware diagram of the project the main thing is the raspberry pi 3 and the 7 inch touchscreen so if you look in front of me here i have the 7 inch touchscreen and the raspberry pi 3 is mounted inside this case and you can kind of see some of the hardware sticking out with like the ethernet cord and everything and then via the onboard bluetooth on the raspberry pi i connect to this guvie led light that's in front of me and then i also have an audio output the wires coming out the back here and that connects to my headphones so where does this live in my setup so i have it on the on top of my desk it's right under one of my work screens here's a close-up um and i have two different computers here so on the left the monitors for my personal desktop and then on the right i have my work work laptop and this monitor the guppy led lights all the way on the left side right by my wife's desk and yeah okay so let's go to the first solution so the first solution is the piano bar interface so this allows me to use basic controls to stop play and switch to the next song on piano bar so there will be a few demos in here and hopefully the internet cooperates with me because these are actually going out to a relay server which is kind of cool so here's the controls um let's see if the i don't know if you build it i only play a little bit of the music if you can even hear it but you can stop play and pause the music and so if i hit next over here um it'll switch to the next song you should be able to see it over there and you can see a little bit of the details looks like we're not getting too much audio but that's okay that's not the focus um and one thing i liked about this versus just the command line interface for me was that because the command line was always hidden away um i i could listen to a song for like three years and i just i didn't even know what song it was just because i never saw it i never saw the album art and one of the cool things is here if i click on it i can actually see like a closer view of the album art if i want to like get a close-up an earlier version had it much larger but that was just ended up being too distracting for me throughout the day so i switched that wrong direction okay so here's a diagram of how this part of my scenic companion works so starting from the bottom right over here we have touch events that are coming into the piano ui elixir application that i have that's running on the beam on the raspberry pi 3. so whenever i touch the screen that gets sent basically as a message to that application and then that application is connected via erlang clustering to my desktop and the desktop is where i'm actually running piano bar and my music um and so it's connecting first to piano ctl which is another little library that i or application i guess that i wrote and that's connecting to piano bar to control the music and the piano bar is what's playing the music into my headphones but let's zoom in on the connection between piano ctl and piano bar because that's not super straightforward so starting on the left over here we have piano bar writing expecting to pandora music to stream music to my device or to my computer i guess and that's playing music into my headphones but then when an event happens there's a configuration in piano bar called an event command and it'll basically run a bash script or whatever command with input about or data about the event that happens so examples being like song start and song finish and inside that command script command.sh i use the beam notify elixir library which is created by frank it's really helpful which basically allows you to take data that you have in like a bash script and easily get it as a message inside your elixir application so that's how i get notified about um changes as of last week then piano ctl is over here and it's doing whenever it sees a new like song is playing it notifies piano ui to update the ui and then when piano ui is receiving touch events it comes back into piano ctl and then it's connected to the ctl pipe which is a unix pipe and it's literally just doing like a file.right on the play or of the p s or n if they're trying to play pause or stop the song and piano bar sees that and does that command i'll show you a little bit of some of the interesting code in here so we have which i was going to show so first is the dashboard code itself so the dashboard is what is running um the main some sort of thing so if we go to piano ui uh dashboard so one of the things i should show here so just rendering various different texts so one of the things with scenic it is a very low level sort of library you have to do a lot of things yourself there's no like gestures and stuff built in um and so when you're rendering text what i'm providing here is like the exact location on the screen of where to put it so it's basically all like absolutely positioned but you do have some things that are kind of similar to html and css with like the text alignment and base that's based on that um let's see get the current song um one of the other cool things i did so when this is the message that's received when you're updating the song and i have a feature to actually like blacklist some album art if i don't like seeing it all the time so and that's driven by a um what's it called oh like sqlite database so it starts to download here um adjusting the size is really easy because i just adjust the um parameters because piano or yeah pandora has um i can receive like a 500 by 500 pixel image all the time and then i just download it in a download the cover image in a task with finch and then after that i just send a process message back to the process that started the task with the actual image there and then it stores it in a file cache in case it needs to be re-read um later and so the file cache i think is in this project also this is this is just a little file that i think is just kind of cool so you're just storing in this case five files on the file system and you're just checking if it exists so um one of the things is here is i i hash it with 256 and encode it and that's how i get the the file path that i'm storing it in and then whenever we have more than that five files we're just removing the xs files okay and then okay the icon component and launcher real quick so inside launcher we have icon component um so i named this scenic control because i eventually want to make it easily usable as a library um but right now it's just in my um project so this is basically a a like but or like a png button based component so you're passing in two different icons one for the normal state and one for the press state and then like an on-click handler and there's some some init stuff and you're rendering like the the main graph here from this render function which is a little bit lower and then we're doing a push graph on it which is what actually causes it to update the display and there's two different inputs we're handling here so the one here is saying that we're this is handling a button that's being pressed and so it is we do a render where we're passing true as the current the button depressed state and then we're recording the time and and then we render the graph and so when we're releasing which is the zero here we're checking if um the press time um it was well if it wasn't that we're saying we're waiting for this full minimum press time so basically this is so that the button you can actually see the button being pressed even if you're just doing a really quick tap or click so because otherwise you might wouldn't be able to tell if you press the button okay so i use that in a few test driving these buttons here on the screen which is a little bit of lag okay so let's continue so next up is the pomodoro timer so one of the things i like about this is that it's configurable um which means that like okay maybe i'm going to be working for like a normal 25 minutes or maybe i want to work for only 15 minutes that's all time i have before lunch i can just hit the plus and minus buttons there and it'll increase and decrease the time respectively and then also something that i added that i like that i don't know if many pomodoro timers have is what i call a limbo mode so when the timer gets down to the very um the end of your pomodoro it goes into limbo mode because i find that many times i'm actually still working during that and i don't know if i'm resting and i kind of basically don't want that to count against my rest time because i'm not actually resting yet i also the pomodoro time plays like some um sounds for the transitions i find that kind of helped me get into the zone of like okay i'm starting to work and at the end it also reminds me okay like it's time to stop and take a break and it's also logging every pomodoro to an sqlite database on the raspberry pi okay demo of that see if this is still connecting here oh you can actually hear the sound um it's not it's counting over here but not over here there okay so it's just the wi-fi is not the best at conference centers so which is too bad okay so there's that and i can do plus and minus here so i'll just go down to zero to kind of show the transition of at the end of a pomodoro and you can kind of hear that so this is limbo mode so you don't know like what you're currently doing you may be working maybe resting who knows but at the end of that once you're ready to rest there's a little break button over here and so you hit that and then you start your coffee break or whatever it is and you get a nice little coffee cup up there and once you're done with your break you hit reset and then you're ready to start all over again here's a little system diagram of it it's relatively simple the main thing is this pomodoro timer gen server so there's one instance of the pomodoro timer that's running for the whole application and basically any component that wants to receive updates about that basically subscribes to updates and whenever it's ticking or any other event that happens in the timer it notifies all those subscribers of the new state is like by passing extract and pomodoro ui is then responsible for playing the sounds out of the the headphones for those transitions some interesting code i'll go let's see the cool thing about the sound player is using a muon trap to make sure you don't have any zombie processes but i'm just going to use this the scenic renderer behavior which is in my pomodoro timer so okay so this is something i like for scenic because in scenic one of the things that's a little bit annoying at least for me is that when you're doing an initial render you're rendering like your full graph but then when you go to do an update you're doing a graph.modify and you're passing in the id of the component you're trying to re-render so what the scenic renderer behavior does is a elixir behavior that abstracts that away for you so there's four callbacks id and knit and draw and so id is just returning the id let's see so init this is just basic initialization of your component state and doing the initial rendering and inside and then whenever this component is receiving a message by a handle info it's basically just forwarding that on to your actual implementation module here as handle message so you can handle it and that's so that you can either do like a normal or any you can basically can the coin can say okay for this update i do want to redraw my state and it'll it tells this behavior that and the behavior will call draw on it to redraw that state and then update the graph again with push graph and here's the actual draw method so we're getting the id and we're checking if the graph already has that has that id in there if it doesn't this is the first draw so we're just drawing ignore this workaround it might not be needed anymore i'll have to check and but if it does already exist in the graph so there's something with that id then we're doing graph.modify and replacing that instance of the component with the new instance okay and solution number three is the meeting indicator light so this has a few states so it has light light is off and then when i press on it'll turn red to let my wife know that i'm on a call and when i press off it'll flash green three times and then fade off so let's try that here so press on and there it turns on flashes three times uh let's my wife know that i'm on the call the image is not loading that's okay um and then when i'm done there we go i with done sometimes it's like a long long zoom call it can be kind of nice to be able to hit this off button and then the light will flash green and after well after the wi-fi delay at least and then can relax and do something else after the end of the call shucks it's not working there we go thank you [Laughter] [Music] okay so here's the and their fades off here's the kind of diagram for that the main thing is the raspberry pi 3 has a built-in bluetooth module it's connecting to this govi h6001 um light which is actually pretty cheap it does only bluetooth it's on amazon for like 13 i'm using the blue heron library which is which is uh nice to use this is one of the use cases that's um well supported by it at least which is this is it only supports bluetooth low energy which look thankfully is what we need for this there are a few applications that make up this meeting indicator light um i probably could have consolidated this if i wanted to but i i don't know just playing around with different abstractions and ways of structuring projects um so the top level one is guppy phoenix so that's a phoenix interface which you may realize we haven't you haven't actually seen yet so i'll show that in a second uh there's also w74 which is what is actually controlling all the timing of the the light saying like okay this is the color green we want the color red this is how it's going to flash and then govy is a low level library that you can use that anybody else could use to interact with a govi led and okay so guppy phoenix is the next tab over so here's a very simple phoenix ui you can choose the let's see i think brightness next brightness and then color so you can just change different colors here figure out what you want something i like to do at night is to actually set it to like a sort of um purplish pink color so i can because i also use i also play games in my office um at night to relax so it's kind of nice to create a little bit of a different ambiance for that time and now let's not do that okay and let's see some interesting code here i'll just do the common commands i'm gonna be resources okay so this is the part of the core w library that lets you send colors to this um led so there's a bunch of constants at the top fan indicators saying like okay we are sending a command it's always the same for everything we're sending there's different types of commands we're changing either power brightness or color i don't currently use the timer feature and we're actually just using the manual mode but the python code that we were importing this from reporting this from to find the so i figure i'll define it for future use okay so we're sending a color we're passing in basically an rgb triplet so ffff is full white ff000 is red and similar for the other colors one of the interesting things about this bulb is that it has a full set of white leds that are separate from the colored leds so that somewhat leaks out into the api we use to call it so when we're building up this binary first thing is okay we're sending we're going to say it's okay we're sending a color command to it the led mode is manual so that's this first um byte here passing rgb as um three bytes i believe and then we're saying we're not using the white leds because we're trying to do color here and then we actually have to pass the rgb again this is just how they structure it i guess it also has here's like the white things basically this is just building up a a set of like common um numbers but i'm not going to show that today but it's codes in there if you want to take a look later and okay so here's build command binary so to build a binary we're padding it first which is just a little bit lower over here and that's basically just adding zeros on to the end until we get to 19 bytes or nine yeah 19 bytes and then we're calculating xor value and then we just create that as the binary that's all there is to it so overall it's not really too bad in terms of crazy code or anything and i have a few bonuses so bonus number one is asteroids which is not going to display well with the or constraints on the wi-fi but i will launch it you might be able to see something happen um or maybe i'll just have to skip that portion or actually you know what i did um where does that go almost over here i believe okay so this is a version that's running on my laptop so i won't have any delay so i have asteroids a little loading indicator it's just a clone of the 80s game asteroids you can fly around like in a ship and shoot asteroids it's not really the best use of scenic scenic is more for like industrial uis but you can do some games in it but yeah thank you [Applause] okay and then bonus number two is i have nerd's live book running on side of this device alongside everything else which i think is just really cool this just points to an internal ip address yes that's up there okay so here's live book running on top of my raspberry pi i have two demos here first is the simple demo here it's a little bit larger so you can do normal live book stuff so you can do like puts you can do some basic math 3 plus 6 or 15 it's like 18 but the the really cool stuff i think is interacting with the hardware that's already on the device so main one will be like this start meeting command so if i call this i can turn the led on right here and then similarly for finishing so you could use this to like do something more like rapid prototyping of a different code you're trying to do trying to tweak a module and then run it immediately and check the logs if you want the govy stuff is actually really noisy so there's a bunch of connection disconnection stuff here and you can play sounds okay and then the other one is for my pomodoros so on here i remember how i said i am logging all my pomegranates to an sqlite database on the pi so i can interact with that through a live book on here because just using this sqlite3 ectoscale lite3 library i believe so i can fetch all the pomodoros i can look at all the fields that i have it's basically like started at finished at rest started rest finished and total seconds of the power girl so that's i think 30 minutes there i can check like how many i have logged total there's 85 um i can check which ones have finished if they have a finished at and so i can see like how many of those there are and then i can do some basic analysis here to get the min and max but i'm going to kind of skip over that and on values here i'm building up a set of let's scroll down a basic thing of um [Music] to do some analysis like how many minutes has been for each pomodoro how many total how many hours it's been and so i can use that with vega light to create a rather basic graph but this is this one's a bar graph of the pomegranates and how long each of them were so you can see most of them are at the 30 minute mark and i did one that was 35 but you can see it over time and then you can do another other graphs like portraits per day so this one is doing an ordinal um ordinal graph and it and so you can see like how how long i worked each day or this is like combining like multiple weeks because it's not like doing it by time but it's just by so every monday so i've only been logging for like a a week as you can see here and also it's not even capturing the time that i'm resting or limbo time but i could add that with some more analysis okay so a little bit about scenic so scenic was built by um boyd multerer it's released during electrical 2018 it was a fantastic talk it targets iot and desktop devices more mostly like industrial type use cases it's going to be part of this criten operating system he's building that's really looks really cool it's like a security conscious operating system scenic is currently undergoing a large 0.11 update and rewrite um there's lots of changes there it's basically almost like a new thing i think it's getting close to 1.0 so i think after the 0.11 is released it'll be a really good time to jump into scenic if you're interested in that you should join the scenic channel on the elixir slack so i think scenic is really cool partially because i just really want to have other ways to build interfaces besides web-based technologies and i just i don't like running a browser on the pi just for a display and scenic and nerves is less than the electron minutes minimum size which is kind of kind of blows my mind just because scenic unknowns is entire operating system and then scenic is other good things like it's good against errors and has some kind of nice primitives and you can build composable uis on it you can take one scene and then um scale it down and put it embedded in another scene and then like rotate it and it'll handle like the click transforms and stuff for you you may have noticed there was quite a number of arrows pointing at my headphones the reason for that is because i do have it i have like a little audio mixer on my desk that controls the audio for all three computers and puts it into my headphones and i just like having analog interfaces and this is kind of a diagram of that i have these ground loop noise isolators because without that you get a lot of static noise and that took me forever to try to figure out so i figured i'd include it here to help people so hooking up multiple computers you want to look into that so to recap i had a lot of fun use building software using scenic and nerves i'm using some hardware that i already had on hand from that nerves training and it's been able to help my workflow because i don't have to do context switching to try to find like a pomodoro app somewhere or my music or my meeting light controls and i just really enjoy being able to use software that i built every day that said there were some challenges along the way so it can be a lot of work to manage all the dependencies i have some of that is self-inflicted with all the different repositories i created but some of that is somewhat inherent one thing to note if you are trying to run multiple applications on a single beam instance like on this raspberry pi all the applications have to you can't have multiple versions of a library so you can't have phoenix 1.5.1 or 15.1 alongside 1 to 15.2 or whatever so you have to upgrade everything all at the same time and also managing all this elixir configuration because some of my code is more reliant on that than it ideally would be it can be a little bit of a chore because i want because i wanted all these applications built to run independently as well so people can try out scenic or whatever so here's some future work that i want to do i want to auto sleep the screen if i'm not i'm interacting with it have the music auto pause after a few hours maybe rather than just sending these like random commands back and forth between piano ctl and piano ui i want to have some sort of like shared state or something more elegant i guess and then the big one is that i want to come up with some sort of excuse to use membrane in this project because membrane is a audio video processing framework that just looks really cool to play with so also my senior commander can become your senior companion these are all the repos it's all open source there's quite a few [Applause] and i also want to say a big i want to say a big thank you to all the maintainers these are all the libraries that i depend on directly none of these are the transitive dependencies there's a bunch on there you should um check check some of them out and uh thank you for coming [Applause] any questions yes uh i would yeah i would love to do that it's about finding a hardware that is easy to kind of pull in and and use without and there is like a little space on the bottom left that isn't used so that might be a candidate yes um how do you um start it up you have to have a desktop client running first or yeah i turn it i put it to sleep at night um but starting it up is really easy i'll just restart it here so because i'm not actively writing to the sd card it's generally mostly safe to do that but right now it should be booting i can't see it it probably takes about 30-ish seconds to boot up yeah but i'll come back up to the comes back up to launcher because you have the desktop client controller right yeah so i use lib cluster to reconnect those um yeah although i don't know i have to look into it it wasn't working quite right so i actually just have a little gen server i added that pings to try to connect every uh five or ten seconds do you remember what uh strategy you're using i think it's like epmd between the devices nothing i think i would call a gotcha just about learning about how to how to do that i'm just using a basic like rpc call i think so it's pretty naive i guess yes yes so that is going to be coming out soon is what boyd asked me to say but yeah so what this is doing is actually so it's actually relaying so it's going from my from the device here up to his relay server and then to my computer so you could be accessing this device from anywhere if you are logged in to this account here so yeah stay tuned for that it is a canvas yeah canvas look out the window i'll do dishes i find that's a nice little something physical rather than mental so anything that's not mental generally yeah i do generally yes i have a little bit of be about you'd have to it can be a little bit hard to run random binaries on nerves but i think if you were able to statically compile it it should work so in that case it would yeah yes the color picker i think that's the the default color picker i think it's just a default color picker in whatever browser i'm using which i think is safari if i'm correct but this is using tailwind styles generally here okay last call any more questions okay thank you [Applause]

Channel: ElixirConf

Views: 778

Keywords: elixir

Id: wCxMSo3TZjw

Length: 37min 16sec (2236 seconds)

Published: Fri Oct 22 2021