Event driven Postgres with OpenFaaS

[Music] hello and welcome to today's stream on my channel we're going to be talking with lucas rosella from continent going to bring him on to the screen in just a moment and thank you for tuning in this is probably the third stream that we've done now with open files and serverless topics so without further ado let's bring on lucas hi lucas how are you where are you joining us from today uh today's well germany as i've been well since like everyone else kind of stuck where they're at but at home for a short time i was fortunate enough that we had like a small office and i could bike in but the rules have gotten strict enough that i'm committed to home working from home yeah it's very common at the moment and i see that you've got i see a little bit of tissue in there not an open and fast ones you're not letting the side down honestly oktoberfest so under who's if anyone in the comments has got a hacktoberfest t-shirt or an open fash this is actually the new one in the store uh new design that we've put out there loads of different colors and yeah i i think i got my t-shirt this year too and uh openvas has been participating in that program for a while quite a few people that have got connected through it so tell us a bit about um postgres so i think you use it quite a bit at work or you have done and what it would mean to sort of dig into the events what kind of things could people do with that sure so i guess i've been using postgres really since i have been programming real because i went from mathematics to web dev pretty quickly and so what's that 10 ish years now i mean postgres is a pretty easy choice if you're in startups you need a open source database something that's cheap to deploy and i guess the two options are what postgres in my sql and there's others of course but those are the the big ones so if you go to cloud providers you're definitely going to find those there so postgres is just probably by chance of that's what the team i joined was doing and then you just stick with tools that you become better and better at and i find myself on every team that i join being that guy that kind of likes sql so i think about it a lot i kind of keep up with what's going on there and um as far as the events like the the things that naturally come to me are because i'm in this kind of web dev background web hooks are an obvious kind of place to start and depending on on what you're doing i mean web hooks are about sending external tools to you know a notification about hey something inside my app has changed but there's no reason why those have to be completely external so especially as like these service oriented architectures microservices are more more popular you know depending on who you talk to then uh those those external things i mean they could be your own services that you've written just saying hey this is this has changed here's an update uh one of the things i've always kind of dreamed about you know developers a lot of the more front-end focus developers always like hey we want websockets and they just want something to listen to like hey let me know when something has changed and so you could probably create kind of just a simple notification channel to your front end even just saying here's an endpoint i won't give you necessarily all the data maybe you have privacy like authorization concerns but you can at least say hey here is a object of this type and here's the id right so you can do things like this pretty quickly if you could get the the change notifications there's various ways of doing that it sounds like there's a lot of different options that could be used there i just asked the people that are watching in hey uh medi vivec fabrizio technologic um who's using postgres and medi replied back he's still stuck with my sequel and there's actually maybe not quite fair right because i've seen some cool stuff done with my sql so there's a bit of a licensing difference so isn't there as well between the two big databases there and postgres is licensed mit a bit more permissive and then it's my sequel gpl3 oh man i'm not entirely sure i've only used it in passing yeah i used it at university and it's how i learned about relational databases and normal form and yeah there's a new version of it there's a mariadb from out there as well and i guess when i when i first got into it was like okay store procedures when i learned about them it's like a whole new world opened up to me and suddenly i was able to write code and functions and really enjoyed it and then when i went over to postgres i thought it would all just work the same and it was very different um i remember actually doing the blog post on the serverless single page app on openfast i don't remember if you remember that on the blog yeah um and what that one was i'll paste it for everyone we had a little leaderboard let me see if i can maybe share this up and to begin with we were just tracking basic statistics from like github's api and this example let us go and build something that was a bit more event-driven so as github was sending events we were collecting them in a database then able to show them on a on a leaderboard and this is actually still out there pretty sure um yeah i think it's here on open fires cloud so that's the raw data i think it's page and this is probably scaled to zero so be coming back back from devnl um there we are so i just thought i'd show you this since i've got you here if anybody can see that but the second biggest lucas and then derek um and then we we've opened lots of issues we've been commenting a lot as you can see that level of activity so that was kind of fun to build and it reminds me of what you've put together because um you know had something similar github was sending us events and we were putting them in a table then we had a nice way of sort of querying them and showing people what that data meant and so that was probably one of the earliest examples now your blog um is relatively new isn't it i think you read did you redesign this recently yeah i had a so i used hugo and i had and they have great templates in their i guess hugo store um but i had this stitch to try some html and css again so over christmas this was my experiment so i took inspiration from a couple of the templates and uh tweaked it from there okay we've got a couple more postgres users there now i'm a big fan of hugo as well we've been using it on the um in lats website so talk us through like what were you sort of aiming to do with this post what did you want to get to get out of it sure um one of the things i mentioned in there is really the inspiration and this is something that i've thought about or it's come up um you know the engineering the team i'm on right now there's a couple engineers that are always kind of like oh we should do more event driven machine work so it's just a topic that i feel has kind of followed me not following me but like it's always kind of in the atmosphere a little bit and um when i've thought about it in the past uh with postgres and what i did here is using something called the wall this is the right ahead log so this is the tech one of the techniques that postgresql uses for its persistence and for also for replication so when you when people talk about uh having h.a postgres usually what that means is shipping the wall or chunks of the wall or there's different techniques depending on how you're doing it but basically the wall powers all of that so it tells that's where all the changes actually live and what kind of inspired me is that you had this release of fast d and said oh by the way postgres is available like we can actually integrate this in now and i had actually a misconception uh i thought oh if i want access to the wall i need to be right next to it so when he said oh fast d is now like oh now i know everything's together and now i said oh now it's possible one thing i did find out is well technically you don't actually have to be right on the same service or inside of postgres access that wall at least not the technique that we that's used here so that was something i learned but that was kind of inspiration it's like oh i'm there i'm next to postgres that means i can get to the wall that's me let's see if we can do something so it's really just kind of trying something out this is one of the first projects you mentioned wasn't it well to jason i think when you sent that to me i thought where has this been all my life basically i can get json out of out of the writer headlock and also reminds me if there's a project we often talk about open source within our community and sustainability there's a project by ben johnson and he's writing something to replicate sequel light and we're talking about blogs hugo and jekyll i run a ghost blog many of you will have seen that and with that i'm using a sequel light file for everything and so occasionally i do sftp and get a copy of it but he's produced this thing that takes the wow reads off the top of it and pushes it to s3 or wherever you like so there's a couple of places where i've heard about this recently yeah and it's all the json i've heard of in the past as well i forget how old it is but it's not exactly uh you know it's not a one month old project or anything like that and but i never really used it because at least you know there's definitely a chance i misread it but the way i understood this is um and the technique that is shared with what i ended up doing but just with a different project is they use what's called the logical replication and this means that postgres kind of creates a channel inside of itself to ask for all those changes okay um not in json format so that's what this does it creates a logical replication slot so it behaves like a receiver as if there was another postgres on the other end receiving the changes that need to be run in order to replicate itself and then uh it just parses them into into json so you always have to connect that back out of the table yeah so that's what i found is that that wall to json seems to behave like a plug-in so you have to have that on your uh on your instance your postgres instance and then you still need another thing to do that select so that's ultimately why i didn't end up using it because when i said okay i need something that does that and then i started looking around a little more and that's when i fall uh found this wall listener which i didn't use directly i tweaked it a little bit to just polish or make pieces fit a little bit better or just to debug some stuff um but it kind of behaves the same way while the facebook does what it does that's why then yeah is so good what's this written in golang yep so that's the other obvious benefit because the last four or five years have been all go and we'd use a lot of go and open fast so it was very natural for me to look at this and say oh well i needed i forget what i even i needed to just debug one or two things and then i wanted to tweak the way that it behaves with mats so this tool here wall listener um and the version that i i hacked on what it does is actually sits there between your postgres and a gnats sort server and publishes to a nas topic and the original version of this would post to a topic based on the schema and the table name so this is what here is it yeah and is that what we would potentially get then so you've got nats deployed uh and then somehow this connects in and we'll publish those events onto a nas topic okay yeah but so the one of the tweaks i made is instead of going to very different topics i go to a single topic so for demonstration purposes it's a little bit simpler than needing to know your entire schema and binding functions because i wanted an open fast function to read all of the events but if i need to know all of the tables because they each each table becomes a new topic then that becomes a little bit so you weren't put off at all you just were like yeah i know go yeah exactly and did you also make it multi-arch your voice yet yeah maybe next might be the next thing you do yeah and uh i just asked everybody is there any any any gophers and we've got one person knows a little i can't remember everybody's aliases who you are um i'm not sure if technologic is matty from get here or if i'm getting mixed up um and i know there's a few of you here that have contributed to open faz so we definitely got some other gophers in the house so you did that and we also talked about the nuts connector now this is something that might i mean we have nuts in two places in open first let's just open the docks here um we've got some stack picture i mean the first obvious places open fast comes with async support out of the box and that's powered by nets yeah yeah so if anybody's ever deployed a function it normally just looks like this and then the only thing you need to do to make it async you just change the the prefix of the url and the way that works somebody asked about this recently so we take the message we end queue it to nats and then that's it we're done you call it immediately back and if you want you can have a callback url you don't have to have it and then a q worker will pick up those messages and fire them off and so tell us how the nats connect is different i wonder if we've got an image in this one or not maybe don't how's this one different this one's a little bit different because um i guess it abstracts the pattern at least to me it feels like it's taking the same pattern but it makes it more abstract because it's not depending on the http request to like figure out oh i need to put this into a queue and then go to to a function so instead the nets connector knows how to you have a nats on one side and you have functions on the other side and it behaves like that glue it looks at your gnats says oh there's a top subscribes to specific topics and then knows that there are those functions waiting uh for those topics so that's all done by annotations here we are yes this is what you're talking about yeah and um this originally this sdk was built because we didn't really know how to do events in openvas and what people were going to want so i think i put this together originally for kafka and the idea was just like lucas is saying you have each function will have an annotation that says these are the topics i want dot checkout for instance and then as an event comes from some source this one component would then invoke the function as required and so with the async you don't really get this it's just one place things go in they come out they get run but with what lucas has done you effectively are you using one that's topic then for all of your wow messages and then they contain the database name as well in the schema yeah and so then that files off to the function which is good okay so talk about deploying multi-pass um and we also yeah it looks like i was right there yep get here let's have yes we know you're right go you you're very good at go i think if we go back to that leaderboard you find his name on there yeah we should have we should see well i think we can do one better than that it was last year that it's i've got up on the the openvis team page there you are with your nice blue open first t-shirt and uh medy is looking to get back into it again we'll let us know we always need people to sort of give us a hand even if it is just testing out the examples in a new post so fast d then what's the the cool difference between that and just going to kubernetes with open first well i'll start with the the main similarity especially in upcoming kubernetes is at the core they're both using container d right so the the very fundamental technology is the same but kubernetes brings i mean it has fcd and it has this control operator pattern that it's the basically the first example of so it brings that api and connecting multiple different nodes together and fast d is meant to be a much more slim version of that running on a single node just using container d with a very i would say a much smaller simpler but uh still powerful api for that um container d so they effectively match don't they so when you do if deploy to fazdy it's the same as doing it to keep openvas and kubernetes the connectors go in a little bit differently to how they do otherwise how did you find deploying the wow listener to fasd versus having to do it you know i imagine you'd have had to create a helm chart and a repo and all sorts for kubernetes um i mean in art for fastly we actually i think par so let me back up so we had fast swarm for a while that kind of sat in the same place as being this smaller alternative um and so we were kind of used to like this docker compose format and then with fast d i guess recently in the last let's say six months docker compose became its own kind of spec independent from the docker itself and so that's something that we've adopted which means that if you're comfortable with docker compose then you're probably mostly comfortable there's some limitations still it's not a full docker implement docker compose implementation but if you've added or created a compose file and added a service to it then it's really simple and we use docker compose at work for our local dev and kubernetes for the deployment so kind of thinking of both of those is something i do pretty frequently yeah and i thought it was pretty simple to add this we've got 16 people on the live stream now which is great i'm sure we'll have other people watching afterwards if you use like a compose hopefully you find this quite easy to add um now the first thing you've done is you've added postgres and that's in my new ebook as well that sort of talks about how to get all of that set up so we won't go into that in too much detail you've got one customization there though what's that one yeah and this is that one i think there's some other ways of doing it but this is the one that just worked most consistently yeah so if you're you know if you're operations postgres expert maybe they have better ideas like passing as flags or env somehow there's really two um settings right at the end of this uh yeah and this is just the default file and i just tweaked like the two lines so one of them is which line am i looking for i think if you look for a logical if you do a search on that yeah uh that's one i had to tweak or that i specifically came into tweak and i think there was one other setting that's how many uh i have to look again yeah through the whole file but there's like another like maximum number like there's a number of like little slots i think and then we've got the well listener again and then you've overrode its configuration because in open files with all of our functions you'll notice that we're not mounting files we're trying to write everything with this 12-factor app approach of it's either a secret that's a file for confidential data if it's non-confidential it's an environment variable so then when you have a file to mount in it's slightly more work but not too bad with this compared to kubernetes and then yeah okay i mean in this case part of this is coming from that i started wall listener and it had this file format first i think it's it's using viper but i don't think the viper under if if you're familiar with goes one of the packages is viper for configuration yeah yeah yeah yeah and so it can do it can bind to environment variables to flags files and i think it just was set up with files to start with so i kept it pretty my modification is pretty light so we've got insert update and delete those the things that we care about we've got and here i also it's specifically on the the users table so i think if you remove this then you just get everything that happens okay but for this demo i was like hey i know i'm going to write a user's table i'm going to put that in there it allows you potentially to because this little wall is center it's not very heavy you know it's a very small thing you could have different listeners pushing to different topics okay depending on which like maybe one thing is public and something's private you know you can have different uh things happening there and then we've got the host for postgres we've got the port and then that's again because these are components that if i've still got it open here they're just part of the file so then we have your wow listener we've got depends on which we're using to get the order right and then the connector which is the one that we just touched on earlier and it's saying what topic it wants to listen for so if you did have in multiple versions of that well listener we could have one connector per topic or one connector that listens to multiples and again we've got the depends on order there it has a secret for the first gateway so that it can list the functions and find out what topics go where we then got all of the standard bits and pieces that we would get anyway and we've got nats as well so here i'm not adding a separate nets i'm definitely piggybacking on the gnats that open fast already has yeah depending on your if you were to actually go to build a real app on top of this you may or may not want those to be separate so then you you go ahead you install it you go and you're deploying a sample function what language is that one written in i think that one's also in go okay yeah this is where part of the reason why i had to change the wall listener is i remember a while back we um kind of formalized that nats connector and they updated their package their library so that's where was it went from stan the name of the package was stan but now they have a package called matt and stan is of course just gnats backwards i didn't know that i didn't work i did wonder where it came from but yeah so now there's official like nats package called nats and so i had to make that change before everything really came together so people want to know like how to play with with go and with nats go and have a look at the nats connector and the open fazorg and then we have the connect the just the shout out there because we have the connector this function that i use is the sample function we have a a read and a write sample function okay just like testing the connector and that's what this thing is so here i didn't create a brand new function for it i just wanted to prove that it could all work and in that connector repo you can find these test functions which are write or publish a message you have to receive a message and of course the connector itself and they're all various difficulties right so like the publish a message is very straightforward same for receive a message so depending on what you're looking for you can kind of find a bit of everything in that packet in that house back at this point yeah okay well i guess we should build ourselves up for a demo soon um would you like to would you like to take over sure do a little prayer to the demo gods see how this goes here let's see i need entire window i believe and so you're probably seeing yourself and now a nice empty terminal uh so on there okay go for it so this is actually inside of a multi-pass this and ubuntu fasty i created one just before uh this and i installed uh i think i double checked the git was there and then i installed make and i installed fast d so i think if i type history you'll see some of that so i just follow the basic instructions from the fastie using the install script uh double check that everything was working so let's see shouldn't be anything there yet store deploy make sure everything's up and running and happy there we go so this if you are using uh open pass there we go we've probably done this done something similar before uh but the the magic that we want to do and hopefully i can remember my keyboard shortcut i've been swapping between linux and mac a lot recently so it's annoying yeah command and control and the copying and everything yeah so here i have class d which i use for the install and then if i go on open fast wall listener um you said that you had one small problem with postgres uh but i never got a confirmation from you when you did this uh i think i missed a step so was it just adding that uh directory yeah so if you have the if you have the ebit you could open it up and find your way to it it's got all the chamods and mk does everything i think i have it on this laptop but okay by going here the folder i need is [Music] yeah basically all of the ones that we mount are put into a relative directory and uh oh it sounds like you need to to have another go with uh with multipass i'm not sure what it was like a few years ago i only really recently found out about it myself and um let's see i need medi still deploying swarm oh dear wow so let's do this you can't win all of them i like to write make files yes i noticed that um maybe like isn't quite the right word but i like having my notes kind of in script somewhere so yeah yeah this gets me pretty far um but the thing that was this is where my keys are oh this won't work on uh this is the part i don't remember the password on multipass yeah you just have to find it what i what i tend to find things for that itself is that yeah i i ls it in faz d and then it's in secrets so i know i know i need a folder that's quite what you're looking for i'm trying to i'm trying to get to i need to create a folder in here right so in my uh sudo buy it um because the permissions are set up so that you you're not meant to really see what what's going on pseudoapply it yeah to get a root shell and then you can cd in the folder yeah so then i go into var lip fasti and i i just kind of do what i need to do in there so what we found out and you know you and i were chatting about this probably late on a sunday night as we do is that the containers with docker and container d you have to be the same user on the host as in the container to mount a volume um and so yeah you need your data unless it's called pg data i'm going to see if i've got my data yep ebook on my site okay i'll tell you i can give this to you on slack probably just going to be easier if you still got it open this would be like the cheat sheet basically you can copy and paste all of that in okay so that should i think that now we have them i don't know where my slack is anymore there we are yeah i need a both of these it's pg data and we want one for run and that's all the lock files so if you run those commands as sue dash i it would just be easier to paste in i want it there we go and then have you got the run and you've got now and then you would both of those are there i'll show you in a second this is the part that i was following your book and then i didn't write down yeah my my little script for that kind of slightly different so if anybody follows a blog post luke's this one might be slightly different with their names and amounts but i'll i'll add these um in a moment to the repo so that they're looking at your your your side of things you have got pg data and you've got run so you've got that now you need to see how them as as one thousand yeah and so you'll need sudo for that and you can just do the postgres folder who knows is it postgres or postgresql how you actually meant to say that projects spelt both ways everywhere i needed to be capital r for that as well yeah like this i think and then a thousand on the other side as well yeah i've seen notation before where like other sides empty but yeah so i think that gives us everything and okay what else is there to do there is that's too much changing creating the directories that you're going to mount you're not using a token like we use in a book you're using hard coded you've gh owned it so now run the daemon reload and restart first so i put that inside of so i have a make restart for that as well which will restart both of those and then don't really need to restart for this uh i think you're definitely i'm pretty sure you're right uh because you wrote it but i've had i think a long time ago issues where it didn't quite behave overly cautious okay so we've got a little bit of a problem what's gone wrong there you might have to control see it before you get restart again it's that conf file there okay i think i know perhaps i don't think you have the comp file because we just created that folder so now you you just copied it in didn't you uh what's your local folder postgresql or postgres we're in this just yesterday say we still the jury is still out is it postgresql postgresql what should it be guys let's go sequel [Music] so let's have a look at your one right so you've got that that's what we're expecting and then so what do you actually have in var lib this will help there that's where that file was okay so you've got both of them yeah you have to she had shown both of them then actually i i have this up on mine as well um from earlier i was i was able to get this working and i didn't have to change anything uh anything really chris is saying it's up to you apart from chris if it's in a blog post and it's a file name and you really gotta keep an eye on it then but yeah it seems like a cube cuddle you decide right so have you got a valid fasd prosql postgres conf that's my question for you yes you sudo ls right with ql i think if you look at the message there i'm saying i can't mount yeah because you have postpress folder not so it postgresql postpress pokestress there we go i probably forgot to um i think you have a little mismatch on there um i think this one's gonna work it did work the the problem is if you made a change you need to make sure you copy it back so i made a small change to the docker compose but didn't reinstall it all right show us what you got then okay so we have that um what else did i say my readme so the next step of course is we need this installation we need to add a function that's going to receive all these these messages so in in the install in the restart what we've done is we've added we have fast d running so we have a place to put functions we have a postgres running and we have the wall listener running so we have all kind of the pieces there but we don't have any function to respond when that happens so if i grab that and if anyone's not seen it yet this is the other way of deploying functions as if you already know the name of the docker container and everything else you can just do it on one line or you can use the snap camel yeah for simple debug uh functions i like deploying it like this just because it's easy to say like in a blog post or to give to someone over slack or any of these like here's one line that you need okay so this should deploy i was on mute earlier but what i was trying to say was that in fasd the deployments are synchronous so if you've ever deployed in the ui i don't know who's seen this and you're like did i click that or not what's actually happening is we haven't got in there yet to put the animation in the ui but it's synchronous instead of async because there's not a cluster we don't have to find out what node it's going on we just simply download the image while you wait and that's what we just had there and i guess if we wanted we could do the same thing we could create a publish event just to let's see if i remember let's see this is correct why don't we do that if we get time after we've fired off so now that that's all there you have database you have the listener we have function receive so the the last thing in my in your tutorial you had like a test message but i think we should go straight to this one yeah so here all i've done is i created something that will log in to the postgres and then just run some initial sql it'll create the users table for us insert a couple we'll upsert a couple rows and then what we should see if everything's good is that our function receives i said four rows we should get four messages inside of the function so here if i do make what did i say was and if db so let's get our fingers crossed for you here oh i didn't start this is what i ran into as well um man i wish they could decide what they want to call that project everyone's holding the fingers now that that is there and okay here we go so the in in this case you just type the password in yes i typed the password in in the blog um no it's not explicitly stated there but the password in this case is that's because it's simplifies any sort of like okay debugging it's this super secret value super secret so i type that in and what you can see here is it added extension because i like to use uids instead of integers and i created a table added index because i wanted to ensure uniqueness so i could do my upsert and then we have john wick james bond elim garrick and sarah walker all as users of our supposed app and now if i do facili logs if you can push that up to the top of the terminal then i'll put this message on i sell it correctly so i think john wick is uh on netflix i haven't caught up with that one yet i think we haven't seen the last one we've seen the first two i don't think the last one is there okay so something did not happen so how about getting the looks of the wow listener maybe the logs of the connector as well journal tpl how do i list should be what would you want the looks for something yeah should just be wall listening yep and then the the connect you can look at the connector as well so journal ctl and then you want a t i need to run that possibly and then open first colon wild listener there we are okay couldn't die the db but what time was that that was six minutes ago interesting what else have we got in there uh that's connector any any output on there how's that looking we go down to the bottom we've got lots of little pings going on capital g takes you to the bottom and oh interesting i wonder if that's all out of open faz cluster i'm not doing anything now because there's okay so it looks like you very close to getting it working but the wow listeners upset so the typical thing from the it crowd and if anybody can can you remember what it was on the it crowd turn it off and turn around again right let's see we did we did have um we did discuss this that there was a problem connecting but the order was the main culprit so in that docker compose file that's why see if i can make it if you can't get this work and now i'll switch over and show people that uh that it that it works on mine at least a little bit earlier okay so here we go so this is connected so it was probably just a hiccup okay yeah we'll let you off for a hiccup so can you do your db insert again this time ah look at this received wall message and you can see it acknowledging the message you can see it so now it's did something with it worked right let's see your connector and it's just connector you called it you didn't call it that's connector that threw me off we're getting more prayers here from cosy so the nats connector i mean here you can see it received the the message oh we got the message of doing something um [Music] receive receive event got the message what well done i wonder if there is just a hiccup here with okay just the order in which things are starting to click and you know if this gets a bit technical for folks this can all be done on kubernetes as well exactly how you spelt your function you're not done i you've done i before e but not after c either oh man oh man and our logs our logs thing doesn't tell you perhaps everything oh that doesn't exist you know that's a mistake i make all the time with receive i should just learn my lesson and not use receive as a as a name i before e but not after c that's where you got a memory so here we go yeah we've got the data so have we got john wick in there yeah you can see it started with wick um and then james bond garrick and walker and that i think if you come up you can see the same order that it produced here that's awesome and the way that i designed it is every time i run this so if i if i stop you can just keep running it because it's doing an upsert okay and you it should only change the update of that time and that's to be fresh and then we can look at the logs we get some fresh events here uh 1550 which is it's after that yeah yeah that's now okay so given that we've got json that's nice and easy to work with programmatic what's your next step we want to do something with that in the function what what would that i do as a development so at this point there's a couple options so either you're writing an app where maybe it's private and this data is one of the things you should notice is it is pretty um i'll say verbose it's not you know it has the email in here it has a name it has all those values that are inserted so if you had values that you didn't want necessarily publish somewhere so but you're building an app internally so it's okay for it to see all this maybe it just responds to this event directly and does something maybe mutates the data and shoves some more shoves it in another table somewhere um maybe this triggers a welcome email you know these sort of things so you could build something directly on this event you could also use this as kind of like a fan in fan like a pub sub right so you have something that receives the raw event and then it can turn around and make more new events that are uh say appropriate to share as like a web hook or something with an external team so just to put the case in point we know that your schema looks roughly like grab it again this i'm just sharing my screen at the moment if you're confused what just happened um that's what we're looking at and so i can if i get the body in event dot body in the node template i can go p dot action if it's an update on a user record let's say uh yeah let's say it was a user record and then p dot data let's say there's an effective date which is so we ended a customer's sas integration with us the effective date is less than or equal to get date something like that but we don't have effective date we don't have it yet but we could we could have it we're doing imaginary programming here and then what i could do is like you say is some sort of sdk aws ses dot send email um leaving us template.html and then pass in the data dot email into that perhaps and this is pretty similar to something that i put together for um people that bought my new ebook um serverless for everyone else all about fazdy and i ran an offer still there for a couple more days if you bought the book on the fifty dollar tier email you the video for free and i built it with fasti i get the web hook to my raspberry pi once i've got the web hook i then look at the dollar amount and forward it on to email function that uses aws ses so pay 0.12 dollars per goodness knows how many emails they get fired off people get their emails it's all good and it looks a lot like this so what i could do is take those pings from gumroad put them into a database table instead and forget about them and then the wall listener could be triggering the function to email people and if there's some way i don't think it exists yet but of upgrading the amount that you're paying for a book we could then potentially like email you say thank you for upgrading and if you look back on the channel not that long ago i spoke to david mckay and what he was doing was he had this headless cms and this is pretty similar we can turn a database into a cms with an api so one function to get stuff another function to put records then webhooks to fire off other functions when different stuff happens yeah we're gonna schedule another live stream next week probably okay well we'll insert it with one function that might trigger another one to go to youtube publish the link you might use banner bear to get an image for that the open first equivalent update the image to see who's going to be on there so yeah this is this is kind of where it would be next i don't know what ideas people have got in the chat we still have quite a few people watching 18 now did you have any ideas if what you might try this with lucas and do you think it's ready to actually try with real stuff um i think you know when you look at all the pieces that are there so the only thing that's kind of new and young is this wall listener and it frankly is not very complicated so most of the pieces maybe people trust gnats people trust postgres so if you have these things sitting around or maybe not sitting but being used then you could definitely take wall listener if it fits your use case there definitely some rough edges maybe you could say that would make it fit a little better with like the configuration or you might have to think a little bit more about okay which table filters or which events so i mean these are the things but these are things you have to think about anytime you design something that's kind of event driven which events do i want to respond to and and listen to duplication yeah the duplication so like one of the things you might want to do cassiano said here um having something like this would have helped me some years ago being able to cdn cash in front of something the cash would then invalidate when records are changed i actually heard from an open company using open files that's what they were doing um they were generating html with something like hugo micro segmented the website but they're not using this yet and they might you know after watching this they might just give it a try or that for that use case this i think would just work you know i wouldn't have any concerns with that at all with uh all that code that's there so i guess we want to wrap things up what uh what's your sort of closing thought and has anybody got a couple of questions we probably get a couple in about fazi or what lucas has shown you i guess the the last thought i would like to add is part of the reason i also designed it this way is i looked at i wanted to add as little bit of code as possible i knew i had to add some code and of course i have my function that's going to do whatever business logic so i wanted to add as little bit of code as possible so part of the the design was oh we have nets connector that's why i went with is there something that turns the wall and connects to gnats can i hack on top of that and i think one of the things that's really nice is if you do have mysql mariadb any of these other databases i think the pattern can definitely be adapted and i think that all those databases have something that you can say is equivalent to the wall and then some sort of api like logical replication that postgres uses that you can connect to and do exactly the same you can just plop in your database and stop instead of postgres in the diagram and then put some replacement on top of wall listener and then the rest of it kind of fits together so i also kind of thought about like how do i do this in a way that makes it pluggable which is why like wall listener doesn't call a function directly or anything like that i wanted to reuse kind of the ecosystem as much as possible yeah so if you have alternatives my point is you can do this too right you can fit this to whatever tools you're currently using there's somebody chris just asked if they're similar for my sequel mirror db etc and i think you mentioned that did you mention that in your post uh i think i mentioned at the end of my post so when i was doing the research there's another project i looked at called lapidus i don't know where the name comes from but it also supports postgresql but then supports maria to be mysql and okay so i wanted something a little bit i wanted something very small not saying that they're big but when i found wall listener being in go it just i don't know for me it just clicked and i was like this is what i wanted this is what i was looking for but there's definitely other similar projects cassiano is using some red bin log and got json out of it so yeah it sounds like this is something people i think people think it's useful i certainly um i don't think i'm waiting for a use case whereas sometimes that that can be the the case with this sort of thing i think it's um you know let's see when a problem arises does this a good fit is there something that we can do to sort of capture the imagination of the community maybe it's a follow-up post something like a a contentful style cms where we got a little crud api everybody does a credit api but nobody really has events and web hooks coming out of it so yeah definitely be pretty cool to see that did you have anything else you you're going to do with this next and do you want feedback on it yeah if anyone has any feedback any you know they run into any issues then i'll definitely clean up the post or clean up to read me that uh the wall listener is definitely something i want to like i mentioned maybe tidy that up a little bit right now i think it has travis ci but it doesn't have anything to like automate builds like just a little cleaning the house so that it's you know if something comes up and it needs to be improved that it's easy to share again okay thanks lucas and thanks everybody for tuning in uh i hope you enjoyed this format we'll be doing it again uh as long as we get people to join us and share what they've done so if you've got a cool idea or a use case or something you want to chat about come and chat to us in slack and slack.openfazio and lucas's blog is linked in the description below lucasrosa.com is that right yep yep and you can get us both from twitter if you want to reach out just that way too all right thanks everyone yeah thank you for joining us and we will see you there thank you as well it's good to have you here and we will see you soon goodbye

Channel: Alex Ellis

Views: 639

Keywords:

Id: nGOZLOjg5Zs

Length: 62min 4sec (3724 seconds)

Published: Tue Feb 02 2021