JS Monthly #18 / Storybook & Ekko / September 2021

[Music] [Music] so [Music] [Music] do [Music] so [Music] you [Music] [Applause] [Music] then [Music] [Music] foreign [Music] this [Music] [Music] so [Music] [Applause] [Music] hello everyone and welcome to js monthly september meetup and my name is aries marco yanakis and you can find me on twitter at aries marco i'm the organizer of js monthly and also organizer and founder of ctjs we're always looking for new people and if you are interested in joining us you can email us at the jsjs monthly london gmail.com we're always looking for new faces and people to help us to organize meetups three uh word tops we organize again with hubstack i would like to thank our sponsors for helping us programming this amazing event that is coming up livingspec [Music] [Music] who is going to give us a small introduction to a competition of the irani i'm gonna stop studying and he's gonna tell us a bit about his uh his competition and he stops hey guys um let me just can everybody see this yep um so i'm steve i'm i'm the lead technical recruiter uh at b murray um i've uh and and we've decided to kind of take a somewhat novel approach uh to not just um our hiring but also i just wanted to be able to participate in this amazing community so we've actually put together a coding challenge um as a way to offer a macbook pro um to somebody who can kind of demonstrate uh their ability to to complete one of our challenges um so uh the challenges is a technical test that we put together in collaboration with hired um and the winner will get a macbook pro on completion so how it works is um the the test is about 45 minutes um you know you submit your uh your um solution um our engineers internally will assess uh people's solutions and pick um a short list that will be invited to chat to the team um and then we'll select a winner um at the end of the challenge so i'm really looking forward to people's submissions um if you have any questions i will be on the slack channel um for anything but it's pretty self-explanatory um the link uh and the landing page will go up and i think harris has that and we'll share that and uh we'll also be at the js conference as well and would love to chat with you guys uh thank you very much [Music] thank you stephen and that was um yeah i would like to have an m1 laptop i think everyone wants to have that so you know go through your note skills and and get that laptop and yeah we have also a free free workshop introduction to nosql databases that is from our sponsors the data stacks sorry that probably you cannot see my the image is a bit blurry but it's on the 16th of september at 1 00 pm uk time and with that we have also another free workshop uh building a full stack up using an xjs and prisma by mahmoud i said it yes it's a free workshop you should come and learn more about the prisma and xjs i think they just recently connected the prisma with the sql server and mongodb so it's a lot of good stuff to learn and if you wanna brush your typescript skills we have like a starter typescript starter and it's four hours and it's paid but it's gonna look quite a lot and then we have the advanced typescript workshop that is with mobx creator and miscellustrate and that's eight hours intense workshop and no more adverts and i'm just gonna go through to the main course and i'm going to pass you to caterina bellu all the way from greece my my lovely country and she's going to talk about the storybook you're up with an ex so i'm gonna post you to katarina hi everyone let me start sharing my screen oh first of all thank you very much for having me this is very exciting for me and i hope to come to one of your in-person events as well because i love london so can you see my screen yes we can perfect so let's get going give me a moment okay so today i'm going to talk to you about two things storybook and an x and how you can add sort with your app to your app with the help of an x um first of all i'll leave a few things about myself quickly i'm a software engineer at narwhal and i'm a core team member of nx and i'll explain what the next is for those who don't know what it is i'm also the co-founder of the angular athens meetup i'm also a gd for angular web technologies in the google maps platform i'm a speaker and instructor a mentor i mentor women who want to get into technology or who are already in technology and i'm also a cat person and i climb mountains that's me at the top of greece so today as i said we'll talk about starbuck and nx summary what is storybook why use storybook how sorbet works what is an x um then we'll see how an next helps with storybook development and then we'll see a little demo which today i'm going to do live i don't usually do it live that was i usually record but today will be a special day so let's see what is storybook wait let me set my timer because i tend to forget and just go on and on speak okay here's my timer so what is starting um if you go to their website uh you will read the definition of it opens it's an open source tool for developing ui components in isolation well thank you very much what is that so with storybook you create a story for a component where you import the component and write or describe various scenarios or use cases for it what does this mean it may sound like an e3 test so you create a small file which is called a story you import your component and you write a few different use cases i said which will which will use your component so this is called a story and it integrates with most front-end frameworks including react view angular and svelte but i'm not sure you all understood well what it is so let's see first an example of how a storybook looks like and then we will get into more details so that all of you will perfectly understand what it is so if you go to this site here to the storybook site and you will see some live examples uh if we go to the guardians storybook for example what is it it will be so how to start the the concept behind storybook is that an organization or an application will create let's find something else let's find a drop down we'll create a list sort of a catalog of their components and their ui libraries so that it will be able to be reused among their teams their various teams and the cool thing about that is that they can add various components in it and these various components they will be able to be launched independently and if we browse this we can see for example what a drop-down looks like so if i for example if i'm a developer oops i don't know why i got a 404 if i'm a developer at guardian for example and i want to develop a new feature someone from the team will send me this link which will have all the the catalog of the of the elements that i have available and i can go here and they can see the navigation and we will see how you will be able to also change some colors and see how it works so more or less it's uh it's a tool for creating a catalog of your components but your components leave isolated from the rest of your application but let's see some more to make it a bit more clear why would someone use storybook first of all as i said you can develop entire uis without needing to start up a complex dev stack so imagine a scenario where you have a say an input field in a form and the form has various pages and that input field is at the third page of your form and you want to see some um some changes in the styles of that input field from validation and you have multiple validations so if you normally if you would want to test all these various stages you would have to fire up your application and then fire up your server and then start putting data in the form and you put data in the first page of the form the data is not correct you cannot go to the second page you go back you fix the data you go to the second page you go to the third page you finally arrive to that small input field that you want to test you don't like the ui of the you don't like the styles of it you change the styles sorry okay um you change the style of it and you have to refresh your application and start over and i guess this is very tiring for somebody too and it's very frustrating so with storybook you can just take that input field that small component of that input field add it in this storybook environment and when you you just have to launch storybooks local server and it will show up and it will show up and it will have the single field and you will be able to change the styles and see it changed live without starting to fire up your whole dev stack to see it also it can serve as we saw as a documentation for your your ui library or system which is again very very useful if you work in a large organization and you're handed the task you can go and see your ui library your ui system and you can also grab a few ui components see how they look see how their different stages look also you can create your component library as i said also you can rely on your tests and write a little less unit tests you can also as i said easily browser trigger different state and you can do visual testing too which is all very nice right and you will use sort before that so how does storybook work because i never explained all that but i haven't said how it works so if you're just using storybook with the storybook cli it has a cli you just do mpx sb storybook n8 and what this will do it will so say you have a react or an angular application and you run this command it will install all the required dependencies like it will start install the storybook packages and any other things that sort book needs it will set up the necessary scripts to run and build storybook and it will add the default store book configuration and it will have some boilerplate stories to get you started write some default stories the step two is you just set a story file right next to your component file you import your component and start drafting out scenarios so let's see how this looks in an example react layout or an angular layout you would have your test components or your tsx file then you would have your test files your css files so just like that you just add a dot source.tsx file the same thing goes with angular you have your spec file your html your css and your dot source cs file so in our example that we're going to see today you would have and this is from the next workspace which we'll see in which we'll see eventually you will have your button for example because you have a custom button that you want to build and then you will have the this the test the scss and then the stories file so if your component your react component looks like this you have your button props which is the color of the button the background color the label and whether it's disabled or not and you have your tsx which is just some symbol tsx that i made here uh we just background color disable the props deposit props in what your story for that will look like uh you will have to to export the default uh your about your component and then you could create a template for all the stores you you're going to make and in the template you're going to pass the components with its props which are arguments now and then the primary is the first story that you will write so you buy the template and then you pass the arguments that you want to use so for example in my first my primary story i want my buttons to look a little bit red i want to have a label and the button i want to say click me and they don't want it to be disabled so when i fire up my local server by running npm run storybook this will show up so i will have my button and i'll have a canvas here because this is how it's called actually let's see this slide i think um wait let's see this live let's see if this live demo will work they never do today okay i'll give it up okay perfect so let's go to where where do i have my browser here here no i go to localhost 4400 and i make this a lot bigger in a second [Music] right so this um so this here is you can see we have the arguments the color label disabled so now i have my button which has a color a label and disabled and it looks like this and i can click it and it's a live element as you can see and i just needed to to start the local server book server i didn't have to fire my whole application and now what's cool about this is that i can change the colors and everything live right so if i want a green button i can go like this and i can change the text and i can say that i want it to be disabled and i'm not sure that you can see because the text is very slow small but also as i'm changing these controls here not only is the ui and the my component changing but also the url is changing which is pretty cool because storybook integrates with cypress and you can trigger different states of components using the url and when you pass it into cyprus you can make sure you you have your e3 tests uh use your storybook components in your stories so you get to write lesson two tests as well so let's continue now after this right okay so we saw these screenshots all and the step four is that you can build storybook as a static web application and you can deploy just as we saw in the guardian website where they have built their storybook and they have they have it published as a web application so you can do that and share it with your organization or whoever you want and to build it you just run on npm run build storybook so what is an x that i've been talking about on and on nx is an open source tool first of all and it's a suite of powerful extensible dev tools to help you architect test and build at any scale integrated seamlessly with modern technologies and libraries while providing robust cli caching dependency management and more so what does this mean let's see first of all nx offers the nxcli the nxcli will help you generate apps it will help you test the maps around them or organize them you get commands like the create nx workspace which will just generate a workspace for you and in that workspace you can generate applications you can generate libraries and you can import and export you can export these applications and build them but you can also import each application into the other so it also works as a good monorail tool [Music] um [Music] sorry [Music] so if you're familiar with the angular cli some of these may sound familiar to you but it's much more than that the other thing that nx does it is it has a lot of tools first of all it provides you with the dependency graph what nx does is it's um it analyzes your code base and it recognizes the dependencies between your applications in that way it provides you with a dependency graph and this is useful because when you you know you get to never build or test the same code twice because it only rebuilds or re-tests applications that are dependent on each other i wish the dog would stop arguing uh also it has the affected commands which also by your code by analyzing your code base uh exactly with the dependency graph it only rebuilds and retests the affected code um we also provide a number of executors which means that some scripts that you get to run for example to migrate code so if you want it makes upgrading from one version to another very easy not only for nx but for other applications or for storybook as well but even for angular for react if you're going from one version to the other there are some scripts called migrators that are run automatically and not only do they update automatically your dependencies but they also may change actual code that you wrote in order to adjust to the version that you're targeting also it has integration with prettier also it has some generators generators mean uh that generate code that helps you move things around and change your organization uh you get to run commands simultaneously and also it provides you and we will see that in the final demo with a plugin for vs code called nx console that you can do all these things not using the cli but using a graphical user interface also it offers a lot of plugins with third-party tools like storybook as i said or like cypress um so as i said it integrates also with jest with cypress with prettier with eslint and with storybook um the generators that we talked about there's a generator for anything you want to generate like you can generate applications you can generate libraries components and generate tests but you can also create your own uh generator so for example if you have a custom way that you're writing components for libraries in your organization you can create a script and then run it through the annex cli to create your own generator basically also it provides you with workflow management uh you can add all the tools you need and use them from within an x and you can organize your tests your builds and your ci and also we offer the nx cloud which the great thing about nx cloud is you can expand so i didn't mention the nx cache so the nx cache cache is the results of each command that you run and this is how the affected commands work because if you have some cached results and nothing is affected it will fetch the results from the cache nx cloud takes that one step further so it basically saves the the results of your commands to the cloud the nx cloud and if uh if your whole team uses the nx cloud if any of your team members has already built or tested similar the same code actually nx will use the results to speed up the command instead of rebuilding and re-testing everything from scratch so now it's not only your machine's power and your messenger class that it's used but it's also uh other people's um commands and yeah work that is used so how does an x help with storybook development everything we talked about let's bring it together i think the most important things that nx will do for you is that first of all it will create stories for you so if you want as you saw we just wrote a story where we imported the react component and we took the props and we adjusted some things and we added some inputs to the props so what the next will do for you is it will take your component it will analyze the code and if it's angular it will understand your imports or if it's reacting will understand your props and based on that it will create a sample sort for you with some default values or some sample values for you so it will write the code for you so you won't have to do anything so if you want to get started with storybook and use it in your application it will just do the work for you and this is very very useful for example because if you're in a large organization and you don't really care to write a complicated stories and you just want to generate a list and like catalog of your components you can just uh run the nx command and it will just analyze the code and generate the source for you and you will have a ready storybook to to publish or to share also the cool thing another cool thing is that you can manage the storybook through the nx ecosystem so we saw before that you manage storybook with starbucks cli but uh if you use it through an x you will just have to use one cli you will have to only use the nx cli and this is not only cool because you will only have to remember one type of command the nx command but it will also add the storybook to use the cache and use the affected commands in the dependency graph which are all very helpful so yeah i think about the cache also it will generate t3 tests for your source which is very handy because it's not only to test for your search it's actually to test for your components so if you have stories um for your components and x will just generate the two tests and they will be there ready for you to use them what's more because there's always more uh as i said it's an integrated environment for all your commands and it's it has a seamless instant integration with your ci and you can use an x to to build you can use lx build to build your storybook so you can take advantage of the cache and the affected commands so it's everything integrated also it works for both applications and your libraries and in in annex it works only for angular and reacts and also we have active maintenance and support we're working very closely with the storybook core team so we're up to date with any updates they're making um in their code and any updates they're making in the way they're right in their synthetics so we we mainly stay on top of all the changes that are there so in order to use it in an nx workspace and if not i'm just familiar with the next i really hope you try it out um you just do yarn ads and you add the normal storybook package and then you just for example if you're using react you you would just do annex snare will react server configuration and then give you the project name that you want to use and we'll navigate we'll quickly navigate on the next workspace after that so that's those of you who are not familiar with the next will and understand what i'm talking about so this script that nx provides will install the required dependencies they will set up the necessary scripts to run and build storybook they will add the default server configuration so up to there they sort of mostly do what the storybook cli would do as well but nx will also generate stories for your components as i said and it will add the scripts to your uh workspace configuration files to help you serve and build storybook within the nx ecosystem so again this the component that we saw before it will analyze this component and it will speed up this story that we saw before and then you have commands like um the stories command which will which it will generate stories for the components in your project that do not already have stories also it will update any stories for your components that have changed so for example if you have a component and you generate the story for it and then you change your component and you run the stories generator again it will update that story for you and the annex build storybook command it will build your storybook and it will take advantage of cache and the dependency graph and the affected functionalities and then for example this migrator if somebody were using store book version 5 this migrator will would go and migrate all the configurations to storybook version six we similarly have a migrator that would uh actually migrate actual uh stories files from version six to version 6.3 for example so that you would have to do minimal work manually so let's uh spend a few minutes to see this in action um so we saw this in action so now let's see what we're going to see now in action is wait so we have this component here that we have already seen but let's see how we can use the annex console in order to generate a new application and then generate a component for it and then add storybooks for it so if you're using vs code you can add bmx console plugin and here are all the commands that are available so when i click generate i will i want to generate for example a react application so let's name my application js monthly oops this is a dry run by the way js monthly and i want this style to be a css just for the fun of it and and then you have some other options that you may use and i'll leave you to see this on your own time when you install the nx console plugin but when i click run you will see in the uh console here that it went and created a new application for and here are all my menu changes so it generates an application and it also generates an e3d application so in my new application um i do have one default component but i want to generate also a new component for my application so if i go and generate an react component um what will be named the component listening component london my imagination is limited as you can see and i want the styles to be a css and when i click run it will go in here and it will generate a london component and in the london components if i have some props katarina string and alone with my cat is a number so welcome to london hi props tarina um stay for props alone days okay so now we have this component here and now say i want to generate a storybook configuration for this application these components so if i go again to the annex console and click generate and they say storybook configuration the project name was just monthly and i wanted to to configure cypress to run the cypress computer generator which will generate cyber stories for me and i want to generate some cyber specs and generate and also generate stories for my components and now if i click run it will go in here and it will add the stories file which as we saw inverse the component and has two arguments here malone and katarina and it also has a storybook folder in it because these are some default server configuration files that need to exist in every storybook so if i go to my oh and it also generated the stories file for my default component let's remove this so if i go here yeah it doesn't have props there's always something that's breaking um i guess if i do like this okay yeah whatever i just wanted to run so now all i care about is this template here so now if i run [Music] if i add a name here say catherine again sorry for this and start stay for 10 days and if i go again to my next console and i go to js monthly here and i open it i see that i can run storybook for it and when i click this [Music] come on okay it runs at my local server and if i open it here i have my name here that will change and i will stay for 20 days so yes we saw that with the next we just click the number of buttons click like 10 buttons and everything pops up and you have already made application android made storage book for you and we also have some ready-made tests for us which they to test let's see how they look like um [Music] london so they do test for london they would look like this and because we saw that the url takes in the parameters we can add different things here [Music] different values to our parameters here and around the two tests and make sure that um for example we would want our paragraph to contain whatever we put in the arguments here and it would save us a lot of time so let me go back to presentation so to wrap up quickly i really hope you were excited by this presentation so by this demo actually the presentation the demo so to sum up storybook helps you create a catalog for your components put it simply um but not only that circle also helps you quickly test out various states and scenarios in isolation nx helps you use storybook by integrating with this tool system and nx helps you use storybook by generating stories for you um yeah this future steps for next are a bit updated but the the thing that we always want is to have more migration and customization schematics and we're already always addressing bugs and keeping up to date with new story versions so some links that you may find useful are the links for an x and for storybook and i'll give you now the links to my slides which you can open and you will find all the links that you need that are in this public um and that was it hello oh you hear me okay [Music] thank you for telling me that's great thank you guys thank you js london uh alex are we going to the q a okay normally he gives me like a heads up but i know it no that was great and i've been using like a storybook for quite a while now and it makes like a big difference in the way you write a code because you you write your components and separate to the application then you can use it so many times in all the projects and that's what we saw in your presentation and have you got any any cases where projects you move to when you move to storybook that your project speed up and you find out that it made a big impact um i don't have personal examples no but regarding that but i don't know some colleagues who work on large organizations with uh with big big code bases and the generators and the helpful tools that the next provides makes their work with storybook a lot easier sorry my god is trying to get yeah i could suggest one of the dogs i think the well they were getting excited with the the the storybook and i could hear them yeah [Music] it was funny but yes the what were your struggles like someone that starts uh learning storybook or someone that is actually new to frontend and javascript what would you advise them [Music] so someone who is new to storybook or someone who's new someone here because i think just to rephrase it a bit storybook would be useful for someone that doesn't have a lot of experience and starting using storybook could be a good tool for them to use right potentially yes because it would make it easier for them to to see their what they're building live easier than firing up their whole system um but yeah it would also depend on on what they were trying to build because maybe for smaller applications sorry would not be that useful to go to go into the trouble to add storybook if it's just for a very small application for example with not many components that's what i would ask you when would you because i'm accustomed with it nice would you like to tell us a bit about your meetup group and how you uh you would like to invite some people from london to come and watch some of your next meet up oh sure so our meetup has been going on for almost four years now three and a half years i think um and we we we tried to to organize it every two and a half months when we were doing in-person events now we haven't done an in-person event for for almost two years of course due to covet but we are continuing doing the online events uh our next one i think is by the end of the month where we will host actually the angular world tour and i guess maybe we will have the next one in october but we're still doing baby steps and we don't know if when we're going to go back to live events unfortunately yeah i know but um but i do miss live events especially with the meetup because you really get to see the people and you get to interact with the people it's much more okay yeah if you need any speakers from london just let us know we can always of course different courses thank you very much for uh for being with us and yeah after this talk your talk is going to be also going on the pusher like they have a big library of videos that are nice and so yeah people can watch the video after the talk and um yeah no that's great thank you very much properly and have a good night and yeah we will keep it here thank you very much aries for having me and good luck to the rest of the speakers thank you okay we will move now to i think those people are probably there in london or in another country i don't know but yeah we will move to hear more about echo and how it went building it and yeah we're gonna move to alex and true and will so if you are there on stage now uh alex yeah i can okay hello hello um i'm gonna mute my camera and my microphone and stop my camera and that will give you states and you can present to us echo thank you very much okay cool well hopefully everyone can see everything and hear me all right and i'm sure someone will let me know if otherwise but super excited to be here today we're going to be talking about building echo and echo is a real-time message processing framework and we're going to dig into exactly what that means um and just a quick intro my name is will and i'm here with drew and alex today unfortunately dory wasn't able to make it but as you can see we are spread out all over the world and it was super fun to try to learn how to collaborate remotely which is something pretty much the whole world has tried to figure out over the last year and a half or so um yeah and just a quick teaser uh this is a super simple chat application and it's just to show kind of a contrived example of something you could build with echo and all of the real-time capabilities are built on top of echo in this example and something unique you might notice is that on the angry channel all of the actual text messages being sent back and forth are being transformed and so that's one of the key things we're going to be talking about later just a quick overview we're going to be talking about real-time web applications what exactly we mean by that also why they often need dedicated real-time infrastructure and then also why they often need real-time middleware and what exactly that is that's kind of a not a super common phrase after that we'll introduce echo and then we'll go over some of the engineering challenges we faced and then we'll wrap up with some resources in case you want to learn more about echo and jump in since it is all open source so first off let's talk about real-time web applications um so real-time web applications are literally everywhere they're the live dashboards we see they're all the apps that show you the driver location or the like package location on a map um and then there are also all of the like chat features and everything these are just a couple of examples but the whole real-time feature our real-time features are pretty much everywhere now um when it comes to trying to define and classify real-time web applications it really all boils down to the user's perception and this is a quote that i think sums it up pretty well and it says a real-time interaction has a human angle because one of the collaborators in any real-time data exchange is always a human hence the human perception of a real-time event is very important and this is kind of a weird definition but really it's just saying that real time in terms of web applications is like completely focused on just trying to make things happen as quickly as possible and so really the only metric that matters is the human perception and this kind of leads to some interesting insights um so the first one being that we have to shift our idea of what our of how we think about data being transferred and so going back to kind of a traditional with the data and nothing actually happens until that request is made which is why it's called polling data the client is pulling data to itself and this is just me just from the data push pattern and so in the data push pattern you usually do start off with a similar request where the client is requesting data but the difference is that it's requesting that new data be sent over as soon as it's generated so in a data push pattern as soon as the data is available it's pushed to the client and if you think about real-time web applications pretty much everything follows this data push pattern um otherwise you'd have to like refresh your page to see if there's new chat messages and that'd be a really poor user experience another interesting pattern that kind of arises when talking about real-time web applications is a shift away from the kind of server client model and a movement or a shift towards the idea of publishers and subscribers and so the key difference here is that now we're not talking so much about the actual like underlying hardware that's being used and instead we're talking about the roles that are that the different devices are being used for and so on the left we have a single publisher and it's sending messages to the subscriber on the right and just a quick clarification also in real time messaging or in real time um web applications we use the term message for pretty much any data being sent it could be actual like text messages but it could be just any bits of data so it could be like geolocation data or any of those things but again you calling it the messages once again kind of operating a little bit higher layer of abstraction instead of talking about the actual implementation details um and expanding on this idea of pups of messaging patterns you can also have one-to-many so we have a single publisher publishing to three subscribers on the right and then you could take this even further and do many-to-many um and this looks a little crazy this is actually totally possible this is basically like a peer-to-peer network and something like webrtc totally supports this in real time this actually isn't the most common for most web applications most web applications are centralized um and so what's more common is to actually have kind of a central hub to manage real-time connections and so in this example we have a central hub that all the devices are connected to and then so a publisher sends a message to the pub or to the hub and then that emits it to all the subscribers and then last but not least um so far we've only been talking about devices as being a publisher or a subscriber but it's actually very common to have devices be both publishers and subscribers and if you think about again like a chat example or something anytime you send a message or a publisher anytime you receive a message you're a subscriber and then so in this example we can see that any device can publish to that pub sub hub and it gets submitted to all relevant subscribers and it's kind of big level picture of pub sub messaging which is pretty common in the real-time web application space now we talked about why having a central hub is helpful for just keeping things simpler especially if you already have kind of a centralized web application um so a hub makes sense but the next question then that follows is how do you actually implement this if you want to build one yourself or start implementing real-time features and so it kind of makes sense to start off by talking about the actual communication protocols since um with real-time features it's all about sending data very quickly so communication protocols kind of make sense and if you start searching around for like real-time communication protocols you're going to run into you're going to run into a lot of these different types of things so there's a bunch of http derivatives such as server sent events long polling and http streaming there's webrtc which is that peer-to-peer protocol we talked about um and then there's also web transport and web sockets and so all of them have different trade-offs and so uh when we go through them kind of makes sense to compare them and so the first thing they all need data push because we established that's pretty essential for real time messages and they all do support that but the next thing is if you want to support bi-directional communication where devices can be both publishers and subscribers that kind of rules out the http derivatives um these some of them there are some hacks and workarounds to do bi-directional but it's not really the standard use case for them next up if you're already like if you're adding a feature to a web application that's already centralized web application where it's hosted on a server somewhere it makes sense to follow the same pattern um versus trying to set up a whole peer-to-peer network for the real-time feature and so that kind of rules out pub or rules out webrtc for most applications um just because that had a lot of unneeded complexity and last but not least um you want something proven in production most likely and so that rules out web transport which is super interesting but it still is an emerging technology so it's not as battle tested as the other ones out there and that leaves you with web sockets which should make sense websockets is definitely the most common way to implement real-time features in web applications and so let's say you actually are going to jump in and add a real-time feature and so it's actually surprisingly easy to get up and running you just open up your application code you can add a websocket library such as socket io that's pretty much it like you could get a demo up and running in an afternoon pretty easily now the complexity or the challenges happen when you actually want to deploy this into production and you want it to be a robust scalable solution and so you're going to run into a lot of different challenges so some of the challenges are if you're adding multiple real-time features so things like chat boxes and then like location data and dashboards and they're all different real-time data and managing a bunch of different types can be very challenging also if you have to horizontally scale your application managing real-time connections and data across different servers can be very very challenging and then also a lot of times real-time features scale or have different scaling needs than the actual application that they're integrated with and then so building stuff things that are able to accommodate both can be challenging since they fluctuate at different times and so because all of these different challenges are very common and arise pretty quickly with real-time features it's often a good idea to actually have dedicated infrastructure specifically for your real-time features and that's actually what these three companies do so these are some of the leading companies out there that provide real-time infrastructure as a service and this is great for like anyone who doesn't want to build things from the ground up and want to get up and running very quickly and have a lot of guarantees and reliability these server services are amazing at that and they all pretty much advocate the same idea they all advocate the idea of decoupling your real-time infrastructure from your application now if you start digging into this even further and start actually implementing different tools and features that are real time you'll probably run into another kind of need or pain point uh sooner or later and that's the need for real-time middleware and the idea behind real-time middleware is just the idea of wanting to do some sort of business logic on the actual data the real-time data in transit and then so examples of that so for instance when you open up the uber app it doesn't show you every driver in the world it just shows you the ones that are right around your area so there's some filtering going on and then similarly a very literal filtering going on is that with chat messages like in-game chat and whatnot they often filter out profanity so again you have messages going between different users but somewhere in the middle some business logic is being applied and profanity or anything offensive is being filtered out um another example is for chat bots all the different messages you're sending out it's getting fed into some crazy aiml stuff and doing analysis on it and that's or that's uh that's leading to the actual responses you get from the chatbot so again we see some uh some sort of logic or some sort of things happening to the actual data being passed around and we dug into this quite a bit and we found there's actually quite a few different categories out there um so there's message transformation and that's where the actual data in the message is being changed then there's message enrichment and that's where data is being added to the messages um and then there's like message filtering and chat bots which we talked about and then also a lot of times messages are fed into some sort of analysis or they're used to trigger alerts and notifications and so in all these cases you're kind of moving away from just sending the raw data between users and you're doing something else in the middle of it and so this kind of need or pattern this is recognized by pubnub and abele and they actually have some interesting quotes where pubnub says you see everyone publishing down to their servers doing a small bit of processing and then publishing the message right back out it doesn't make sense and abele has a similar quote where they say a common requirement in real-time messaging applications is to be able to insert some business logic into a message processing pipeline and so we thought it was interesting and it kind of makes sense why this uh this pain pointer need was happening where people were using these third-party services so they were separating out the real-time infrastructure from their application infrastructure but if they needed to do very simple different changes to the data in transit then all of a sudden they would have to send everything to their own servers and that's tightly coupling it then they'd have to go right back into the issues of scaling their own services to meet the needs of the real time services and you kind of right back into that tightly coupled system and so interestingly both pubnub and abele after they recognized that they kind of came up with different solutions so pubnub came up with pubnub functions and these are serverless functions that are actually built and deployed inside the pubnub dashboard and then ably took a different approach where you can use a third-party service function provider like aws lambda and that'll integrate into your ably pipeline and we love pusher and we've dug around and we haven't found an easy direct drop-in for this at this time but these are we found it very interesting that these two pubnub and ambly did recognize this problem but they solved the problem in very different ways so we thought there was something interesting there and so we kind of looked into it a little further and we tried to map out what the current landscape of options was and so if you want to build real-time features and you want to add them to your applications you have three options you can either use a third serve or a third party service you could use an open source solution or you could do it yourself and so it really depends on what you're prioritizing so if you want something that's easy to set up then third party services are by far the easiest so that's the pub nubs the ables and the pushers of the world they work really easily and you can get up and running very quickly now if you want complete control over your data and your infrastructure then it gets a little trickier so there's definitely open source solutions out there um you could always do it yourself um so it just depends on what you're optimizing for and then probably the most interesting one we thought was if you've recognized this need for real-time middleware um then it gets a little tricky because not all the third-party services support this and they the ones that do support it in different ways and then you could always do it yourself but that's a pretty big undertaking and then at the time that we looked at this there were no open source solutions so there was no open source real-time infrastructure solution that supported real-time middleware out of the box and that's actually what led us to building echo um and with that pat thanks well um so uh i'm drew and i'm going to talk a little bit about what echo is how it works and give a little demo so echo is an open source framework and you can use it to deploy your own real-time infrastructure with that real-time middleware and there are four main parts to echo so the first is the echo server and that's what manages the real-time messages for your applications next is echo functions and those are what provide that real-time middleware for in-transit message processing and then we have the echo cli which makes it easy to manage echo functions and spin up that echo infrastructure and tear it down and the last part is echo client and this enables you to build real-time applications on top of the echo server and because echo's open source and self-deployable you have complete control over this infrastructure you can customize it to your needs and you don't have to rely on a third party to manage your real-time data so that's what echo is and we're going to try it out and spin up some infrastructure with our cli tool so the echo cli tool provides us with several commands that we can use you can add a help flag to the end of a command to see its documentation and here we'll run echo init and we'll spin up a new infrastructure so the cli gets our aws credentials and it uses those along with aws cloud development kit to spin up echo infrastructure in our aws environment so this normally takes 10 or 15 minutes but we've sped it up here and it'll also create an echo directory in our current working directory and it's not no it's worth noting that you can tear down your echo infrastructure in a similar way using the echo teardown command so we've spun up our infrastructure here and let's go look at a diagram and see what it looks like so this is our echo infrastructure you can see the echo server on the top right and this is basically a node application deployed in a container to aws elastic container service on the left side there's an application load balancer and this distributes the websocket connections to our echo server on the bottom we have an s3 bucket and an elasticast instance and we'll learn a little more about the role that those play later for now it's important to know is that it's this echo server that's managing the real-time messages for our applications so um let's set up a real app a real-time application now that uses the echo server to manage the real-time messages so recall that when we ran echoinnet it created an echo directory in our current working directory in there we have echo apps and echo functions and we'll look at echo functions later within echo apps we have a few demo applications and these are designed to help a developer become familiar with building an application on top of echo so we'll be using the chat demo application here and in there we have a basic html file and this html file runs a couple of scripts the first is our echo client which is stored on a cdn and then the other is the script js so in this script we're creating a new echo client instance and this will allow our chat demo application to connect to that echo server that we just deployed so that go client instance takes a handful of parameters an app name a host a json web token and a uuid so we've set our app our app name here to chat demo and we'll generate the host and json web token values with our cli tool and then the uuid is optional so no we're not going to worry about it here so back in our cli tool we can generate those host and json web token values and we'll do that by running the echo jwt command and we'll pass in the name of our application as the argument so we get a few values here and the first is the echo server endpoint and we'll paste that in here this is the endpoint of the server that we just deployed which will allow the echo client to communicate with the echo server and send it real-time messages so the next value that was output is the user.json web token and we'll paste that in as well and this is the value used by the echo server to verify that this is indeed a valid echo client and the last value is an admin token and this is for an application server that gives it special capabilities to subscribe to a special admin channel and get status events and we don't need that functionality here so we won't use it um so we've got our chat demo set up and um we've seen that we can use the echo client to communicate with the echo server so now let's go test it out so to do so we'll open our html file in two different browsers and we'll simulate two different clients in this way they're both subscribed to four channels you see on the top boring angry backwards and robot and when one client publishes a message it's sent out to our echo server which then emits that message back out to all subscribed clients so in this case both clients are sending and receiving the messages and so now let's go look at our echo infrastructure and see what it looks like with these clients connected to it so you can see that we now have these two echo clients connected to our echo server and when one client publishes a message out on a certain channel it goes to our echo server which then emits that message back out to all subscribed clients so we've seen how an application can use the echo server to manage real-time messages and now let's go ahead and deploy some real-time middleware to process these messages in transit so if we go back to our chat demo app and we switch to the angry channel we'll type a message in here and when we publish it we expect it to be transformed capitalized with exclamation points added but we can see that that's not happening and so the reason is is because in order to process our messages in transit we need to deploy echo functions so recall that this echo functions directory was created internet we have some demo echo functions that are designed to go along with our demo apps and help a developer become familiar with how these real-time middleware works so we're going to use the demo angry function here and you can see that each function has a specific file structure and format and that's so it can be deployed to aws lambda so this function in particular it takes the message payload capitalizes the text and adds a few exclamation points so now that we've seen how these functions work and how they're set up we're going to use our cli tool to deploy them so with our cli tool we can change to the echo functions directory and we can check the status of our functions and we see that they haven't been deployed so we can run the deploy command to do so and what this is doing is deploying these functions to aws lambda where they can run alongside our echo server so we'll run the status command again here and we see that all those functions have been deployed so there's one more thing we have to do we have to run updateassociations.json and what this does is lets the server know which functions are associated with which channels and we'll talk a little bit more about that later so back in our chat demo we'll see if these deployed echo functions work and they do so now our messages are being processed in transit it's important to note that we didn't have to refresh the client um or the browser rather we didn't have to update any client-side code and we didn't have to update any server-side code instead we're using these echo functions that are on our local machine deploying them to aws lambda where the message processing is done so the benefits to implementing the real-time middleware in this way are are many and we'll go into some of those later but for now it's apparent that this process makes it really easy to create echo functions deploy them update and destroy them with our cli tool so we'll go ahead now and take one last look at the echo infrastructure and see what it looks like with that middleware so we can see on the right our echo functions deployed to aws lambda and when we send a message on the angry channel it um it gets sent by the server to the angry lambda for processing and the same happens to the backwards channel and the robot channel so the server knows which lambdas are associated with which channels using that associations.json file so that's pretty much what echo is and how it works and now i'll turn it over to alex and he's going to talk about some of the engineering challenges that we face while building echo true so the engineering challenges we faced while building echo fell into three broad categories the first was how to implement echo functions so that's our real-time middleware and specifically choosing the technology that we were going to use the second challenge was how to manage the data for linking specific echo functions with real-time messages and the last was how to make sure that echo could scale up as load and demand increased so let's start with the first engineering challenge implementing echo functions as was mentioned a few times above we chose to use serverless functions to handle the load and i'm not going to go into all of the detail on here and i'm sure many of you are familiar with the concept and even maybe the implementation of several of serverless functions but the core reason why we chose this was their flexibility their ease of deployment so obviously that had implications for us being able to implement the functions via the cli tool and of course the infrastructure side of things which i'll talk a bit about later um you know in terms of their being able to handle bursty traffic so imagine again we have a chat app that's using the echo server and filtering out profanity from all of its messages and we also have a geolocation based app that needs to hydrate real-time messages with directions obtained from the google maps api these two apps are both routing messages through the echo server but their needs are unrelated and they have bursty traffic at different times we could we could say now if we use serverless functions as our real-time middleware we can scale the profanity filtering up without affecting the directions processing and vice versa several of those functions also provide flexibility they can be reused across multiple apps and when functions are added or updated the rest of the real-time service is not affected but let's go back to our use case that will talked about earlier which is many publishers and many subscribers needing to process messages in transit on demand without any interdependency or need to keep track of state our goal was to provide a flexible framework to accommodate changing business needs and for these reasons we thought serverless functions were a good fit for our real-time middleware and as we will mention before this also happens to be the way that ably and pubnub both handle message processing in their services now let's move on to the second engineering challenge how to manage the echo functions once we had determined where our riddle real-time middleware was going to live we needed to figure out how to manage it specifically linking middleware with real-time messages since clients publish and subscribe to channels it made sense to link specific channels with specific echo functions so if you're making the chat app that uses a profanity filter you can create a channel and have all messages being published to that channel get processed through the profanity filter all subscribers to that channel will receive the process message we decided on using just a simple json file to store those channel middleware associations and here is a sample of that association.json file that's being used to store the associations it's organized by application as you can see those applications correspond to the demo applications that you get on a baseline install of echo each application has an array of channels and each of those channels contain an array of echo functions to be used for that channel echo will process the message using all of the specified echo functions in order before returning the process message to subscribers and when it came to making that file available to the cloud deployed service the json file lives on an s3 bucket which is an aws cloud storage service the developer is responsible for manually updating it locally and then updating it with the echo cli tool we also want to be able to update the echo server directly at the same time that we update the new json file to the s3 bucket this involves sending a basic put request to the echo server with the json file as the payload this is all handled as part of the cli tool and happens automatically when the developer uses the echo update command the last of the engineering challenges we want to mention relates to how we set up echo to scale horizontally we wanted echo to be able to scale up and down as needed and of course some of this was achieved by simply choosing the appropriate infrastructure and policies on aws the main part of ecco that needed to be able to scale were the echo servers we needed to be able to support a flexible number of users connecting to the real-time servers as well as an increased volume of messages being published in order to be able to scale flexibly a pretty attractive option for horizontal scaling was to package up our server application as a docker container and then use aws's fargate service to scale those server tasks up and down according to how text the particular task instance became now fargate scales according to rules that take into account how much cpu and memory each container instance is using it's not always completely transparent to use but it did handle our core problem that we were trying to solve wanting to horizontally scale our echo server functionality there were some other complexities to actually making this happen in practice but this is the general approach that we took and with this infrastructure setup we did some load testing using artillery.io to validate some of our main use cases our first test scenario was that we wanted to make sure we were able to sustain reasonably high volumes of messages published through the echo server to this end we tested and learned that echo is able to sustain 50 000 messages sent per minute each of those messages is also in turn transformed by lambda functions that's our serverless middleware for our test this amounted to a total of half a million messages and lambda function invocations sent over the course of 10 minutes during the test aws elastic container service spun up two additional echo server containers to deal with this load for the second test scenario we wanted to make sure that we were able to support many subscribers receiving messages echo is able to sustain six hundred thousand messages received per minute published to a single channel we were pretty happy with these numbers and actually these loads didn't max out our server capacities or cause failures so we probably could have increased the load even further but it didn't seem like our use cases demanded that and that's echo an open source framework allowing developers to easily add real-time infrastructure and in-transit message processing to web applications and by this point i hope we have shown you how flexible echo is to work with the possibilities available to you are many as you can see from some of the examples above the combination of a real-time server with serverless functions as a kind of middleware for in-transit processing and messages offers a rich pallet of options we look forward to hearing what you build with echo you can test drive our code and demo applications by visiting our github repo and our ka study is available to read on our website along with a fully hosted version of our chat demo app ready for you to try out even right now live and with that we welcome any questions you might have yep yeah um so uh you mentioned niko what is the the actual cost of someone using egg so echo is i mean it's open source so it's free so the costs are essentially whatever your infrastructure costs are for spinning it out um you spin up a redis cluster as part of the deployment so it's not um it's not as cheap i guess it's running it off a simple kind of a slightly more simple ec2 instance or something but it's yeah it's it's equivalent occurrence as you said the occurrence on an aws right yeah so echo has to make you know if someone is using echo does it mean that echo is paying for a aws hosting no it's self-hosted so um whoever whoever the person is it's you know someone has a web application let's say and they want to have real-time function functionality you put in your own aws keys uh and access access codes and so on and you deploy on your own aws stack so you pay for the deployment costs and um just to reiterate because you mentioned some of the competitor companies and we have also streaming the event and how do you and can you go back and and re-emphasize um what is the the difference between uh you know if what the ability and or pusher or the i don't remember the other name forgive me for that um yes can you emphasize that that part i could take this one um so for what they they have a lot more than three engineers working on it part time and so it's a and for both pubnubly and pusher all those companies are like fully flushed out robust systems um but for those ones you do like you are paying some cost to them and they're managing all of the infrastructure and everything so if you want full control of everything that's the one little niche where um you can get that with echo but it is like a very small open source project so you don't get all the guarantees that a large company has because those are like fully fleshed out products so they have things like um different things to handle like inorder message delivery they have encryption stuff they have global global globally deployed things whereas like echo is going to be whatever aws region there are um so this is not even it's not competing in any way it's just those are kind of those are the full-fledged uh commercial options and then if you want to do something on your own with a smaller application you can get up and running and have full control if you want to do it with echo it's kind of a different thing can i ask you sorry yeah we don't have a lot of them but it's quite interesting um can someone deploy this on azure or any other platforms rather than aws i just mean yes right at the moment the cli tool which drew works on mostly like that it's hooked up kind of pretty pretty tightly into the aws ecosystem that's not to say that you know it wouldn't be wouldn't be possible to to deploy it on on gcp it's just yeah we we don't have that engineered at the moment yeah and can you give some examples of people that use the echo and some use cases you might mention it but just repeating it now here somewhere yeah yeah unless uh well or alex know of any use cases that i don't um we just finished building it a couple of months ago and uh yeah not sure of anybody who's actually picked it up and used it for their way back yet so it was uh using it yeah yeah maybe after this presentation okay yeah we can speak about doing a work so you can show people a bit more on how to use it um properly because it was quite a lot of stuff i liked playing the the talk i think in the beginning it was giving a lot of information is like how someone can use the sockets whereby to see not everybody knows all these things but then you went a bit more deeper and i lost you at some point so and yeah that no no no yes yes put it out yeah what was the most interesting part while building it so i guess i can start um for me i was kind of responsible for the infrastructure deployment piece um and that was a world i hadn't really engaged with too much um and so just figuring out not only like what the little pieces of the aws ecosystem we would need to like reliably use to reliably kind of have our have our service hosted and scalable but then you know dealing with what pieces of our service would start breaking when you have it deployed on multiple different docker containers for example so that was a real kind of learning curve um for me and i found that really interesting to to get into if if you so what are the next steps like what what would you like to to further develop what you know after after what you've done [Music] i mean i think there's a million little things that are on the list i know some of the big things we talked about were be really interesting to get message encryption built in so that when the actual like message is being passed through the echo server everything be fully encrypted um also like different things like in order honestly it's like the whole giant list that all the commercial companies do really well they're all just kind of the big wish list things for long-term development goals is this backed by by any like um as a like a startup or not so are you looking to get packed up by someone that is looking [Music] yeah sure why not yeah okay good good and yeah so anyone that wants to help these people to expand further you know um how do they reach to you [Music] yeah the echo dash real time so e k k o dash real time dot com is the website and that has links to the github repo all of our contact information on it and also some kind of a walkthrough of it if you want to actually jump in yourself how to get up and running with the sample apps and deploying it to your own aws um account yeah okay that's great and well i have to say goodbye now but it was a great one and uh yeah it sounded really nice when i saw the the title on our on our form that you you added it in and yeah i think it's going to be good to present it also in other places like you know not not conf and other places but uh thank you very much and have a good day and night because you're in split in different places right i'm speaking right now i'm speaking from the netherlands and drew and will are both in the united states where about in the united states i'm currently in chicago and true and i'm in colorado okay so you're spread around okay thank you very much guys and yeah keep keep up and if you need anything promoting let us know thank you thank you very much for having us thank you bye [Music] foreign [Music] foreign [Music] so [Music] [Music] so [Music] so [Music] my [Music] so [Music] my [Music] [Music] so [Music] [Applause] [Music] [Music] foreign [Music] [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] bye

Channel: Pusher

Views: 412

Rating: 5 out of 5

Keywords: developer language, language code, libraries, help, tips, forum, meet-up, meet up, learn code, coding education, coding hints and tips, lecture, coding lecture, learn about code, learn a developer language, amazon alexa skills, developer conference, node.js, javascript, backend

Id: LvQy4Jmxat0

Length: 96min 19sec (5779 seconds)

Published: Fri Sep 10 2021