JS Monthly Lunchtime Meetup #06 / Logging / July 2021

[Music] um [Music] [Music] foreign [Music] [Music] [Applause] [Music] [Music] so [Music] you [Music] [Music] do [Music] you [Music] [Applause] [Music] so [Music] well [Music] so [Music] uh [Music] [Music] do [Music] [Applause] [Music] do [Music] [Music] foreign [Music] hello everyone this is the js monthly our july meetup at lunchtime it's one o'clock in uk and we today we have another meet up before we start our meetup with our special guest i'm going to take you through my name is aries marco janakis you can find me on twitter at aries marco and i'm organizing a js monthly and also i'm organizing ctjs conference we're always looking for new people so if you if you want to help the community we're a community meetup you can email us at the js monthly london gmail.com and you can involve help help us and help the community i have a special announcement today we're gonna be doing the first outdoor and in-person event in london it's gonna be happening at the beach bar next time in london on the 7th of september there are only 70 tickets available at the moment so grab yours and come and join us in september i will now pass you to a special guest thanos volcanics is quite well known for his community contribution and he's going to be talking about why logging is important we're going to have a q a after the the talk and for any questions you can post them on and also on our youtube on the youtube feed thank you very much enjoy hello everyone my name is thomas i'm very happy to be here with you one moment to share my screen and get started right away so iris can you see my search screen uh yes i can cool so today we'll talk about why logging is important and how logarithmic can help you with that logarithm is a logging library that i've built and we will see what it can do and how it compares to other popular libraries on the node.js ecosystem so before we get started a few things about me i've published nearly 50 packages on npm i've got over 200 open source repositories and i have contributions on major and small open source projects like mongoose equalize socket and a few smaller ones so i um i'm quite passionate about open source and i've been doing that for over 10 years now it's been a few couple of years that uh i consider myself semi-retired and focused on gardening and quite recently i have developed a interest on decentralized violence which is an entire subject a different subject entirely so uh let's see what we're going to talk about today today um we're going to see why logging is important what are the logging requirements so as you are required to set up logging uh on your system or your infrastructure what the requirements should be what and how to log and where and when to log and then we're going to have a quick pass-through on locality and how it compares to other libraries so this is quite straightforward logging is important because having a log in the right place at the right time and with the right kind of information can really save you from hours of frustration troubleshooting and most likely save your uh you know your life production system from burning down and staying and having downtime for hours having a very bad day um so let's break this down what logging helps you with it helps you with natural debugging as you're developing error tracing while on production and local development as well performance troubleshooting i wouldn't say that logging excels at performance but it can give you hints if you have the correct logs at the right place as to whether you have bottlenecks in your system accounting accounting is the act of monitoring everything that happens in your system everything that changes every change on your system and not the trail audit trail is accounting when you store it on an immutable uh data store and we're gonna see in depth what we mean by that later down on another slide and uh security so logging helps you with security as naturally you would log suspicious events and attempt breaches now let's start with the requirements and those requirements really kind of apply to all uh you know all sizes whether you start with a single instance on a single server or you have a big infrastructure the you know the natural outcome of your application as it grows is that you're gonna have multiple instances and probably multiple operating systems and multiple stacks so those requirements are kind of cross across you know all of the sizes of your infrastructure so first off uh you know one of the requirements is that you need to take under account that multiple instances of your application are going to be running so this is about scaling you know horizontally your application your application will have multiple workers which are going to work on the back end and perform you know reduced tasks and things that need to happen in the background you will probably have multiple stacks and runtimes so you know might have a server that runs on go another server that does something in java and uh and you're gonna have you know basically a multi-stack environment and it's most likely that you're going to be using multiple services and databases like you know postgres nosql database radis all sorts of things and it's quite likely that if you grow enough you're gonna run on multiple platforms that is multiple um providers like you might be in between the migration from aws to google cloud or you might have some service running on azure to satisfy a certain customer all things can happen so that also needs to be taken under account now when you are thinking about how you're going to consume your logs you need to you need to consider the following requirements as well which is it makes sense to have a common logic aggregator so you won't need to hop between systems to view your logs right you would expect that all of the infrastructure of the systems across your infrastructure log to one single system and where you can view the logs and it would make sense for that aggregator to be querable and while most aggregators are basically text search engines it would be way more convenient to have your logs in a json format so that you have a specific schema that you can query against okay i don't believe i need to go more in depth about why this is important and you need to have a common logging schema and so this is very important and that's why i want to repeat it and it makes a whole lot of sense for all of the applications no matter what the stack they're on to use the same properties when they are referring to a time stamp okay so you would be able to query easily based on this timestamp or based on the log level and you know every system wouldn't have its own different log level system or different schema of logging things and you would be forced to jump through hoops to be able to interpret logs between different systems so having a common logging schema is of paramount importance let's now see what and how to look and let's start with what and we start with who am i and who am i is going to be explained like you know who am i my node.js application what instance in my running what worker am i where am i is um you know where am i running on what server exactly what data center and what i buy what is my performance what is my function metadata and flags and tagging so this is actually the mid of the logging message which is uh okay we've covered the basics who am i where am i what am i now let's give you the actual data that i want to log this is what we call metadata and flags and tagging we're going to see later on what we mean by that it is basically creating boolean tags and flags to easily being able to filter and query for the logs that you care about so let's see what we mean by who am i who am i means you know what's my run time is it no does it go is it php the hostname that i'm running on uh the exact hostname my process id is my process name you know how was the application launched was it npm start or note dot or node app slash starts whatever how did this came to be where and where am i and this is this has to do about uh you know logging level the severity is it an info message or is it a warning or is it a fatal message that brings the service down what module was the log did the log originate from and that is what file in particular produced that log and more specifically what function in that file produced that log and this is important so you want when you're trying to debug your application and you see an error you don't need to go search across your entire code base to find where this log got produced from you have this information right there and then and you can directly go to that file and do what you have to do because without saying that we need a timestamp now flags and tagging flags and tagging is nothing else other than having boolean flags basically that uh create tags for your log messages and you might target log messages as being security related so you know you have a failed login or something that is really you know you should raise your awareness around security you tag that log message with the security equals true flag equally when you when it's an error condition you have an error boolean so you can easily query for that and uh similarly an audit and we're going to see more examples about an audit later let's go into the how to log um pretty straight forward you need to have the ability to pretty print pretty printing is the act of having the logs uh pretty printed in a human readable manner so this is a human readable log you can see the date here you can see the log level you can see where the log came from it's from this particular file and the log message itself booting services and when you're a production you need this to be outputted as a json so this is the same block message when it's been echoed on production which we can see it has way richer information like all the things we talked about host name process id the process name and all other stuff that you may want to attach and when you're testing you would expect that all of your tests are passing on your continuous integration machine so you basically mute your uh logs so they they won't produce huge artifacts and in case something breaks and breaks and you want to debug it then you can manually enable tests for ci but typically from experience it doesn't need to happen that often so let's talk about where and when to log and it makes sense to uh to start logging logs on your application boot up and that means that you know as your application starts to bootstrap itself you create the first log should state you know who am i where are my node environments operating system running on what user am i running on that's important so for instance you don't want your application running as good and you want to uh make sure that your application is running as a specific user so you log that and you know that that's the case models and service booting services putting up so basically as your application bootstraps it starts to connect to services before it becomes ready uh log all of those steps it's one by one so you know i'm connecting to postgres i'm connecting to redis i am connecting to xyz api and i am warming up the cache whatever you do as steps of bringing your application in a ready state log them each one of them so when something breaks it's easy to understand you know at what point your application broke and how to fix it and you know exactly where to go error handlers so there are uh global error handlers on node.js that you can capture like uncode exception or uncode problems rejection when the application goes down c gain sick term you know all of that stuff put a lot there look for those and put a lot there you have an http server whatever that is there is a certain global error handler for the http server log that as well so you know what's happening there and uh throughout your code base wherever you have you know try catch statements naturally all of those need to log and typically they shouldn't bubble up because you would best practice is to handle the log on the spot and part of handling the log part of sorry handling the error is logging it accounting audit at security so what else do you log throughout your uh application's lifetime only mutating operations any operation that basically writes on your database mutates the state like update the user profile save a different email create a new product whatever of your application is whenever the state changes whenever you have a right in the database log that you know this user did this change at that time previous value new value depending on the use case you want to log more or less and this this uh accounting becomes an audit trail when it is uh saved on an immutable store and what's an immutable store well again it's up to the it's up to you to figure what this is but for instance a simple example would be to have um an s3 bucket that you know has only write permissions does not have update or delete and then automatically that becomes you know kind of an audit trail because the store now is immutable but you know all of this is going to be dictated based on your infosec policies and your mileage may vary i'm coming from uh security aware business and that is why the reason why i created locality that's the reason why i dived so much deep into logging and this is important for security aware applications and when you're in that kind of regime it might be uh required that you also log the reads so uh you know this user access this information that might also be important for your compliance requirements and naturally to easily capture what needs to go where like what needs to become an audit trail and what it pertains to security the flags had you know we talked about the boolean flags before now one thing to be aware of is personal identifiable information and considering uh european and california's privacy laws and considering the fact that it's most likely the case that your logs are going to be aggregated on a third party like locally or something else and you cannot guarantee you know the safety of those logs or the retention or the fact that they remain on sovereign land like the data remains in europe and doesn't travel across continents so if you're not sure tokenize the data basically so convert emails ids and names into a hash and md5 string and work work like that so that was the theory now let's go into uh the logging applications and let's start with the deep dive on locality an application i authored because of the requirements that i had had a very high uh at the very demanding security wise business so this locality location is a versatile and extensible logger for node.js what does it do differently well it prints log messages naturally that's a standard thing but aside from that it can extend the login schema to fit your needs it has built-in overrideable serializers i'm going to see exactly what those are you can also have custom serializers it supports middleware like express does like true middleware and you have multiple middleware where you can transform the log or you can perform different actions and again we're going to see this in more detail uh locality enables full manipulation of the output so you can rewrite the entire schema to fit your needs remember that is a requirement that all systems uh log with the same schema and that is what is the case right here it supports asynchronous operation which is a unique feature of locality and we're gonna see why this is powerful and very useful and locality can also be used on open source libraries and composed on the root project which is something that has an open source contributor i have suffered with for a very long time and going to see why and locality also has some helping features that automatically detects the module filename and path that the log originated from so let's see an expanded log of logality so you can see the level is expressed both as a string and as a number they both have their use cases and they are both important to exist you have the message here and it's either an event or a context that's going to be populated depending on what triggered and what the data is of the logger of the log that is being produced and as you can see here we have the application name we have the source where this log originated from and system information is automatically also populated all of this is automatically detected by locality for you you don't have to do anything previously that was a json log expanded this is the pretty print output the logging level uses the syslog rfc system for severity levels so we start from debug info notice up to emergency pretty straightforward and this is how this is the basic signature of uh locality so you have the locality instance and then the methods are based on the severity of the log so this in this case that's info the first argument is the message and the second argument is any data you want to pass through this is basically the metadata and here is where serializers kick in and what serializers do is that they look for a specific key like the user key and then they take the value of that key which is the udl in this instance which is uh initials for user data object and serializes it in a way that that makes sense for logging so for instance this is a built-in serializer and this is a user similarizer whenever you input a user object with a user key with an object on the building serializer this is what it will do it will extract the id and the email and there are a couple more built-in serializers you have a serializer for javascript errors so if you pass an exception locality will automatically break it out into name message and then it will give you a very nice stack trace that is serialized without new lines and finally there is the express serializer where you pass the request object of express and you get some very nice information out of that already out of the box you can have your own custom serializers say you are an eshop e-commerce website and you have an order and you know naturally all of these objects are their schema is standardized and that's the whole you know convention here that the order objects don't have the same amount of the time and it should throughout your application so this is serializing the the order object so we're being more explicit here this is another id so we take the id this is the skuid so we take the sku we can perform transformations like we can have multiplication between the fields we can do pretty much anything this is plain javascript here so the next time you log anything and you pass an order item under the order key this serializer is going to run and properly format the object for you now you are also free to overwrite the building serializers so if your user data object doesn't have id and email but it has different names that's totally okay in this particular example we are adding a first name and the last name on top of the id and the email so whenever we pass the user object this is how it gets serialized now let's look into middleware and as i've said before midway are a very powerful feature of locality it allows you to interject business logic in the logging lifecycle so the the usage is pretty straightforward you have a use function and then your callback the callback receives one argument which is the log context analog context is the entire log message represented in a native.js object which is mutable so you can you know delete certain fields you can change them shift them you can do whatever you want basically with the log message before it goes downstream to the output functions and here is where locality really excels at which is it has this unique feature that with a single streets during instantiation of locality you can make it operate asynchronously and that means that all of the middleware are operating asynchronously and that enables some very very interesting um applications where you can now start distributing your log messages to multiple endpoints so for instance you can see here that we perform a database right of the log and for instance this could be there could be a conditional here checking for a boolean the audit boolean that we talked about before and if the audit boolean is true then you know i want you to write this on the database as well and what this may be is up to you again this is just you know showcase or if it's a security kind of log then i need you to send that log on my slide channel as well so you can have all any sort of business logic here interjecting into your log messages and i remember that these are very nicely formatted messages as we have serialized them in the previous step right serializers come before middleware in the life cycle of the log naturally that means that you're gonna have to invoke all of log messages with the await yeah you're going to have to because they execute a sequence link it's required that you wait for them the keyword is a way to make sure that the everything executes properly and last but not least is that you can interject on the master outputs so this is how locality is instantiated we require the library right here and then we get an instance which accepts a single argument an object of configurations we pass the app name we say whether we want to pretty print or not we say whether with this instance is a synchronous or node we just saw the example of asynchronous and uh then we can define and overwrite the output and basically what happens here is that we are overwriting because as long as we are not returning the log context from this function locality will not continue any further action if we return to local context in this function then locality would proceed and and use its own output functions but now since we don't do that locality will take no further action because this is a signal to locality that do not plug this with taking care of that and you don't need to bother yourself with outputting anything and that's the last feature that i talked to you about which is locality can be used in libraries so this is a third party library imagine this is mongoose or this is equalized but you're requiring and sqlize then exposes the locality instance which you pipe to your own instance and then with the use of middleware as we saw you can intercept the log messages coming from mongoose or sqlize and apply whatever business logic makes sense to you this has been a very thorny issue because you know major libraries require some sort of logging so that you know and understand what's going on and the only way for logging libraries to pass information upstream to the application required in them is through errors and that is a very brutal method right so the legality let's have a quick look at uh how lugardi compares to all the other popular libraries out there and i've chosen three of the most popular ones winston with 6 million downloads per week pino pinot has been hosted a lot of times by this meetup and probably are very familiar with it and bunyan was one of the classic vloggers of node.js in the past so all of the libraries output has json which is very important and very nice all of them pretty print except boonian which requires another library to transform the json output and 3d print it so it's not included custom log levels is not supported by locality and bunion but i guess if you're really determined with custom block levels you can intercept the master output and transpose localities levels into whatever you see fit serializers winston doesn't have serializers bunion does pinot kind of has it gives it a check mark because it it's got the same concept middleware locality has material as we've seen winston has milwaukee albeit in a very complicated way that's a general remark around winston is that it's fairly complicated as a library to work with bunion doesn't have middleware and pino barely has midware in the sense that it only has it only supports a single mirror and so you can't you cannot have multiple middleware with pino mutating json schema we've seen why this is important and why this is a core requirement for the logging libraries because you cannot get you should get married with the schema that the authors propose right because then there's another library on go that has a different schema and you need to have parity between the schemas so uh that's only supported by instant and locality output destination you can choose another destination on all of the libraries mutate the output the master output function that we talked about this is only possible on missile and legality a sequence operation as we've seen is only available with locality file name detection again it's a locality on the feature speed optimization is clearly uh pino is the winner here that's the entire value prop of uh pino and the reason why it was it was built around it's it's a very fast logger that's you know that's the reason why you use pino and library composibility is only available in locality all the other libraries have a system of producing child loggers which only works on the same application so to recap logging is important for uh debugging and troubleshooting security and auditing excel is there performance monitoring less so but it can help common schema and appropriate logs can make all the difference in your entire infrastructure and you have many choices for a logger choose what is appropriate for you not what is more shinier and that's all since i don't have a live crowd i'm going to upload on myself thank you very much and i am open to any questions that you might have harris yeah that last slide because i've seen it before uh for me it's the key slide from the whole talk where people can can see why you know it's so important to use and i mean people just have to get a coffee or a beer or thanos because he helps them to find out what is the the best solution and in greece probably they have to get him a coffee because that's what we drink most of the time and but yeah i mean that was a good talk and let's see if there any questions on on on youtube and we're waiting and [Music] how did you come up with building this building locality and so the requirements were from the security aware business that i was working with and we needed two requirements that basically could not be done by any package that i completely transform the schema so it meets a certain you know it meets a certain schema that we had in the infrastructure that would work with all of the stacks across the infrastructure and the other requirement was being able to flag and store audit log trails as they happened so you have such well-structured information it made sense to leverage that in creating the audit logs versus building a different uh service that basically did the same thing has serialized information and stored in a different data store okay and then and i am probably a bit of repetition but it's good to know um how was the thought of using these creating a something from scratch rather than using something that already exists how did you take that decision and what was right that chart that you saw was with them all the different systems that they didn't have something that made you build something from scratch yeah i mean that goes back to the matrix this that i had with features and the reality is that logality didn't wasn't built in a day so it started like four years ago as a very simple uh way to log a schema and provide a pipe to save the data on a different endpoint as well it was very very you know custom made and very coupled and took the course of four years to make it into a library where everything is decoupled and it's more of a generic use rather than you know verticalized structured only for our use case okay so it can be used now much easier than all the other different solutions like [Music] well i wouldn't say easier but it's like yeah it's more of a generic logger right now rather than a specific logger to our particular needs and it just so happens that out of that you know some interesting features came out like the middleware where you can have intercepting uh the live cycle of a log and the asynchronous operation that allows you to pipe the log messages through different uh mediums like slack or you know location data store no that's a good feature um no that's very good and and how have you seen the the usage so far have they been using intellect you know some projects that have it uh to to be frank i'm not monitoring it but we can check it right now yeah it will be good for people to know where it's being used in some different kind of cases i have an open source uh [Music] library that is linked from locality that uses locality and uh so that people can see how it's actually using production the usage is uh rather small but it's not something that i've been actively uh promoting as i see now on nbm it is at 55 downloads per week so that's very smooth very small but with the help of your meetup maybe we can raise them and you're looking for contributors and people to help you to improve are there anything that in particular that you are you that you are looking in terms of help i am looking i'm not looking for any help in terms of development i i'm mostly looking for participation in the project people using it people coming out with feature requests or issues and moving the library forwards okay now that's great it's good to know okay yeah um so yeah so it can be used for uh for uh both back end and front end right because you said uh is it is it i believe so i haven't really uh tested it on the front end but uh i intend on doing that fairly soon and if anything needs to be worked out i will do it on spots okay that's very that's perfect and there are no questions on the on youtube um but um we will be checking the defeat because people can watch it after this and when they have time yep yeah i will check it too and answer any questions in the comments perfect uh thank you very much as always thank you thank you for the opportunity and we're very excited to have you every time and we look forward to have you in the future as well thank you very much thank you have a great day [Music] foreign [Music] do [Music] so [Music] so [Music] [Applause] [Music] my [Music] one [Music] so [Music] [Music] so [Music] so [Music] so [Music] so [Music] so [Music] [Applause] [Music] my [Music] [Music] foreign [Music] you [Music] [Music] so [Music] so [Music] then [Music] so [Music] foreign [Music] so [Applause]

Channel: Pusher

Views: 174

Rating: 5 out of 5

Keywords: developer language, language code, libraries, help, tips, forum, meet-up, meet up, learn code, coding education, coding hints and tips, lecture, coding lecture, learn about code, learn a developer language, amazon alexa skills, developer conference, node.js, javascript, backend

Id: S_AzdcJhM9s

Length: 59min 22sec (3562 seconds)

Published: Fri Jul 16 2021