Logging in Django and Python Applications - Handlers / Formatters / Better Stack aggregation

logging is essential for building dependable and maintainable software applications with a proper logging setup you can record events and diagnose issues that occur in your applications and build an audit Trail for your software and by building this Trail you can help diagnose issues and you can understand runtime behavior of your application and a good logging system will allow you to solve critical problems very quickly in your application for example you may email administrative users when a problem occurs and that problem can then be fixed very quickly and logs can also be analyzed and charted to get further insights into your software using log aggregation systems we're going to see that later in this video so this video is going to be a deep dive into how to do logging with d Jango web framework applications we're going to start with a brief overview of logging in Python and then we'll see how to build out the logging functionality in jangle using one of the settings that the framework provides and we're going to see also how to integrate that with an external log provider later in the video so let's get started and we're going to start by looking at the logging module that's built into the python standard Library this is a module that defines functions and classes that Implement a flexible event logging system for applications and libraries so this package standardizes logging in Python and it provides you a lot of utilities that you can use and it can be imported as it says here from the standard Library we're going to start in this video by looking at a sample application just a script in p py so I'm going to open VSS code and I'm going to walk through the code that we have here and we're going to understand what comprises the logging module in Python so we're importing the logging module and also the CIS module from Python and then what we're doing is we're creating a logger object we're using the logging modules get logger function and we can pass any name we want for our logger I'm just calling it my logger here now the loggers are one of the key objects that provided by the logging module and what I'm going to do is go back to the doc documentation this time for Jango and there will be links to these documentation Pages below the video what we're going to do is look at what Jango calls the cast of players in the login configuration for the web framework now Jango extends the python logging module that's the one that we just looked at here and it provides its own way to configure the logging module that we're going to look at later in the video but the cast of players is what we're going to focus on now and we have four main objects that are involved in the python logging module we have a logger object and that's what we just saw here this is kind of like the root object in the logging system we Define a logger and we give it a name and what we also have is handlers filters and formatters so let's quickly describe each of these objects the loggers they are the entry point into the logging system and each logger is a named bucket to which messages can be written for processing and loggers have this important concept of a log level the log level describes the severity of the messages that the logger will handle and by by default python has five log levels we have debug which is the lowest level then we have info for General system information we also have the warning log level which describes a minor problem so you might want to log those out and analyze what those minor problems are but we have some more serious log levels as well we have error and that's a major problem that has occurred in your system and we also have critical for describing a critical problem so each log that you write to the output will have one of those levels and if you need to you can also extend these in Python and add your own log levels as well so the logger is the entry point into the logging system if we scroll down we come to the Handler section the Handler is the engine that determines what happens to each message in a logger so it's going to describe the loging behavior for example writing a message to the screen or to the console or writing it to a file or sending it to an HTTP endpoint to an external service you can do all of that with python and the Handler is basically going to describe where you're going to send your log messages to as well as handlers we have filters these can provide control over which log records are passed from the logger to the Handler we're not going to cover filters in this video if you're interested in that we can do a followup on that topic we also have formatters which are quite simple these simply describe how the log record needs to be rendered as text and a formatter consists of a string containing log record attributes if we follow through to the link here we're taken to another page in Python's logging module and there are attributes for things such as the time at which the log has occurred and we also have things such as the level number the message that's actually being logged to the system and many more variables or attributes that we can attach to our messages so what we're going to do is we're going to now look at that python application that we saw in Visual Studio code we've already defined the logger object what we're doing in this application is we're also defining two handlers here we're defining a stream Handler that's it's going to write to standard out and that's why we're importing the CIS module at the top so we can specify the stream that we're writing to here that is the standard output stream on the system and below that we have a second Handler this is a file Handler and it's going to write to the file that we provide here as an argument in our case it's going to be logs. text and what we do is we take the logger object that we created and we're going to add each of those handlers to that logger so we're adding the console Handler on the first line here and then we add the file Handler so the important takeway there is that a logger can actually have multiple handlers and it's going to send the log output to all of the handlers that are added to that logger and above this we also have a formatter which is currently commented out we'll bring that in in a second what we've got at the bottom of this file is a function called division that's going to try and divide the first argument by the second argument and you can see that operation here within a try accept statement of course the potential problem there is that we might try and divide by zero in which case we're going to send a logger do error message out to our handlers and what we're going to do in this block at the bottom here when the python script is called we're going to call the division function and pass in as the denominator zero so that we're going to see what the loging do error message is going to Output here so let's bring up a terminal here and what I'm going to do is run this program so we're going to say python logs. pi and you can see we get this basic output message here division by zero error with parameters four and zero now if we want to customize the message that we're get getting output by the logger what we can do is we can go up to this section that we've commented out here and that's where we Define a loging do formatter object and we're passing a particular format string to the formatter and we're referencing some of those log record attributes the name the current time when the log is generated the level name and also the message so let's try and re-execute this but first of all we need to actually add the formatter to our handlers so we're going to add it to the file Handler and also the console hand Handler here and we do that by calling the Handler's set format or function and passing that in as an argument let's re-execute this script at the bottom and then we can see we get the formatted log message it contains the name of the logger and that name is on the left hand side here then we get the time when the log was generated we're getting the level which is an error message and that's because we are calling logger do error within this function here and finally we're getting the message at the bottom here that was output to the log file now you'll notice that this is on the console here but we also have a file on the left hand side called logs. text and if we open that we're going to see the same output in this file as we saw on the console and the reason for that is because if we go back to the program and we scroll up here we're creating two Handler objects here and adding both to the logger object so we're going to send the log output to both of those handlers so both the console and the file now before we go on it's worth noting about some security implications here you don't want to be outputting logs containing things like email addresses and passwords or credit card information or anything like that so you need to be careful what you're putting in your logs when you're outputting these messages be careful not to include any sensitive information in log files and always assume that the log files can be read by anyone so that's a very quick overview of Python's logging module we've seen the logger object we've seen the formatter object that allows us to determine the format of the output and we've seen the Handler objects that we've got here and these allow us to send the output to a particular location what we're going to do now is move on and see how to use logging in a jangle application so I've just added a jangle project here and this has some very simple configuration and what we're going to do is we're going to see how to use the settings.py file in order to set up logging in ajango application and we're going to see the different settings that we can use and later on how to integrate it with a log aggregation service let's start by going to Jango documentation on logging again we saw the cast of players earlier in this video what we're going to do now is we're going to going to look at how to configure logging in jangle Now by default jangle uses What's called the deck config format so in order to configure logging you define a setting called loging and those settings are going to describe the loggers the handlers the filters and the formatters that you want in your jangle logging setup as well as things like the log levels and any other properties now what we're going to do is we're going to add that setting of logging so let's scroll down a little here I'm going to copy a couple of the statements here I'm just going to copy the top of this dictionary and we're going to go to settings. Pi and let's scroll right to the bottom of this file and we'll paste those in there and we'll close off that dictionary so because we're using this dict logging format that has a version which is just one by default and that's the only version that exists at the moment we have an variable here called disable existing loggers which we're setting to false and normally you would want to keep that as false in a jangle application we're now going to start defining the actual logger objects and handlers within this dictionary if we go back to the documentation we can see how to do that we have a key within this loging dictionary for our handlers and you can see we have another key here called root as well so we can Define any handlers within the handlers block and we can also Define loggers in this block here and this setup is going to configure the root logger to send messages with a warning level and higher to the console so why is that happening in this setup we have the root logger here and we're adding the handlers to that within that block and we're setting it to the console Handler which refers to this Handler that we Define in the Handler section above and that is going to be a logging Dost stream Handler and that's something we saw earlier in the script that logs the output to whatever location you provide as the stream to that class and the logger object also has the level set here to warning what that means is that only messages with a level of warning or above are going to be output to the handle so we will see warning messages in the log and we're also going to see the error messages and the critical messages because those log levels are higher than the warning log level so let's go back to jangle and we're going to define a block for our handlers now and I'm going to scroll down a bit to give us some more space here so that's going to be another dictionary this is a nesty dictionary and we can name our handlers here I'm just going to call this one file and that's going to Output to a file Handler so what we do within the file block is we Define the class of the particular Handler that we want to use what we're going to use is the loging do file Handler class and then we can set a file name of course if we're logging to a file we need to tell D Jango or rather we need to tell the file Handler where do we want to actually send those logs I'm just going to create a log called general. log and we can set a level here as well which I'm going to set to let's say debug so that we see basically every message that's being logged so just to recap we're setting up a file Handler here and that's an instance of logging do file Handler and it's going to Output those logs to the general. log file now after the handlers let's define another key within this logging dictionary this one's going to be called loggers and that's going to Define any logging objects that we want to use within this jangle application now you can provide a catch all by providing an empty string rather than creating a named logger we just create an empty string that maps to a dictionary here and we can provide again the level for the messages or the logs that we want to Output again we'll set that to debug but a logger needs to have a Handler attached so that it knows where to send those messages so we're going to set a Handler's dictionary here and actually that won't be a dictionary that will be a python list and we're going to set it to the file Handler that we created above within the Handler section now I'm going to reference another page of jango's documentation just to explain this catch all empty string that we're defining here within the logger dictionary let's go back to a page here that I'll link below the video and this is about how to configure those logger mappings now you can see in this documentation that they're using that empty string here and if you look at the paragraph below here you can see that the mapping name determines which log records it will process but the configuration with the empty string is unnamed and that means that it's going to process records from all loggers so what we're doing here in this setting by defining an empty string is we're telling D Jango that we want all of the log outputs to be sent to this particular log configuration that we have here you can get more granular than this and we're going to see examples of that later on now what we're going to do is save the settings. pile we have just a recap added a loging dictionary that's going to tell jangle how we want to configure and Define the loging in this application what I'm going to do in this jangle project is go to the views.py file and we're going to go to the top here and I'm going to import the logging library from Python and once we've imported the logging module I'm going to create a logger here we can do that by calling the modules get logger function and again we pass a name in there for the logger as you can see in VSS code when you call this function it's going to return a logger with the specified name and it will create that if necessary and in order to give the logger the same name as this module we can use this statement here this is double uncore name double uncore and that is a python variable that evaluates to the name of the current module so that's going to evaluate to core. views within this jangle application core comes from the name of the app in jangle you can see that on the left hand side here we have an app called core and the file that we're in is the views.py file so we're giving the logger a name that is relative to the file in which it's being used and you can do this through all of the files in a python application and then you'll get a logger that's kind of specific to each module and that can help with organizing the logs that you've got when you come to analyze them later on so that's has created a logger object here we're going to use that object here within the index view that came with this starter code so what I'm going to do at the top of that view is just log out a dummy message we're going to use the logger and let's just use the info method here and we can pass a message into that method so I'm going to pass just a basic message here that we're testing the logger and what we're also going to do is import from Jango its user model and we're going to try a look up here that we know is not going to return an object because that user does not exist in the database so we're going to define a try accept block and I'm going to try and look up the user by an ID so user doobs doget and let's pass a primary key of one in here that we don't have any users in this dummy application so we're not going to get a result here so this exception is going to be called or rather the exception block is going to be executed here and what we can do within that block is call the logger do error function and we can pass a message into that function so I'm going to say user with ID and then reference the ID that was passed in does not exist and we can pass a second argument to logger do error and in fact it can take as many arguments as we provide as placeholders in this string so we're passing the primary key of one that user does not exist and that's going to be defined as an error log so we have two logging messages here a logger doino call and also a logger do error call if this look up to a user does not exist so let's test this out and save the views.py file and we're going to first of all run the python manager . Pi migrate function and that's going to generate the database that we're going to use that's just a local SQL light database once we've done that what we can do is run the Django server and we're going to visit this index URL here so the URL is just the root path in this application so what I'm going to do is go to Local Host 8000 here and when we go to that in the browser you can see a message of hello here what we're going to do is see if our Handler and our logger are working by going back to vs code and we should see a log file that's that's been generated within the core application or within the project sorry and you can see we have a file here called general. log and if we load that file you can see we're getting all of the log output because we use that catch all empty string logger here we're getting the Jango logs for example this one from the stat reloader but we do get our message here where we test that logger out and we also get the logging do error message where the user with id1 does not exist now these are pretty Bare Bones messages at the moment when you include logging you really need to include as much useful information as possible to help you diagnose where and when that problem has occurred so what we're going to do is go back to settings. Pi and we're going to create better log messages by defining a formatters block so this login configuration in jangle as well as the handlers and the loggers we also can Define formatters that we can then add to our handlers so let's define a formatter here we'll call it a simple formatter and we can Define the the configuration for that formatter in here what we need to give that is a format and I'm going to reference some of those log record variables that we're going to look at in a second so the level name which is going to be info or error or so on we'll output that level name and we also want to Output the log message itself and what we also can do is Define the style of the output here with a key called style and we can just pass a single curly brace as the value for that output and to see why I'm going to go back to Jangles documentation and there's a section on the logging howto on configuring a formatter and you can see we use those log record attributes such as name and level name and also the time of the actual log output but if we go down here we can also see the style keyword and this curly brace means that it's going to use the string do format style in Python whereas a dollar sign there will use the string. template formatting so if you want to refer to these log record variables just using the curly braces and then the name of the variable you can set the style to that curly brace now once we've created the formatter which we've given a name of simple what we're going to do is go up to our Handler here and this is the file Handler and we can add a formatter to the Handler here so I'm going to add a key of formatter and we can paste the name of that formatter which is just simple and then we can save this and we're going to test out this application Again by going back to this page and if we refresh the page and go back to the general. log file you can see we're getting a slightly more informative set of messages now we have the log level as well as the message now we might also want to know the time when these logs occurred so what we can do is go back to the formatter and I'm going to add another one of these log record variables here and it's going to be ASC time basically the current time at which this log was generated if we save that go back and refresh the page when we go to general. log you can see at the bottom we're now getting the time stamp when these logs were generated both the info and the a error log and we can Define as many formatters as we want here so as well as the simple formatter I'm going to paste another one in here and let me tab that over so that it looks better we have one here called verbose where we have many more variables in the format of this particular formatter so as well as the time and the level name we also have the name of the formatter and the module that was generating that log message and then we also have the line number as well here using the line number variable and finally the message its output as well so we've called this one verbose what we can do is scroll up to the file Handler and we're going to change the formatter that's being used to verbos here and we can go back to general. log and what I'm going to do is go back to the page refresh the page and now we can see we're getting a more verbose message here beforehand we just had the time the log level and the message what we've got now as you can see at the bottom is we also have the module and the name of the logger that's being used here so this is the name of the logger core. views we have the module and also the line number as well so this is a more verbose logger that allows us to get more information from our log files so if there was ever a problem in our application a log like this one is going to tell us a lot more information than some of the logs that we set up at the beginning here so let's scroll to the bottom here and in fact for now I'm going to clear out this log file and save it what I'm going to do is go back to settings.py and as well as the file Handler we can also add other handlers to to this application so I'm going to add another one here which we're going to call the console Handler and that's going to be another dictionary here and I'm just going to copy some settings in so for the console Handler we're using the stream Handler class instead of the file Handler and that's going to Output these logs to the terminal basically we're setting the level of those messages to debug and we're also setting the formatter to the simple formatter that we defined here so not the verbose one like the file is using we're now using a different formatter for our new console handl now in order to use that we need to go to the logger section and actually reference that Handler within here so we're going to copy the name console and within this list of handlers we can paste that in and that's then going to send the logs to all of the handlers that defined within this list so let's see if we get some messages on the terminal now when we go back to our page and refresh the page if we go to the terminal you can see we are now getting some messages and this is using the simple formatter so we're only getting the time the log level and also the message so we're learning a bit here about how to use jango's logging setting here which is using the dictionary to define how we want our handlers or loggers and our formatters to handle our log outputs what we're going to do now is we're going to look at configuring log levels dynamically using environment variables so what I'm going to do is stop the Jango server and we're going to install a library just now that's going to allow us to readin settings from an environment file now actually let's have a look at why we might want to do this we have here a file Handler and we've hardcoded the name of the file that we want to log this output to and that might not be what you want to do you might want to have this determined by an environment file and you can set that up per server and Define the name and location of the files that you want to log your output to any kind of hard coding of strings like this is generally not the best practice when it comes to these settings and another thing we can do is move the level of the logging out to an environment file as well and that means that we can maybe specify a level of debug or info on development but for production you might want to only look at warnings and above or vice versa the point is that we want these settings here to be configurable from an environment variable file so what we're going to do is move these hardcoded settings out of settings.py and within this application I'm going to create an environment file a do end file and I'm going to paste in two settings that we want to use here a jangle log level which we're going to set to debug and also a Jango log file let's specify a new file here called Jango logs. text now in order to actually reference these values in ajango settings. pile what I'm going to do is scroll to the top and we're going to need to import a module here or rather install a new module I'm going to go to the documentation for this module it's called Jango Environ and this package allows you to use the 12 Factor methodology where you can configure your application using environment variables that can be changed very easily for development vers production versus testing and so on so what we're going to do is scroll down and we're going to go to the installation section and I'll leave a link to this below the video we're going to copy the PIP install command and we're going to go back to vs code and install that into our virtual environment once that's installed we can then use the library so let's go back to the documentation and the quick start is going to help us out here we import an Environ object so let's copy that into the top here just underneath the pathlib import and then once we've imported that what we're going to do is set up an inv object here so let's copy this again and we're going to paste that in at the top here we're setting up the N object that's going to allow us to then read the values from this n file and in order to read those values we are going to copy one final statement it's this one here and I'll copy the comment as well actually and we can paste that into the file we need to do this below the base directory and that's because we're referencing that base directory here and we're using path lib the base Direct is an instance of a path lb. path object so we're going to use the path Leb syntax here to reference that n file that exists in the base directory so what this is doing as it says in the comment it's going to take the environment variables from thein file and it's going to load them into our inv object that we've defined here and we can then reference these values by the key name for example Jango log level so what we're going to do is we're going to go back to settings and we're going to do that down at the bottom where we're defining these loging settings so the file name here we want to get rid of the hardcoded reference to that and we're going to use the EnV object and reference the value that's coming from our environment file that's the jangle log file setting so let's paste that in there and we also want to remove the hardcoded level here we're going to change that and reference again the name of the variable in the end file so let's paste that in there that's the jangle log level and we're going to use that in multiple places throughout this configuration so let's copy that and for the console Handler we're going to use that there as well and also within our loggers setting we're going to paste that in there too what I'm going to do now is just test that this is working so let's start up jango's development server here and then we're going to go back to this page and refresh the page if we go back to vs code you now see that we have a new file here called Jango logs. text and that is the file that is being referenced in thein file here and we're reading that in in the settings and we're applying that file to our file Handler here so this is working the setup is working and we're getting our logs now in this file what we're going to do now is look at a new concept and that's logger namespacing so far we've been using an unnamed login configuration just with an empty string when you use a named login configuration that will capture the logs only from the loggers that have a matching name now what we've done before in the views.py file if we open that now we have defined a loger here and we're giving it a name of cord. views basically calling it the name name of the module so if we only want to capture the logs from that particular logger what we can do is go back to settings. Pi and in the logger setting instead of using an empty string we can use the name of the logger that we want to Target here with these settings so we're going to say core. views is the name of that logger and that allows us to Target only the logs from the views.py file so if we go back to our application and refresh this we are still going to see the same result here if we go to the Jango logs we're still getting this output but but if we were to add logging in one of the other modules here and try and log that out that would not be sent to this particular setting here because this is only going to capture the logs from the core. views logger now we're going to demonstrate something new here that's logging hierarchy I'm going to copy these settings and I'm going to paste them in here and I'm going to remove the views part of this logger here we're going to just call that one core and the second one is called core. views so we now have two separate log defined in this settings.py file now as we said the code. views logger is only going to capture logs from the views.py file because we've given the logger in that file that particular name but the core logger on the other hand is going to capture all of the logs from the core application including logs from the views.py file if we go back to the documentation on this we have a section here on logger hierarchies and propagation and as it says loger naming is hierarchical a loger called my app would be the parent of my app. viws and unless specified otherwise logger mappings will propagate the records that they process to the parent loggers so a record from a logger called mya. views. private that will be handled by both the my app and also the My app. Views loggers because of that hierarchical relationship so what we're going to do is see that in action now we have created these two loggers and the core logger can be thought of as a parent to the core. views logger which means that any logs generated by core. views are also going to be sent to the core logger let's see a demo of that now what I'm going to do is go up to our handlers here and I'm going to define a new Handler just for demonstration purposes here and again I'll need to T that over we have the Handler called file views and that's an instance of a file Handler and we've changed the file name where this Handler is going to Output its logs to and for the level I need to reference the value in the environment here so this is our new hand Handler it's called file views we're going to use that Handler here within the core. views logger so let's paste that in there and this time the core. views logger is only going to use the file views Handler so let's run the server again and we're going to test out what is going to happen here if we go back to our page and refresh the page we can go back to our D Jango application and we're going to go to that particular views file Handler if we click this file this is the file where our new file Handler is going to send its logs to and this is just the one that was defined here in the core. views logger so that core. views logger is sending its output to this Handler called fil views but remember this relationship this hierarchy the core logger is a parent of core. views so what's going to happen is if we look at the jangle logs. views. text we get these messages here but these messages are also going to be in the jangle logs here and just to make this a bit clearer I'm going to remove all of the logs from both of these files and if if we go back and refresh this page and go back to these log files you can see the views logger has both of the lines but this one also has the lines and that's because the logs that are generated for this particular logger are also being sent to the handlers to find for the parent so you end up with the output from this particular logger also being sent to the parent where it's also sent to that parents handlers now you can disable this Behavior by adding a key called propagate and setting that to false in the child Handler so this time if we save settings.py go back to our page and refresh the page if we go back to the Views logger here we can see we get the two new messages but now if we go back to the Jango logs. text and this is the one that's attached to the parents file Handler we no longer get those messages because we have set propagate to false in the child logger so there's a hierarchical relationship between loggers based on the name of that logger and you can use the propagate key here in order to control whether or not you want those logs to propagate to the parents or not now we've covered a lot of ground here and how to use logging in jangle and python in general what we're going to do now is bring in a log aggregation service and that's going to allow us to send these logs to an external location where we can use a service that's specialized for dealing with these logs and analyzing them searching for particular items in the logs and also creating dashboards and charts and the service that we're going to use is this one here it's called better stack which has a logging platform for log management and this platform is going to allow you to rapidly search through the logs that are being generated in your application if we scroll down we can get some more information here for example here you can see that better stack is going to aggregate all of your logs into structured data that you can then query with SQL so you can send logs from all of your servers your apps your Docker containers your kubernetes clusters and so on send them all into to one place using this service and then you can search and create dashboards based on these logs and the data that's in them now what I'm going to do is go back to the top here and I'm going to sign in at the top let me try and make this a bit bigger so we can see that going to click sign in and I'm going to sign up now with my email address now once you've logged in you'll be taken to a page like this after filling in some basic details we have a page here on sources and as the info message says here sources represent services that you want to collect Lo from and you can integrate your existing stack by adding a new source so let's start by doing that just now we're going to click the connect source button and we're taken to this form here we can give the source a name I'm just going to call this Jango test and then we're going to set the platform after that and you can see there's many Integrations here with things like Docker and kubernetes and Gen X and there's Integrations with Vero and other platforms we're going to scroll down to python here and we're going to select Python and click create source so we've created The Source here and it's got an ID which is called Django test that comes from the name that we just set and we also have a source token we're going to copy that by clicking this here and that's copied to the clipboard and we're going to reference that now within our environment file so let's reopen that end file and what I'm going to do here is paste in the value here and we're creating a variable called better stack Source token and we're setting that equal to this value here which is the token that we got when we created that Source on better stack and you can see some other information here we have a data retention period of 30 days and that means that the logs that are generated here are kept around for 30 days before they're deleted and if you need greater retention you can sign up for a plan that will include that what we now want to do is go back to our Jango application and we want to reference this Source token in the logging settings and we're going to define a new Handler here now in order to do this we need to install another package that comes from better stack let's go to some documentation here this this is a page on python logging and it contains some steps that we can use here now there's a package called log tail python so what I'm going to do is copy the name of that we're going to go back to the terminal and we're going to run pip install and we're going to call it log tail python here that's the name of the package once that's installed we're going to then use that in our application so if you look at the setup here what we're getting here is we're adding a Handler that we're calling log to let me make this bigger again here so we can see it and that log tail Handler is referencing this class called log tail Handler and we can see that we're inputting the source token here where we have a key of source token so we're going to copy this new Handler called log toil and we're going to go back to VSS code so let's open settings.py and within the handlers block what I'm going to do is I'm just going to remove this file views Handler for now and I'm going to paste in that log tail Handler here and we're using the class log tail Handler that will only be available ailable after you run that pip install command and what we also need to do is replace the source token here with the token that we have in our environment file so let's reference this better stack Source token and we can paste that into the EnV function and of course the final thing that we need to do here is we need to add this new log tail Handler to the loggers that we have here or rather the handlers that are defined for those loggers so I'm going to add it to both of them let's add log tail to the core Handler and also let's paste it in here and replace the file views Handler let's now start D jango's development server and we're going to go back to our application now and refresh this page and let's now go back to the better stack console here and I'm going to go to the live tail and we're going to see if we're getting these messages sent to This Server now you might need to wait a second but you can see we're getting some output now for example the error logs where the user with id1 does not exist and also this info log as well and I'm going to make this bigger again so we can see exactly what's going on here I'm going to click the info messages for this and we're going to see that we can actually filter down the logs to just the info messages so this is very useful and you can see at the top that it's added a filter here where the log level is set to info and of course we can change that for example if we wanted to look only at error messages we could change that to error and we get back this here where we only have the error logs in this case and we can look not only at the level we could look at something like the severity which is an integer between 1 and five in the case of these python logs we could look for all logs that have a severity of greater than two and if we search for that we get back again only the error logs if we change that to one we're going to get back the info log as well which has that severity level of two so we can do these operations here where we can do equality checks or greater than or less than checks and filter down the logs very easily by using these checks and you can see as well we have have two sources this one comes by default when you sign up for better stack but we have the second one here called D Jango test and you can imagine if you had a complex system with multiple Services you could have multiple sources and you can also filter these down by the source for example if I click Jango test here it's only going to give us back the logs from that particular source so as well as aggregating logs from multiple places you can also of course just look at one particular source and we you can do things like string contains searches if we click this bar at the top here what I can type is the message so we can actually search through the message that's being sent in the log if I click that we can Define what kind of operation for example contains or does not contain and there are also equality operations at the bottom I'm going to look for any message that contains user and if I execute that by clicking this at the top we only get back the logs now that contain that message where the user I ID does not exist so it's very easy to search through the logs using better stack I want to very quickly show one final thing before we finish the video I'm going to click the dashboards tab at the left hand side and we're going to create a new dashboard here and by default we'll use a blank dashboard to start with we can add that and then we can start creating charts in a dashboard and this can be used to aggregate the logs that you're getting from your different sources and create visualizations and charts that help you gain insights into to the events and issues that might be happening in your system so I'm going to click create chart here and we are going to get back this here we have an SQL expression and we can select a chart type on the right hand side now I'm going to select a bar chart just for the demonstration here and as well as that I'm going to change the source here to the Django test Source that's the application that we have been working on in this video once we've done that we can click run query at the bottom and we're going to see a chart appearing here and this chart is basically aggregating the number of logs that are coming back at each Tim stamp so we can generate charts and save those charts to our dashboard so if I click save chart here and we go back to the dashboard section I've got a new dashboard here that I can click and we see this chart and this chart will update in real time and give us some idea about the number of logs that are coming into our system so if I go back to our application and we refresh this we can go back to the dashboard and this is going to update after a short period of time and it's going to show us that new log that is coming into the system and you can see that here I think that's just updated now we have the new log that's been generated at the current time and again we can configure these charts and change them as we want and we have multiple chart types here on the right hand side so we can build complex dashboards and we can have line charts bar charts pie charts Scatter Plots and heat maps and so on let's show One Last Thing Before we finish this video we have this bar chart here that is aggregating the logs that are coming in what we can also do is flip the SQL here and do some log filtering to customize the chart even further now if I only wanted to look at the events in the system where the level is equal to error we can add that log filter and run the query again and it's going to generate the charts and you can see the value has went from two to one so before we were getting two logs at this time stamp one was the error log but the other one was the info log and that info log is what you see in the views.py file here where we write testing the logger out to the system so every time we go to that page we get an info message and also the error message because the user does not exist if we go back to the better stack dashboard here you can see that value is down from 2 to 1 and if I remove the filter here and rerun the query you can see there's now two logs being generated when we remove that filter so there's a lot you can do with these dashboards to build a picture of the events that are happening and being generated by your system but that's all for this video this has been a quick dive into logging in jangle and in Python we've demonstrated a number of Concepts here including the main objects in Python's logging module and also how to configure the logging dictionary setup in a jangle web application we've also seen how to send our logs to the better stack log aggregation service where we can then collect those logs and analyze and search through them in order to to get better insights into what's going on and what might be going wrong in our systems so thank you for watching

Channel: Better Stack

Views: 4,346

Keywords:

Id: XSwIUnGXrwY

Length: 42min 32sec (2552 seconds)

Published: Mon Nov 20 2023