Logback with JSON output and Custom fields

all right let's talk about log pack logger uh with the Json output and custom data so the setup I have as of now is a very simple API which is going to serve on slash hello endpoint and this is going to return the hello string and in the dependencies I already have the log back classic and the configuration for log back is very straightforward we are using console appender and that is it so it's going to Output the logs in the console and let's start the application and see the current setup for that I'm going to open the terminal because we'll utilize the curl command so our server has started and if I hit the slash hello endpoint I receive the response and I receive the logs in the console mentioning request received that's what we are logging here so let's turn this string based logging into the Json one so for that first thing we will do is we'll go to the dependencies and I'm gonna copy two dependencies and paste it here so these are the two dependencies and I'm gonna comment out the logback classic log back Json version I'm using it's mentioned in the griddle.properties file it's zero one five and now I'm gonna import the grid I will build file next thing is to configure the log back itself I've got the configuration for that already prepared so I'm gonna paste it here so let's go over it um so I'm using two variables here that's optional thing it's not related to what we are doing in this video so what I'm doing is basically is reading the log level from the environment variables if it is present then set it to that value otherwise by default it will behave with the info level and for the locker so by default we are going to write the logs to a file inside slash VAR slash log and it this variable is being utilized somewhere here so we'll create this app logs.log under slash VAR slash log by default unless this is present in the environment variable and for the appender we are using rolling file opener with the file name this and other settings like a rollover pattern and all that stuff so for the layout we are going to utilize the Json layout provided by the logback itself and we are not going to use the pretty print because that will increase the file size a lot shouldn't be done in the production and will have a new line after each log all right and that is it so root level either the info or from the environment variable and the file is the appender we are using all right and let's get back to our application controller and let's restart the application now so now you can see here I don't have any app log file here but if I hit the curl command again IC attempted to append to run launch Charter okay we have some error and this guy says don't have permission to write this okay so I'm using WSL and it doesn't have permission to write to this location so I'm gonna utilize the locker environment variable so we'll go to the Run configuration and I'm going to supply the logar as equal to dot I thought I had it but let's rerun the application so we are basically saying log back to create the file in this current directory itself okay so let me check okay still still oh okay we changed the Run configuration but I utilize the same configuration again let me stop and rerun and apply the new configuration and this time we do not see any error [Music] and if I hit the curl command again and reload oh we already have the file here okay and here are the logs let me bring soft wrap and minimize this project navigation so these are the two logs produced and you can see we already getting the Json output and by default we get timestamp level of logging and the thread which is logging and the logger name and the message which we are which is the actual log okay and the logger name is coming from whatever you set in here all right that's pretty cool right you're getting all the required information timestamp and everything but what if I want to add more details for example uh my hello API is deployed as a multi-region API in AWS cloud and I want to know from where the request is coming which machine is serving all that stuff okay so how do we add basically more data to our app logs for example [Music] um um I'm pushing all these logs to say elasticsearch and I want to see which application has put this log into my cabana right I want to filter on the basis of the application name so I want to include the application name into this Json which is not happening as of now so how do we do that so if you remember we are using Json layout so we are going to extend this particular class and create a custom Json layout out of it so let's do it uh we are going to create a new file okay so I've already got that code with me I'm gonna copy the class name so that I can create it here okay and I'm gonna copy the code I'm going to paste it here so what we are doing basically we are um extending the Json layout and we are overriding its add custom data to Json map method and in in it we are just adding three more properties to it basically the application name which is coming from here that this is the hello API and the region where the app is deployed and the Machine name which machine is has served these requests for which the logs are coming into the kibana so these two environment variables will be basically configured in the container settings so we'll pick it from them in the actual deployment but as of now I don't have these in configured in my IDE we can do that but as of now they will just spit out not available okay so let's try the this but before that what we should do is we have got the custom lockback Json layout and I'm gonna copy the reference and we will update the logback configuration here that use this particular Json layout we'll restart the app and this time in the app logs we should see the new layout Json layout working so let's hit our API one more time and you see we are getting the application name the region and the Machine name pretty cool so until now we can add custom data but what if we want to add a request specific data for example let me get back to the hollow controller and say after receiving the request we did some work here a lot of work and then we said Okay um foreign response but we cannot just add request ID to each and all the log we log statements we are gonna have in our application right so but we really want to attach particular log statement with the particular request ID for for the convenience sake what we can do is and what we should do is we should put this into the MDC context like this put and we can mention that this is the request ID okay and we can put the request ID here now what will happen is if I rerun and do one more thing just to be sure that every time you are done with the request do clear out the context because you do not want to have this request ID for something else so do clear this out now say see here we are not explicitly logging the request ID but this should come in our logs so let's try that I'm gonna hit shift F10 to restart my application and we'll switch the tab to the app logs so it says okay shutting down let's hit the curl again [Music] so this time if you can see uh the request ID is not part of my message which we actually logged but this is coming as part of the MDC key which is again a level 2 object with the data we have stored in the context you can store I mean different stuff here for example the IP of the requester and more information related to this particular request so that you can actually see in the production are in app logs that this particular message was related to this particular request ID right and this is this is how it is useful and you need not to there can be like two three or tens of uh data stored in your context and this is coming by default with each of your logs you don't have to put explicitly in every file so that's one thing right and what next so you can say that okay um this is cool but I want this request ID to appear in one level above which is the default I mean I I don't want this as a separate object okay for that what we will do is we'll update our um custom Json layout and if if I press Ctrl over the methods that we can override you can see here let me type MDC okay we can set whether to include MDC in our logging output or not so we can totally avoid that but how do we include that in our map so basically this map so let's take the data out of it first what we are going to do is we'll say event Dot get MDC property map this will give you everything stored in the MDC context as of now and I'm gonna put it into sorry I'm gonna put it into the current map so we basically we are shifting everything stored in the MDC out to we are basically copying not actually shifting now we are copying the properties from MDC into our main map so now if I rerun this it will print the MDC two times one at the level zero and one at the MDC key itself so let me shift back to the app logs and hit it again one more time 11 line so you can see here request ID is coming under the MDC and as a level zero key as well so we successfully copied the data from MDC out into our root level of the Json but we now want to avoid MDC getting included in our main map so for that uh we'll introduce a Constructor here and we'll say okay this dot set include mdc2 false now if I rerun this I think we have reached our final State this is what we want and if I rerun the curl command again um okay there was a delay but now you can see that we do have the MDC fields at the root level and the custom data from the app level details everything is perfect and you can put anything you want so there are only two rules the data you want to to appear in all of the logs irrespective of the API endpoint uh the data which is at the application Level you put it into the this map directly and the data which is tied to a particular request or a particular method a particular endpoint you do it you do store that data into the MDC so that is it so we saw in this video how to configure a Json output format for log back and then we saw how to put custom data and rearrange the data out of the MDC context in at the level zero of the output Json I hope you find this useful and till next time take care foreign

Channel: theRDnotes

Views: 2,863

Keywords: Logback JSON, Logback Custom Fields, Java Logback

Id: usKrJZ7LcfE

Length: 15min 39sec (939 seconds)

Published: Sun Jul 31 2022