Structured logging, Serilog, Seq, Rust, and .NET 8 with Nicholas Blumhardt

hey everyone John Bristow here with another  episode of inside devops if I could speak properly   in this episode we're very fortunate to be joined  by Nicholas boomhart who's the CEO founder of seek   data lust rather gotta get my terms right you  may know Nick he's he's heavily involved in in   the.net space he's built a whole bunch of things  there he's he's responsible for uh Siri log he's   responsible for auto fact and a whole bunch of  other things uh within the.net libraries and   Frameworks Etc uh Nick thanks for taking the time  to chat with me I really appreciate it hey that's   awesome it's nice to be here John yeah we've  crossed paths before uh many times and uh you know   in a in a day a long long ago in the galaxy far  far away we were both at Microsoft and I guess my   first question is what can I blame you for in the  in the bcl's that uh the The Base Class libraries   of the dotted framework because you were you were  a member of that team is that correct I was yeah   I actually spent two students at um uh Microsoft  working on the the BCL team way back it's amazing   how how long.net has been a thing for no I know  right that's going back quite some years um yeah   originally I I joined Microsoft to work on managed  extensibility framework off the back of autofack   which is you know ioc componenty stuff and um yeah  I spotted a blog post by christophelina in the   BCL team yeah you know if you're we're going to  build a something like this uh you know if you're   interested in this space reach out and uh yeah and  I sent I sent him an email I think way back then   um and then and then and then what did you what  did you touch because I want to know what what   you touched there so I can put on my blame on  you so somehow yeah so somehow Memphis was a   really interesting project because it pulled  together all these people who'd come from   um like uh it was in Visual Studio particularly  like a lot of people from this kind of component   these component engineering backgrounds building  IDs and things and smooshed them together with   all of us who'd been out in these Scrappy open  source projects building these experimental ioc   containers so I mean you can't really blame me  for meth but it's probably uh one of the things   I spent the most time on yeah there's there's  a few of you smattered around I've seen at   conferences I'm like wait a minute wait a minute  we gotta talk after cutting my teeth on that   um and seeing how hard it was for us back then  to actually get shipped software out the door   and then how Frozen it was because we shipped  things in the.net framework and once they'd   gone out that was it the apis were immovable  and unbreakable for all those maternity um so   I I had been using nuget like when it was like  before it was kind of Taken under the Microsoft   umbrella um and I and a bunch of the other folks  on on that team were really eager to start pushing   not I mean asp.net has been shipping on nuget for  a while but the BCL itself at that point wasn't   um wasn't really this uh broken down kind of chunk  of uh These Broken Out chunks of packages yeah or   shipping independently um so I I actually  you could kind of blame me for some of the   um uh I PM the first BCL packages that were  released on nuget oh there we go one of them   was the uh the second version of meth but  the other one was really interesting um uh   it was TPL data flow which is uh something that  I wasn't involved in the development of but the   the owner of that one had um had just picked up  another role internally and was was going off to   um you know with this completed project ready  to ship in the Box behind them and I saw this   opportunity to uh uh to you know keep one more  thing out of the box and make one more thing   a kind of uh um fast moving you get package  components so at the at the you know 11th Hour   TPL also uh you know not TPL the um the base  library but the dataflow library on top yeah   yeah yeah I've just mentioned this yeah if some  folks but like I love that I love it I love TPL   when it came out I was like this is awesome yeah  but the folks involved with data flow I think were   um were hugely distressed that I didn't  want to ship this in the in the box of the   framework and that we're going to put on this  nuget thing but hopefully hopefully history yeah that was a that was a really cool  interesting experience um I think this   is a really cool library but yeah um awesome so  anyways I mean like like as I mentioned earlier   I mean you're better known for for the open source  libraries that you've been working on historically   like Siri log for example Auto fact folks you  don't know did you did you want to give the the   sort of I mean I I can't imagine too many people  watching this haven't heard of these libraries but   um no zero log actually hit my radar a long time  ago and I was like this is awesome structured   logging fantastic here we go so so yeah uh so  Auto fact was the first of those but yeah it's   probably I mean I feel like a lot of that is  water under the bridge in a way and that like   I think Auto facts obviously really widely  used today um yeah but we've kind of been   through digesting and internalizing what all  of this ioc dependency injection stuff is now   um sorry sorry log I think um you know um I mean  autovac introduced me to how interesting it is to   collaborate with people online on open source I  mean it was one of the first things that I really   worked in that mode on um stereolog is just about  to turn 10 uh in just a just another month's time   um it's definitely the one that I feel like it's  um it's both you know seeing this really wide   adoption but still not really being fully digested  um yeah it's really surprising still uh I think   one of the one of the original aims of sarilogue  was that it's a logging library and it looks like   every other logging Library you know you write  events using a log dot something interface but   behind the scenes it's got this kind of tactic to  smuggle these structured uh you know structured   properties and things along with your log events  to um you know it's to make diagnosis uh you know   Diagnostics easier at the other end um I mean a  lot of I think a lot of us a lot of people out   there still really kind of consume these sorts of  libraries without seeing um just how incredible it   can be to have a rich event stream kind of flowing  out of every application basically for free um so   I think I think sterilogue still has so that was  a long way to go um oh really but what more things no no yeah no no no no no not a long way to  go in implementation but I think um you know   the ideas that necessary log have um uh one one  of the interesting people I met while I was at   um Microsoft in was um involved in the design of  Microsoft extensions logging which the framework   obviously now uses and lots of people can  consume as well um and and Lou I believe and   Daniel playsted a couple of other people who are  involved early in that project uh were also aware   of how stereolog was representing events at the  time and um able to make it compatible so that   you know Microsoft extensions logging also has  this structured structured data um capability   that stereolog has but I think the the challenge  is it still looks exactly like I'm just writing   log events into Strings like console writeline and  so I mean the the thing about structured logging   is that it's it's actually a completely different  process to um you know telling that narrative   story by producing text logs it's um it really  is like every individual event it might have a   way to be formatted out so you can read it in a  human friendly kind of way um but everything about   structured logging is logging is all about like  keeping those events rich with correlation info   so that you can you know you can easily slice that  huge stream of data in different ways to to find   the you know the thread that kind of leads to your  you know the cause of your bug or your outage or   um yeah yeah you know those flame charts won't  rate themselves you know what I mean so those   who know no but uh anyways yeah yeah I didn't  want to spend too long on on your history but   like I mean it's it's still interesting  to talk about because of the impact right   well the funny thing I've seen in the last  probably in the history of that project is   like 10 years ago a lot of this devops kind of  territory was almost a bit subversive right in   that yeah like like no one ever came to a project  with budget to automate deployments or testing or   builds or to to build runtime information like  instrumentation into into an app it was always   something that developers had to kind of like  uh you know squeeze in on the you know the the   crumbs of the of the time left um that's  right yeah which is which is interesting   and now I'd like only only 10 years later  it's kind of like the the imperative to make   you know everything automated and repeatable and  everything instrumented and observable is coming   from like the sea level so uh it's it's kind  of cool but I also think like as as developers   we really should be careful not to lose our  kind of claim over that territory like sure   um I mean structured logging Hotel traces  metrics the single pane of glass things they're   all interesting and important um right you know  outputs of this observability observability trend   but like the techniques we've always used to make  apps observable are just so much broader than that   um like one of my favorite examples is like if  you're using postgres and you want to know you   know what the database is doing when it runs a  query you don't you don't go and look at its logs   really and you don't go and look at like a fancy  dashboard with a you know a whole description of   that process you look at like um a query plan  that you get from something like explain and   then your SQL statement or explain analyze and  you get your SQL statement with timings and I   think like that's just that's just one out of a  million different techniques that developers have   come up with in the last probably 50 years to make  apps observable and and the focus on things like   traces and logs metrics right now I think I'm  I'm not trying to get too cynical but it's   just that you know that they're common enough  that they're easy to commoditize and a lot of   products have kind of um you know focused on  that common theme that is applicable across   a wide range of scenarios and apps but but  really I think as developers we still need   and and Ops people as well we still need to be  kind of creatively thinking about how to build   you know observability of internal State into our  apps and systems um you know I think it's I think   it's uh it's cool to see the the process going  I mean you I'm sure you've I'm shooting exactly   the same thing in in devops like it's amazing  that there are these common tools that we can   just take off the shelf now to do so much  but a lot of what makes devops so amazing   and successful is really the practices and how  developers you know uh engage with what their   they're doing yep yeah I like to say that devops  is is just is like people say people in process   before tools but I I like to say it's people  process and tools I think and is important there   um I think that provides a really good segue  to talk a little bit about data lust and seek   um yeah you know so I wanted to get to this one  um but yeah you know you guys you guys have been   into space for a long time yeah yeah yeah well  I mean come on I mean I'm a technical guy what   do you I'm gonna gravitate to the prettiest  pictures come on so um but you guys you guys   have been in the observability space for for a  while now and and with seek which is a product   that we use quite quite a lot and I absolutely  love because it gives me a just a huge leg up   in terms of understanding what's going on uh under  the covers the way that I like to think of seek is   um you know like if you think about logs  on an application by application basis   um they're pretty easy to grok when you have a  single application but it's but that's not how   applications and services are built today like  today it like your application resembles almost   like an orchestra and each application and service  is represented by a different instrument and yes   you can you can realize like an orchestra can play  a beautiful Symphony but that's not how logging is   like logging logging if anything it's like it's  like when the orchestra is noodling you know when   they're warming up and you can't understand a  single note or all that that's what logging's   like to me and so that's it and you need to turn  that noisy instrument down at the mixing desk you   know I was yeah 100 so so basically the way that  I think of seek which is you know a reference   the site here so the way that I like to think of  seek is that like I'm a huge fan like I said and   the power of seek for me is when you have this  boatload of log data coming in from a variety   of different sources like an orchestra everyone  playing it once you got lots of noise lots of   activity and then you simply walk up you write  a query and then suddenly you can start you can   instantly You Hear The Melody of the music being  played in your app and so tell us a little bit   about seek yeah it's uh that's that's a really  good um description of what we try what we try   to achieve with it it's um I like I think one  of the one of the things that I like we've kept   about C I mean I started building C because they  um as uh you know scratching and it's kind of uh   project back when I was a consultant because I  felt like there was there was just a there was a   gap between kind of like log files and then on the  one hand and then something that I had to have you   know a Consulting team in my office for a month to  implement on the other hand like I I really wanted   something and I still want something that's very  like that feels a lot like a log file it gives me   the gives me my data you know open up seek and  I just see uh everything that's pouring in yeah   um pretty much in in real time and then I can pick  out something in that stream that's interesting   because I've either filtered down a bit or or  just spotted something you know probably of   someone's giving me a kind of a bug report they  mentioned an exception message I can search and   find that exception but then from there uh see  you can see in the screenshot there actually with   um all the properties attached to that expanded  event you know yep you can yep anyone you can   pick any one oh I know I use this all the time  I I I do this like I love this so much like I   know everything you describing because like I  do that and I do that and I do that like I use   the signals all the time I'm like filter out  everything else and I write queries and I like   the little mini charts you put at the bottom like  I love all of it like it's fantastic yeah sorry I   didn't mean that yeah I I asked you the question  no I'm like no no this is awesome yeah yeah no I   think it's just being able to I mean just being  able to pick any any kind of aspect of an event   and then go and follow that and find all of the  other events that that have that same property   you know like a request ID or productivity I  think that the other thing that that makes sick   quite distinctive in this way is that it doesn't  really prioritize any of those things in that like   it doesn't it's not got a special schema for web  requests or for SQL queries or anything built in   it's like whatever you attach to your log events  that is the first class data that seek works with   um and so instead of instead of kind of  having an off-the-shelf interface that   has all of that kind of nice you know  request timings dashboarding type thing   um and and then you have to dig through  to find your data as kind of the other way   it's like you create the developer interface  that you want and then you start to kind of   you know surface things in the way that suit your  app because yeah I think I think if you're um   you know if you're building a I don't know I  don't want to say throw away up no one builds   throwaway apps like actually every time the the  surest way I'm sorry yeah paraphrasing XKCD I   think the surest way to to keep an app alive  forever is to you know build it as a throwaway   yeah yeah but I mean there's lots of things that  are common to all apps you might want to sort of   uh just use instrumentation that comes out of  the box in the framework say and get some idea   of what requests are being handled and everything  else but I think for things that you're investing   in over a long time like particularly big big  systems with microservices and lots of components   it's really um it it's really worth your  while thinking about your own goals for   instrumentation and Diagnostics like if your  whole world revolves around you know uh Pisa   you know then uh pizza delivery say then you  know the whole uh I know that that whole domain   like being able to trace like um orders that were  cooked in a particular oven you know and adding   an oven ID is uh is kind of like I mean that's a  that's a draft example but you know fine you're   I know exactly where we're going because it's not  only like you you guys you guys make it easy for   us to be kind of like lazy in a sense when when  logging right because you you you parse it all   you allow us to query it just like in this example  here where basically you can construct queries the   result sets come back like insanely fast and to  your point I mean you can log as you can almost   log as much as you want like it's it's bananas  how much you can log in here like within reason   obviously but yeah eventually eventually you pay  the piper I guess that's right yeah yeah but it's   fantastic because to your point like sometimes  when you use these logging libraries like you get   kind of you get kind of antsy because after like  they're they're addictive right you're like I'll   just log everything input was this output is that  and like you like you start playing log levels and   all that stuff you're like wait wait wait but  then you use a tool like seek and it's like no   no it's okay it's like you can still do a lot of  logging and you don't have to worry so much about   you know having too many logs because a tool like  seek will help you kind of grock what's in there   yeah yeah definitely I think uh I think the  nice thing that you've got as well is if you   that you've got an insiders if you start building  in that kind of instrumentation early early on in   the process like as you're Building Systems  you can kind of course correct like initially   you just you speculatively log whatever you  think might be interesting and then as you   as you go through a few outages and you fix  a few bugs you start to get a feel for what   actually pays for itself and what doesn't and  you can kind of tune that data collection and   um you know what what you keep  and what you what you filter out   um I don't mean interrupt but that was something I  wanted to ask you like when you talk to customers   about logging do you like do you have to  sell them on the idea of like listen you   really should think about this way up front like  don't add this as an afterthought you know it's   important as part of the software development  life cycle observability is part of the you know   an important part of a devops strategy whatever  you want to call it you know yada yada you know   where I'm going with this but basically like I'm  imagining you're you're having to not only sell   the product but also sell the idea of like no log  like it's important right yeah I think um I I feel   like it's probably there's probably also a bit  of self-selection that happens there with uh with   customers in their way because I think you I  think if you don't think about these things up   front you quickly find yourself in the position  where all you can really do is just try to   um I don't know I guess it is a garbage in kind  of garbage out thing but you can you know you   just have to create a huge search index over a um  over a whole pile of kind of questionable stuff   and then I don't know I think you I think yeah if  you don't build it in from the start you kind of   miss that navigation aspect of it and you tend to  end up with just like I know there's something out   there with this keyword and that I and I'm going  to be able to track down an event or the events   that have that but that you can lose the overall  kind of um uh the the flow of your like it's just   a it's just a disjointed pile of data if you're  not um you know if you're not building those   things in I guess correlation is one of the things  that makes that um uh imperative if you've got   lots of systems like you really want to be able  to move from uh one system to another by following   you know you can obviously follow things like  request IDs or hierarchical Trace IDs from from   one service into another but like say say you're  a you know you're instrumenting uh deployment   automation system and you want to actually be  able to follow like a a release well you've got   to really make sure that or or an individual  deployment you've got to make sure that all   the nodes that are cooperating in that deployment  use the same deployment ID when they're you know   recording log events for jobs they're doing as  part of that deployment that could be that could   be important yep yeah definitely definitely  if you um yeah if you don't build it yeah sorry I I didn't interrupt so I was just going to  say like so when you've got all that data in there   now you've got the ability to visualize it and one  of the other things that I like is obviously the   ability to make pretty pictures I'm a big fan  of the uh the charting capabilities of seek so   this is this is a very profitable Cafe we're  looking at uh that's right this is that yeah   that's a lot full test data generator um yeah yeah  and also the alerts capability is something that   I absolutely love as well we actually use this as  ourselves so um alerts are really really important   for notifying you hey something well I won't use  the phrase like this doesn't happen with us but   if something were to catch on fire per se you know  or there was a scenario a set of a set of uh logs   that are coming in that you're like this should  never happen uh and then they start happening   the thing about the thing that really strikes me  about seek is that it gives you true observability   and I find that nothing is better not there's no  more magical feeling as a developer than observe   bring your apps running in the wild and what I  mean by that is you know how people talk about   observability and it's like yeah yeah you know  you're logging in on your Dev environment and   that's not real world but when you start logging  back in a in a live environment you start seeing   things coming in and then things catch on fire  and then it gets progressively worse and worse   you're like wait wait stop you know so I think  yeah yeah so I love this like it's fantastic   yeah I think well the other thing about I  think log events as a basis for observability   um you know not to go too too deep into it it's  just that like if you're if you're instrumenting   uh if you're instrumenting a program you  you pretty much you pretty pretty quickly   find out that the only thing you can really  attach to a running program meaningfully   um is event oriented right like you uh you  don't have this kind of a you can't really   bung some kind of state into a live running  application to kind of collect info you just   have to you have to surface information  about your your um app through events   um is the least kind of intrusive and disruptive  way um and the cool thing about events is that   there's sort of a layer below or a you  know a level below a lot of other kinds   of uh data you can collect at runtime like  you can use events to build up metrics say   um you know for example you can count the number  of orders to kind of get an order and you know an   order count metric without ever adding an  old account metric explicitly to your app   and it doesn't really work the other way around  you know if your app is instrumented just with   metrics you you can't go back and and get all of  the detail about each individual order out of that   like events are kind of the most fundamental thing  and it's similar with traces too like you you can   Surface a start event and you can Surface a stop  events and you can correlate things between them   um but you can't you know it's you can't really  go the other way you can't you know you can like   a trace is a kind of a a higher level construct  than yeah so you know you definitely need to kick   things off with a correlation ID otherwise it's  like oh yeah and it's like the way I like the way   I think of observability tools is like you know  there's that phrase like if a tree falls in a   forest does it make a sound I'm like observability  tools will not only tell you that the tree fell it   did make a sound and here's the tree that fell and  here is the other and this is where it fell and   this is where these other trees are about to fall  you know sort of things so yeah yeah I love them   yep yeah I think I think the cool thing about  them yeah the the neat thing about like logs   uh specifically say is though once the tree has  fallen you can see all the all of the individual   you know uh squirrels and chipmunks that were  crushed that's right do you have that really   that really fine granddaughter and you know if  you've got if you haven't got logs you really um   uh you're really left trying to kind of uh second  guess what the art effects of these kinds of   events were you know yes so so when coming back to  seek I know like seeks got a great web interface   and it's great for all those reasons the thing  that I've been playing around with um quite a bit   actually and I've really kind of gravitated to is  of course your CLI which is another way you have   apis as well and you know but um you know uh we  we recently had octopus we recently redid our CLI   uh we have one that's based in C sharp we recently  did we recently redid it in go uh wow that was an   interesting experience yeah we had some fun doing  that so we learned some new things that was good   um but you guys you guys have a CLI too and um you  know this this is obviously like giving you kind   of I think I think I don't know I'm gonna just  kind of go a little bit off the uh the the stand   on my soapbox here but I think yeah I think that  there's there's an I think we're in a Renaissance   of CLI and and terminal uis I think that for far  Joy like this is my presidential address far too   long we have ignored we have ignored the terminal  far too long my brothers and sisters comrades we   have we have ignored the CLI and we are having a  a return to the CLI so we've got the spectrum of   apis on one end on the other end you've got your  web UI which is typically web because let's face   it it's the internet and then somewhere in the  middle it's kind of like lost like there's this   no like space that no one kind of exists but like  that's where the CLI really shines I find and so   I've been watching like I've been watching the  work that um folks like the the charm folks have   been doing they've been building out a terminal  UI layer which is fantastic um we used a framework   called Cobra for for go um there's all kinds of  libraries that are out there for like Python and   for uh for c-sharp has a has a set of libraries  for doing this you guys have your own CLI as well   which is fantastic like is this the primary way  in which like once once you get folks past the   web you are they're like yeah yeah okay great and  then they become Advanced users like does everyone   gravitate to the CLI yeah that's interesting  yeah I mean uh the CIA the CLI at the moment   I mean actually we we actually have a CLI under  our CLI we are CLI is like several layers Turtles   all the way down we've got the 60li and then we've  got the the CLI of seek itself and then underneath   that the database that seek runs on which is also  a custom event store has its own CLI and we live   in there so we certainly do [Music] um uh at the  moment the CLI for us is a big integration point   for you know automatically I mean uh provisioning  you know someone someone will set a a process up   that when they deploy a new application they you  they trigger the of course the CLI and that goes   and creates a new API key for that application  to log with right um you know those kinds of   scenarios really strong but I'm yeah I'm with  you on the Renaissance thing like I've I've   actually been spending a lot of time recently  playing around with a new shell called New Shell   um which which kind of promises to be a nice  integration uh point for all of these different   like new cli's that that speak Jason or speak  other structured uh formats I think if you're   if you're from the windows world I mean Powershell  that will sound like Powershell and there'll be a   familiar case a notion but um but yeah New Shell  maybe has a little bit more of a um I guess a   traditional shell feel makes it really easy to  go and take output from things like um uh no   that's uh you're almost different shell yeah okay  sorry I mean like there's so many shells out there   they're like yeah oh my Posh and like there's like  terminal.dev and like there's like who doesn't   have a shell these days yeah I know it's actually  an end you for for new show oh my apologies yeah   and I'm sure I'll have to all right this will  be fun let's see uh let's see if we can find   it on the Fly here so new is it this or this or no  yeah that's right ell okay let's search all GitHub   let's see oh there we go sorry yeah oh you're  yeah that's because it's built in Rust that's   right we're gonna talk about this don't worry  but yeah there you go okay you're you're playing   you're showing me your hand well before we're  gonna talk about it so all right so tell me about   this new Shelf well I mean I'd probably could  I couldn't do it justice but in New Shell yes   uh Michelle I think is is somewhat like I guess  partial's concept of uh you know between commands   piping objects data carries its structure yeah  but unlike yeah yeah and like I think a lot of   the ones we've seen before this uh new shells  based on tables rather than on objects and it   just happens to fit really well with a lot  of our operating system data um probably   it's it's you know it's um it's probably a  bit deep to try and dig into all of the um this makes total sense to me I hadn't I hadn't  heard about this so I'm gonna I'm gonna have to   research this thank you very much now yeah I'm  gonna waste my time on this no I think I think   once this terminal Renaissance is fully uh you  know really realized we're gonna have to glue   all of these cli's together with something that  speaks the you know speaks the language that uh   I know I know everyone's gonna it's crazy like I  I mean I didn't mean to talk about this with you   but it's like it's I'm glad that I brought this  up because I'm not the I I I feel like I'm the   only one talking about this sometimes but I see it  everywhere like everyone's using console command   line Terminals and clis are the new hotness like  GitHub has its CLI one password has it CLI people   are building these cli's in a variety of languages  it's crazy like what's what you can do there   it's funny it's funny because we're also running a  lot more code now in the like in the especially in   the microsoft.net windows world I guess that I've  that I've come from historically like we're we're   running a lot more code and environments where  there is no GUI you know uh where like I think   10 years ago we all deployed uh machines that you  could remote desktop into and uh click around them   with a mouse and now I find you know I've got  a tiny little prompt on a graviton machine or   something like that on AWS and like if it's not  got a CLI there's nothing I can yeah I know it's   like it's like dead to me I don't want to speak  yeah yeah yeah um so yeah so getting back to a   little bit around seek so I'm going to challenge  you a little bit here so you guys dropped a recent   update uh last week so we're recording here on uh  February 23rd where roughly so about two weeks ago   as you guys dropped 20 23.1 and you have this very  um I guess ambitious bold arrogant I would say   dare I say a ready to rock your world so um we'll  talk a little bit about about seek proper like in   terms of the tech stack but I'm curious to know  what how how is my world about to be rocked yeah   well uh well I I hope it will be rocked by your  own your own log data that's now within you know   um yeah I know the the interesting thing about  seek and the way it's evolved uh as a tech   stack I guess is that it was it was built  really with this user experience golf first   um and I mean that was just the night that  the naivety I guess of the the founding uh   Team where instead of um no I mean it was a good  thing in a sense but instead of starting with like   we've got this amazing data store it's column  based we can you you know we can issue this   particular kind of query really easily so can  we build a model for Diagnostics around that   you know we or or you know we've got this amazing  text index like listening was back when elastic   was kind of um first imagined I think like  uh seeks kind of the other way around where   the goals initially were always almost all  experience driven um and then you know the   the back end of it was just a servant to that  like uh what kind of what kind of storage   technology is going to support this kind  of interaction model and um so in the very   beginning it was um it was sort of a looking like  it was very hard to find things off the shelf that   that provided anything like the kind of log file  like Time Warner stage to talk about Flair now   yeah well no uh see yeah that's it seek was  initially kind of uh uh hoping like I I hope that   it could be built without building essentially a  database and storage engine yeah you had you had   to do this you had to write your own cryption  algorithm didn't you no thankfully no nothing   that crazy yeah but no about um about five years  ago I think we realized that um no one else was   going to do it for us there's just not much  investment happening in in like the uh I mean   there's Kafka which I think is um probably well  known as being something that's highly like event   uh Centric but most of the other tools that for  storing data at that low level all rely on some   kind of schema and all make it very difficult  to navigate in a Time based manner seek is all   about like incremental time orders search so if  something is recent you'll find it before you   found before something that's old and that's just  how we that's how we want to work with log data   um yeah so we started building I guess the lowest  level of that the storage engine and rust uh   about 2018 uh and that was our first foray into  rust which was uh uh yeah I was drinking from I   was drinking from the fire hose it was amazing  experience but it was like more like a trigger   from the water slide oh my God like rust was  rust has been CR I can't tell you it's already   interrupt but like I can't tell you how much I've  been reading about rust just going bananas like   everyone it's there's this term I read on Twitter  it's called the Chad Tech stack I don't know what   that means yeah but but rust it like falls falls  in that camp of like this is awesome like this   is the best thing like I mean what's been your  experience with Russ like personally like have you   found it I know that like strong like it's hugely  strongly typed really like really great for uh you   know uh perf uh very small module sizes um really  really secure I mean like I've heard countless   um anecdotes about it um what's your take on it  yeah so uh so initially yeah like everybody I had   kind of uh un unbridled enthusiasm um and I think  I've probably developed a more reasonable view   okay over time like uh my colleague Ashley  actually just gave a really great talk at   um at one of the local conferences uh here in  in Brisbane sort of pointing out just how absurd   some of the complexity that you do need to uh  confront when you're working with rust can be   um but I think I I think if you have the  kind of problem that rustic sells for there   there are very few uh like accessible  Alternatives so when you when you're   so we're building um essentially a database  right and we deal with tons of concurrency and   uh a whole lot of data moving um you know the  latest iteration that's in the in the recent   um 23.1 release you know we've got a a parallel  query engine that uses data flow and so we've got   you know readers pulling data out of buffers  and channels Distributing that to workers   and stuff being aggregated and uh and like all  this has to happen like in a in a database uh   with a kind of a reasonable ceiling on resource  usage and a predictability that I think once if   you've got a garbage collector involved it can be  very hard to to achieve um yeah uh or at least or   at least your def you're out of The Sweet Spot of  garbage collected languages once you're into that   right World um you know so so rust has been  extremely um robust for us it constantly catches   our concurrency but bugs before they are bugs uh  memory bugs before their bugs you know it just   um it won't let you write buggy code in a lot of  different ways um I don't think we could possibly   have have uh shipped what we've been working  on and shipping without it uh really we we get   up to a certain layer where you know uh where  say at the at the top top of things you know   Seekers like a web application that's serving web  requests and it's um you know uh it's dealing with   open ID connect and it's dealing with you know  uh encryption you know different kinds of uh   plugins all those kinds of things and there like  we use C sharp and we're currently using.net seven   and it just it just it excels because you can  just concentrate on the details of the the problem   rather than the details of how data is moving  like like in those lower layers of the stack   how the data is moving around is the problem you  know but once I'm handling a web request I just   want my web request body deserialized into a nice  Json structure and I don't want to think about   anything particularly much except for the you know  the business logic of maybe you know adding a new   user or whatever that you know the web endpoint  is handling and I think that's where I send   sharpened and I guess I would imagine go and the  garbage collected languages Java they all really   um I think are likely to you know remain  better choices than rust at that at that layer   um but you're in good company I mean like this  is the this is your what you're stating here is   that you know you're you're pretty much aligned to  all the popular languages like this is the state   of the octaverse uh and uh yeah so you're seeing  like the fastest growing second fastest growing   language in uh GitHub is Ross still rust oh yes  in the top top one is a configuration language   so it's probably yeah yep that's uh hashicorp  configuration language so there you go this   one always this one always just is bananas to me  like I'm just like HCL really it's fastest one but   I guess a lot of people are doing infrastructure  as code which is awesome but um if you read this   report or you read uh SAC overflow survey Etc rust  is always ranked as like the most loved and the   most wanted to be used etc etc so I mean you guys  are you guys have definitely taken a good choice   and you said 2018 is when you decide to move so I  mean you saw the writing on the wall there I guess   yeah from from that point we've kind of kept on  moving rust up the stack until we've reached a   point now we're kind of we're we've reached a kind  of a balance between the C sharp and Dot net side   and the rust side that works really nicely for us  so which one's going to win which one's going to   win for you guys are you guys are you going to  are you guys gonna put a mandate down and say   okay it's all rust or all C sharp no no I think  I think we'll be hybrid C sharp and rust you know   for those for the level future yeah it's a I'd  love to see more of the.net world Embrace rust   and and probably more of the rust World supporting  you know those higher level consuming languages   like like c-sharp and the the.net stack because I  think you know it's it's uh is is pretty unlikely   for any one application that just one of those  languages is going to be the the best choice to   me but yeah I don't know interesting um I I wanted  to ask you if you if you're keeping track like you   mentioned you're on dot you just recently went  up the.net7 had you heard about like dot net a   preview one just dropped have you heard about this  is there anything that you is there anything that   you guys need like I like the look of the yeah I  like the look of the freezable um container types   that are in there actually there's um uh there's a  couple of new types like a frozen dictionary and a   frozen yes okay where you can you know you build  a dictionary or a set and then you freeze it and   it optimizes itself internally for lookups from  that point onwards right yeah because that pattern   seems to sharpen a lot of our code you know where  you don't really need all of the the kind of uh   both sides of insertion and you know and look up  that you get with something like a dictionary all   the time like lots of types tend to just get you  know um a kind of a build a building stage and   then a use stage and I think I think we're gonna  see heaps of use out of those types so yeah really   okay this is the Frozen Frozen dictionary you were  mentioning earlier yeah yeah I think that I think   they're super exciting um I am interested too in  uh doing more native aot but I've I've been uh   this is this is uh maybe a bit cynical too but  like I've I've been kind of burnt by ahead of   time compilation sort of features that I really  um I want the confidence that the code I write   that compiles is going to run as much as possible  I think rust doesn't support me for that actually   like if if rust code builds it generally runs like  uh and again is the problem it'd be nice if if.net   went further and further in that direction I'm so  with you because this is part of the reason one   of the reasons why we went to go for our new CLI  because it was based in.net before was because the   installation instructions we had built in C sharp  and we had the comp we tried to compile it down as   as much as we could and you still had to have all  these libraries run along for the ride if you're   running on Linux for example and it was just we  just wanted something that was fast small and like   Run Anywhere it could run anywhere and I think I  think what you're speaking to is what one of the   values of rust I think to a certain extent go is  also it's kind of interesting you got the you got   the crabs versus the uh the Gophers fighting it  out for popularity so um I would argue that the go   hits the sweet spot for me in terms of it's right  in the middle it threads the needle there are you   just fast enough Etc I mean the library's a little  bit bloated compared to rust I mean rust can't   beat their the size of the binaries are insanely  small it's incredible but um yeah you know asigo   as being like very much in the same kind of um um  Tiaras C sharp though two and that it's like uh   if if you're worried about sort of controlling the  you know exactly when memory mem copies and things   happen like Go's not really serving that sort of  needs I think if you've got that if you've got   the opportunity to to choose go over rust maybe  because you're not dealing with those kinds of   concerns so much like I think I can I can imagine  go being pretty comfortable and ergonomic that way   too but um um yeah and I guess the I I'm not I'm  I'm not keeping track of how good the ecosystem   is around Russ with with its libraries and such  but I would imagine you're you're feeling pretty   confident in what's available there for you for  for your team to use is that is that a fair or   something yeah well yeah rust has amazing  um great ecosystem so very much like nuget   um I you know what it's just a great time to be  a developer though because so many ecosystems now   are effectively complete where you know uh it's  not that you have to you have to run Java or net   or you might not be able to interrupt with some  kind of you know enterprisey standard it's like   actually now if you choose JavaScript or rust or  go or python or net or Java you know you can you   can be pretty sure that you'll be able to interact  with most operating system features and most   you know services out there it's really it's  really cool but I'm glad that we didn't know and   that was just some you know homogeneous everybody  in the world runs JavaScript kind of uh yeah yeah   I do like competition is good I think that's what  we're saying so yeah it's kind of interesting GPC   wipes them all away the other one I wanted to  ask you about and I know we're kind of jumping   around here but um this is uh this is one that  I've been watching for a while you mentioned a   little bit about this where you've got you know  um you've got metrics logs and traces Etc this   is definitely in your camp but open Telemetry is  one of those things it's a cncf project it's it's   got a lot of broad reach Etc um I don't know it's  certainly under the auspices of cncf I think it   is yeah you've got a lot of vendors involved Etc  what are your thoughts on this I mean I think it's   I think it's an awesome initiative um especially  for products like seek right like we're building   um open Telemetry log ingester that'll mean that  you know any framework or ecosystem that's got   open Telemetry support we should be able to get  high quality structured log data in that way   um yep I like I'm cautious of really putting too  much hope uh in what in a new standard reducing   the total number of Standards uh like I do note  that there's a lot of uh there's a lot of kind of   careful accommodation in open Telemetry uh for all  of the existing Protocols of all of the existing   vendors who are involved in the efforts I think  that I could be nice if eventually we reach a   point where everything speaks a single protocol  um but I I think that you know at the moment um   it's it's pretty unlikely that we'll switch away  from accepting like for example gelf is another   standard for receiving structured logs that's been  standard basically standardized and available in   um a number of ecosystems to date and uh I think  one of the problems that you that you also have   with really broad um interop efforts is on the  surface it looks like there's a lot of common   ground but behind the scenes like I mean we we see  we have to ingest logs from all kinds of different   um applications or using similar  protocols like using Gulf for example   um and it's like actually actually getting really  consistent uh useful data that looks the same   out of all of them is still I don't know I  think we should we should probably not get   too distracted by the transport and I think  the the next step after after a lot of this   stuff is widely available is going to be back to  focusing on the quality of the data that we put in   um like like having your logs or your traces or  metrics or whatever else is eventually added to   open Telemetry consume consumable anywhere  over over that kind of protocol is awesome   but if you're not writing useful log events or  if you're not controlling the volume of the data   that you're logging you know uh is still going to  be a much less useful tool than than if you are   you know I guess it's like unfortunately on or  not as as developers we can't kind of hand off   responsibility for producing quality data in our  applications like like you can you can get sort   of uh tracing of web requests say out of the box  but if your domain is like deploying software or   delivering pizza or whatever else you really  need to put the effort in to think about you   know how you're going to instrument that domain  in a way that serves your goals rather than just   producing a kind of um you know a tick box on  the the list of you know uh technologies that   you're going to have supported in there um  yep yep yep I don't know but then again I'm   I'm probably uh I'm probably a bit jaded you know  I'm reminded of uh soap and how we were gonna have   interoperability between Services based on and I  see the the Ws star you know extensions and those   kinds of things yeah like every time I see a uh  you know a big standardization effort like that in the end there's no substitute for just good  systems engineering at all at both ends of the   pipe right like the right you can standardize all  of the interop and the protocols and transports   to your heart's content but if the data that's  flying across them you know is is not kind of   aligned or yeah yeah you know I I would I would  agree up to a point I would say that the Ws star   or what people call the Ws Death Star um the  specification uh moment was the thing that they   didn't do well which I think the open Telemetry  folks are doing is they didn't make it practical   for developers they published a ton of specs and  no implementations yeah it was like what do I do   with this thing this is useless and so people were  kind of left in their own devices whereas the open   Telemetry folks they're writing custom uh sources  and syncs and all that sort of stuff and they're   allowing you to get your hands dirty basically if  you're doing C sharp or go or rust or whatever so   there's a slight difference there definitely we  have learned we have learned a few things in the   intervention that's right yeah you got to get you  gotta give me the best you got to give me the bits   I can't do anything if you don't give me bits so  yeah yeah totally I just I guess I just mean that   um like at the end of the day you know um it's  all down to practice is and Tool you know not   just uh not just tools hmm yep yeah people process  and tools yeah I think you nailed it on the head   um just wrapping up here obviously if you guys  want to you know check out what uh Nick and his   his band of Merry developers are doing you can  go check out Dana lust at datalost.co is there   a reason why you didn't get the.com come  on you know what what is this yeah we do   it we do have the.com but we just saved that  extra character every time you type it yep   I like it uh you're also on uh you're also on  Twitter there so you can follow along if you're   interested in hearing Nick's thoughts and uh  anything else you wanted to kind of wrap up on   what's next what's next for what's next for seek  what what can customers expect what are you going   to look at next what is there anything you want  to share anything you want to divulge uh well you   know agile agile is never done yeah okay so I mean  more polish no yeah and uh and also you know we've   um uh we've just built kind of like another yet  another kind of uh I foundation for a whole stack   of stuff that we've had in the pipeline for a  long time so it's going to be a bit easier for   us but yeah that's awesome yeah right um but I  definitely yeah I think just um I don't think I   think we should totally expect a lot more activity  and interest still in this instrumentation world   over the next coming years like I think um I  think we'll start to once we've I guess once we've   um internalized that it's kind of part of the part  and parcel of doing software engineering much like   much like we have with them cicd and automated  testing like yep I think I think then we'll we'll   really start to see some of the diversity of of  tools and techniques coming through so I think   it's uh you know there's an exciting 10 years  ahead of like things that we might not have been   positioned now um and how we how we're going to  instruments and and manage apps so I don't know   I think it'll be uh we'll be a lot we'll be there  and we'll be along for that ride I hate I hate   this you I'm ending this like Robin Williams at  the end of Goodwill Hunting he's like you stole   my line and so anyways I think that's a great  place for us to wrap it up thanks a lot Nick I   really appreciate chatting with you and uh I can't  wait to see what you guys are going to do next

Channel: Octopus Deploy

Views: 1,184

Keywords:

Id: AHw15plsuN0

Length: 52min 41sec (3161 seconds)

Published: Tue Mar 07 2023