Rubyfuza 2020: Tina Heiligers - Keeping performance in check- real time!

[Music] [Applause] [Music] well if I fail I'm going to feel great because that is actually you know I can't speak talked before before mine is rather innocence of my journey through life because yeah it just ended up that way so I don't know if anyone read my my profile or had a look at me you know look me up on LinkedIn but as you can tell I've got a South African accent some of the words I speak may sound American others may sound British others may sound like complete garbage it's um it's also you know it leads to my title here you know what what am I doing with a PhD in physics and working as a software engineering elastic it was and never the goal in my life my goal also started off as by the time I'm 30 I want a PhD in physics well I got there then what well then we ended up moving emigrating the first time so my journey through life from being a physicist to where I currently am and is continuing is reminiscent of my bookshelf okay without the the the shelves that have got the mace and the cats and the dogs on them I didn't cut those out it's at the top left here we have two physical copies of my PhD which I got at in MMU University and professor Yellen and I started off my working career working in an analytics laboratory for element 6 to be as industrial Dimond had a ball there but then of course the bottom shelf shall we move yeah why not you know then mark and I ended up in the UK and I was not allowed to work for you because I was getting residency and I don't have a passport so what do I do you know I'm away from family I'm away from friends first time ever and all I had was a laptop now I'm one of those people that can't be bored because then my brain just takes over I become that brain in the jar and unfortunately the joy is attached to my body I figured well let me just fall into my default state I've been doing it all my life let me study but now what do I study and realize look I've been working with with data all my life in science I may as well learn some of the more modern tricks of the trade instead of using Excel I'm sure everybody's had a run in with Excel and I picked up you know one of the the obvious languages which was are down here supposed to be really good and for the first time ever I typed some code into a terminal I thought I was this Rockstar but unfortunately not before too long I don't know who's had experience with our before our keeps everything in memory all right so the bigger data sets you work with more complicated algorithms you're working with yeah it Charles the memory and then literally I ran out of RAM what do i do do I give up no that's not me I moved on to the next best thing you know there's always this this warmth between our versus Python so I thought okay I've tried or I know what the my limitations are there so let's try Python why not so then we move on to some of the Python books and by that stage which we'd actually landed in the u.s. so the US was our our final destination yeah right and I started looking into where can I work because I you know but not before too long I'd be able to get a work permit then the next bomb shot in life hit me that all those years of studying and this wonderful PhD that I've got are useless because I need one more piece of paper and one more piece of paper I couldn't get in five years that's US citizenship all of the research in Arizona in my previous field is import-export and Department of Defense so it doesn't matter if you've got qualifications coming out of your ears you need to be a citizen now I'm not gonna wait around for five years because by then the the whole community is way advanced what can I do I can carry on with software engineering and I started volunteering for a data analytics company as a foot in the door but at the same time I wanted to build something that I could share my life and my experiences with with my family you know social media is is okay but my sister doesn't want to post pictures of a three-month-old baby on Facebook and then have the whole world look at them right and luckily enough what happened is in emacs in inbox the specialization online specialization came across and the capstone of that was exactly a tourism side yeah you could put photographs up take blog posts and share stories of your life the only catch was wasn't ready surf like okay he's a reboost he'll help me nudge nudge and I started that and boy did I fall in love with that all right this was a really cool you could you could really make things happen you didn't have to start with these huge data sets you could provide the data and I found that I was enjoying that a lot more than the data analytic stuff that I was doing but I also realized it's taking so long you know teaching yourself after a year and a half and I've gotten to the point where I was still scared it's a new language but I wanted to become serious about this and I did what most people then do so you apply to a boot camp right it's supposed to be a fast-track into a job I applied to a whole bunch of boot camps got into them all and then had to decide which one do I go to so I took the list to to someone who's been in the field for a very very long time and he came back with an answer that surprised me in being a rubyist all these talks and online talk about you know how bad JavaScript really is and he said good with a JavaScript boot camp so I thought about that for a while but the reasoning was there and I bit the bullet now keep in mind I did the entrance exam for this JavaScript boot camp in Ruby so I knew very very little of that but I went anyway and packed my bags bags moved to New York for three months it was grueling has anybody been to a boot camp or know anyone who's been to boot camp they were right at the back okay there are grueling it's not for the faint-hearted but you persevere it's three months versus ten years right and two weeks before the end of the boot camp I got a phone call from someone in Arizona would I like a job but you know how lucky can you be you being offered a job to come and work for someone they need you of course I said yes that's when I also realize that what we do and what we do it on is very very different to the system that we're actually working with and inevitably when that happens in production we have a spinning beach ball or a spinner or a timer that never times out and inevitably it works on my machine what do we do right we end up opening these if we're lucky enough to have all of them now we've got two problems the original problem and the one of many windows we're gathering all this information all this data but no single system can tell us what the problem is we as human beings have to use our brains to put the pieces of the puzzle together and figure out what went wrong hopefully we're all smart we've got an interconnected network of smart people or we just lucky but we figure it out after how many eight hours of problem solving and pizza or salad or cigarettes or coffee or beer we figure it out but in 2017 I thought to myself there's got to be a better way if we're as human beings have a brain that can sufficiently piece those pieces of the puzzle together and find the problem why can't we have a solution that will help us and of course there is a solution that's why I'm here today right in 2017 elastic launched their APM solution how many of you know about the elastic APM solution one two three okay well it is reasonably new it's um as the market goes but it's more than just an application performance monitoring system right what the solution gives you is it gives you all the logs all that data that feeds into those mini windows which is problem number two it gives you metrics which is also data and it gives you performance monitoring together and the magic is it does it in one click of a button how does this how can that be well the elastic APM solution uses the versatility of the elks tech elasticsearch log station Cabana where it uses elasticsearch a single data store so you've got all your information in one place that helps and it uses Cabana as a window for you a single window to look at to help put all this data together the other nice thing about it is you don't have to pay a fortune right it's for free if you want to try it out you can use it what does the solution give us right so in a nutshell these are just some of the highlights I think are really awesome it gives us response time for requests you don't have to look at timestamps and then figure out how many minutes how many seconds between the this call and the next one it monitors out the Box unhandled errors and exceptions it gives you a visual graphic of the call hierarchy which call it the next one can help you identify code bottlenecks and in some situations you can drill down to the code level right there in the Cabana dashboard you don't have to go and put ten commands in the command line or debug you can figure it out right there and of course it's distributed across the stack that your instruments the primary thing that I do want to say is APM is made by developers for developers every single person that works on the APM solution right now that I know has been in that original situation so they feel the pain we know the pain and we'll help you solve it yes you can hold it on prim it's a free license you can have a cloud version and of course it's got an integrated search bar which is what elastic search is known for supported languages I wouldn't be here today if Ruby wasn't supported I think I'd be kicked out at least yeah there are there are a host of other languages and frameworks that are supported I know I spoke with Stu so he's interested in in the Django framework and Java go and a bunch of others that are in the making but today I'm going to show you with enough talking for me an actual demo so this is the bit where I need the Internet so please bear with me I hope it goes alright and I'm going to hopefully not stumble too hard and switch microphones hello fantastic first I'm going to take some water I'm incredibly nervous in the quiet look at the crowd is so quiet come on yeah I don't want to miss this up thing the first thing I'm gonna do is actually hop arts I don't want us to be staring at the paint dry or the walls dry or the wind blow outside while I download distributable so I'm going to cheat and go onto the cloud what am I doing wrong now you can see that okay great so the elastic cloud you can get to easily cloud elastic car you can set up a free trial all right no questions asked 14 days you won't get an email in your inbox until day at least day 13 and I'm going to start up a deployment let that spin up for a little while takes about 3 to 5 minutes to start up and then I'll jump into the applications that I'm going to give you the live demo of instrumenting with with APM so when you hit the dashboard for the first time you probably won't have deployments unless you've already got some but really just follow the steps create the deployments I'm going to give the deployments a name maybe Fiza it's call this one I practiced a few times as you can tell and I'm gonna go with TCP we've got support Amazon TCP and azure and then a region this trips me up I'm so used to going to us but we're in Africa now so Europe I suppose and I'm going to stick with the defaults that are given so you can choose different stock versions if you've been paying around on a 7 2 or a 6 8 you can choose those visions if you want to I'm out the box going to select the defaults the optimize of deployment hi I oh sure why not that's what you recommend and at the bottom here you'll see with a current configuration this is an estimated cost if it wasn't a trial so as you play with your deployments later on and you change the configuration you don't have to hit submit and then wait for bull and then you'll get an actual an estimate of the cost and I'm just going to create that again you can do all of this on on your own machine if you want to but I'm going to do that and I'm going to copy the password since we are told it won't be shown again please take heed of that warning it will not be shown again but you can reset your password and I'm going to do something and save it in a super safe place which I do not recommend at home and while that spins up you'll see an estimated time of about three minutes let's have a look at the application that we're going to instrument so that is just my we've got a rails back-end it's a super simple API and it's powered by a react front-end so we shall start those up this is nothing to be blown away by please don't expect the moon on the earth it is really a simple demonstrative little application which is up beans coffee distribution company I love coffee and I clearly have not had enough yet this morning what it really does is it's got a dashboard we've got a list of selling products and some stats now this is dummy data of course one in a real life situation would hope you make a bit more revenue than zero dollars and that your profit is a bit more than zero dollars but let's just go with it and we've got a list of products different coffees which ones are sold out there's obviously quite popular and if we click on a popular coffee we'll see Wow alright it's popular besides its selling price this must really be really be good let's see who's bought this we can maybe spam them like guests becoming we're getting more coffee and we see a list of custom customers down there if we want to try and find some contact details we have the email and all of those are of course you can find them from than that bar and that's it okay simple back-end simple front-end it's now trying instrument what would happen if we were coming across some errors and failures so let's take a look at our deployments still busy spinning up okay getting there we can see thus far we have elasticsearch has been installed and configured for us we have Cabana set up and we have a p.m. the cabana one is a migration index pattern and this is to help us move data over from all the stack versions so you don't have to worry about it it's doing what it needs to do to help you start up started a minute ago and there we done all right it was probably like slightly longer than three minutes was anyone timing that I wasn't and let's have a look at the deployment so here we have the entire deployment we've got elasticsearch we've got Cabana and we've got APM if you want to look at the instances that you're running the different nodes that you're running you've got a master node right or master eligible you've got high i/o which is gonna store all your data then you've got cabana and APM that's what we asked for that's what we got Marvis what we gonna do now well we want to know how to instrument APM you could go read that up in the docs but what is really useful is if you actually just launched IBM directly in the cloud it'll give you the instructions step by step instructions of what to do I'm going to zoom out a little bit so that you can see there are a gazillion instructions no they're not we have another spinning beach ball cabana is a monster she needs to start up so bear with her it is all JavaScript busy compiling but she does log in this is where we paste in our password that we saved in a super safe place hopefully that still on my clipboard you know I will not save this all these tools that help us and just actually get in the way all right at least now we know we we have a live situation we're detecting that we don't have any we don't have any agents configured so let's go and have a look it's going to look the set of instructions while we do that okay well that was quick enough now here you'll see a list of all the agents that are supported please choose the agent for your language and your framework otherwise you might get some errors or confused or yeah so start on the node we're going to instrument our server first so let's go to Ruby and rails and here we see the whole long list of instructions of which there are three of them plus we have some really cool copy snippets okay let's follow these directly and sure alright so step one is install the Jim let's follow step one but let us first actually bring down service as of next week you won't need to bring down your your application all right I'll get to that in a moment but as of right now we do still need to bring those done I'm going to go into Visual Studio code because I am a JavaScript developer and head off to the rails at add that in and this is the one thing I do love about rails apps just go bundle and it's so quick I don't know if anybody's compared NPM with bundle it's like NPM you walk off you go and grab a cup of coffee maybe made some lunch yarn is slightly better you can fit a snack in but bundle don't don't even like blink so we followed step one right what's next so it's configure the agent and we need to create a file config elastic APN yellow and copy and paste that snippet in so I'm not gonna try and remember hard to type that because I'll probably make a typo just copy the snippet and go and create the config elastic APM yeah more file so where is this yellow stick APM yeah more you can tell I'm not familiar anymore with the work structure of around where we shall paste that in all right now we need some code to run so there are few things we do you need to uncomment and the first one is the service name but it's already prefilled in for us the secret token that's there already you don't have to remember it and I'll serve a URL of course you can type these in for yourself but why all right that was step two all right let's move on what's step three load cabaña objects okay I feel like the Cabana objects confirm overwrite and now we can launch a p.m. but before we do that I'm going to firstly tell rails not to swallow my errors because I'm running this in dev mode and to do that I shall go to the environment dev and change that to false all right nothing's gonna break OPM will still work but you're not gonna figure out where any errors that one exceptions okay because what rails will show you that yourself the next thing I'm going to do is go ahead and instrument the react app quickly because I want to actually show you you know from the front-end if there is the full stack trace from there and to do that I shall go into once the smoky runs away from me there's no rum agent now there's one extra step here and that first step is please do not do not skip the step is read the docs there are so many different frameworks for your front-end you know you can't give a list of instructions for all of them you might if you're running react you might be running react 15 you might be running react 16 the structure is different I am running react and there are only a few little knobs that you've got to turn and so I have not skipped that reading the documentation is just I've already done that and step two is very similar to the rails application install the agent right I'll do that and because I've read the docs I know there's one more library I need to install and that's to run an instrument the routers in react let us go to the react framework and install a PM run and at elastic a p.m. run react thank you I was watching if anybody was actually no that was just me being silly [Music] are we dancing it's only two packages hopefully this won't take too long so we don't have to go for lunch yet and as any good developer would do I'm going to ignore all those warnings because it's just development mode and it's just a dummy app and all right we have now done step two let's look at step three Oh No we also have to configure this agent now you'll notice here the instructions are not telling you exactly what file to create and where to create it because that depends on your framework if you're working in angular if you're working and react the two of us here different beasts if you're working in angular I feel sorry for you no I still have to work in some angular so I I feel your pain but we shall copy that snippet and go into the right code base and within the source folder create a new file and I'm going to call this run dot J yes and paste that in there right now you're not given a service name you don't have to give a service name but I kind of want to know what I'm looking at so I'm going to give it a service name run again the server URL is given to you you don't have to remember that long string and the service version so if you want sourcemap features give it a service version otherwise there won't be any and you can see it's it's just a string so you know sure service version one what are the next instructions the cabana objects but wait we installed another library do you want to instrument the react routes and I need to tell the JavaScript code that I added a new file those aren't or automatically you know detected so let's go into the code and do that then in the index don't you love the way JavaScript just has an index in every bloody folder and you're looking for this index and that index and you got to find the wrong the other magic is that you have to import your APM right at the top of your index J is file alright APM needs to know about the rest of the stack in there done now theoretically it should know about it I'm going to not be pedantic and address the semicolons that's a bit silly then in the app which is where my and my rats are configured I need to import the APM rats so import IBM route from that's elastic and then go ahead and change a few of these it's not to the not found all right now we hold our breath no that is it apparently the instructions are done so let's start up a server start a front-end and see where we are okay nothing too critical their start in p.m. and hopefully we are still Oh what look you know are we still working so the agents didn't break anything okay I still have a list of products we still don't have any Jamaican Blue Mountain Vienna roast and all our orders are still there we have all our cast customers already right so things seem to be working we can go back to the dashboard and magic let's go and see what happened in the cloud keep honest thoughts that begin there we have everything so we did in the cloud was startup deployments followed the instructions and the service names came through so we have a service which is our Ruby agent we have our rum agent and that's jeaious base please know it's like J's base it's J's something are we not gonna try and infer what it was built by and just a tabular just a source small summary of the in of the data that we've seen thus far average response time transactions per minute errors per minute bear in mind all right it's just me little Kippy standing here hitting my own code so we're not going to get a lot of information out fortunately I've got a few scripts which I'm gonna run now some that are hitting the the back end directly and others that are taking over the browser so when we go into that it's just run these these are gonna run yeah 5,000 is fine it's so that's just cycling through a list of the api's that we've exposed there and a few mock array of the eyes that are thrown in just see what happens there then flourish is written thankfully by a certain somebody in this crowd that was helping me out so we don't all have to get this at the same time that is a selenium driven let's go okay that's really just gonna cycle through the pages and find all the links and click them as if we simulating thousands of customers hitting this this application and now we can go and have a look at our date one thing that might trip you up when you first hit a p.m. is you thinking like I've been gathering all this information I'm trying to send it to you why can't I see it the time picker by default is said to the last 24 hours so if you don't change that we've been working here for you know literally two minutes change that to 15 minutes if you don't if you do want to gather information on the fly set your refresh interval 10 seconds is fine and we can hit start now let's go and have a look at what what actually information we've got here let's have a look at our back in first right the top graph that you see on this dashboard now this is a Cabana dashboard it's specifically built for APM you can go and configure them yourself but this is what you get out the box and here we start seeing how our data coming in and it's split by yeah controller post grace which is our database the view clearly nothing because it's run by react okay we've got a react front-end then we've got some pretty graphs of transaction duration and requests per minute can hover over them to get more information out of those and a lovely table at the bottom which is your waterfall hierarchy of the transactions that are taking place the statistics here you can dive into you can also configure if you want to but let's just have a look at what's going on here have a look at the coffee controller then you get some top-level statistics of the time spent by each span in a in a percentage so you don't have to go and calculate that yourself you have your duration distribution and your trace sample data at the bottom so you've got an HTTP call goes to your coffee controller and goes templates what might be slightly more interesting is if we now go and have a look at the full stack trace between our front end and our back end yeah no we won't get anything here because we don't have any statistics on it and quite honestly I think the team just forgot to like collapse that by default but again we have information coming in all right so we have our transactions you can see that the graph slowly checking in that's all our information coming in and we've got the the different transactions got the impact is also showing us there so let's have a look at the impact the route change well it's just me in South Africa so we can't sorry that's a different map I really am was much better on Tuesday night it's like never mind yes here's a fun statistic right this is what we all wanted to see right so we're going from our front end which is the products selling us then you're sure we we attain the API it's a get request that is going to our products controller on our server that's forwarding the great request to Postgres we've got a summary here select from products and then select from product kinds this is an example of a full stack trace with zero custom instrumentation it's all out the box if we wanted to have a look at one of these HTTP requests you click on it get out a whole bunch of request information that you can do your own little bit of data fishing from and we told ok it's not found not modified and what version of Chrome we're running but perhaps we instruments it to have a look at critical errors so let's go and do that let's have a look at some errors on the front end and we see right ok we actually finding an uncaught type error cannot read property load settings this is J's it's a static bundle but at least we get a hint of where in the bundle this code went wrong if you wants to look at something metadata that's there too you'll see the error ID that you can go and then trace down and some information about the the agents now there are there's one more example I'll show you if we have a look on our server side and we have a look at what some of the errors are that we are catching we'll see we have a few unhandled with Bridget and handle rejection errors and these are fiercely problematic in our code so let's go and have a look into one of them and this is where you get examples of actual code level 2 all done now not clearly this is instrument this is you know it's mock but if you use out of the box instrumentation without custom configuration this is the type of stuff you'll get and I've ran out of water that is basically what you get I mean the more information you have the more time you can spend in here you can follow the links to go and look at your raw documents and discover it's all Cabana everything is an index patent so you can do what you would expect to do in cabana and add your custom dashboards you can go and create canvas presentations that you can print out to sent to your boss right there from your infrastructure you don't have to then go and play with a different different application so I guess we can move back into the talk then so that was the demo thank you all for not using too much internet that actually started up quite quickly and what did we do I mean it was really three simple steps for both the back end and the front end you install the agents you configure the agents and you load your Cabana objects the documentation then gives you hints and insights on how to add your own custom configuration but what I just showed you is what you get out-of-the-box that's a QuickStart guide there are a few catches that I found right that you will find in the documentation but you know that I found setting this up so the order of your configuration is important if you set an environment variable doesn't matter what you do in your yo-yo or your config or your default it's always gonna listen to your environment variables so if you're figuring if you if you're coming across the solution like why is it just not finding this thing check your environment variables that tripped me up the other thing was you know Tyrells not to swallow your errors you actually want to capture those and the APM server URL that you're given if you're not using the copy paste option make sure that you've got the port there the client-side if you happen to be using react you need to install the correct libraries doesn't matter what framework you're working on just go and have a look at the talk it should be a quick start guide and you need to import that code so what I showed you here was the real basics of of the APM solution which is the real user monitoring and the application monitoring of course you can go further using the arc stack right you can add server level monitoring you can add logging it's all integrated in Cabana with one click it'll tell you exactly what to do to add those it that instrumentation and we can do that because it's all in an in index all your data is stored within a single data store and customized dashboards and of course we all want to know you know I don't want to go monitoring this 24/7 I want a slack message or an email or ping my partner or colleague who's on call and you can add that with alerting using watcher what's coming next week well the first one as I said you know you can set API keys in all Feathers teams really excited about that the other two things of - there are noteworthy here are the sequel summarizes being moved to a LexA based approach so the summaries that you see in your waterfall will become even smarter more intelligent and in certain situations faster - then of course there's G RPC support if you want that grape is also supported if you choose to go that way I just don't think we've gotten to updating the documentation yet sorry about that and of course it's all open this is the magic of elastic all our code is there for you to look at and snicker at and ask for future it's all there it does make life as a coder somewhat interesting and challenging but it's fun nothing is hidden from you you can download the the font of the code you Soph that's all folks [Music] [Music]

Channel: Rubyfuza

Views: 39

Keywords:

Id: z8XpttsI8eQ

Length: 46min 47sec (2807 seconds)

Published: Thu Apr 02 2020