(upbeat music) - Hello, hello. I mean I guess, I guess if
we get stuck in Minneapolis that's where we're
supposed to be anyway, so. Right, I mean is that why you were late? You were scouting locations? (laughing) Okay hold on, I gotta
do this real quick okay. This is actually for
expenses, to prove I was here. (audience laughing) (mumbles) We're on? I'm on? Okay, hold on one sec. Hold on. No, come on. (audience laughing) One sec, one sec, cheese okay. Okay all right, okay there we go. All right. (audience laughing) Let's start, let's start. So this is a just in time,
just in time presentation, not to be confused with Justin Time. I'm Aaron. (laughing) So, I am an extremely, extremely
horrible procrastinator and instead of working on my presentation, I kept putting stuff and
like doing actual work and putting stuff in
slack and Eileen suggested that I titled my talk Things I've been Doing
Besides Preparing my Talk because I get a lot done, when I am supposed to
be doing something else. (laughing) Anyway, I get extremely, extremely nervous about giving presentations
and I'm always paranoid that I'm going to go way, way too fast and I won't have enough slides. So the other day, like the
other day DHH made this tweet he said, "I'm just going
to say it I hate Foo, "Bar, and Baz as
prototypical variable names "it's time for a change." And this made me laugh a lot. So, I made a little video
about it and I tweeted about it I'm like, "There's one slide done yes." This is what I'm doing,
working on my presentation and then, and then, and then DHH likes and I'm like, "Yes,
there's two slides done." (audience laughing) I'm getting like four or
five slides out of this. (laughing) So, I think the presentation
is going okay so far. Anyway oh, hello there. My name is Aaron Patterson, I may look different on the
internet than I do in person this is what I look like online, if you don't recognize me, some of you may have noticed maybe a cat picture coming up on your screen, I don't know if you recognize this. But Jason and Kevin I'm not sure why, you should have accepted
my Airdrop, I'm not sure this one too, geez. Gabrielle come on, just trying to send you photos of my cat please. (laughing) Anyway, I work for a very
small startup company called GitHub, I don't
know if you've heard of it. It is the only legit company
I've ever worked for. (laughing) I really enjoy using git, but I will not force push it on you. Now I make this pawn at every conference and my co-workers told me that
I really need to branch out. (audience laughing) But I had to say to them, "Look, look "I'm just really committed
to these git based puns "not even not even flogging
me is going to stop me, "in fact even if you reflog
me that wouldn't stop me." (audience laughing) Anyway, if you don't
like these get based puns we could switch to SVN,
but I just don't know what the git diff would be. (audience laughing) How long do we have left? (audience laughing) All right, so. (laughing) I love, I love titles and at GitHub I am a level 5 engineer. (laughing) I am level 5, and despite the fact that I have played
Xenoblade for over 90 hours I still don't have enough XP to level up to level six engineering. So, I'm going to be
working on this every day and my boss is here in the audience. I'm going to be working
on Xenoblade every day until I can get up to
a level six engineer. (laughing) So, I guess I should go to real content. No, not yet. I love, I love local businesses. I love local businesses
before I visit any place I look up the best local
businesses for that place so that I can go help support them. And I was inclined, to learn more about Pittsburgh. Come on. Come on
(audience laughing) Come on, anyway so you may or may not know this, Heinz is a local, local business
so I bought some ketchup this hashtag local yes. These are really exciting, I wanted to support local
businesses even more so I picked up some Heinz
Texas barbecue sauce that is also hashtag
local this is exciting. Then I went and picked up some Heinz beans and these are actually amazing because there's 57 varieties of
beans you may not know this, it's 57 different varieties. But anyway, I thought this is really neat but unfortunately I looked
at the back of the can and they're actually made in England, so they're hashtag not local, so I apologize for that
I was trying real hard I'm not sure what the deal is there. So anyway, I love local businesses I hope you got the point. Pittsburgh people, yes yes. Come on, I know there's like
five people in the audience that got this one and I'm sure
you're laughing real hard. (laughing) Thank You Evan. (audience laughing) The end of RailsConf where
I waste one of your hours. (laughing) So the day before yesterday, I learned. I've got 99 problems I
guess they're all yours. (audience laughing) Anyway, I love I really,
really love coming to RailsConf and it's really exciting
for me to be here. I like to visit everyone, everyone here I feel like it's kind
of a homecoming for me like I get to see all
the you know old faces and new faces, meet new
people see other folks on the Rails core team,
do a lot of mingling. So, I wanna catch up everybody on what I've been doing
over the past year, so this past year I
actually, I became an uncle which is exciting for me I've
never been an uncle before. This is my niece, this is me touching her face. (audience laughing) Surprised my sister let me do this. (laughing) Since this past year I actually became extremely famous in Japan, this is true and if we have time at the
end of the presentation I will tell you exactly why, but I made this tweet right
here it is one character one character and it garnered 81,000 likes which from a social media perspective is like a really good deal,
like characters per like is like very, very efficient here. This is worth my time, so any of you social media
managers out there look at this one, look at your likes per character I mean this one's like great. Anyway, so this is well
worth your investment. I decided to become a
thought leader again, every year I wanna be a thought leader but it just doesn't work out, but the only role model I have
for thought leadership is DHH so I decided the first
thing that I should do is get a headshot like his,
so I've got this new headshot. (audience laughing) You may notice the resolution is a little bit off on mine, but that's because I've got an older Mac it's still at a 480p camera, so I'm sorry. Not quite there, I'm still working on it but anyway I follow him on Instagram and he has this amazing desk, so I thought "Well, "I wanna have an amazing
desk like this too "I'll set it up like, I can
become a thought leader too "if I have the right desk." So I wanna show you
like I put together mine like this, this is DHH's
office, and here's mine it's exactly the same you
can see, it's perfect. In fact one thing I noticed
when I took this photo there there's actually that photo. (audience laughing) But really, I'm actually really, really good friends with DHH just check out this selfie
that we took together. (audience laughing) In fact, in fact he invited me over to his house the other day, so I went and I took a
photo and this is it. (audience laughing) Okay yeah, it's a little dark isn't it? (laughing) All right, this is just a test slide so I was doing a test slide here I just wanted to see if
this would actually work. I don't know if you can see it, there's something right there in the slide it's right between the S and the W, I'm not sure if you can
see that right there but it is a zero with a
space, that is character that's between those two and we're actually gonna talk about that, we're gonna talk about that in a bit this is just a test slide. Anyway so, this tweet really annoys me. (audience laughing) So, I have like really weird
conversations in my head like I think I have a problem
focusing or something, I don't know I have a weird conversations with people in my head, I think okay we're gonna talk about this, and this is how the
conversation is going to go and I kept replaying it
over in my head and I'm like you know what, I have to I
have to get this down somewhere so I made a video out of
it, this is the video. (laughing) Anyway, so this is what's
going through my head like I'm having this
conversation in my head. Unfortunately like, DHH is
the last one to say something so anyway, I'm thinking about this I'm thinking about this Foo, Bar and Baz is prototypical variable names and then I'm thinking about code and Ruby and I don't know if all of you know this but a valid identifier in Ruby is anything that is not a space basically. So this code works. I'm defining a method
that's named smiley face and I'm calling the smiley
face method there at the bottom so if I run this it'll
work and print out neat. This also works like I've
got a method with smiley face and it takes keyword arguments, and the keyword is a green heart. Defaults to a train and
you can call it in passing keyword arguments and it prints out that little trolley thing there. So this is what I was thinking
about is Foo Bar in baz because they're really just you know representing something I thought, "Okay well, what what else
can we do with this code?" Here's another example. (audience laughing) Can anyone anyone see what's
going on with this code? I'll wait, I'm gonna wait for you to type it out on your laptop you can give it a try, it
actually works, this really works. No okay, all right. So I'll explain it right
here at the very top the name of this method is
actually a non-breaking space so there's a non-breaking
space right there, and down at the bottom there
is another non-breaking space so I'm calling the method
with non-breaking space and if we look at this in
vim you can actually see it. On the right is the source code in vim, you can see the non-breaking
space character there but when we cut the file
and actually run it, it totally works fine. So let's extend the
program just a little bit we're gonna do one extension to it here. In this case we have a method, the name of the method
is a non-breaking space and we have a parameter
given to the method that is a zero width joiner. So right there that's our, name our parameter we're calling puts. So, here it is again in vim
you can see like right there and of course this is Seattle style there's no parentheses. So, you can see in vim we've
got the non-breaking space and the other space, and it totally works fine. So what I'm proposing today is that instead of using Foo and Bar, we replace Foo with zero width joiner, we replace a bar with non-breaking space. So all right, with that let's
let's write some production. Production ready code. (laughing) I dare all of you to put that
in your app code tonight, wait don't do that, don't
do that, don't, don't. Okay, all right so today, today surprise, surprise we're going
to talk about performance. I always give performance talks, we actually just had our
performance reviews at work. (laughing) My boss is in the audience. (laughing) You did a great job of
my performance review. (audience laughing) Thank you, anyway so we're gonna talk about performance today but I don't wanna talk
about the speed of your code I wanna talk about the
speed of your development like your actual development, getting features done and
actually making that website that you need to ship. Now, the thing is I
was thinking about this and we're always getting faster, we're learning new tools,
we're learning new technologies we're learning new techniques. But unfortunately, it
seems like as time goes on your team velocity gets slower and slower. It takes longer to add a new feature, it takes longer to ship your code. So why does this happen? This seems paradoxical,
who is this mythical man? Why are we concerned with his month? (laughing) But seriously why are we slower? Why are we slower to ship
these features over time, and what can we do about it? Well, there's different
ways that we can deal with some sort of situation like this, one of those ways to deal with
this situation is to rewrite. We wanna rewrite our code,
we wanna chase that dragon, that dragon that we felt that high when we wrote Rails new and we
can get all of those features out the door really quickly. Wanna throw away all that code and say, "Okay, I'm gonna begin again." And feel that exhilaration when you can ship new features quickly. But if you're doing this all the time, if you're just rewriting
something all you're doing is going down the same path
over and over again. Maybe the weather has changed
but the places you're going are exactly the same, what good is this velocity that you get if you're just running down the same path over and over again. Now, the day before yesterday I learned that if you don't wanna deal with these hard problems in your program you can just hire someone to fix it, hire someone better than you to fix that scaling issue
in your application. Now unfortunately, I cannot afford that, so I have to pick a different solution I choose to refactor. We could we could choose
to refactor our code, we could make performance improvements and pay off technical debt etc, etc. But the problem is, it seems like we have these two different arcs juxtaposed against each other
as application developers. It seems that we either
have to be software writers or we have to be a PhD in computer science to work on an application. Now in my opinion there has
to be something in between we can't just only be software writers we can't only be PhDs in computer science, we have to go deeper into the application we to go down this cliff so
that we can actually solve these more difficult problems. Now, as we learned day before yesterday, I am the only one who can fix this. (audience laughing) And we've recently learned that, that doesn't actually work out. The truth is that you're the only one that can fix your application code. So the Generic features
that Rails provides gives you a great bootstrap
you can get going early, we give you all the things
that a generic web application would require to ship a
feature but as time goes on your application is getting
more and more unique. You know your business
logic, you know your business I don't know your business,
I can't help you with that. So I think this is one of the reasons that application development
starts to slow down over time as we have to get better
at our particular domain. So those generic features and Rails give us a huge, huge boost and I think this is why I
like to focus on Rails a lot, this is why I speak about
profiling and speed improvements with Rails is because,
I want people to be able to take their Rails application and live with it for a long time, I want them to have a successful business not have to run into these
scaling issues as often. So, I love Rails and I
want Rails to continue for a very long time, so this is why I study
these particular things. So today we're gonna be talking
about profiling and tuning we're gonna specifically look at Runtime and Memory management, which sometimes I like to call time and space because it makes me feel
like I'm on Star Trek. (laughing) It makes me feel a
little bit more important than I actually am. So, let's look at some stuff we're gonna look at types of profilers and how to use them, how
to build them and basically how to improve our application
performance with them. So there are two types of profilers that I typically deal with, one is called an exact profiler and one is called a sampling profiler. And to figure out which profiler you need when you're profiling your application, you just need to ask
yourself a couple questions. The thing that you're trying to solve, if you're at thing is how many? How many of something? How many method calls? How many object allocations? If you need to know an
exact precise number then you wanna use an exact profiler. Now if you don't care
about an exact number like let's say you need to know, what is the slowest method? What percentage of time do
this particular thing take? You wanna use a sampling profiler. So, if your question is how much? Use a sampling profiler. So in order to figure
out which one you need, just ask the question out loud. What is the thing that I'm trying to do? What am I trying to solve? So let's take a look at exact profilers, we're gonna actually
implement an exact profiler so that we can understand
the way that it works. This is an example of an
exact profiler written in Ruby it uses the trace point API. This trace point basically what it does, is it hooks into the call method and every time a method gets called once we enable the trace point, our hook method will actually get called. So we listen for the event
and every time we get an event we just keep track of that
an increment encounter. And you'll see at the
bottom of this code here it outputs the result which says okay benchmark me was called once, fast was called a thousand times and slow was called 10 times. Now if this code wasn't so
simple you might look at this and think, "Oh, the
place I need to optimize "is the fast method, or
the method called fast. "That's the thing I need to
optimize is called the most." But obviously when we look at this code we can tell the slow method is a slow one, so this is a particular case where we wouldn't want
to use an exact profiler, this is misleading us. If our goal is to speed up this code we don't wanna use an exact profiler in this particular example. Now the other profiler
is a sampling profiler and this sampling profiler what it does, is it stops your program at intervals, regular intervals and it asks the program, "What are you thinking about?" (audience laughing) So every so often, it asks
what you're thinking about it just records it. So every so often, we have a
program here that's running so time is along the
bottom our program runs. It stops we record what it's doing and we keep doing this
at regular intervals until we're done sampling it. Now the idea behind this
type of profiler is that let's say you have a slow
method, if your method is slow then the probability
of being in that method any time you stop the program is higher. So we can tell that if we're in this one particular method frequently, if that shows up frequently
within these samples that's probably where our slow code is. So we can write a sampling profiler and Ruby here's an example of it. What this profiler does
is it starts up a thread and it asks the main
thread at an interval, what are you doing? What are you doing? What are you doing? Every so often, and you
can see at the bottom we have our results from that and in this particular example we have almost 6,500 samples
inside the sleep method and maybe one sample
inside the slow method. So we don't know how many times
the sleep method was called but we do know that
when it's being sampled it's very frequently in the sleep method, so 99% of the time we sample we know it's inside the sleep method, so that's the thing
that we need to target. Our fast method didn't even
show up in this profiler. So when do you use exact versus sampling? It depends on what
you're trying to measure, again if you're trying
to get absolute counts using exact profiler, for example the number
of objects in the system or the number of times a method was called or API or whatever. Now, if you're trying
to reduce time or space then you wanna look at
a sampling profiler. If you're not sure, use
a sampling profiler. Sampling profiler is what you need about 99.963% of the time that is with one sample. (laughing) Check my math, it's wrong. (laughing) So let's take a look at some
Runtime Profiling and Tuning, we're gonna talk about tuning your runtime to profiling your runtime
and tuning your runtime. So my favorite tool for
doing runtime profiling is a tool called stackprof, this thing uses MRIs built-in
sampling capabilities so it doesn't spin up its own thread. You can actually use a
timer inside of Ruby, Ruby has a timer inside the C code not inside the Ruby code, that lets you sample every so often. So stackprof just uses
that built-in capabilities and it looks very similar,
the API looks very similar to the one that we wrote earlier. We just wrap up our code inside this block execute it, and when you execute it it saves a profile to a dump file so I've specified profile.dump here and when we're done with that, we can just run stackprof on that and it dumps out all the
sampling information. And this looks similar to the information we got in our profile,
are just much nicer right so we get kind of the same
information, 999 samples. Again, we can't tell how many times the slow method was actually called, we just know that it's in the
slow method very frequently. So, we can use this to
benchmark Rails boot time so let's benchmark the Rails boot process let's talk about the Rails boot process. So let's take a look at how. a breakdown of the Rails boot process. If we look at it like this, where time is moving along
to the to the right there we can divide it into a
few different sections so at the very beginning here, that's where we hit enter,
we hit enter right there. Then we start requiring files and once we've loaded and
required all the files we start what's called compilation
we're compiling the app. It's not really a compilation phase essentially what it's doing, it's taking all of your rack middleware and putting it all together and
then handing that middleware off to whatever your web server is, in this case we're using unicorn. So after compilation occurs we hand off the application to unicorn. Now after that when you
get your first request it has to compile the views and then actually execute
your business logic. So, if we look at this from
a little bit higher level we can say that these steps here these requiring files in app compilation, we can say that that's our time where we're not ready for request yet. We're spending time here
and we're just not ready to respond to a request, and now the first request that comes in it's gonna spend time compiling reviews and then executing the
business logic for that page. And then the second request
is only going to execute the business logic. So if we wanna profile these things now we know what to talk about, where we're actually looking. So knowing this, we can
figure out what we need to benchmark depending on the task. So, what we're going to do now is we're gonna look at
benchmarking a script, just a startup script. This is an example here we have, we just load config environment
this is how you in benchmark that very first slice, before we're actually compiling views. This is going to load all of
your applications information all of your I don't know models and stuff if you're using production mode maybe it's lazily loading in development, but this is how you can benchmark that overhead for your scripts. Now, if we look at the output
from this you'll see okay it looks very similar
to our previous output but I wanna call out one thing in here you can see that we're spending some time in garbage collection right there GC. So we can see what percentage of our time is spent in GC, and we're gonna look at, how to improve that
time a little bit later. So, run time tuning, I'm gonna talk a little bit about tuning your Runtime, now the thing is Ruby will try to execute
your code as fast as it can so there's not really much
Runtime tuning you can do, the best thing that you can
do is to write faster code. (laughing) Obviously, we want the Rails framework to be as fast as possible, but at some point we can't
make the framework any faster so you need to look at your own code to see what's slow about it and the most advice I can
give about that is say like, okay use these tools, find
where your bottleneck is look at it and say well, is there a faster way
to do the same thing? Can I do this thing in a faster way? Do we need to even do this thing so much? Maybe I only need to do it once or twice, or maybe I can defer this
until a later moment, maybe I don't need to do it right now. So these are questions that
you should be asking yourself as you're trying to tune your application. All right, let's take a
look at memory tuning. We talked about tuning Runtime let's take a look at space, actually using memory in your application. But before we get to memory tuning we need to talk a little bit about memory, how memory is handled in Ruby and I hate this so much,
this made me so mad. How did David know that
I am going to talk about Garbage Collection in this presentation? Am I that transparent? Why? (laughing) He said that in his talk, and I'm like, "No, how do you know?" Anyway, so let's talk about
our Garbage Collector. The GC has two responsibilities
in your application, one it frees memory and this is the thing that most people think about,
the garbage collector doing it's freeing up your memory. But the other thing that it does for you is it actually allocates memory. I think people don't
think about this as much, and I think the reason
is because of the name it's called garbage collector, and you think, "Oh, I'm
just taking stuff away." But it's actually in charge
of allocating your memory too. So, what is MRI's GC? I'm gonna tell you what it is, we're not gonna go through
all the algorithms and stuff but I just wanna give
you a high-level overview so that when you have time later you can Google these words. Because that's really
what's most important is knowing the keywords
to Google, seriously. So, Ruby's garbage collector is a mark and sweep garbage collector, it's generational, has incremental marking lazy sweeping and uses a
free list for allocation. I feel really, really
bad for the translator. (laughing) I'm sorry. (laughing) Anyway, so the generational part of this just is a optimization
on top of Mark and Sweep it makes your GC faster. Incremental Marking and Lazy Sweeping mean that we have shorter pause times, so when MRIs Garbage
Collector is concurrent it is not parallel, and what that means is as your program is running
every once in a while has to stop and do something. Now, these incremental
marketing and lazy sweeping mean that those pause times are very short and what that does is it increases the throughput of your program, so you can actually run
stuff through it faster. So, one question I think is interesting is when can the GC collect garbage? Or when does the GC collect garbage? When does that happen? Well, there's three three main places if you just call GC.start
it will collect garbage though it's not guaranteed, and none of these are guaranteed
these are just locations where it could collect
garbage, so I said can. So, if you call GC.start it will do a collection, maybe it's not guaranteed
but it will don't worry. Now, I know there are case
when you allocate an object, if you need to allocate an object the GC may do a collection. Or if you allocate memory via malloc, the GC may do a collection. So, allocating objects
and allocating via malloc are two very different things and we're gonna talk about the differences between those in a minute. So, this leads us to what
is an object in Ruby? In Ruby, an object is a 40 byte chunk so it's a chunk of memory
that's 40 bytes wide. So, let's say we have some
code that looks like this here at the bottom, it's a string that says, "OMG this
is a very long string." Okay, now this string
is obviously too large to fit inside of a 40 byte chunk, so how do we actually store this? How can we have a string that's this long? The way that it actually works is, rather than storing that string inside of the Ruby object itself, we have a string object that
points at something else. So, we have a string
object, a Ruby string object that points at a C string object. Now, the garbage collector is in charge of allocating these Ruby objects and malloc the system malloc, is what's in charge of
allocating these strings. So, we have a Ruby object that points at a C object, so in this particular case we have two places where
our GC could have happened. Now, Ruby doesn't allocate
one object at a time instead it allocates a page. So an object is a 40 byte chunk and when you say give me an object it pulls that object out of a page that contains many objects. So the GC won't actually allocate one, it'll allocate a page
and then it'll give you one of the objects inside of that page. So, in this, particular example
we allocate our Ruby string our string is just a ruby
object inside the page and it points at a different string that was allocated via malloc, and in this case we have two places where the GC may have run. So, a Slot is a location
where a Ruby object may or may not be and this
Slot term is important when it comes to tuning
the Garbage Collector because we allocate in terms of slots, well in terms of pages
and those pages have slots and this is the way that we tune it, is by the number of slots
that we have available. so we can control the
total amount of free space in the program, in the
way that we control it is via this term slots. So let's look at some tuning parameters. GC tuning is done via
environment parameters and we're gonna take a look at some of them and what they do. So this is the first one, first one we'll look at is
called RUBY_GC_HEAP_INIT_SLOTS I don't expect anyone to remember this, I promise I will put these slides online so you can just google it later. But we can see how it
impacts the process here if we run Ruby, you can see
the number of available slots with no tuning is about 1800, we have about 1800 slots available. Now if we say INIT with 600,000 slots and we run it it has about
600,000 slots available. So, with this particular
parameter we can say I want this many objects, or
this many free spaces declared when the script boots, or
when the program boots. So, let's take a look at that profile that we looked at a little bit earlier. This is an example here
we have our app boot, this is the default case that we saw previously, no changes you can see how I ran the script at the bottom of the slide there. Now, it's spent about 16%
of its time in allocation 16.5% of its time in GC. Now if we tuned the slots
and say we want to init with a bunch more slots, we can do this we just say let's set
that environment variable to 600,000, you can see
the command at the bottom and in this case we only
spent about 8.3% of our time or 8.5% of our time in the GC. So, we can see kind of an impact on that when we go to boot our application. In the top one there's no tuning and it took about 4.1 seconds
and in the bottom example it took about 3.5 seconds, and this is a default application,
default Rails application where I just randomly chose a number. And the nice thing about this is that we didn't have to change
any of our application code and we were able to decrease the boot time just by saying hey I want you to prepare a bunch of room for allocation. So, we didn't need to
change our app code at all. You can use this, if
you measure the objects that are allocated in your system you can come up with
a good number for this and then decrease your boot time just by tuning this particular parameter. The other thing that we can
do is we can control growth and these are very long, very long. So, we can control the growth of our heap via these particular parameters, and again don't need to remember these the important is
that we're applying a ratio to the amount of free space
that we have in our application. The GC tries to maintain a
certain amount of free space for you to allocate into, and we can specify those ratios so we have a minimum ratio, a maximum ratio and a goal ratio. So we have our mean and our max and we're trying to hit the goal so kind of a window there. Now, let's take a look at a test program we have a program here,
this is our test program and it just allocates a
whole bunch of objects it allocates 600,000 objects. Now, what I wanna do is, I wanna study how many times the Garbage Collector had to expand its heap, while
we're running this program. So each time we allocate a new object we may have to expand that heap and we wanna look at how many times we actually had to expand the heap. So, if we set our goal
to particular values we can study how many times
I had to expand the heap by checking the GC count. So, if we tweak the goal
numbers a little bit and then graph it, we can
see a different behavior and this is an example of that behavior. Our goal, the blue line our goal is set to .2 and our green line the goal is set to .7 now the interesting thing about this is, we're able to hit that 600,000 mark in fewer GCs with a larger goal. We're essentially telling the GC, "Okay I want you to be aggressive
about allocating memory "then allocate it, and
then expand into that." Since we were aggressive
about allocating memory we're able to hit that goal,
or hit that 600,000 mark sooner So unfortunately this means that we have more memory growth, we're not being conservative
about our memory growth. The blue line we're more
conservative about memory growth but we have to spend more time in GCs or using more CPU time. So it really depends on what
your application is doing and what you're trying to tune for. If you wanna tune for fewer GCS, you may want to increase your goal. If you want to tune for lower memory usage you may want to decrease your goal, and you can actually
try this at home again this slide is literally
just for being online later so don't like write this
down or anything please, like you wrote down the example code with a non-breaking space in it. (laughing) All right, so let's take a
look at some memory profiling I saved this for last because I actually have a story
to go along with it. At work, somebody pinged me at work and we have an application at work it's called the GitHub enterprise, and basically what it is
it's a version of GitHub that we ship out to clients and they host it in their data centers. It's just I guess stored
in a container or something I don't really know all the details. But, one of the people
on the team came to me and said, "Aaron I've noticed that "as time goes on the
application that we're shipping "is consuming more and more memory, "it's getting larger and larger. "Can you help with this? "Do you know why this is?" So I said okay well I'll take a look at it and see if I can figure
out what the problem is and the first tool that I
reached for was stackprof because is my favorite tool to use. And this is an example of using stackprof with object allocations, we can use a sampling profiler
with object allocations. So, I ran this code I
required our environment and took a look at the output from that and this is what the output looked like it told me that we were spending, this isn't actually our application this is a sample app. But it said oh we're
making a lot of allocations inside of the require method. Now, the problem with this
type of profiling is that even though we made a lot of allocations in the require method, those objects aren't necessarily retained, they may not live for
the life of the program. So, if they don't live for
the life of the program we don't really care, it's not contributing to
our bottom line so to speak. So in this case, we actually needed to switch to exact profiling. This is a case where
we said I need to know how big is the heap? I need to count exactly
the number of objects that are in memory right now. So this is a case where you
want to use exact profiling. And to do that I use object space this is my favorite tool for
doing exact memory profiling and the way that I use
this is via Heap Dumps, so there's a way to get a heap dump and I will describe it a bit here. This dumps your heap to a JSON file and in this case I'm
just dumping it to a file named my dump JSON. This line up here, the
second line what it does is it enables allocation location
recording that's fun to say. Any objects that are
allocated before this line we don't know where they've been allocated the Garbage Collector does not keep track of where they are allocated. After this line when we call this method it tells the GC, "Hey every time an object "is allocated I want you to
record where it got allocated." So anything that's
allocated after that line we can find out where it was allocated and though that information will actually be inside the heap dump. So, after I enabled that I said okay let's require the application booted up after we required the
allocation I say okay I wanna clear out they do it, run a GC I want you to clear out any
objects that are temporary that aren't going to live
for the life of the program, then dump out the heap. So, once the heap is dumped out we just have this JSON file,
and each line and the JSON file represents one object,
one line is one object. I know this is hard to read, so I've made it a little bit nicer here. this is an example of an array we can tell it has the
type that says array it gives you a list of all the references so those are all the objects
that it's pointing to. We can actually see the
allocation location of that object and finally my favorite is the
memory size of that object. And these JSON files make for a very extremely easy analysis. So for example, let's say I wanna know how many objects are alive? Just how many objects
do I have in the system? Very easy, I just count
the number of lines in the file and I know. So this example had about
185000 objects, done. Okay, so how about I wanna know what file is most popular. This is one tool I really like to use JQ if you don't have the JQ
tool, get this it's amazing. Basically, it's like a
really great grep for a JSON. so I can say hey, I want to extract a file just the file field, I'm gonna sort it and get unique them,
then I'm gonna sort those by the number of counts and
then just give the top 10. When I run this I can see okay, well the top allocations came from active support dependencies.rb so I know it came from that file. Okay, I've focused in on that file now I can use JQ again and say, "Well, I only care about allocations "that are inside that file, "so I want you to filter
out all the allocations "for that file and then give me the line." then we run through the same, basically the same command again sort them, get the unique
account, sort by account and then get the last 10. So we can see here okay
the most popular line inside that file, is line 283. So I know okay, I need
to go look at that file, look at that line and now
I can kind of get an idea of why we're allocating so much stuff. So we can also answer
other interesting questions like how much memory does
active support allocate? So I just say okay give me all the records whose file contains active support extract the mem size from that, pipe that to a little off command and sum the total and we can see that okay active support allocated what is that? Like around three Meg's okay. So back to our performance
problem at work. we ran into a thing, I wanna
talk about hidden references this is the thing that we ran into. We're trying to find what
has references to what, basically this heap dump contains all the information of
the entire object graph we can make a graph out of it so let's talk about references. This is very simple, we have an array it references two things
this is obvious right? We can see okay, we have an
array it points at two symbols now here here's another example of some references in a
program we have a lambda. This function foo it returns a lambda and that lambda sums the two
values that you passed in right so the variable m contains two references and I promise this isn't a trick, there are no non-breaking
spaces in this code. (laughing) So all right, here's a question for you here's similar code the
Foo method returns a lambda how many references does this lambda have? You don't need to answer, I will tell you the answer just think, how many references of it does it have? It actually has two references. The reference is the number A and B and there are two reasons why
this lambda has references to those variables one, Ruby doesn't do what's
called escape analysis so it doesn't look at
the Lambda block and say, "Oh, I'm not actually using those things "there's no reason to keep track of them." And the other thing which is basically an implementers nightmare
is a thing called binding. (laughing) So if you asks the lambda for its binding that binding object is the environment in which the lambda was created, you can actually ask the binding, "Hey what local variables
do you know about?" And it says, "Hey I know about A and B "they are local, when
the lambda was created." Then you can say, "Hey binding "give me the value of A and B." So we actually have these references this lambda contains references to A and B but you can't really see it when you're just reading the code. So, I was able to find
these particular locations unfortunately I didn't
understand this code like I could point to it and say, "Hey, that's the problem there "we're allocating a bunch of stuff there." But I couldn't really fix it, so I worked with other folks on the team and they actually rolled
out fixes for this that eliminated those hidden references and we were actually able
to drop 200 gigabytes memory from our production application. So, this is like really awesome and I praise the person who
did this a lot it was amazing, a really amazing change. So I wanna talk about
some tools that I want and then maybe some thought leaders stuff and then we'll wrap up with another story, and maybe two stories
depending on my time. So let's look at the
Rails Boot Process again unfortunately so far all we've been doing is looking at looking
at analysis of scripts, like maybe running DB migrate or other scripts that you
run with your application. We haven't looked at anything else. What about app compilation time? I mean when we run this, it's easy to know okay this is how much our scripts are spending inside of
just loading the allocate or loading the application, but what about that
actual compilation time? I wanna know how long does it
take to restart a web server, I wanna optimize that,
how do we deal with that? So how do we deal with benchmarking
the rest of these things we only looked at that one. So, today we're gonna take a look at some new rackup options that I've added in order to deal with
this particular situation so rack we merged this into
rack a couple weeks ago this is one of the things that I was doing instead of working on my talk. In newer versions of rack
you will able to say, "Hey rack, I want you
to dump the heap before "or after you come compile the application "but before you give
it to the web server." Or, "I want you to profile the application "up until you give it to the web server." So we're going to build
this tool into rack for you so that if you wanna know
why is it taking so long for my application to restart,
or be ready for requests we can actually have this built into rack and give you some information, so you know how to identify
problems in your code and we've written the
benchmarking code for you. So this just leaves us
with two other issues, our warmup time and our run time so we're able to profile script boot time, we're able to profile
compiling applications but we're missing two components. Now last year Eileen
spoke about system tests and this is a really important thing so besides the fact that now
you can do entire system tests this is an important
refactoring on Rails itself and means that we can actually run real requests through the application. So, something that looks
more like a request that you would see in production. Before we had system
tests, most of the stuff this I mean we on the
Rails core team know this and maybe most people did not but a lot of the integration tests were basically just BS. (laughing) And I want to say that,
I'm trying to say that in the nicest way possible. They would actually run the
data, run these requests through only your application it would not consider any middleware that you had in your application, and this is the advantage
of the system test is that they actually run the data all the way through the middleware into your application and then back out, so you can get a more real-world idea of performance of your application. So now that we have that system in place for running these system tests. (laughing) Sorry, I said I'm a little
scatterbrained right? Okay, so now that we
have this system in place we can run real requests through and actually benchmark those, so we have a script for doing this at work it's called this as bench app
we use this for benchmarking our actual requests at work, and this is what the output looks like it's very easy to read. (laughing) So, I wanna tell a little
story about using this tool. We had a, this is amazing, this was amazing we had a page at work that would allocate 13
million objects per request. (laughing) 13 million. (laughing) And I mean this page would
return in like a second so anytime anyone tells me Ruby is slow, I'm like what? (laughing) We did 13 million
allocations and we're able to render that page in a second,
like this is pretty good. (laughing) Anyway, I was able to get it down to about 300,000 objects for requests using these tools so we could monitor an actual request going through the application and get a real benchmark with it, and I wanna share the graph with you because it's extremely satisfying, so this is the graph and you can see where I pushed the production
there it's very, very nice. So, the thing that I wanna do, the thing that I want to do with Rails 6 is that I wanna make performance easy. So we Rails, we make generating applications easy we make connecting to databases easy we make writing routes
easy, generating views working with forms, writing
tests, all of those things. But I think that now it's time
for us to do the same thing with application performance analysis, I think we're falling down in
that particular area of Rails. Yesterday Eileen said something that I really so first off, her talk was amazing, amazing! I said to her your slides
look really, really great and she said to me, "You
should go back to school." And I was like, "Whoa!" (laughing) "Look I, yes I did drop out
of college but that hurts." (laughing) But yesterday she said she wants to make Rails
scalable by default, but I think that this is really important this is a really important thing. But when I think about making
Rails scalable by default I don't wanna make just the application itself scalable by default, I wanna make your team scalable by default I wanna make it so that not
only is it easy get started it's easy for people who
don't know much about Rails web applications to get started that's the way it is today. We can easily start a Rails application but I don't think that the
Rails team is providing as much support as we can mid-career. So that's where I wanna start improving is making sure that we can
actually provide tools for you to move from beginner on to more advanced. So for those of you like
me who cannot hire somebody to fix their performance issues, you can just instead of
rewriting everything in rust you can say like, "Let's
take about five minutes "let's just take like
a couple minutes here "and run this one command "and see what it says our bottleneck is "and maybe we can just fix that instead." So I wanna do for application generation or I wanna do for performance what Rails is already done
for application generation. And I think the first thing
that we're going to do is start by taking the
tools that we use internally at GitHub for doing
performance improvements and pushing those into open source, pushing those upstream. I have two minutes left, I
wanna tell one more story I have one story to end with, and this is not serious at all so you can sit back and relax. For those of you that
weren't here last year I need to do a little bit of ketchup (laughing) So, last year I gave a presentation that was much more of a waste of time than this presentation. So last year I spoke about Crap Cata and it was basically what it was, the talk where I presented this data here that I had acquired Crap Data. I talked about doing, analyzing this data and you know different types of analysis that you could do and 40 minutes later I revealed that I had acquired this data by a Raspberry Pi and a Motion Sensor that I had put together,
where I had mounted the motion sensor above
a litter box as such, and my cat would get into the
litter box and then come out and I recorded all this
data and made a log and this is well, I
guess the cat made a log. (laughing) Poop jokes yes. So we had about 100 grams leftover. (laughing) So, it was literally crap data but anyway the actual reason that I
built this thing is because last year around this
time, we were moving, we bought a house. Like the day after I got
home from RailsConf last year we had to pack up all of
our stuff and we moved. So we got a house and my wife said to me "You have a closet full of electronics "if you do not use these electronics "we will throw them away." And I said to myself
what can I do with it? (laughing) What could I do with a
scale and a Raspberry Pi and a motion sensor? (laughing) And this is what I came up with. So let's talk a little
bit about home automation because I mean the first thing that you do as a new homeowner I think, well at least is the first
thing I do as a new homeowner so I'm like wow! I've got a house, I need to
like automate everything, need to automate all of the things. So we have a toilet in the house as you do and unfortunately we have
that like blue water in it, okay it's got like that
blue cleanie water in it and the thing that's
really bad about that stuff is it's basically Tide Pods for cats unfortunately my cat isn't
as smart as a 13 year old and he tries to drink that stuff. (laughing) So I'm like okay, I need a way to solve this issue, the way I did it was I
decided to automate my house I bought one of these, this is a door sensor, basically
it's a magnetic sensor and on the inside it
has a little reed switch so that's our magnetic
reed switch on the inside. Now what I did is I took a tilt sensor, this is a tilt sensor I put my hand there for scale. (laughing) And basically I bypassed the read switch with the tilt sensor. So we have the Z-Wave sensor where imagine this is a side view, I've mounted the tilt
sensor on it like that bypass the read switch such that when the Z-Wave sensor tilts like this oh so this is our on position. So when it's like that it's on and then if you rotate it
like this it turns off, so it's as if you've opened
the door for example. Then I decided to hook it up to a siren. (laughing) And I took a video to demo, I think it's running we'll see. So that's the siren
there, there is a toilet when I lift the lid you can
see the siren will come on. (laughing) And when I close it, of course yes good job editing
Aaron yes it shuts off. So now I can tell when the
toilet is open and closed. So, if nobody's in the bathroom we know to close the toilet so
the cat does not go in there. Now aside, like one
happy accident from this is that I actually have
a log of all the times the toilet lid is open and closed. (laughing) And so I decided that Cambridge Analytica doesn't
have enough of my information. (audience laughing) All right, anyway so. (laughing) I mean I wish I had implemented
this earlier so that our cat didn't drink any of that
blue water and get sick, but the truth is hindsight is 20/20. (laughing) Thank you very much, I hope I
can see you again next year. (audience applauding)
