How processes get more memory. (mmap, brk)

need more memory where does malakut memory from let's find out C programs typically get their memory by calling malloc and they release their memory by calling free when they're done malloc is like this magical fountain of memory and it just keeps handing out memory whenever we asked for it we just malloc 500 malloc 100 malloc a thousand we just it just keeps handing us memory but how does it work in my last video I talked about using s trace to try to understand how software works where you don't have the source code for it and that's convenient cuz that's sort of what we're trying to do today so let's take a look at malloc by using s trace so let's start with a simple program it allocates a bunch of memory in this loop that's just to help you see what's going on and then we're gonna run we're gonna compile it and then we're gonna run it using s trace and we're gonna see what system calls it makes of course you're gonna see all the Lib C stuff up at the top what we care about is all the stuff down here at the bottom you'll notice the right calls the printf uses to print stuff to standard out you also notice some BRK calls that don't happen every time through loop but they do happen sometimes in this case BRK is actually where the magic is happening but before we dive into BRK things are a little more complicated and so so what happens if i change my code so that i allocate bigger blocks let's say I'm trying to allocate 5 megabytes every time through the loop now we'll compile it again and now we run it through s trace you'll notice that the BRK calls are gone and they've been replaced every time through the loop with an M map call so now M map is actually where the magic happens now I'm going to focus on Linux in this video things are gonna vary a bit between operating systems if you're using Windows you can use virtual Alec or virtual Alec X which are kind of like a map and which we're not going to talk about in this video but B R K and M map are the main syscalls that are gonna be used for memory allocation on Linux and on other UNIX based operating systems so let's look at how they work first of all a quick reminder about how memory is laid out for a process you get the stack up here at the top you have the code the Global's and the heap down at the bottom you have some libraries in the middle and a whole lot of space in between the address space is huge on a 64-bit machine you can address more than 17 billion gigabytes of memory which of course is more memory than you have and this is all part of a bigger topic which is virtual memory and virtual memory is too big of a topic to include today but I do want to just mention that it's out there we'll try to get to it in a future video but the relevant part for today is that the top of the heap is often called the program break now I'm not sure why it's called that I sometimes think maybe they should have called it the program cliff or the program line of doom it marks the top of the heap if you try to read and write memory below the program break you're gonna be okay if you try to read and write the memory that's right above the program break you're probably going to second so moving the program break up allocates more memory moving the program breakdown D allocates memory basically expanding or shrinking the heap and that's what BRK does BRK essentially moves the program break I guess be rk4 break anyway you're moving the program break with BRK there's also SBR K which is really just a wrapper around BR K which allows you rather than specifying a new address for the program break it allows you to specify an increment allows you just say move the program break up by a certain amount of memory or down by a certain amount of memory here's a quick example that uses SBR K now this may be a little bit counterintuitive because SPR K actually returns the address of the previous break so it's gonna move the break by some amount but it's gonna return you the address of where the break used to be so when I say SPR k zero that's basically saying tell me where the break is if I say sprk 4096 that's saying move the program break up 4096 bytes and tell me where it used to be notice that I moved the program break up by 4 K I could have tried to move it up by less but it's effectively the same thing because modern virtual memory systems use paging and the page size is 4 K and so I could try to move it up by 10 bytes but effectively it gets rounded up to the next 4k page also knows that if I try to store something in the memory that's after the program break like right here which you'd never want to do but but if I do you notice that it doesn't need a sec phone so what about M map a map is similar in that it requests memory from the kernel it basically says I need more memory in my address space to be usable but rather than just allowing you to bump up or down the program break it gives you more options the best way to introduce you to a map is to show you an example so here is one this is probably the simplest end map example I could come up with and maps two pages of memory and it prints out the address of those pages you'll notice that M map takes a number of different arguments let's go through what they are the first one is a hint I specified Knowle because I don't really care where it puts the memory this is basically my way of telling the operating system just give me what I just need memory just put it where the second is the is the size of the block that I want allocate so I specified the page size as 4k you can request sizes that are not multiples of 4k but it doesn't really make any sense because you're gonna get a multiple of 4k one way or the other and you notice I can specify protection so I can actually say I want this memory to be read only or write only in this case I want it to be readable and writeable but I could specify just one or the other then there's the map private and map anonymous flags which essentially describe they basically tell the kernel on how we want the memory to be managed how we wanted to be shared do we want it to be shared at all or do we want it to be private to a particular process and then these last two are actually useful for memory mapped files and that's something that I'm gonna save for a later day memory mapped file IO is really cool I just don't have time to cover it today so let's let's change a few things here first of all let's say that I do care where the memory goes if I care where the memory goes I can specify a particular address as the hint and you'll notice that it tries to satisfy that request it tries to follow my hint but blocks of memory are always gonna be aligned to the beginning of a page and because pages are for K the bottom 12 bits are always going to be 0 so in this case it tried to follow my hint but it wasn't quite able to so but it did give me the page that contains that address that I requested so close enough and of course I can mount more than one page if I want like I map four pages right here and you notice the addresses are spread out a bit more because the first block is just bigger and now just a little caution I show you these functions because I want you to understand how it works I want you to be able to see how memory allocators actually work I want you to be able to see how they request memory from the kernel unless you're writing your own memory allocator I don't recommend using BRK because it's used by the system's implementation of malloc and so if you're not careful you may be fighting with malloc and calloc and realloc and free and that's a fight you're probably gonna lose so if you're gonna use any of these I recommend use em map which is a great option if you want to write your own memory allocator if you want to do something custom or if you just need a big block of memory and you think for some reason that you can do it better than malloc does because then your memory allocator can coexist fairly nicely with the system allocator and it's not gonna cause problems as that's all I have for you today and map and BRK you don't use them a lot but it's useful to know what they do and how they work just in case you find yourself in a situation where you want a little more control than malloc gives you I have a lot of ideas for other virtual memory related videos that I'll make in the future but for today that's all until next time I'll see you later [Music]

Channel: Jacob Sorber

Views: 33,167

Rating: 4.9797101 out of 5

Keywords:

Id: XV5sRaSVtXQ

Length: 6min 50sec (410 seconds)

Published: Mon Apr 16 2018