- [AJ] Hello and welcome
to the SANS webcast Introduction to Window's
Memory Analysis. I am AJ Boyle and I will be
moderating the course today. The instructor is Chad Tilbury. Before I turn the
time over to him, I would like to cover
features of the interface for those who have not
attended a SANS webcast before. The screen is split up
into three main regions: The participant's list, the
direct message chat window, and the white board which
has already been loaded. The participant's list
has several columns. The first with the
little hand indicates someone has raised their
hand to ask a question. To raise your hand,
click the hand button right below the
participant's list. The next column is
a yes, no indicator. You can click the toolbar button with the green
check or the red X to indicate a yes or no
answer to the question from a moderator
or the instructor. The next column
is the microphone. If this icon has a yellow
background, someone is talking. The microphone will be locked, so only one person
can speak at a time. If you have a turn to speak, you can click the microphone
button in the audio section. Do note that the illuminate
software will buffer speech, so if you experience
network delays, playback will speed up
causing the chipmunk effect. The next column is
the chat indicator. Anyone who is typing in
the messaging entry box will have a yellow
background here. I will cover how messaging
works in a moment. The next column
with the pen icon is the white board
editing indicator. I have disabled white board
editing for participants. The last icon before names
is application sharing, which has also been
disabled for participants. Finally, the list
of names you see are your classmates,
moderators, and the instructor who is also a moderator. Let me explain how the
messaging system works. To enter a message,
point your cursor at the entry box
below the main window. Before sending, use
the dropdown to select who the message will go to. Valid choices are all
participants, only moderators, and only selected participants. You can select the names of
other students, the instructor, or moderators to send a
message only that person, but do note that moderators
will see all messages, both public and private. As mentioned, the white board
already has the slides loaded, the instructor will
advance the slides as he moves through
the course material, and it will automatically
advance for all students. If you experience technical
difficulties during the class, please send an e-mail to
webcast-support@san.org or notify your moderator. Thank you for your attention. Now let me turn the mic over to your instructor,
Chad Tilbury. - [Chad] Hello everybody. Hey I actually have
two computers going, so I am trying to
manipulate both my keyboards and mouse at the same time. We'll see if I can get this hack situation going
a little bit here. Let me know, give me a thumbs
up or hand clap or something if you can hear me okay. Very good, thank you very much. See a few of you, fantastic. Well I've got, I've hacked
the system a little bit because I've got a
presentation with an intense amount
of animations in it and I'm hoping I'll
be able to use that. I realize that given the
fact that we're going across the undersea cables
for many of you that it may not be
the best option, but we'll see how it works out. As long as everyone keeps
happy, I'll stick to this. Otherwise we can
jump back down to the kind of cast version
of the slide. Before I get too deep,
I do wanna mention hopefully you noticed when
you came into the webcast that there was a
link to download the
PDF of these slides. So worst case, grab that pdf, you can follow
along in that way. Well I don't wanna spend too
much time on introductions, but I do wanna say thanks to
AISA for getting the word out, and sponsoring this webcast. It looks like we have
a great showing here. I have to admit that
I'd probably prefer
to be in Australia giving this
presentation in person, but well it saved me a
long plane flight anyway. And I see I have
quite a bit of friends I've seen before on
here, so hello to you and hello to any new
friends in the making. One thing I do like
to point out is that, feel free to ask
questions as we go along and as well as shouting
answers out to the, to the chat room,
that's probably the
best place to do it. I'll be keeping track of
what's going on in the chatroom and that's actually why I have
two computers up and running, so I can watch that chat
and kind of get a good pace. Now, as I said, memory forensics is actually
one of my favorite topics and I'm very happy to kind of
present this to y'all tonight. I won't spend a whole lot of
time on introducing myself, but I got my start
in the Air Force with the, an organization called Air Force Office of
Special Investigations and I've done a wide variety
of forensic type of cases, but really my specialty
is in incident response as well as counter espionage. I currently serve as a
incident response consultant. I've spent my time
over the last few years doing a large number
of cases varying from small to medium
sized businesses all the way up to
Fortune 500 companies facing things like a lot
of our banking theft hacks or APT type intrusions. And it won't surprise you,
I'm also a SANS instructor and I wanna just point out that I have my contact
information here, so if you have any
questions or comments or war stories to
share, please feel free to reach out to me. I'd love to hear from ya. So we've got a lot
to cover tonight. The first thing is why do we
care about memory forensics? And over the last decade or so that I've been doing forensics, I haven't seen anything close to the value of a technique
like memory forensics. It's raged onto the scene, really starting around
2006 time period, and I believe it has made
a revolutionary change over what's available to us and in a large number
of investigations. Now, a lot of what
we'll be covering today is going to be focusing
a little more on kind of the intrusion type
investigations or malware type of cases you
may be running. But just keep in mind, memory
forensics is so important because memory is the, really its the
critical choke point that we see on a
computer system. It is the bridge that
everything passes through from the disk all
the way up to CPU. So we can get virtually
anything out of memory, from the standard
URLs or web traffic that might've been
going on in the system to open network sockets
to, in some cases, complete copies of
registry keys and values. A large amount of information
on hardware configurations, even be able to find
things like web USB keys or Google Devices may have
been played from that system. One of the really interesting
places of research that's going on in
memory forensics is this idea of passwords, particularly as we
get into the world of full disk encryption
or encrypted volumes where you may actually need to go back to a memory image in
order to get that information. And of course the final
thing that I've got up here is this idea of malware. And malware is really probably
easiest to find in memory, which if I had one
object or one artifact to actually look for
with inside a system that's been infected
with malware, I would almost certainly
pick the memory image. I'm gonna drop out for a second so I can get my
pointer set up here. Just give me one moment. Well, these slides we're
gonna be talking about are really about
doing the analysis, but I didn't want to
sweep by and not cover kind of the most critical
part of doing the analysis which is really the
idea of first we have to get our memory image So we're not gonna spend
a lot of time on it during this hour. I did wanna point out
one of the newer tools that's come out
regarding memory image. Does anyone actually
use this tool? It's called DumpIt
from MoonSols. The reason why I've
pointed it out is because it is by far the easiest memory accessing tool
you're gonna run across. It literally has no options. Wherever you run it from, it will show you how much
memory you have to dump, it'll ask you if
you wanna continue, and it'll just dump it
to the extreme location that you ran the
application through. It's a fantastic application
to throw on a USB thumbdrive and may be deployed
throughout your environment or given your
instant responders. Literally all you
have to do is walk up to a system that's
up and running, plug the thumbdrive
in, run one executable, it'll dump the memory image
back to the thumbdrive, and then send it
back for analysis. It's a really really neat tool, freely available from
the MoonSols website. Now this is really an
introduction to memory analysis. We could get really
deep in the weeds and this slide is, what
I call the one slide kind of down and dirty in that. While we don't really
need to understand a lot of these
objects in order to use some of the new tools
that have recently come out in the memory analysis realm, I really hate to have anyone
actually running a tool without at least knowing what's happening
underneath the hood. I think that's a critical thing. We just don't want
people punching buttons. Really want you to
kind of understand what we're seeing and
why we're seeing it. So while I don't wanna
go into the depth that you'd need
say to bring WinDbg and start tossing
memory yourself, I do wanna just
cover just a couple of the key artifacts or
pieces of memory analysis that we'll be
taking advantage of. One of the things
that really stirred me to make this presentation
is the fact that now we finally have
some incredibly robust, incredibly easy to use
memory forensic tools that make all of these,
or give us access to all of these objects
now in a very nice, easy to use interface unlike
we've ever had before. So, we no longer
have to necessarily draft up into
WinDbg to see this, but we should at least know
kind of what we're looking at. So the first thing we have to do when we get out memory image is we have to find some context. We have to be able to tell
where all the other objects are, and that usually
entails finding some sort of an artifact
or a kernel reference that allows us to then
trace to the other memory structures that we're
gonna be interested in. In Windows, the most
popular thing to find is this idea of a
Kernel Processor
Control Region, or KPCR. Now this was relatively
easy to find in the XP days in that it was at a fixed
address within memory. Once we got into Vista,
Windows 7, 2008 world, all of a sudden
it started moving. And some of you may
be familiar with address space layer
randomization, well that actually affected our memory analysis tools as well. And that's why when
we, when we get down to seeing the various
tools that we can use, you'll notice that
there's a wide gap between operations systems, that different tools
actually support. Some of that is because
every operating system, even every server stack
may actually confuse or make it difficult to
find a good reference point and actually be able
to go and pull out all the other memory
structures we care about. So the first thing we
do is find context. Once we find that context, we can piggyback off
of that in order to identify critical things
that we care about. One of the most critical
is gonna be this idea of Executive Process blocks
or the idea of finding the running processes that
were in that memory image. Those running processes
will then allow us to find what we call the
Process Environment block, which is going to show
us the loaded DLLs or loaded libraries
for that process. We'll also be able to the parse
what we call the VAD Tree. That VAD Tree keeps track
of all the memory pages or memory sections that now are, that belong to that process. And then there's a
lot of other artifacts that we'll see as
we go through here. Things like file handles
for instance or threads. But also the final thing that
we really wanna focus on, this idea of these
kernel modules with the ability to kind
of patch the kernel. We commonly will see
these as drivers. And of course our
memory analysis tools will need to go and
pull the information of those kernel modules
out so we can identify which drivers have been
loaded so we can see if maybe any malicious
ones, maybe part of the problem on that system. The next step after we identify where all our key
components of memory are is to really kind of dig deep. Now we could probably do
a standard memory analysis just with steps one and two, but if we're investigating
kind of more advanced malware, we're gonna see that
advanced malware can take steps to hide
itself, even in memory. We're gonna see that
it's almost creepily easy for malware to unlink itself
from the process list, or unlink DLLs so they
won't be identified as being loaded in
a specific process. So we're gonna need to again
find kind of key signatures for all those memory structures, and our memory analysis
tools, it isn't just gonna simply
search all of memory for any kind of outliers
that are out there. Bring them up
allows us to kind of include those in our analysis. And then the final
piece will be for us to, of course, then use our brains. It'll be pooling all this
information together, hopefully in a nice,
user friendly way so that we can very
quickly parse through an intense amount
of data looking for kind of the key piece
which will help us solve what happened
on that system. So if you were familiar
with standard forensics, you'd think of a file system. Standard forensics very often is concerned about file systems. And of course, the key things
in a file system are files. And so we spend a lot of time concentrating on
allocated files, unallocated files,
metadata about files. If you wanna take that analogy into the world of
memory analysis, instead of files
think of processes. Processes are by far
the most important structure within a memory image. So one of the first things
our tool are gonna do are go out and find those
executive process blocks, and they're gonna
look like this. Actually the executive
process blocks are tracking memory in
a doubly linked list. You see that we've smss.exe
and lsass.exe and service host. Do you notice that
there's a forward link and a back link to all of this? That's our doubly linked list that we simply find, we
start up and follow each link in order to identify all
the processes running, or at least all
the processes that haven't been somehow
manipulated by malware. And then I've blown
up the lsass.exe just to really
represent this idea that each one of those process blocks is gonna have a
wealth of information that we can use to
identify whether it is legitimate or maybe evil. We'll see things
like the process ID, we'll see what it's parent was, we'll see where it
was sitting in memory, we'll have timestamps
associated with processes that can be very useful. In addition to a
wealth of other things that you'll see shortly,
like our DLLs that are loaded and our threads that
are part of that process and handles to files or pieces of the registry within Windows. But think of that
process block as kind of a structure
that holds kind of all the information we may need. You can almost
think of it as the metadata structure
around a process. So as we go through
this presentation, I'm gonna introduce some of the newest kind of tools available. I've picked the
freely available tools primarily because
that opens it up to the widest
audience view able to go out and grab these and
actually play with them. And the simple fact
is we are very lucky the free memory
analysis tools out there are really best of
breed and it's amazing the capabilities they give us. Without these tools, like
Redline for instance, we'd literally be
back in the Stone Age. We'd literally need
individuals that, that can code an
assembly and that are complete Windows internals
experts to be memory houses. However, with these
tools all of a sudden, they've really opened the
field and allowed kind of the rest of us who aren't,
who don't think in Hex, to actually be able
to do memory analysis. And I start with
Redline because I think it is by far the easiest
tool to actually use. It has the best interface
and it's a fantastic tool for really starting to learn
what can find in memory. It has the best support of
any free memory analysis tool, you know in conjunction with
its kind of sister tool, which is called Memoryze. We see it as support
for just about any memory image you're
likely to run into, both 32 bit and 64 bit, which is kind of a neat
distinction with this tool. It's put out by a
company called Mandiant. They do a lot of kind of
instant response type of work and they've kind of
built in a lot of these heuristics and ways of
quickly identifying malware. And they also included kind of a quick and dirty walkthrough, almost like an idea
of a step-by-step way to identify what
to look at first, then what to look
at next to that. And in addition to
being easy to use, it also has a great ability
to kind of be extended. It has some built in
signatures, although I'll admit, the number of signatures
built in are quite small, but the nice thing is
you can add your own. So as you find malware
working on a new environment, you can create
signatures for that and continue to
search for that in any additional memory analysis
that you do in the future. So I highly recommend
that you take a look at Redline if you
haven't already. It's download address was at the bottom of that last slide. So once we start
on memory analysis, very commonly we'll start
looking at processes. And while we're
looking at processes, I'm starting to try
to identify ones that just might not be right. I'm looking for the evil ones. So what we're seeing
here on this slide is all those things that you
might consider looking at in order to identify
the bad from the good. So for instance, what's
the process called? Is that something
that you're used to seeing on that server
product or that work station? Is it even spelled correctly? Sometimes as we'll see,
a slight misspelling could help it kind of
hide in plain sight. What path is it running from? If I see a binary
running from something like Windows System 32,
I'm probably going to trust it a heck of
lot more than if I see a process running from
a temp directory or under some sort of a
user app data folder or something along those lines. So just simply seeing
where it was running from can help us identify
maybe those outliers. Also identifying what
process spawned it. Was it spawned by
the proper process? And some of this comes
with practicing knowledge and knowing, for instance,
that svchost.exe processes are always spawned by
the services.exe process. If you see one that's not
spawned by services.exe, well that's a problem. So being able to
recognize kind of these patterns can
really help us. We also get an ability
to see what the original command line that
spawned that process was. We can see the arguments, we
can see if it makes sense. If it actually looks legitimate. And then the final thing
that I see a lot of people kind of not pay as
much attention to, is this idea that
every process has a timestamp associated with it. If we happen to get lucky enough to find a malicious process, that alone can be
enough to help us blow open our case
because all of a sudden I have a malicious process
and I have a start time. I know when it first
appeared on that system. As such, I might be
able to leverage that to then go to my
files system analysis and start to identify
everything else that happened
around that time or my event log or my
registry analysis, really tying all of these
individual artifacts with their own timestamp
together and looking for things that happened
about that same time. So start time can also
be very very helpful. So let's start with an example. So this is an example
of sobig worm. Sobig worms spread through email attachments and network shares. And as you see on the
left we have Redline, and I've run Redline
on a memory image that's been infected with sobig. We see along the bottom
a list of processes and then up top on that you
see Investigative Steps. Those investigative
steps are actually that guided analysis that
I talked to you about. The idea of kind of being
able to just walk through kind of a series of
steps that will help you hopefully identify what
shouldn't be on the system or identify whether there's
a cause to me concerned about whatever
you're looking at. Now when we look
at the processes, you'll notice that
one, well actually two are what we call Redlined, in that they have a red
marker next to them. This means that they
have a very high what we call malware
rating index, meaning that the
software believes that these are likely to be
malicious processes. Now the Memoryze.exe turns out
to be a legitimate process. That's actually
the program I used to acquire that process,
so if you see Memoryze.exe very commonly that's
going to be the memory acquisition tool. It could also be
something like DumpIt or Win32DD or WinDD or
whatever memory dumping tool you typically use,
you may see it here. And the reason why it
actually gets Redlined by the tool is that
if you think about how memory acquisition works, you
need raw access to memory. And very commonly
the way you get that on our newer systems
is to load a driver. So a driver has to be
loaded, it has to be loaded at the kernel
level, and it gives raw access to an application. Now that's very similar to
what malware tries to do, and as such it kind of
get flagged along with the other piece of malware, which actually turns
out to be winppr32.exe. So if we didn't know that
winppr32.exe was bad, we might look a
little deeper into it. And this is kind
of the details page that we get when we
click on a process. And just looking at
this, because I have a little bit of experience, I can see all kinds of evil just from the process details here. Thought I had some
arrows that would pop up. Well first of all we have our
malware risk index of 100. Redline rates applications or
processes from zero to 100, with 100 being the most evil or the most likely to be malicious. So the fact that this has
100 risk malware risk index, all of a sudden
makes me think hmm, at least Redline
is fairly convinced that this is a bad process. Oops. Well for some reason my
pointer isn't coming up, but the other items which would make me a little bit
interested in this are if you look at the path, notice that this executable
is actually running from the Windows directory? That's not a guarantee that what we're looking at is malicious, but it's a little anomalous. Most of our processes are likely to be running from the
Windows system 32 directory. There are exceptions, of course. For instance, the
explorer.exe process runs from the Windows directory, but an admin might
see that and say, hmm, that's a
little interesting. The other really key one that I would recognize is the SID, notice the security
identifier done there. Those of you that
are Windows jockies will see that that's
actually a SID for a user within a user context or a
user security identifier. Now, of course, users
can spawn processes and often do on systems, but at least I know now that if this process
is purporting to be something like
a system process, it doesn't have the
right security identifier to actually back that up. That's another
item that at least might tell me that, hmm, I might wanna look a little
deeper into this process. Now we had that big
long list of processes and what to look inside, or what to actually focus
on when we're trying to identify anomalous processes, but Redline actually thought
of something pretty clever. It thought well, if
we have all these kind of rules of
thumb, for instance, the fact that
svchost.exe should only have a parent process
of services.exe, or should always be running from the Windows32 system directory, or should always be running
under the system context, not a user security
identifier context. We know all of these things. Why don't we just
package those up and create something called
a malware risk index? So it's kind of
a heuristic check that looks for a large
number of things. It'll even look for items like command shells being spawned
from specific processes as well as if you're
familiar with DLL or load order hijacking, it'll also do some
rudimentary checks for load order hijacking, which is kind of one of the
newer things we've seen, kind of more advanced
attackers use in order to very very
cleverly hide their malware. If you're not familiar with it, do a search for DLL
load order hijacking, there's some great blog
entries on that out there. You can also do
things like search for evidence of code injection. If you're running
it on a live system, it'll even do digital
signature checks. So it will go in and verify
the digital signatures of all the executables that
have been loaded into memory. And finally it will
do what we call kind of least frequency
of occurrence checking. It will look at how
many occurrences of each DLL have been loaded. And if it's a very low count, that could indicate that
it's a little bit anomalous or it may not be
something you wanna trust. We'll talk a little
bit more about least frequency
occurrences shortly here. So let's see another example, let's see an example
of a piece of malware that's commonly used
by someone like APT or an advanced attacker. Instead of really being
incredibly advanced from the stand point of how it hides itself in the file system, it actually takes a very
rudimentary approach, which turns out to
be very effective in 99 out of 100 cases. And that's the idea of
hiding in plain sight. The idea of trying
to name itself and keep all of those tactics
that we talked about so far as similar to the
original as possible so that an instant responder
just easily glosses over it. So here is our process list
for this piece of malware. Now does anyone
actually see kind of what looks malicious in here? I'll be honest, I
actually didn't catch it the first time I was
reviewing this memory image. I kind of blew right by it, yeah you see some of
this .exe with all caps, that actually turns
out to be legit. Looks like Justin has nailed it. It's right there, it's
actually not svchost, and one of the reasons
why hackers love svchost is because there's
usually about four to six of them running on
any given system. So it's really easy to
just add another one in and it kind of just blends in. But in this case, the
attackers actually instead of naming it svchost,
which its real name is, they just transposed
the two letters so we're seeing
that scvhost.exe. And here's our
particulars related to it. And I was looking at this, once I went through
all the processes, I didn't see anything
that jumped out at me, and then I said well, since I know attackers
often will use svchost, let me at least look
into one of those. So I kind of looked
down the row on each one and when I got to this
one I did identify that something was
off, for instance, look at the parent. The parent is explorer.exe. Now that is a dead giveaway that something was run under
kind of a user context. Because explorer.exe
is your GUI. You wouldn't expect a system
process to typically run with a parent explorer.exe. The parent process
path is also C:Windows. Which is kind of interesting. Well actually, that's
not interesting at all because that's where
explorer.exe runs, but it was actually
it's an application path is system32, so that's it. That makes sense. And then of course we see
the security identifier is under kind of a user context, which you typically
wouldn't expect to see on a standard svchost process. So I looked at all
those and I swear, I still didn't catch it. I was still looking at it like, I know this one's bad
I just don't know why. I was thinking about things like well, okay so svchost, we
know it's in system 32, maybe you know they did
some sort of replacement of the original DLL,
you know trying to maybe circumventing
Windows file protection. Which we all know is
relatively trivial to do, but then I was thinking
that would almost certainly make the system very unstable, because svchost.exe is
such an important process for so many things, so I was still
beating my head around and we'll actually shortly
here how I found it, but the moral of the
story is there's actually always more than one
way to find things. Justin found it immediately. Yes, dyslexic accuracy, nice. I like that. (laughs) But absolutely, but
one of the things that I really like
or tried to always make for myself as
I'm doing forensics, is a backup plan. I never wanna have
to get perfect every time I iterate
over kind of a review. I always want to
have enough processes and enough checks in place that if I miss something
the first time, I actually catch it
with some other tool, or some other kind of technique. And we'll see what techniques
I use very shortly here. So here's the
various things that I kind of found as interesting. You can't tell from this, but actually the start time
of this service host or this scvhose.exe process, was different than the
rest of the svchost.exe. That means it actually
started after them and very commonly one of
the things that you'll see, you'll be able to tell, is whether the process
started or boot or whether it
started after boot. And you can tell that
because you'll see a large number of
processes all started near the same time, and the rest started in incremental
times after that. And as such, this one
happened to not start at boot, it started sometime
after the initial boot, which is another red flag
that something was up. So another way that we
could have found that is by looking at it visually. Now we have way in Redline
to view hierarchal processes. And if we look at this, we'll see everything
running under services.exe, including all of the
legitimate svchost processes. Look where our scvhost.exe
process is run. Look at it this way, it's
very much of an outlier. Something doesn't fit. And these should be the things
you're kind of looking for, the ones that just,
hmm, why the heck isn't that svchost process
with the other ones? Well, in this case, it's
because it's not named svchost, and it was actually
spawned under a user context in explorer.exe. But in this case, the
hierarchal view shows you that. And we'll see the volatility
has a means to do this as well as each binaries
responder tool actually added this capability in
November of last year. So most of the tools that we
cover, well actually all the tools we'll cover
in this presentation actually now have
this capability to do things hierarchically. So kind of a neat way to very
quickly look for anomalies. So it turns out
the processes are really important. They're just one little sliver
of information available. There are a heck of a
lot and in some cases thousands of other pieces
of the puzzle that are all part of an individual process. So we have that big
executive process structure. It's going to have pointers too. Things like our
loaded libraries, things too like handles
which you can think of as pointers to system resources, any files or registry, keys or values. We may also see
things like mutexes. Mutexes and
semaphores are used by processes in order to maintain meek
access to a resource to make sure that
they don't compete for a resource that maybe can
only be used sequentially. What malware often uses
it for as a means to mark it's territory. We'll see that malware
commonly will will send a mutex so that once it
compromises the system if another piece of
malware, like another kind of similar worm tries to compromise the system again, it will first check for that
mutex because the mutex or it's also called mutants, the mutex or mutant is
not, is on a system, it will just silently die, making sure that it
doesn't repeatedly comprise this thing's system, taking
up too many resources and eventually causing
it to maybe crash. This is the old problem if
you remember way back to the Morris worm that did that. And then the final thing we
might look for is things like network sockets. Network sockets are
interestingly enough tied into individual processes and will
be able to leverage that, to view what kind of network
activity a process is doing. So let's go back to
that Sobig memory image. You remember that it
Redline that winppr32.exe, we saw it was running
from the Windows directory and had a user security
identifier related to it. The reason why that
malware risk index was rated 100 by Redline is this. It turns out that Redline
recognized a couple of commonly known mutants for that
piece of malware. Now, taking a look at these, you're
probably looking and saying hmm, I don't think I would
have recognized those and I'll be honest, I wouldn't
have recognized those. That Sobig is a little,
I think it's 2008ish. So I haven't seen
Sobig in a longtime and I certainly wouldn't
have been up to date on what the individual
mutant was for that. However, the great
thing about this tool is it has some
built in heuristics, so I didn't have to memorize
what the mutant is for Sobig, it actually found it for me. In this case I got lucky,
it found it, said it was a malware risk index of 100 and made it very clear that
I should focus my efforts on this process than
many of the other 40 or 50 processes that
were on that system. Looking at a different
piece of malware, Conficker. So again, running Redline on Conficker
we get a malware risk index of 100. You'll notice that the
rest of the process details are actually legit. This process is legitimately
called services.exe, it's running from the
system32 directory where we would expect, it's parent services.exe, it was actually
started to correct time so everything is
legit about this. Even the security identifier
you'll notice looks like a system security
identifier and not like a user context based shit. The reason why
everything looks legit is everything is legit. In this case, Conficker injected
itself into a legitimate servicehost.exe process, in effect camouflaging itself. However, Redline identified
a couple of mutants that were well
know for Conficker and then elevated the
malware risk index in that case. Now again, I can't say I
would have visually
recognized those mutants myself. Here is the output of opening up that
servicehost.exe process and looking at the
specific mutants that are tied to that process. Now, while I
wouldn't necessarily maybe recognize those mutants, one of the things
that really is nice about Redline and also
memorize when we get to it is it has something called this least frequency of occurrence. Most of your elements, our memory structures that
you'll see within this tool can be sorted by their
level of occurrence, how many times they're
referenced throughout the memory image. Ones that are referenced very few times tend to be more likely to be malicious. The idea goes all
the way back to what we call the
root kit paradox. This is an idea that malware wants to run but it also really wants to hide and also the fact
that malware should be the least common
thing on your system. By definition malware
should be anomalistic. As such individually
relied upon, especially kind of mutants or maybe it's just like
registry keys it's added, or drivers, those should
really only be referenced by the malware and
by nothing else. As such their occurrence
rate should be much smaller. So in this case,
there are about six very small referenced mutants, three of those
six turn out to be evil mutant related
to Conficker. Those two that Redline
found and then that third is actually a randomly
generated mutant, it's based on the process ID. The Conficker just creates this on the fly. So actually three out
of the six mutants were actually bad, so
just focusing on those, maybe grabbing one
doing a Google Search might have led us
also to Conficker, even if the initial
signatures weren't built in and that's important
because I don't want my analysis to only
rely upon signatures because as we know
from the Ami virus, that kind of problem, if
you're only signature based you're actually gonna be
missing a heck of a lot. So having techniques like
LFO to go through the sometimes thousands of pieces of a process and sort
them by order occurrence and looking at
just or focusing on the ones that actually occur
the least actually is a very very powerful concept. The next tool I wanted to
introduce to you, Memoryze. Now Memoryze is essentially
the same as Redline. They both use the same
backend analysis engine. They just have two different
kind of GUI interfaces. So Memoryze was the original kind of free GUI
memory analysis tool. It's unclear whether
Mandiant is gonna continue to support both or if they're
gonna move all their development into Redline but in effect, they have
the same analysis engine, they just have two
different front ends. So Redline is the
newer of the two and I think it has a much
more kind of understandable front end. The front ends of Memoryze is
something called Auditviewer. So you actually have two
pieces when you go to download Memoryze. You need to get
Memoryze and Auditviewer in order to use it. It also has the malware
rating index, it also has the least frequency
of occurrence. So essentially the same idea
and just a little different way of looking at it. And it's actually more built for reverse engineers. It was kind of a team was led
by a guy named James Butler and Peter Silverman,
two extremely smart guys who were really kind of
legends in the field of memory analysis and I think this first, like
at least Memoryze to me, you can really see it
was a tool that they made for themselves, they
made for people that immediately understand
the kind of internals of what's happening in memory and then Redline was
a subsequent effort
that said well, hey let's open this up to
the mere morals out there and give them a little more usable interface, at
least in my opinion. So here's what
Memoryze looks like and this is getting back to how I actually found
that scvhost.exe. So I was scratching my head,
I knew there was something wrong but I just didn't
know quite what it was. So what I did is I started
going to the handles and I got to the loaded DLLs section for that
process and I saw this. So I sorted by occurrence and I thought, but I still didn't put it
together that it was named scvhost. My eyes just still
saw it as svchost but I started looking
a more and I said well if it is actually
scvhost there should be much much more, many
more occurrences than a single occurrence. I should expect this to be
occurring 50 or more times because servicehost is all over an operating system. So the fact that there
was only one occurrence, made me look at it deeper
and I finally got the big slap in the back of the head, said oh my god, it's
actually misspelled as you guys picked up
pretty early in the game. This is a great example of
us layering this idea of options. I don't
have to be perfect, I just have to continue
to work through my process and even if I miss it
in the process with, something might be least
index of occurrence or my heuristics or
my known signatures may be the key that
unlocks it for me. So then we're getting
into network artifacts. Network artifacts or
sockets are actually tied to processes. These for a long have been a
big part of instant response. We can often get some very valuable information
by looking at what network connections
have been going on. If you think about it, similar
to malware wants to run and wants to hide, malware
also wants to communicate. It often doesn't do malware
much good to compromise the system if it can't
link back to it's attackers or it's commanding
control server. So as such we usually
get very lucky at finding network based artifacts. One of the great things
about memory analysis is that we're not only
gonna find evidence potentially of the current network connections that
were happening when we took memory image, you can find evidence
of residue of old network connections
that are just being literally marked as unallocated
and are floating around in memory but our memory
tools are smart enough to search for those
and pull those out. So when we find
network artifacts, what we're really looking for are again, anything anomalies. We're looking for strange
ports, just like we used to, an idea of looking for
anything that's communicating on a strange port or
maybe listing at a port that would indicate maybe
some sort of a backdoor. Certainly identifying
things like the IP addresses or domain
for these connections are going to. Are they going out
side of your network? Should they be? are they going to things
like known blacklisted IPs? I can't tell you how many
IPs I've thrown into Google from a memory analysis
engine and seen immediately that they link back to
something that's been blacklisted by Conficker or timed back to, I don't
know, something like the (mumbles). Then I already have a
nice piece of information that tells me probably what my malware
really is on the system before I even get to that point. The other key thing is that while identifying
backdoors are great, commonly what we're seeing
now in more modern malware is most malware I've seen
is speaking it out now. Doesn't really open
up a port to listen. It typically will be beaking
out to the commanding control servers, mainly to evade things like filtering and to look like legitimate process going out on port 80, or SSL for instance. So that's why
looking for these old processes or network sockets
that are sitting out there when we identify that we
can help to the commanding control center. And then finally, should this process really
even be communicated. Sometimes you'll see a
process, I don't know, like services.exe or Lsass.exe that's actually communicating
with the network socket open. Well that should be a
big red flag that that shouldn't be an issue. So if we look at these, this network listing here,
this is actually Metasploit. This was a meterpreter module. Those of you that have
dealt with Metasploit know that it is designed
to be extremely silent. There are very few clues
that meterpreter gives us when running on a system. But one of the really
interesting things is that we just got lucky and one
of the network connections for that meterpreter
process actually identify it and we see that
there's a port 4444 which should be kind of a clue of something that
looks a little strange and it turned out
that that was tied to a servicehost.exe process at process ID 1012 that
had been injected in with the meterpreter. So even the most
stealthy of malware can sometimes be undone by
things like network artifacts. That kind of leads us to
our next tool which is HBGary Responder. So Responder is a
commercial product that HBGary's had
for a longtime. They recently, I think
about nine months ago, release a community edition. It's a GUI tool. It's a nice free interface and it's obviously not as
fully featured as their commercial tool, for
instance, it doesn't have additional VNA which has a
lot of built in heuristics and signatures. It has interesting things,
we'll see that it'll automatically try to
recover URLs, the domain as well as usernames, passwords and even memory mapped files like images or documents
that relates to a process, which is kind of a new feature. It does have a big limitation
that it will only work on images of six gigs or less. So just be aware of that and
a link for this tool is here. So let's look at HBGary and in particular, I'm gonna look at Z botter Zeus. So Zeus is an interesting
piece of malware. It is not stealthy by
any stretch although, if you're looking at it from
just the system perspective as a user you'll probably never
have an idea it's running but in a memory image,
man, Zeus sticks out like a sore thumb. So looking at the network
artifacts we see that a strange port, maybe 29220
is an uncommonly seen port, could be legit, maybe not. Since it's tied with
an svchost.exe process that kind of makes me look
a little deeper into it because I know that
often malware will want to play or kind
of utilize svchost to camouflage itself. I looked at the internet history for this process and found the following. You see the highlighted items. For instance, I saw
an interesting IP that was embedded
within that process. What really got me was
this idea of the 75.bro, wasn't quite sure what that was. We also see maybe some
banking site information. Now it's used as a
credential stealer and or kind of
banking site Trojan. So that makes sense. You could do a search on
75.bro or the IP address. It ties back to Zeus
almost immediately. So that was a really
neat indicator that helps me very
quickly figure out what was up with the system. So that same memory
image though, one thing I like to
point out is that these tools, while they
kind of do the same thing, they're all cleared differently. So if it's an important
case, you're just not finding something, consider actually running
the same memory image through multiple tools. Some of them do, well,
all of them kind of have their pros and cons
or what they do best. Now for instance, I find
that the Volatility project, it's network scanning or
network socket scanning is probably the best of breed and as such, I ran the same
image through Volatility and it actually came up
with some old connections and in fact, we actually
see the connections to that IP address and we actually
see the ports they're going out to. So we've got servicehost.exe beaconing out to a
foreign IP at port 80. Hmm, that is a
classic indicator of kind of a outbound beaconing
of a commanding control server. And we don't that with
any of the other tools, actually volatility was the
only one that found that (mumbles) which is kind of interesting. So now we're gonna get
into code injection. Now code injection is
interesting in that the most advanced malware
out there is commonly using code injection. We're seeing it everywhere and it's an incredibly simple really and effective way
of cloaking malware from the files perspective
but interestingly enough, while it's probably one of
the more effective methods from a file system or from a user perspective, it's actually incredibly easy to find code
injection from a memory image. So the way DLL injection
works is we see a couple of Windows API functions there. Commonly, something
with administrator
or more importantly the set debug privilege
within Windows will go out, target
a specific process, will allocate space
within that process, load malicious payload in
there, commonly something like a DLL and then run it and it
runs under the context of whatever process it
was injected into. Very nicely kind of hiding
under a running process. So you run a process
list on a running system, you don't see anything
out of the ordinary, you don't see that winppr32.exe that we saw with
the Sobig malware. An alternative to DLL
injection is something called process hollowing. With the way this works is that the malware starts
a new process, hollows it out or
carves out whatever code was part of that
original process and then just
stuffs it's code in. So it's using kind of
the overall process container to run
new malicious code. The good news is that there
are techniques that have been developed out there
that make code injection really easy to detect and all
these tools that we'll talk about have the ability to search
for it and to identify it. And really the way to doing
is they're looking for memory pages that
belong to a process but are unnamed and marked as executable or commonly
read write executable which is an uncommon
thing for a memory page. Then the final thing you
can look for in these pages is to take a look within
them and see if they contain something like a DLL
or an executable. So I think Justin asked, how
does meterpreters migrate, inject itself into
another process? Well very simply what
it does is it either uses an exploit or
just something like escalation in order to
get admin privileges. Once you have admin
privileges on a system, injecting yourself
into any process is fairly trivial. So then it's literally just
pick the process you want, go through the process
I just described and now the meterpreter is running
directly within that process. So let's look again at Zeus. So Zeus has a ton
of DLL injections. So here is one variant of Zeus. As you know there are a lot
of variances of all these malware but malware in
Zeus will copy itself to a specific file within
system32 directory. It will not only
inject one process, it injects almost every process. Then it creates an auto run path or entry in order to
maintain persistence against reboots, creates some additional
directories that throw some of the files in. It has a very well known
mutant that we could look for and it does a ton of hooking. It is very much a kind of a busy piece of malware. And of course Zeus is, as
we've mentioned before, is a credential stealer. It's looking for everything
from protected storage to certificates that
you have on the system, to any passwords and
usernames on a user system and it is extremely effective and extremely
pervasive out there. So let's imagine that you're
looking at a computer, you think it might be infected
by Zeus, you're not quite sure but you decide to grab a
memory image and load it into Redline. Now as you see, the pro tip, Memoryze has
something interesting. Any process that it sees
that is likely injected will actually be shown in red. Hmm, I wonder if this
computer is actually infected? (laughs) Actually enough, also Spyeye. If you're feeling
with that, Spyeye does much of the same
kind of techniques. It injects into just about
every running process available. So this is an obvious and the real
question here is, man, where do we start? It looks like every single
process is injected. And we might start up
at that svchost.exe because you see that actually
has a number next to it. That's the malware rating index. So that is the item or
the process that Memoryze believes is probably
the most malicious. And we can look within there
and we can actually see the memory section. We could also just export out that process and
all those memory sections and start to do our analysis
from that perspective. So Memoryze makes
it incredibly easy. How about Stuxnet? I think everyone's probably
familiar with Stuxnet. Stuxnet is largely thought
to be one of the most advanced piece of malware
we've seen in the while. It was the malware
thought to be targeting the Iranian nuclear reactors. So here's an image
with running Stuxnet and this is actually
a HBGary product and looking at it you may see something
anomalistic. We see actually three
lsass.exe processes running. Anybody wanna guess
which one is the legit and which ones aren't? To be honest, at this point
it would be a complete guess. You could guess the lowest pick and that is actually
the case here but that's not always the case. So it turns out that
these two processes are the illegitimate ones. These are the ones that worked, installed or initiated
by Stuxnet, hollowed out and then it's code was
actually placed within there. And as you could see there
are memory analysis tools quickly show this anomaly. One way we could tell whether
which ones are legitimate or not is if you look at
the additional information about the process. So for instance, the loaded
DLLs of the legitimate ones we see here on the left and of the illegitimate. You notice that
there is much fewer as you load the DLLs on the malicious lsass process. You also might notice this. One of the DLLs actually doesn't
even have a name. It's what we call an
unnamed memory section. This was a name that just HBGary's product gave it. So it's just a memory,
looks like it has a pe file port about executable
and it's at this address. That's a great example
of an unnamed memory page that's siting out
there tied to a process which is almost
certainly malicious. So one of the most advanced
piece of malware in the world and we found it in maybe a minute or two. Which leads me to the final tool that I want to
introduce, Volatility. Volatility is probably my favorite
of all the tools. I think Redline is a fantastic
tool for everyday analysis but when I'm getting down
to hardcore analysis, nothing has the
flexibility of Volatility. It's an opensource
based framework. As such, there's a wealth
of different plugins for the framework. Up until very recently,
it only worked on XP systems. With the new version
that's now been expanded to a wide range of Windows related systems, one of the only drawbacks of Volatility right
now is that it only currently has 32 bit support. Now 64 bit support is in
beta, it'll probably out by the summer I suspect but just keep in mind that
if you have 64 bit systems, you won't be able
to run Volatility on
them unfortunately. Now if you want to
play with Volatility and you don't want to go
through the process of installing and configuring it, keep in mind of the
SANS SIFT Workstation. You can download it
freely from the SANS computer forensics site. Actually has it kind of
preloaded with a lot of the most popular plugins. So the only downside
of Volatility is that it's a little bit
of a learning curve. It's not GUI based, you
have to kind of understand what the plugins do and you
have to have a little more knowledge of what's
happening behind the scenes of what those memory
structures and items actually are and
how they interoperable. Once you get there, it
is a fantastic tool. My favorite plugin for
Volatility is called Malfind. It's actually written by Michael Hale Ligh who
wrote also what the Malware Analyst's Cookbook. Fantastic tool. It essentially goes and tries
to detect code injection. So running this on Stuxnet,
you see the command line at the top. It immediately finds
those two lsass process that we saw before. You'll notice that
they are paged, execute read write. You look on the right
side of each process. You also see how it tries
to give us a little bit more information, exactly
what's in that process. Those of you that are used to doing file
signature analysis on this will notice
that MZ header is the file signature
for portable executables. As such it looks like
this is a page that is unnamed, just part
of a running process and has a loaded, it
was marked as executable and has a loaded DLL or portable executable file and all very strong factors
indicated as malicious. The other cool thing
that Malfind does is it dumps them out and you
can see the location, it dumps to location. So then we have these to
actually take a deeper look into to run strings on or maybe to upload to something like virus total
or to run against our own antivirus product. Interesting enough at Stuxnet, Stuxnet's been
around for a longtime and I was shocked
when I ran it against virus total and
like two out of 43 antivirus defenders
actually found it. And if you look at
that Comodo entry, the reason why Comodo found
it wasn't because it was necessarily Stuxnet,
it was because that the executable that was
in that memory page actually was packed with UPX which is a commonly
used hacker for malware. Can't explain why. I expect they're getting
better and better each month but this just shows you that memory analysis can
often find things like our traditional
defensive techniques may not immediately. So putting it all together,
we have all these items that I talked about. I know I'm going kind of
rapid and rapid fire here. We only have an hour. I'm trying to stick as much
as possible to that hour. But the bottom line is how do we actually
put it all together and the way I've got
this presentation is going through the steps
in the order that I would typically look. The first thing you do is you identify processes
and review them. Now that's a given. If you find something
that's anomalistic just paste on the parent
ID or the process name or the process command
line, well hey, that's the easy way to find it. Next step might be look
deeper into these processes that you just aren't sure about. Reviewing things like their
loaded DLLs or their handles. Of course, the problem with
this is that there could be a thousand or more of those and a lot of them aren't
recognizable to the average mere mortal. But doing things like
digital signature checks and they're built into
Redline and memorize when you went on a live
system can be helpful. Also least frequency of
occurrence is, I think, a life saver. And that's why I like
those manual tools so much is because they have
that capability. They're the only ones that
have the ability to do LFO kind of out of the box. Next step I would
do is something like looking for evidence
of code injection. So we saw how that works and
how powerful that can be. Then look for network
connections or ports, looking for anything
anomalistic, maybe looking for streams within the
process for any HDP or FDP connections or any type of IP or
domain name that we can do. Sometimes you can
use things like regular expressions
to do that or you saw the HBGary
tool that automatically pull that information up. Then I would look
for signs of hooking. Now, we don't have enough
time to cover hooking. You'll notice that there
is a bonus section, the end of my slides that cover kind
of how these tools actually identify hooking. Well, hooking is a
very common to rootkit and is also relatively
easy to identify using memory analysis. To be honest, it's the easiest
way to identify rootkits is identify what they
have hooked on a system. And then finally, you probably
have a series of processes is if in fact this system
we're looking at is in fact intruded upon or has malware, you probably have one or more processes that you still
believe may be malicious, the final step is to export them out. One of the great things about
this tool is you can literally just export out the entire
process or individual memory sections and then
continue to do our analysis. So AV runs strings
against them or eventually, pass them to
someone like a reverse-engineer and put them into something
like a disassembler and try to review the code
and identify what it's doing. So there are some resources
that I want you to be aware of. This is a definitely Windows memory
analysis 101 class. I've tried to keep
it at a high level but also I wanted you to realize kind of as Chris is saying, this idea that it really is nothing to fear. Really, it's getting
to the point where for much of the malware
you're likely to come into, if you get a good memory
example, you have a pretty darn good chance of finding it. Do you wanna learn more or I can't say enough about that
Malware Analyst's Cookbook. It's the best book I've read
in the last couple of years. I think it should be
on everyone's shelf. Fantastic, it actually
has a quarter of the book just related to memory analysis, specifically
Volatility memory analysis. So if you wanna
learn Volatility, probably the best
book available. James Butler and Greg Hoglund. James Butler actually
was involved with Redline and Memoryze. Greg Hoglund, of
course, with HBGary and the Responder product, they wrote a book on rootkits which is a great one
if you're interested in those kind of more
advanced malware and then the Windows
Internals book is a must have when you're trying to learn what all of these things,
like executive blocks are and process environment blocks
and all the things that I glossed over due to time and
list investigation. Oh, I'm sorry, in
this presentation. And then finally if you're
familiar with the SANS courses we have a pretty
robust curriculum. Our 408, 508 and 610 courses all cover different
elements of memory analysis at their core
fundamental levels. So if you're taking one
of those or you have taken one of those probably
know about that. Here are some links. I won't leave this up
for too long because I'm pretty much out of
time but get that PDF. These links are fantastic for really getting your feet wet
in memory analysis. Many of them go
through step by step how a given memory
sample was analyzed. Also the tools, the
Memoryze and HBGary tools as well as Redline all
have good user manuals. So don't forget to look
at the user manuals, this can be very helpful. And then volatility has a
great wiki if you look at the bottom reference there under
the code.google.com link. And as mister Parker mentions,
we do have some classes coming up. These is the forensics 508
is gonna be offered as a mentor based course
out of Sydney. So if you're in that area
definitely look that up. We'll be teaching 408 in Singapore in March
and then in Brisbane in the May timeframe. So we're starting
to kind of ramp up our forensics curriculum
in Asia Pacific, love to see you at
one of the courses. Of course, if you
have any questions about this presentation
or about the classes I'd love to answer them for you. Here's my email address. Don't forget that we
have the bonus section. And I'll stick
around for questions. I realize that some
of you I've been told around your lunch breaks. Lucky you, I'm about
to go to bed but I'll stick around
from questions. I did see one question
that I didn't forget, was actually asked by
Justin and he asked, said he mentioned
rootkits in hiding. Is it possible that malware
rootkits to affect a memory dump? How trustworthy are
the live memory dumb on a compromised box? And my answer to that is that a live memory dump is
typically pretty trustworthy. Everything in our field
is measure countermeasure. And so every time we
get an edge and I think very much in the
memory analysis part, we have an edge right now. These tools give us a distinct
edge over malware creators. However, there has been
research and there is a proof of concept for a technique that would be effective
potentially against taking memory dumb. If you're interested,
it's called Shadow Locker. What it does is
something pretty clever. It monitors processes and memory and if it sees a process
sequentially accessing memory, it will assume that that
is probably a memory dumping tool and will
try to page itself out. So it'll page out the
malware, most of the malware to the page file, hoping to actually evade
that live memory dumb. Now I don't know if
any malware in the wild that's been
discovered with that, unless there's just
something running around that's very custom. I don't that there is anything, at least that we're aware of that currently does this but it is something to think about and while I didn't have
time to cover it here there is a new cutting
edge of forensics and that's this idea of live memory analysis
and I didn't mention it. This idea that typically when
we think of memory analysis, we think of running it
against a static memory image that we took from a system,
we then put out some, drive bring over to our
friends at Workstation. However, the better
or more robust way is to actually doing a
memory audit on the system that's running. The reason why is that you
get access to the page file among other things. You can also do things like
digital signature checks that you can't do from
a dead memory image and you have a better
chance of heuristics and anything actually
catching something. So that's a little bit, very much
in the cutting edge. I think people are still
gonna have to get really used to that. The best tool for doing live
memory analysis is Redline. And I actually have
a post up on my blog at ForensicMethods.com, you can do a search on Redline, there you'll see that there's
a post on the pros and really there's not many cons to doing live memory analysis
and that's where I see things moving and that has a
very good chance of defeating that shadow walker
technique of paging out because it's also taking
into account the page file, which can be very useful. And I think I answered
the question about what tools would do
live memory analysis. The only tool that I know
of are Redline and Memoryze. Redline has a
really neat feature. You can dumb your
Redline to a thumbdrive, run it on the command
line so you have a much smaller footprint
on the system and you'll actually do
not only a memory audit but it'll also do a memory
image so you get the best of both worlds. You get like a complete
memory audit as well as the memory image, then you come back and
import your memory audit back into Redline and you can
do your analysis then offline and it's a much more
robust way of going about. That's where I see a
lot of research moving. Any other questions? Well, I appreciate
you all coming. This is a great showing. As I said, I'm excited. I guess it's a little, just a few weeks and I'll
actually be in Singapore so I'll be kind of nearby. If you're in Singapore, I'll
be giving this presentation live, actually, in
Singapore at an ECCI event that's tied to the SANS event. So if you're in Singapore, hit me up and I'll
give you the details. It's also on my blog and I'd love to meet you in person. And again, thanks to AISA
for giving the word out. And thanks to you all
for taking the time. So find the bad guys.
