- [Jason] Hello everyone
and welcome to Memory Forensics for Incident Response. I'm Jason Keeler with
the SANS Institute and I will be
moderating this webcast. Today's featured
speaker is Hal Pomeranz with Deer Run and
the SANS Institute. Before I turn
things over to Hal, the Q&A portion will
take place at the end of the webcast but
please feel free to submit your questions
at any point by using the chat window. Right now I'd like to introduce our featured speaker,
Hal Pomeranz. - [Hal] Hey everybody. I hope your Monday isn't
treating you too badly. I honestly expected
fewer people to be here, it being a Monday and
all, but I'm delighted to see everyone. Let's see here, so I wanted to kind of explain a little bit where this talk came from. Last month I was asked
to give a presentation for a group that, they're primarily
interested in audit, maybe IP management,
they weren't as, sort of hardcore tech
meanies as the typical SANS audience, and
I wanted to say something interesting
to them about memory forensics. Because I sort of
like the challenge of trying to present a
deeply technical subject to a bunch of people
who don't sort of grovel around in the
internals as much as I do. And what I came up with was this notion of creating
a presentation that was about, you know,
what is the 20% that gets you 80%
of the way there? What are a few simple
heuristic things that you can do
with memory analysis and memory forensics to detect, you know at least
malware infection so that you can, you know alert the incident
response team at your organization or
whoever deals with that for you that, yes
we have a problem, yes the system is infected,
and we need to take action. Because modern malware
is difficult for a lot of organizations to even detect. We're kind of falling down on, just sort of basic,
is it infected? Because modern malware
has frankly become quite adept at
hiding itself from typical endpoint
protection schemes that most enterprises are using. So that's really the
goal of this talk is, something that you
could take back and give to, sort of the
frontline people, your systems admins,
your IT operations people and say, hey if you think
there may be a problem with the system, check
these three or four things really quick and
here's how to do it. And you don't even really
need to understand the sort of deep technical stuff, although we'll talk
about that a little bit as we go along as well. Throw your questions
into the chat window at any time and I'll
try and respond to them as I move along through
the presentation. So, why memory forensics? Because everything goes
through the memory. Everything happens in memory. You may have malware
that comes in as a packed and obfuscated executable,
but in order to execute, it needs to
be fully unpacked in the memory. All the actions that
that malware does on the system making,
network connections, modifying data in the
registry, all of that stuff, that also is reflected
in the memory, and if you know how
to look at the memory in a systematic fashion
you can actually, very rapidly detect, sort
of classic patterns that malware uses
over and over again to affect your system. There's also lots of
other stuff in memory that we talk about in
more detail in other SANS classes, you know,
chat, email, encryption keys, all of this other stuff,
which we're not going to go into this presentation,
but there's just a huge treasure trove of
information once you start learning how
to unlock the memory of a system. So, modern malware, these days it's very good at hiding, as I mentioned from, sort of traditional
endpoint protection kinds of things. It can have a very
minimal impact or footprint on the
disk in the system. In fact, we're starting
to see malware that is purely memory
resident, or at most, there's a small
loader that lives on your hard drive as a sort of persistence mechanism,
but the actual functioning malware
is downloaded from somewhere else
into the memory of the computer. And so sometimes
analyzing the memory is the only way to
see what's going on with the malware. Also particularly nasty Packers and things like that can
be hard to get through. When you're doing
static analysis you may be able to dump
the unpacked executable directly out of memory
and analyze it there. So there's a whole
bunch of stuff that, the malware when it's sort
of in its dormant form or just sort of living
on the dead disk drive of the system is hard to detect but when you look at the memory of the system, it
lights up like a Christmas tree. You're sort of taking a outside, almost you know
hypervisor like position to analyze the system
and as a result it's much more difficult
for the malware to hide from you,
and certain behaviors of the malware make it pop up when you start looking
at it with these memory analysis tools. So memory forensics
then is just like some traditional dead
drive computer forensics. It's imaging that is
being able to capture a memory image
from one of several different locations. We'll talk about
that a little bit. And then being able
to analyze that image in a sensible way. Now back in the olden dark
ages of memory forensics, I mean you were kind of
hunting around and doing string searching and
it was gross because you didn't have any context
of where these things that you were finding in memory, what process memory the
string hit was part of or any of that. The good news is that,
particularly in the last, I would say, three
to five years, we've started to see
some really, really amazingly useful
memory analysis tools coming out, and I'm
going to talk about a couple of different
ones, mainly it's Redline and the open source
Volatility project which are just amazing. I mean, they pack a
whole bunch of useful analysis that's sort
of done for you, so it's almost like
having somebody sort of standing over your
shoulder and saying, "look at this, look at
this, look at this." I mean you don't
need to have this deep knowledge of
Windows and Charles to figure out what's going on. And so you use these
tools to look for stuff that's interesting,
you know for some value of interesting. And I'm just going
to quickly point out, some of the stuff that you
should be interested in as far as looking
for malware and signs of a malware
infection on a system. All right, so you
know, step one is getting the memory
dump somehow, right? And there are many
tools out there, some free, some
commercial, for dumping the memory from a live system. And in this conversation
I'm just going to talk about Windows. There are similar tools
for Linux and Mac, but I'm just going to
focus in on Windows malware for right now. So there's a bunch
of different tools out there, some of which
are listed on this slide. DumpIt is a easy to use tool from MoonSol's. MoonSols is actually
the company that was put together by Matthew
Swuiche who also wrote Win32 and
Win64dd which you also see listed here. But DumpIt basically
takes the functionality of Win32 and Win64dd
and puts it into a sort of nice wrapper. So you basically you install DumpIt on a thumb
drive and then you plug the thumb drive
into the system whose memory whose
image you want to dump and it dumps the
memory image to the thumb drive so that
you don't end up you know potentially forensically corrupting the disk
of the system by dumping the memory image there. So DumpIt is cool, I mean
obviously you need to have a thumb drive
that's large enough to handle the memory
dump in the system so if it's a server
and it's got 32 gigs of ram in it, you
need a thumb drive, a 64 gig um drive to hold the memory image plus
the dumping program. But that aside, DumpIt
is a quick and easy way to take a memory image
from a live system. Win32 and Win64dd
are nice as little command-line tools. You can run them over
PS Exact remotely. They are capable of
dumping the memory image over the network
to another machine, so that's useful. The Redline tool that
we'll be using for analysis is also capable of
taking memory images. You can also install
Redline as a portable agent on a thumb drive just
like DumpIt, so it can dump the memory and
analyze the system from the thumb drive. There are a whole bunch
of other tools out there. HBGary has a has a
free tool, F Response, which is one of the
tools we teach and Forensics 508 is a
nice enterprise tool. It's a commercial tool. The enterprise forensic tools like encase Enterprise
and things like that now have the ability
to dump memory. So there are lots of
different ways for dumping memory
from a live system. In general you're
looking for something that dumps memory
into what we call a raw dump format. It's basically just
a byte for byte copy of the memory. To the extent that
people are using virtualization you
know I mean if it's one of your virtual
machines that's possibly gotten
infected, getting a memory dump from a
virtual machine is usually really straightforward. Best if you can at least
temporarily suspend the virtual machine. But once you do
that, there's usually a file in the directory
where the virtual machine is kept that is
usually a memory image of the system so you just
suspend the virtual machine and copy that file
it's the ,vmem file under VMware you know and so on. The one exception
actually is VirtualBox which unfortunately
the .sav file the VirtualBox uses
for storing memories it's only a partial memory dump. It's actually only the
active memory pages in a virtual machine
that at the moment you suspend it, and
unfortunately that means that the VirtualBox
dead save files are not useful for
the kind of memory analysis tools that
we're going to be using. So for VirtualBox
virtual machines you'll need to
actually sort of treat them like a live
system and use a tool like you know DumpIt
or win32 and 64dd to dump the memory
of the system. But for all the
other virtualization environments you know, again, suspend the machine
grab a copy of the memory file and away you go. And that's you know
how we do a lot of testing is you know
we sort of test virtual machines and infect
them with various pieces of malware
and then you know capture the memory
image that way. And it's just a lot easier. The other thing I
wanted to mention real quick before
we actually get into the analysis is, you
know in a lot of cases, I deal with law
enforcement a lot, and so very often I'm dealing with just
a dead disk drive and I don't get a memory
dump along with it, and you know that
it's too bad really but unfortunately a lot of law enforcement agencies they don't do live forensics. So one thing I'll say is that if it's an image of something like a laptop where
there's a hibernation file, that hibernation
file is a compressed memory image from the system at the time it was
last hibernated and there are tools you can use to convert that hibernation file into a raw memory
dump that you can then use you know something
like Redline or Volatility on. Volatility in fact has a module that will convert
hibernation files. Now I will say
that my experience with the Volatility
module has been sort of hit and miss,
sometimes it works and sometimes it doesn't,
but you can always try it and if it works, great, you know if
it doesn't work, there's a commercial
tool from MoonSols called hibr2bin which
converts hibernation files and that one always
works in my experience. It's a commercial
tool but frankly, it's not that expensive and you know the MoonSols people
deserve your money money so hibr2bin
works works well for unpacking hibernation files. I mean this could
be really useful. A colleague of mine was
working on an investigation where they found a
TrueCrypt container in the disk image that
they were looking at. They didn't have the TrueCrypt passphrase or keys
to unlock the volume but they were able to unpack the hibernation file, get
the TrueCrypt keys out of that memory
and actually unpack the volume that way. So you know, like I said,
this kind of memory forensics is useful for a lot
of stuff besides, you know, malware analysis. So you know getting into
encrypted volumes is another useful thing
that you can do with with memory
analysis, and with hibernation files like this. So just wanted to mention
that before we roll along. I had a question. Can malware prevent
a tool like DumpIt from getting the true
contents of the memory? You know, sort of like can
the malware hide itself from a tool like DumpIt? We certainly haven't
seen malware that does that yet, it's hard
for me to imagine how malware would do that. I mean, if I suspend
a virtual machine in the middle of a
session and then grab the .vmem file, it's difficult
for me to imagine how the malware would
completely hide itself. Now we'll be talking about
some kernel level root kits that, you know,
do some hiding, in a process hiding
and things like that. And you know,
potentially the malware could crap up the in-memory data structure that's
used by the system to make it more
difficult to find, but you know the malware
has to be careful about that because it
doesn't want to render the system inoperable either. Got another question
about you know, what tools should
you use for which version of windows and
which service pack version? Generally speaking,
all the tools that I mentioned work
across all sort of modern windows variants. There can sometimes
be a brief delay after a new windows
release comes out where the tools
need to catch up, but in general the
tools track, you know and work on all
Windows versions from at least you know XP forward. As you can tell by
the names, win32dd is for dumping 32-bit
systems, win64dd is for dumping 64-bit systems. But generally they work
across all windows variants. All right, so let's
talk a little bit about some malware
techniques and ways in which you can detect them
with memory forensics. We're going to start
with DLL injection which is a super
common mechanism for malware to infect the system. And I just wanted to sort of quickly explain what
DLL injection is. Again, you know, it's
not absolutely critical that you understand
what's going on here because I'm just going
to show you a couple of heuristics that you can use to detect the DLL Injection even if you don't
understand the mechanism. DLL injection is always
a sign that something is bad on your system. So you know, it's one
of these red flags that you can quickly
look at on a system. All right, so before we
talk about DLL Injection, let's talk about what
happens normally, right? So you've got a process which is the blue box here right? And it's happily running
along doing whatever that process does,
maybe it's a web browser or something like thatm
and it needs to interact with the operating system,
it needs to interact with the operating
system kernel which is the orange box at
the bottom of the slide. So generally
there's a whole API, a Windows API, that
process is used to interact with the
operating system. They might call,
an API call that, I don't know, let's
just say it reads some values from
the registry, right? That library call
that the process calls interacts with a dll,
a system-level dll in order to talk
with the kernel. So the library call
invokes a routine out of the dll which
in turn communicates with the kernel, says,
"hey give me this value out of the registry" and
the kernel returns that value which
is then you know presents in some
nice way back to your application. And so, that's the way
things normally work. You want to talk to
the operating system, you go through some
routine in the dll which talks to the OS
kernel to get the information that you need. So when we talk
about DLL injection what you got is,
malicious software has infected the system
and achieved some level of privilege on the system. What it does that it proceeds
to infect other normal applications that are
running on the system. And so DLL injection literally means that sort of unsuspecting applications that
are run by the user are infected with
new code through the malicious software. The malicious software
literally pushes a whole new evil
DLL, if you will, that's the read rootkit
box on the left hand side of the picture,
it literally, the malware literally
pushes that evil DLL into legitimate user processes. And it does that through
sort of a non-standard loading mechanism
and because of that, that injected code
is actually very easy to detect when you
start doing memory analysis. Now it's not enough though for the malware just to
push the malicious code into the unsuspecting process, it also needs to patch or what we call hook, the
legitimate routine in the system DLL so
that anytime the user process calls, you know ntdll, in this case, the
routine and ntdll it is in fact diverted
into the malicious code that was injected by
the attacker, right? And that patching
or that hooking is also detectable
when you start doing memory analysis. And when you see that happening, particularly when you see that happening on a large
number of processes on the system,
something has gone seriously wrong, right? And so way you detect
the DLL injection is one, detect the injected code because it's not
loaded in the normal loading mechanism and
two, detect the patches or the hooks that are made to, you know, multiple routines
in multiple system DLL's. And the combination
of those two things or any one of those
two things allows you to easily sort of at
least know that something bad has happened to your system. Analysis beyond that
point requires additional deep knowledge, but
you know it's very easy to detect that
something has gone wrong on the system
just by looking for these things. So let's talk
about how we detect the bad stuff. So we're going to use a
couple of different tools. We're going to use
Redline which is a Windows GUI based
tool that was created by Mandiant. And let me just disclose before we go any
further that I do a lot of subcontracting
for Mandiant so I'm not just, you know,
pimping the mothership. I actually think that Redline is a really useful
tool and a great way to get started with
memory analysis. But I do have a
relationship with Mandiant that I feel like I
need to tell people. So we're going to use
Redline to take a look at injected code. Redline has a whole
bunch of other additional functionality
that we won't have time to get into
but I can at least give you a quick Redline demo. And then we'll be
using Volatility which is a command
line based tool for analyzing memory. And the different
tools do different things better, so it
kind of depends upon which you're more
comfortable with, you know, GUI versus command-line
and also what you're looking for. So the first malware we're going to look at is Zeus. I've been doing a
bunch of investigation related to Zeus malware and ACH fraud in my
professional life, so 'm sort of, Zeus is
an old friend for me. And I have a memory
image of a system that was infected with Zeus, actually a virtual
machine, so that I can actually show you
what, this is an older Zeus variant, but
it'll give you an idea of how this kind
of malware works. Now Zeus is a predominantly
used for banking and ACH fraud. It is a Trojan that wants to
steal banking credentials. It also has capable
of doing what we call web injection to intercept log on's for so
standard banking sites it's a keystroke
logger, it has lots and lots of different
functionality. But we'll just be
looking at you know how you detect Zeus once
it infects the system. So let's take a look at Redline real quick. And I'm actually just going to go ahead and
bring up the Redline demo so here we go
we're going to share this application. How this is going to work
with 400 participants, we'll just see. So let's see, let's
bring up Redline. Okay, so if things
are good you all should be able to see
the Redline window. Can I get an affirmation? Like raise your hand
or say something in the chat window. Everybody seeing this? Yeah, kay good. We're getting some pinging. Okey dokey, so... Welcome to Redline. Redline is a sort
of multi pane GUI. If you look up in the
upper left hand corner you'll see investigative steps. And that is basically
a guided analysis. The folks who developed
Redline are trying to guide you through the process of analyzing a memory image. And we're not
going to go through all these different sections but you can really
sort of walk through those steps and it will lead you by the hand for
analyzing the system. Down the left-hand
side you see a list of all the processes. You see a little circle
next to each process. Redline has a scoring
system and basically tries to heuristically
detect badness. If it flagged any
processes as bad, they would show
up as red in that left-hand window. And generally when
Redline flagged something as bad and shows it red, you definitely have a problem. What's interesting
about this Zeus infected system is
that none of the individual processes
have reached Redline scoring threshold for a bad process. So... Let's go take a look at one of the investigative
steps which is, I'm going to go with
the review memory sections DLL. So I'm going to skip
the first couple and jump right into
reviewing memory sections. And you'll notice, if
you look over on the right-hand side, you
can see different views. And I've chosen to view the
injected memory sections, okay, so I've clicked on
review memory sections dll's and then I
selected injected memories sections on
the right hand side. If you see anything
in this window, right? I mean, if you take
those two steps, review memory sections
and view injected memory sections and
you see anything listed in this window
as being injected, your system is infected, right? I mean, you know,
if you talk about, sort of simple heuristics
but you know you don't have to understand
what's going on, just find the evil,
if you see this, you have a problem, right? Because normally
you shouldn't see injected code in this way. Now, I wanted to make
the screen small enough that would fit on
most people's screens. So there's a bit
more to the right. If you scroll, the
bottom scroller in the middle, you can
actually see that, pretty much every process
on the system has a bit of injected code
and this is very typical of a Zeus infection. It injects into
essentially every process on the system and it uses that to keystroke log,
capture credentials, and stuff like that. So like I said, if you see anything
on the screen, you're in bad shape. What's also interesting, if I scroll it back to the left, you'll notice that there's, the second column
is the name column and there's nothing
in the name column for these injected
code segments. That's also not a
good sign because it's another indicator
that the code was not loaded in
the normal fashion. If the injected code was
loaded in normal fashion you would see in the name column a path name to a DLL, right? And I can show you
that if we click over and we look at one of
these processes like, I don't know, Explorer. And we look at the
sections in Explorer you can see in the
name column here most of the code dll's
and things like that have hard drive path
names associated with them all except for the injected one which is not so much, right? So yeah, the fact
that you don't see a backed by disk dll
here for this segment of code is another
sign you've got a significant problem. So these are the kinds of things heuristically that
you can look for, you know, just real
quick to get a sense of whether something
bad has happened on your system. There's a whole
bunch of other stuff that you can do with Redline. And we'll talk about hooking. I actually prefer
to use Volatility to look for hooks, but
you can use Redline to look for hooks as well. With Redline it's kind of more art than science though. So I'll show you
hooking with Volatility in just a little bit. All right, so let's go
back to the slides here a little bit. So that was just a quick
Redline introduction. And let me catch
up on the slides. These are just some
slides I put in for later which kind of
go over the stuff I showed you in the live demo. We had a couple of
questions as well. I got a question about,
can you use Redline to perform live memory analysis or does it only work
on static dumps? Red line absolutely
can be used to do live memory analysis. In fact, it actually
sort of works better if you're on the live system
because it can do things like check DLL signing
and things like that to look for evil on the system. So again, you know, in creating a Redline portable
agent and you can read the Redline
documentation which basically means
installing Redline on a USB thumb drive so
that you can use it for live analysis
is actually a really useful thing. There's another question about, how does it, you
know sort of detect that the DLL is injected? I'm actually going to
have some more to say about that as we move along, so I'll come back and
answer that question again, if people are confused. And there's another question
about, should you use memory analysis
just for infected systems or should
you also use it as part of behavioral
analysis when you are investigating malware? I think it's appropriate
in both and you know when you're
doing malware analysis using memory forensics
is a great way to speed up your static
malware analysis. I mean for one thing
I mean if nothing else you can get the
unpacked malicious executable out of
the memory of the system and you know
then you don't have to fight your way
through the unpacker in order to be able
to start looking at the code. And I should mention that one of the things
you can do having identified an injected
memory segment in Redline like I was
showing you in the demo, you can right-click
on the injected memory segment and
Redline will dump that memory segment
out into a file for you that you can then load into, you know, whatever,
Ida, whatever you want to do for static analysis
and start looking at the code that way. All right, so let's
talk a little bit about Volatility. So Volatility is
open source utility. It's created by
Volatility Project. There's a whole bunch
of really smart people who are working on Volatility, Aaron Walters, Andrew Case, Jamie Levy, Brendan Dolan-Gavitt. These are all people whose names that you
probably recognize. And the cool thing
about Volatility is Volatility is really
an API more than anything for
dealing with memory. Basically it's an
API that lets you easily write Python modules for investigating memory. And so that makes it
possible for people to rapidly extend the
functionality of Volatility just by popping out new modules. And if you've been
paying attention for the last month,
in the run-up to Volatility 2.2 the
Volatility project has been doing their month of Volatility plugins and there have just been some
awesome new plugins that have come out
in the last 30 days which are really,
really cool and the latest version of
Volatility you know it's not just it's not
just for Windows anymore. There's Linux support
including support for android mobile
phones which is pretty neat. And they just
announced MAC support in the last few months. So that is super, super useful. So we've got this great
tool now that's community supported that, you
know, is cross-platform and because it's
written in Python it's also cross-platform
in the sense that you can run it anywhere, right? As long as you've got a Python interpreter and the
appropriate modules, away you go. I mean I personally
use it on the UNIX machine but you know that's just me. I'm the UNIX guy, right? But you can run it on Windows or wherever you want. So, yay Volatility. And I'm just going
to run through a quick Volatility demo on the same Zeus
memory image that I was using for Redline. So let's go ahead and share my putty window here. So I can actually get
onto a UNIX machine and make myself feel better. Okay, all right, so here we are. And if you can't see this window raise your hand. No just kidding. So let's just do a quick
run-through on Volatility. If you're confused and
you don't know what to do about Volatility
you can always get help by doing a vol.py dash, dash, help. I'm going to pipe
this into less so that we can take a look at it. So Volatility , dash, dash,
help gives you sort of the basic command line
syntax, but if you scroll further down
it actually gives you a list of all
the different modules that you can use and
there are literally dozens of modules
and this is actually the old 2.1 version
of Volatility. This isn't even the
latest and greatest release candidate
which has even more modules in it. But for example, API hooks which is one of
the modules I'll be showing you, is a
good way for detecting those hooks in the normal DLL's that I talked about. But there's a whole
bunch of stuff in here that's useful. You can get, you can dump out, you know, basically
the same information you see in net stat, you know, network connections. You can dump out
registry information which is really cool. You can dump out
process listings and a whole bunch of
other really useful stuff that unfortunately
we don't have time to get into. The basic syntax for
Volatility is straightforward. So I mentioned
that these injected code segments of, you
know, are loaded in an abnormal fashion. And because they're loaded
in an abnormal fashion you can detect the
injected segments because they don't show
up in certain data structures or certain
lists that the operating system
maintains when a module is loaded normally. When a module is loaded
normally, the existence of that module is populated in several different
data structures in the kernel memory. There's a Volatility module
called loader modules that lets you look at
those data structures and detect the injected code because the injected
code is loaded but it's not listed
in one of those lists. So let's talk about
how you would run a Volatility module
like loader modules. So it's always vol.py, right? And then the name of the
module you want to run and we're going to
do loader modules. And then you need
to give it the name of the memory image
that you want to run it against and I've
got my Zeus.vmem files of Zeus
memory image, right? And again, the output
of this can be rather long, so let me type
this into less so that we can actually
sort of see it a screen at a time. All right so Volatility
loader modules goes off and it
interrogate those data structures that
I was talking about. And you can see
sort of towards the middle right side of
the output, inload init mem, and inmem
columns, these are the different data
structures that Volatility is looking
at to see whether the loaded module
appears in those lists, all right, and
basically those should always be all true, okay? If you see a loaded module that has a false in any one of those columns, that is generally a sign that that module
has been injected okay? And you can actually
see the base memory address where the injected code resides
and all that stuff. So heuristically
what you want to do is run loader
modules and look for the false's, right? And if you see any false's, that's probably injected code and that's certainly
a bad thing. So, it's Unix, right? So we can do that easily with a little bit of
command line kung fu. So I'm going to run
loader modules and I'm going to grep for the
false's ok so show me any lines in
the loader modules output that contain
the word false, right? So away we go. And on a normal
system this should return no output, right? You should get no
lines of output. But again, you can see just like we saw in Redline you see it almost every process
on this system there's injected code, right? So, you know, again,
if you're looking for a heuristic for looking
at whether your system has bad things
happening on it or not, you know, Volatility
loader modules type to grep false,
if you get any output bad things have happened. So I mentioned that
Volatility also has, the way I prefer to look
for the hooks that we also talked about
earlier, right? So hooking is when
the malware patches the legitimate DLL
calls so that it calls the evil injected
code rather than the normal DLL call. And Volatility has
a module called API hooks which is really good at detecting those hooks. So I'm going to do
instead of loader modules, I'm going
to do API hooks. So the neat thing
about Volatility is, once you understand the
basic command-line syntax, it doesn't take much
effort to run the different modules. It's always, you
know, Volatility, module you want to run, image name, and away you go. So let's take a look at the API hooks output. So the difficulty with
any sort of memory analysis is that
you've got to grovel through the memory
image and you know find the appropriate
data structures and decode them. And so that's why there's
always this pregnant pause when I invoke
a Volatility module because it's doing
all of that parsing. So here we got the
API hooks output and it's showing you
basically unexpected patches in standard
system DLL's. And there's a lot of output here including some
disassembly of the first you know 20 bytes
or so of the hooked function which can
be useful if you know assembly language,
but let's say you know nothing about
Windows internals. What are we looking for here that's significant? You see hooking module unknown? Okay, that's a bad sign. Hooking module
basically should show you the module where
the jump instruction, the hook is
redirecting you, okay? You can see the
hook address right above it, but the fact
that Volatility is not able to tell you what DLL that hook jumps into, is not a good sign, okay? Again, it means that
the module that it's jumping into is not
loaded in the normal fashion so it can't
actually tell you what would be LL you're targeting. That is never a good sign So if you run API
hooks and you see hooking module unknown,
you know call for help, right? You've got a significant
problem going on. Now you're sort of looking deeper at
the output you can actually see the
name of the function that was hooked, in this case. And, you know, like
this is where you start needing a little bit of you know programming knowledge
and a little Windows internals knowledge. As you scroll down through
the API hooks output, you can actually
see the different functions that are being hooked, and if you understand
what each one of these functions does, it can give you some indication of
the functionality of the malware that's
on your system, right? So we see ntdll NT query
directory file, okay? So that's basically getting
a directory listing. So the fact that the Zeus
malware is hooking that function would
indicate to me that it has some ability to
hide files from the Systems Administrator, right? So, if it's hooking
things that deal with interrogating the
registry, then you know it's capable of hiding
registry entries and things like that. So in any event, if you know more about
Windows internals you can actually
formate the eyehooks, get an idea of the functionality of the malware, okay? So, you know, loader modules
look for the false's. API hooks look for hooking
module unknown, right? So those, your sort of
simple base heuristics and then the last
Volatility module that I want to show
you in this case is the MAL find module. So MAL find is a
little bit different from the other modules
in that it looks for the injected code, but it also is capable of and will dump out
the injected segments for you automatically
so that you can then analyze them. And so because it's
dumping out the injected code, you need to
give it a directory that you want it to
dump the code into. So I'm just going to tell it to dump into my temp
directory, so that's what the extra minus D flag is. And I'm going to let
mal find go and I'm going to talk about
what it's doing while it's going. So basically mal find
looks for injected code by walking what's called, the virtual address
descriptor tree or VAD tree, okay? And basically
injected code shows up in the VAD tree
with a particular marker that is unique
to injected code. So mal find basically
just walks through the VAD tree, which is a
list of the virtual memory pages in the system, looking for this particular marker
that's indicative of injected code. And when it finds it,
it dumps the memory segment out for you
automatically into the temp directory
and now my temp directory is full of these dump files. You see the .dmp files? So those are the
dump memory regions and you can then suck
those into IDA Pro. You'll probably need
to rebase them a little bit which is why
the memory offsets appear in the file name. And then analyze the code in the injected segment. So the combination
of loader modules, API hooks, and mal
find are very good for quickly zeroing in on injected code either by examining the sort of normal internal data
structures that are used by the loader,
which is what loader modules does or by
walking the VAD tree, which is what mal find does. You can quickly home
in on this injected code which is difficult
to find, you know, other ways, right? So let's go back
to illuminate here and I'll see if
I can catch up on some questions as
well for people. Can Redline read vmem's? Yes, it absolutely can. Vmem is really nothing more than a raw memory dump just like you create with
DumpIt or Win32dd or anything else. So there's nothing magical
about a .vmem file. They're just normal
raw memory dumps. Does Volatility work
directly from dumps, from DumpIt? Yeah it should just
be a raw memory dump. Okay so, there
are a couple other questions but I'll
take those at the end if we have a little bit of time. So... We've got some summary
slides that just summarize the different Volatility modules
I went through. I wanted to talk, before
we run out of time here, about another rootkit
technique which is process hiding. Process hiding is
used by what we call decom, direct kernel
object manipulation is what that stands for. A decom style rootkit
basically gets loaded gets system
privileges, and then, because it's achieved
elevated privilege, can actually manipulate
internal kernel data structures. One of the most common data structures for these decom style rootkits to manipulate is the linked list of
EPROCESS structures that make up the
process list in memory. So every process on the
system is represented by one of these EPROCESS
structures that live in memory. And the EPROCESS
structures are linked to one another. They're actually
linked in this sort of bi-directional daisy
chain, alright, so every process points
to its neighboring processes in a big ring. And so when something
like task list wants to list all the
processes on the system, it just starts with
one of the processes and walks around the
daisy chain listing each EPROCESS
structure, all right? Now when a decom
style rootkit wants to hide one or more
processes what it literally does is, it
removes the process it wants to hide from
that daisy chain. So in this case the
decom style rootkit is hiding winlogon.exe. So, it marches around
the circle until it finds winlogon.exe,
the process it wants to hide, and
then changes the pointers with what
we call the forward and backward pointers,
or flink and blink. So it changes those
pointers to basically remove winlogon.exe
from the daisy chain. Okay, so now when
you know processes like task list or whatever,
enumerate the processes on the system, they
won't see winlogon.exe because it's not in the
linked list anymore. Now you're probably thinking, well how the hell
does the process run? that I mean it's
not in the list of processes, how is that
process ever scheduled in order to actually function? Well, it turns out that, process scheduling on Windows
is not done in terms of processes. It's done in terms of threads. So each process that you see in this picture will
have multiple threads running inside it
and those threads are scheduled and managed
by a completely different set of data
structures in the kernel. So even though the
process has been removed from this linked list, the threads within that process are still scheduled
and still execute and still run as normal. You just can't see
the process, right? So, you know, the attacker
starts up a packet sniffer or a remote
command shell or something like that and hides it using this mechanism from
the systems administrators. It's very common decom style rootkit technique. But again, doing a
little memory analysis, you can easily detect
these hidden processes. And we think about,
how would you do that, right? I mean, obviously
we can get a list of the processes
by walking around the linked list, right? But that won't
detect the ones that have been hidden by the rootkit. Well, because we have
the memory image we can grovel through
the memory image looking for the
process structures. The process structures
have a very, sort of particular
structure and you can detect them in memory even when they're not part of
this linked list. So what do you do then? Well, so what I do
is I get the list of processes the normal way by following the
linked list and then I walk over the
entire memory image looking for EPROCESS structures and I compare the
two lists, right? And if there's a
process that shows up when I exhaustively
interrogate memory that's not in the list
of linked processes then you have a hidden process and you've got a problem. Now back in the olden days, you had to run two
separate Volatility modules for this. In fact, there's a
whole command-line kung-fu episode
that I wrote about how to do that. But in the newer
versions of Volatility there's actually a
module that does exactly that detection mechanism. And I can just quickly
show you that module. It's called PSX view. And what I've got, I've got
another memory image here from the fu rootkit which is
a popular Windows rootkit. So I'm going to do vol.py PSX view on my Fu rootkit image. Now, we'll take a look
at the output real quick. It takes a little while for this module to run because
first it has to enumerate the processes
the normal way and then it has to
scan through the entire memory image looking for EPROCESS structures. But just like loader
modules, right, if you see any
columns with a false, generally speaking,
you've got a hidden process, right? So for example, oh look we've got a hidden command shell on the system. We've got this wmiprvse process Also, you know, hidden process. So when you see something that shows up as true in the PS scan column but when you
you know look at the thread proc or you
know other columns and it shows up as false there, you've got a hidden
process, right? So, you know... PS list, this column here, PS list, is the normal
walk around the daisy chain, right? PS scan is the one
that enumerates memory. So when you see
things that are found by PS scan and not
showing up in PS list, it's not a good sign, right? So again, it's a quick way to detect decom style
process hiding. You don't even need to
know what's going on. Just run PSx view and
look for the false's. When you have a
discrepancy between PS list and PS scan, it's
always a problem. There's never a case where this should happen
in normal behavior. So when you see the
discrepancy between those two columns
raise the alarm and start analyzing the system more closely. So... Let me just finish up real quick with, this is just a summary. Here's good old PSx view. So the reason I put
this presentation together was partially
to get people to stop, as a matter of
standard practice, just pulling the plug
on the system and imaging the hard drive. As I mentioned, I'm doing work with law enforcement right now that's still the standard operating procedure for a lot of law enforcement
organizations. But if you can, if the system
is live at the time you begin interacting
with it and you have the tools, I beg you to please, do a memory dump of the system. Even if you don't
have the ability to analyze the image. Somebody later is going to want that memory image. Somebody like me. So please, you know,
try and do a live capture of the
memory on the system. Now, you know, the
reason law enforcement agencies are worried
about doing this is they say, oh well, you know, you're modifying the
state of the system, you're corrupting the
forensic integrity of the system. Well okay, yes. I have to run a process on the system and yes, that changes the state
of the memory on the system and makes
various other changes to the system, but you
know, Ovie Carroll who, aside from being
a SANS instructor, is the head of the
DOJ CCIPS Crime Laboratory, right,
so you'd think this guy knows what
he's talking about. He has a very good analogy
which I like to say here. Imagine the memory
of the system as being piles of paper stacked up, floor-to-ceiling in a big
room somewhere, right. There's one door into the room. The room is so tightly
packed with these pieces of paper that
you have to blow the door off the
hinges in order to get access to the room. And yes, blowing the
door off the hinges will crap up 5% of
the paper in the room. But are you telling me, you're not willing to sacrifice that 5% to get the other 95% which is critical
information that will make your investigation
successful? I mean, it's ridiculous. Of course you're going
to do that, right? So yes, you will
consume some memory by running a memory dumping tool but the value you
get out of that is worth it, right? Totally worth it. So please, you know, if you get nothing out of this,
please, you know, take a memory capture
in the system. So I'm going to try and catch up and answer questions for people. This presentation is available from the last URL
that you see on this slide so, deerun.com tilde hal. You'll find the PDF of
the slides from this presentation at that URL. This presentation is
sort of a high-speed collision of the
SANS forensics 508 and 610 classes,
which I and many other fine SANS instructors teach, we've got a bunch of
opportunities for you to take these courses
if you have some year-end budget burning
a hole in your pocket. Alyssa Torres who's
in the chat right now is teaching it in
Chicago in a few weeks. I'm following up in San
Antonio a few weeks later. I've got some
forensics 610 dates and you can go to the SANS website and sign
up for one of those courses if this, you
know, wasn't enough for you there's a whole
lot of geeky memory stuff going on in forensics
508 and forensics 610. in forensics 610,
we'll show you how to actually analyze the
malware once you've identified it using
the techniques in this class. So there's a whole
curriculum of stuff. So we see if I can get
to some questions here before we run out of time. So let's see. Yeah, so we had a question about something I said
earlier as far as hibernation files. So I mentioned that
a colleague of mine was working with a
case where they had an encrypted volume. They didn't have a
memory image but they did have a hibernation
file on the system and the system
was hibernated at the time that the encrypted
volume was mounted. And they were actually able to use hibr2bin to convert the hibernation file into
a raw memory dump and then they used
a commercial tool actually called
passware to extract the TrueCrypt key for
the volume from the memory image from
the hibernation file and they were able
to use that to unpack the TrueCrypt container. So yes, true story,
actually works, incredibly useful, so yeah. Let's see here. So we had a question
about memory images versus the page file. Right, so the memory
image of a system, whether it's VMware
or whether it's a live system is, you
know, the sort of active memory of the
system but certain pages may be paged
out to disk because you know typically
you know you don't have enough memory
then hold all the running processes on the system, particularly on
personal machines like laptops and things like that. So yeah, I mean, sometimes
when you're analyzing a memory dump you
will like for example try and dump an
executable image or something like that
and you will find that pages of that
image have been paged out to the page file
and so you don't get complete
information out of the memory dump. And that's, you know,
just one of the downsides of memory analysis. There aren't any good tools that I'm aware of, for sort
of, fully integrating the page file that
exists with the memory image. And I mean, you know,
that's difficult, right, because memory is
constantly changing and pages are being moved
in and out of memory just, you know, creating
a sort of perfect snapshot that integrates those
two is difficult and usually you can get
enough information out of just raw
memory dump without referring to pagefile.sys. So you're, you know,
it's enough to get useful information. How do you do a
memory dump remotely? Well, you know, my
personal preference is F-Response which
is a commercial tool but if you need to
do it on the cheap you can run win32
or win64dd remotely via PS exec and have them
dump the memory image right over the network
to, you know, a system where you have a
netcat listener setup or something like that. And that is a free
way of doing that. So you know PS exec from
sysinternals and win32 or win64dd and a little bit of netcat and you're good to go. What memory
forensics tool do you recommend for use
with mac memory dumps? That would be
Volatility which now has max support. Idon't know of other good tools for analyzing mac memory dumps other than Volatility. If anybody knows one they
should throw it into the chat. Is forcing the system
to hibernate a good way to get a memory image if
you don't have admin rights? Yeah, I mean, it's certainly one way to do it. Obviously there is
some forensic impact to hibernating the system but any port in a storm. Is it possible to
inject code on a physical host that
is hosting virtual machines and have
it affect the hosted virtual machines
in some bad way? I mean in general,
if I have hypervisor access on a system, so if I've got control of
the host operating system there are many better ways for me though to impact
the virtual machines. And so, you know, once
you get access to the host environment, I can
completely harass the virtual machines. You basically have
to assume that the virtual machines are
completely compromised if I get access to
the physical machine. I would probably attack them
through some other mechanism then you know you're
injecting code from the outside. I'd probably just
project a module into the virtual
machines and then inject it using the
normal injection mechanism or I'd simply, from the hypervisor, steal
whatever information out of the virtual
machine I needed to infect and not impact the
virtual machine at all, that way it wouldn't be
detectable from within the virtual machine. Yeah, people are
mentioning that the HBGary tool does
try and Stitch the memory dump and the
page file together. So if you look at HBGary's responder tool, it
tries to do some pushing together
the pagefile.sys and the memory dump. So you know, this,we get the
question we always get about now we're using GPUs
in order to execute. Yeah, we've seen, you know, certainly we've seen
malware that does that. More the proof of
concept stage in my experience than
an actual reality. We're getting some
MAC memory stuff. MacQuistion from BlackBag. Mac memory dumper. So I'd forgotten about
the BlackBag tool. Ao shouts to Drew and
the crew at BlackBag. Okay, so I'm actually
over time here. So I'm going to wrap it up. You have my email
address on the slide here and my Twitter handle
so if you want to ask additional questions
you're more than welcome to contact me by email. As I said, the slides are
available from the last URL and, you know,
thanks everybody for showing up. I think we had 400
people at the peak. So thanks again everybody and we'll see you on
the next webcast. - [Jason] Thank you so
much Hal for your great presentation and
for bringing this content to the SANS community. To our audience, we
really appreciate you listening in. For a schedule of all upcoming
and archived SANS webcasts visit sans org
forward slash webcast until next time take
care and we hope to have you back again for
the next SANS webcast.
