Sysinternals Fireside Chat - Mark Russinovich | Interview, History, Windows | Microsoft

[Music] welcome to sis internals at 25 i am super excited to be here i've got mark racinovich with me how you doing my friend doing great huh legal driver and cheap insurance now for assistant journals is that right it's great it's grown up it's all grown up so i want you to get into the history of system drills because my understanding is because i i did a lot of research i read the wikipedia page yesterday um that it's changed names over the year tell us about the history well so uh i've always been interested in operating system internals and then writing tools to explore it and to take advantage of that kind of knowledge i started to do that in high school and then i went to grad undergrad went to get my phd in computer engineering and met somebody named bryce cogswell who becomes key to the story in a few minutes but we started working on products for as part of a startup that we came up with idea that involved leveraging windows internals both windows 931 windows 95 and windows nt and so i started to learn about internals and then i said hey i can actually understand better the way the operating system works by writing some tools that will show me what's going on and the first tool that i wrote that did that was something called regmon registry monitor and uh started to say hey bryce hey we can write more of these kinds of tools and so we started writing more tools like that uh filemon followed that ntfs dos is a tool that we wrote that runs actually on dos but lets you mount mtfs volumes that were created by windows 90 by windows nt and we started to continue making these kinds of little utilities we decided hey we're let's publish these so i had a friend named andrew shulman who wrote this book on undocumented windows 95 that was really popular at the time made him unpopular with microsoft but but uh i'd gotten to know him and i said hey can you've got a website how about if you post these tools up for us and so we started to do that and then i'd say here's an update and he'd say uh okay i'll get around to it and a few weeks would go by and he wouldn't post it and so i'm like okay we need to take this into our own hands bryce let's put up a website and so we put up ntinternals.com and started publishing the tools there directly and so that was september of 96 when we launched that that domain ntinternals.com and started publishing the tools and we launched with four or five tools that we developed over that year uh leading up to that so let me ask you a question so basically this was born out of a curiosity for the internals of an operating system you're like i'm gonna write some things to visualize obviously the registry was a scary place back then where you you don't want to change stuff and you you visualize certain things to help you and decide to release it yeah i mean it was really kind of fun because i love the multi-purpose aspect of things that i do where i'm getting multiple uses out of it in this case i was learning about the operating system just writing the tool and then the tool itself would teach me more about it and then other people would find it useful too and so there was a bunch of tools like you mentioned just a couple of them um tell me what were some of your favorite ones to work on that you're you learned the most from well regimen and filemon were both fun because they both showed operating system internals but took two different approaches on filemon windows had this built-in system for intercepting file ios that antivirus drivers would use but there was no example for how do you monitor what's going on and and the documentation for it was pretty lame at the time there's something called the ifs developers kit installable file system developers kit which didn't really talk about doing filter drivers as these were called very well so i had to reverse engineer a lot and experiment a lot to get that running but it leveraged mostly you know documented kind of semi-documented windows uh apis to do what it was doing for regmon there was no like here's how you intercept registry calls i'd working on my phd at carnegie mellon i'd come up with something for unix of patching the system call table for example when you call open file or op the open api it goes into the kernel and then the kernel through a index right and the open function in the kernel is called from through that index and if you replace that index with a different function then the kernel will call that function instead and if you record where it was originally pointing the original open call you could point it at yours do some stuff call the original one get back what the result was and then return this is called system call hooking and so i did that for unix and then i'm like i can do this for windows too and so i came up with system call hooking for windows mt to develop regimen for it so two different approaches and required me to be a little creative about how i could get that kind of visibility into the operating system so you were doing like some crazy pointer arithmetic then for the system calls as they were going to the kernel yeah right that's right that's really cool i look i took an operating system class i didn't know i had 24 hours 26 hours in a day to write operating system code because it was that bad how deep did you have to get into things to make this work because you're looking at the system call table which is literally how things happen in the kernel yeah so a lot of this stuff wasn't documented of course because it's so low level and no no reason to document it for the average developer so i did a lot of reverse engineering um and i reverse engineered a couple ways one is i would have a kernel debugger and it's kind of ironic because i started to use this tool called soft ice which had debuggers for dos and then windows 95 windows 31 windows 95 and windows nt i ended up joining the company that made that new mega technologies but that's the tool that i used and i actually wrote some apis for it and wrote documentation for it after i joined it but that's the tool that i use to set breakpoints and then say oh the operating system's doing this now it's doing that and then i also used static analysis where you can take the binaries and then disassemble them which means translate them into the assembly language and then i could study that and also kind of get a view of where things were and in windows the kind of cool thing about windows is it came with symbols at least for what they call public symbols so for kernel apis that were public that when somebody's debugging something would help them understand what's going on in the kernel those symbols i could map into the disassembly and then i could say oh here's the create function and it's calling the allocate memory function and it's you know i could follow my way through the system that way so you're not just looking at jump calls and addresses because that would be that would be terrible the thing is the private apis weren't didn't have symbols so that i did a lot of that too like what is it called what is this function doing and then i'd give it my own name so i could track what it was going on but i remember when i when i did my operating system class i asked my professor hey can i use c plus and he laughed at me he's like if you get it to work great so you were literally like in c and lower level code to get this to work because there's no other way to do it yeah this is uh assembly language which is you know the in the low-level cpu instructions just in a human-readable form you know it's not binary but the next level up from that it's like move this thing address jump this thing address and it's that low level now as you're debugging these things are you actually jumping through the actual assembly yep and then you're looking at the register set and you're looking at memory to see what changed so we mentioned a couple of tools are there any tools that were inspired other than looking at the system calls for someone asked you for something or you're like oh i have a really good idea about this or any tools like that where somebody asked me for something well actually one came out of me and dave solomon teaching windows internals classes together which i started to do in the late 90s and we co-authored that this uh this book actually together i've got a little picture of it oh nice so i the little story on this one um dave wrote this uh book called inside windows nt and second edition after the helen kuster famous inside windows and t that came out when nt did and i looked at that book i was going to do my own internals book and i'm like wow he's just raised the bar so high he had he had demo kind of samples and demos inside the text to illustrate concepts with tools that came in like the nt resource kit and uh i wanted to be part of the next version of it so i'm like dave hey uh how about join letting me join his co-author i can take some of the articles that i've written and incorporate those into the book and then i can bring the assistant turtle's tools in and add a whole lot of new experiments to the book and uh there's some stuff that he had to negotiate with microsoft that you know uh kind of uh related some tools because i pissed microsoft off a little bit and they're like okay fine you can but the tools are on a cd on on the uh the book but i started to teach windows internals with him too even including going to microsoft but one of the things that we would do is experiments live in the class and you know on the screen and the challenge was that you'd have to make the fonts super big and it's still there's someone you couldn't control so it was really hard for people to see the the demos a lot of times so dave had this tool that came with one of his pointers that uh on a laptop that will let you magnify stuff and so we started to use that and it was kind of clunky like the you know the hotkeys to magnify were clunky and you couldn't do annotations very well in it and so i'm like after a couple years i'm like that that's this is obnoxious i'm doing my own and you're talking about zooming zoom it yeah i use that thing religiously to this day the other thing is that you were making these tools for free but there was a moment in time where you decided there's there's got to be a way to monetize these things and then there's a fun story on the way to microsoft as well why don't you talk about that well so um i was happy just writing these tools and making them free bryce had this idea for this tool that we ended up calling nt recover he's like hey what if we take any you know one of the problems that we saw even ourselves in doing our stuff is you get the system into a state where it wouldn't boot and like how do you get data off that system that doesn't boot how do you go fix it and bryce is like what if we take uh the serial cable that we use for debugging and connect it between the two things and boot the dead machine off of floppy and then have a program running on windows nt that would mount that you have a driver that would make that remote disk look like it's a local disk and then it would mount it and then you could look at it use all your favorite tools to recover data off it and fix it and uh so we ended up writing that and he's like there were recovery parts gold we can sell that so so we decided hey let's start a company and actually this i owe bryce this basically because he's the one that went and you know got the business license he set up the e-commerce site this is 1996 as well yeah and then uh he started copying floppies he got floppy machine copiers and then went to the shipping place and he'd answered tech support meanwhile i was working at ibm at the time i'd already gone to ibm so i was doing this in my spare time but that's what how wind tunnels got off the ground in nt recover just in the first month we sold a thousand dollars of it in the second month it was like five thousand dollars and it just kind of took off and we were advertising the tool on system journals and making a free read one read version you know this is shareware idea that we made on sysinternals uh so a year into that and i was making enough money on the side from my ibm job that i was like uh you know i've always wanted a ferrari i'm gonna go buy a ferrari and and so that's that's really cool and i i look i have stories of because my dad is a doctor right he's retired and he put all back in the day he put all his medical records on some machines and i turned him on and broke him right and he's like well you got to get that back son if i would have had a tool like that it would have been a lot a lot easier to do that so tell me about like the move into microsoft did they just bring all the tools in with you or what was all that like because now your job is very different obviously yeah so uh 2006 the acquisition happened and it was an acquisition um but of both cis internals and wind tunnels they're both legal entity independent legal entities system internals being free wind turns being the commercial software company we had 85 employees at the time so acquisition of both and the technology of both uh one of the things that i really was passionate about and was um keeping assistant internals going in fact uh the day the uh acquisition is announced uh microsoft put up a blog post we put up a thing on wind tunnels uh and cis internal saying microsoft's acquired it and there was a mad rush of people to everybody go download microsoft's going to kill this and so our web server crashed you know that's what they did at the time yeah back in the day yeah so um we had to you know keep it up and running and we're like no no don't calm down it's not going anywhere but um the wind turtles tools became part of a microsoft desktop and recovery toolkit how much dart and part of our backup tool too so data protection manager the assist internals tools i was basically left me and bryce both joined microsoft to continue working on them and so we just did that and continued working on the most part time we worked in windows as day jobs but the first tool that i wrote after with price after coming is process monitor which is one of the it's probably the most useful tool for troubleshooting in fact this is a slogan somebody came up with uh um that uh uh chris dave solomon came up with it actually i think we we can't remember the quite origin of this but uh when in doubt before it was when in doubt run final month regimen really this was the combination of those two plus a whole lot more kit power um so that that became uh t-shirts and that became a thing and we would say when in doubt i would get audiences to say in troubleshooting case of then explain talks when in doubt and run procman before we get into like the current state of the tools and the future by the time and i really want to dig into this by the time people are using these kinds of tools some pretty serious things have happened right do you have any some do you have some interesting stories of customers using these things and the reasons why you don't have to give us names and dates or anything but i'd love to hear some of the interesting stories from customers well the i mean there's just hundreds of cases of examples of customers uh customers users uh using the tools to solve problems and one of the things that became a slogan for assistant turtles and wind tunnels was you saved my butt or you saved my you know yes yes and uh so we actually had a um uh wind turtles ad campaign that was you know save your butt you know kind of thing um but that was so there's just hundreds and i would incorporate them into the case of the explain talks but some of the most uh kind of funny ones were that i one of the tools that i wrote was the blue screen screensaver that would uh like a screensaver when it was like your system was idle would draw a very realistic blue screen and then i took it a step further i'm like i'm going to make it look like it's rebooting your machine too with the windows logo and the scroll you know the progress bar is and then it would crash again it looked like it crashed again and i'd have people that would run this on production machine and then somebody would come by and be like oh this machine's crashing i'm going to reset it and then take down their production machine so not the intended effect and then people use it to play pranks and there's a tool one of the most popular tools ps exec a part of the ps tools that lets you run things remotely it's actually much better than powershell remoting it is like the way to do remote desktop administrator you know administration i'm just poking jeffree snow over there we should we have to it's part of every interview yes it is but ps exact is um still incredibly popular but uh what people do with ps exactly the blue screen screen saver is run it on somebody else's machine and then they would freak out and think their machine was crashing oh my goodness because back in the day the machines were literally under people's desks and if you push the button like the web server would go down yeah kind of thing that's that's funny any other interesting stories or or things that people use your tools for that you were surprised about well uh you know lots of tools like these uh you know tool is a tool and it does doesn't it it doesn't have any inherent goodness or badness but of course people can take a tool and use it for bed good purposes or bad ones ps exact is a prime example of this actually because a lot of hackers attack get on corporate networks and then spread through it and the way that they spread is taking advantage of ps exec so ps exact countless and still even today you'll see um security analysis of malware and infiltration into a corporate network and it's like and then they use ps exec to jump from one machine to the other um so that was surpri it kind of surprised me i guess in you know i didn't kind of uh realize that it was gonna be used that way and then um then it was that's cool okay so let's talk about now uh sysinternals at microsoft there's a whole docs page with tons of different tools one thing that i've always asked myself and maybe you can answer this maybe you can is why aren't they part of windows yeah that's a question that i get all the time and i still see it on twitter's like why aren't these just built into windows uh one of the reasons that they're not built into windows is um windows used to have a very slow shipping cadence ship cadence and so that the rate of refresh on tools and bug fixes and feature editions i didn't want to be constrained so even you know i was like i need to keep these independent so that we can release them and they go across lots of versions of windows in fact the tools for a long time up until probably six or seven years ago supported both windows 95 no maybe older than that but windows 95 as well as windows nt so it was the agility of the release that i didn't want to impact it but the other one is that the windows bar is much higher than you know freeware toolbar yes and it's like um full support you know for slas with customers and customer support and a lot of of machinery goes into supporting a tool that is part of windows from a legal perspective and from customer promise perspective and i didn't want to have these things slowed down i mean obviously very popular with what we were doing the kind of level of support and the kind of level of polish that we had on them and i wanted to preserve that so that's another reason that we kept them outside of being shipped in the box with his windows and that's interesting to mention legal reasons because if you're downloading these tools you're taking some responsibility for what's going on as opposed to you know you have a eula in windows yeah that that promises and does certain things yeah and the ula and sysinternals is like these are you know for your own use yeah as is yeah as is no merchant ability that's the word that we always read in our software so another question and maybe you can't answer this but i'm going to ask it anyways why aren't these open sourced uh another question that i get all the time and on the windows ones one of the reasons that they're not open source is that a lot of them use undocumented apis so i mentioned you know regmon was one example up up until it folded into process monitor but process monitor also uses some undocumented apis and the reason not to make them open source is that effectively documents them and once you document an api you're also responsible for documentation and backward compatibility and that would constrain in some ways the the ability for the platf the core of windows to to evolve um kind of inadvertently right if you don't do it deliberately you're not going to be able to preserve those kinds of guarantees that customers expect out of documented apis that so that that's uh really the key reason now some of them are tools that are useful fully documented apis and the reason i haven't open source them is that open sourcing actually takes some effort like if you're going to open source you need to like embrace the spirit of open source which is i'm going to put this up you can do pull requests on it i'm going to look at your pull requests i'm going to you know so i was like um really it was more of i want to don't want to again burden the tools with with that and also uh not be worried about the line of which ones are using non-documented apis which ones are kind of things so that's that's the reason for that now on the linux side we've taken the you know it's linux it's what's open source so there's some new linux tools and for those we're starting just with those being completely open source and i was going to ask you about it because all the way at the beginning you mentioned that you did some things in linux style systems are we doing things with linux style systems now uh what do you mean we doing things like for example our assistant turns tool is going to work on linux so yes um you know linux used at microsoft in azure it's exploded in fact there's more cores on running running linux uh external cores running linux than than windows cores in azure so hugely important for us to support linux and what i looked at is like a lot of these tools make sense on linux so one of the tools the first one that we made a version for linux process process dump it's called procdump we said hey that there's a good reason to have a process dumping capability on linux that uses triggers like process dump for windows and then you can control where the dumps go and the other capabilities that are part of that tool and so we created pros proctum for uh linux and pro prospectum for windows is actually used inside of azure's infrastructure uh proc dump for linux is gonna be used in azure's infrastructure so it's kind of one of these kind of core utilities that makes sense in both operating system environments and then we're excited to announce uh here at this event the the release of sysmon which is uh a security monitoring tool that i wrote back in 2014 um when john lambert who works in our threat intelligence group writing tools and tracking hunt bad actors came and you know called me up and say hey i need to come over and talk to you about something it's like we've got a bad actor and that we've seen on our network and we need it'd be nice if we had a tool that would let us better track exactly what they're doing and here's what it needs to do track processes track files they do this shredding thing where they um drop a file they use it and then they delete it and they shred it by overwriting it so that when we do forensics we can't see what they ran like it'd be cool if you could have the tool catch that and i'm like well i've got a lot of that infrastructure already ready already um in one of the tools process monitor so i could just start with that and have probably have something for you pretty quickly so two weeks later i'm like here's a tool that does what you wanted and uh so he used it and and found it effective and then we said hey we should make this can i can make this in system turtle's tool and we said yes there are some capabilities that we'd like to be withheld from the public version for now which are all now in the public version like clipboard copy and this file shredding recovery but that was sysmon and so that excited today to announce that sysmon for linux is out um which we've been talking about working on for a while but it's of course completely different implementation for for linux but it has the same purpose and one of the powerful aspects of sysmon for windows is its filtering capability that lets you say i'm looking for this particular signature for malware let me write a configuration file that captures that and that's something that that there's no built-in tool in windows or linux to have that fine grain filtering of what you want to capture with all the level of detail that you capture that is relevant to security analysis and so that's built in with compatibility in the configuration file and the same kind of behavior as uh the cismond for windows now available for linux so future of cis internals i mean are you still writing are you literally going in there writing stuff stuff yeah i do um not nearly as much as i did um for example uh one of the most recent things um i'll add little things every now and then but i added uh to sysmon what's the last thing i just added it's hard for even me to remember i added a feature to it just a few months ago um i bet it was awesome it was it was it was the best feature ever you're not going to want to be without it oh i know i remember what it was um one of the hacker tools is to launch a process it's called process uh this technique is called process halloween where you launch a process like notepad and then the bat the malware unloads notepad and the notepad executable from the process address space and then loads in their malware so when you look from the outside about what's happening it's like notepad is doing all this funky stuff yeah but it's not really notepad so i added a detection in process and sysmond for detecting those techniques of process halloween and i did that a while ago that's one of the most recent things so i still do work on the tools but now we've got a few other developers that are contributing to the tools um that uh inc including people that are working like uh mario and kevin sheldrake um thomas garnier who work on the linux side of things as well which i've also participated in the design of those as well so i do still you know have an active role in what the tools are and where they're going and what we should prioritize and then throw in features every now and then so nobody should rush and try to download them all because they're going away it's still active and there's still going to be stuff happening oh yeah for sure in fact um we've so now we've got actually a few dabs that are full-time on system turtles um up until now it's so it was first being bryce and then we hired somebody to kind of manage the website kind of operations things and then we hired somebody else to work help work on the tools um so now we've got a few people that are just full-time job you know let's keep the tools fresh we try to shoot for release of tools every month of something um so yeah i'm a fan i mean i've been using zoom it forever i use the what was it the pro process explorer process monitor process model that one to to see which text yeah which text file is holding up everything yeah process explorer cross explorer uh and i and i enjoy them thoroughly like i said i've literally used it today and i've used it like almost every day since this has been super cool anything else you'd like to add um well one of the things um you know there's support you can get like there's forums for assistant turtles but uh one of the cool things uh is and it's still a great reference is um this this book that i actually this is not the right version of it um but this this book um this was the first book that i wrote with aaron margosis they finally let you let you write the book like not like the other one right that's right yeah this one and um that book uh which is in the second edition it's called troubleshooting with this this internals tools with aaron margosis who's giving a talk uh here at the seminar a couple talks at the seminar on a couple of tools really is a fantastic like you want to understand the tools and what to do with them and it's got a bunch of cases then explains um that is a great reference uh to help you get the most out of them and just from a pedagogical perspective i wanted to use a five dollar word here yeah like using those tools gives you a really good sense of what's actually going on under the hood which is super useful and it's what got you excited about these things in the first place right that's right yeah no i mean it it's just been it's they've had a huge impact on my life obviously um just the the knowledge i've gotten on the enjoyment i've got out of the the i mean i still today i'll go visit a customer and they'll cut or they go to the ebc and a customer come in and be like this cis internals tools man those are so awesome or i i used them 10 years ago and my career has been you know so much better because those tools are so rewarding to hear that all right so here's a question that that i've had what makes a tool assist internals tool there's got to be some bar right yeah well one of the things that i believe from the start of if i'm going to use a tool i don't want to install it i just want to use it and so that was a principle from the very first tool is i can just copy it someplace and use it which is kind of the linux philosophy unix or philosophy and i'd actually come from working a lot on unix the other one is that it shouldn't leave any stuff behind so no like so it clean they clean up after themselves and try to not not to leave anything behind on your system um and the third one is um in general this isn't 100 true i didn't want to have to write manuals and documentation for them and i wanted them to be very straightforward and kind of self-explanatory and and intuitive to use and so that's been another key uh goal of the tools is no manual required you can run the tool and kind of figure out your way around it uh pretty quickly it's funny you should say that because i'm not even kidding i have i'm on windows c colon util on my path and i literally download the the scissors and i just put them in there yeah and then i just you know call it right yeah and that's that's actually pretty cool because i i've seen that and i didn't know that was a philosophy okay by the way speaking of that that's another announcement for today they're now in the windows store so you can install them they will go into your start menu and they will also be accessible through shortcuts and from the command prompt so you'll and they'll always be kept up to date so that's another uh thing that we're doing today that's actually pretty useful because i every once i was like i should probably change my util directory with the newest ones so that's that's a really cool thing now another question that i've had if because there's so many tools and i've seen a lot of them sometimes i get a little overwhelmed myself which ones are the indispensable ones that people should try out yeah well i think just going through the most popular ones uh will get you basically that list um auto runs which will show you what's configured to start and run on your system automatically so if you want to go get off the crap that's popping up when you log in and figure out how to get rid of it because then there you still like the thing but you don't like it popping up or you're going after malware hunting auto runs process explorer to understand you know what's actually running and find out who's got the fan here yeah who's got the open file um process monitor so number one tool for troubleshooting uh zoom it for i mean i use zoom it myself to zoom in on text on my screen when it's too small it's rather than having to change my fonts and dpi i'll just zoom in you can also get screen clip clips very quickly from it like zoom in on something and do control s and or control c and you can not know that yeah that's cool little undocumented tip where it's documented that little tip there for you um so i'd say that those are the big ones and then if as a if you're working in security sysmon which is a system a security monitoring tool logging tool well awesome yeah thank you so much for spending some time with us mark and thank you so much for watching sis internals is now 25 years old and hopefully many years to come thank you for watching and we'll see you in a little bit

Channel: Windows IT Pro

Views: 14,971

Keywords:

Id: tR22u6H8E5w

Length: 31min 55sec (1915 seconds)

Published: Tue Feb 08 2022