Running Windows Inside Containers On Linux - PSW #728

welcome back everyone to paul's security weekly rather don't miss any of your favorite security weekly content visit securityweekly.com forward slash subscribe subscribe to all of our shows on the network at that link via your favorite podcast catcher you can join our discord server which is an amazing community uh our webcasts subscribe to our mailing list all kinds of stuff we have streaming platforms we're on twitch youtube make sure you go to securityweekly.com forward slash subscribe alrighty so we're continuing the series on uh the container lab that i built which kind of i don't know where it all stemmed from i was trying to show exploits and stuff so i started building containers and i started building more containers then i started putting uh all kinds of different tools and then someone asked hey can you get windows in there and i was like i don't know maybe i gotta i gotta look into that i'm like it's kind of weird because i'm on linux so this could get weird and it's gonna get weird just throwing that out there it is kind of weird um but if you go to uh github.com forward slash security weekly the vault and i put the link in the show notes the volehub lab github repository uh is where i have the there's a stable version out there and then there's my dev uh branch which and i'm trying to get a windows domain controller in there uh and then get them to join the domain together so i'm working on that i'm way better at linux and windows just for one is one thing i learned in this project uh so the other thing is inside of the vulhab lab right i've got the uh http web server i've got trevor c2 i've got merlin and i've got a bunch of vulnerable linux targets as well as a kali linux instance it's all configurable like the kali one you can tell how much you want to install by default or not and so uh that's the the kind of starting point there is an interview coming up with dave kennedy that we've pre-recorded already to talk about trevor c2 and c2 communications in general so make sure you check that out um and so you you can download spin this lab up yourself uh it's similar to other labs you might spin up in cloud environments right but that some of them like you know incur costs for the students so i thought it was kind of nice to spin this up on your own now this is assuming you're running linux as your host operating system i i do intend if someone wants to take this on i accept pull requests you can spin this up in docker on windows i'm assuming that's possible right it would just be flip-flopped right you'd run your windows inside of docker containers and linux inside of docker containers with windows as your host operating system that would be cool um and so i fully i want to get to that point as well um i'm going to show you how to do it in linux using docker with containers to spin up all the things i just talked about including windows which actually runs inside of uh a virtualization layer qmu that that runs inside of the docker container that runs on linux now i didn't i didn't invent this i just want to throw that out there i linked to a a few resources on this one but the one i want to give credit to is the individual whose blog post i linked to on medium that showed me how to do this there's other resources as well and i'll put more resources in the in the github uh repository as well i have them flagged i just got to integrate them so like this isn't my original idea or my original work however i did rewrite uh pretty much a lot of things off of that medium post because by default that medium post did not work for me and according to the comments it didn't work for a lot of other people who read that medium post because some things have changed since that post was written i think about a year or a year ago or so so i actually i was nice i made a comment on that post i said hey great job i made some modifications to get it working uh and you can go to my github and and look in the wind folder in my github repository you get a docker file uh a startup script and a vagrant file that'll make it work for you um so i shared that with the original author i haven't checked back to see if he saw that or commented on it but there was a lot of people lots of questions like i got this error i'm like i got around that error and then like i got this error i'm like i got that error too like what's the and there was no solution so i had to do a lot of a lot of googling on the internet and experimental trial and error uh to get this to work now the other option is um detection lab is great and detection lab also can spin all of this up um but not inside of docker so you can spin all this up using vagrant and lib vert which was somewhat new to me the virtualization layer built on top of linux uh that uses qmu so you can you can spin this up natively uh on linux i does windows support vagrant and all that stuff i'm not it does yes there are windows vagrant systems yeah so you could spin this up and that that's more like running virtual machines rather than containers right um and so a lot of people uh tyler i know you do do this probably on hyper-v right uh there are some people that basically have a way i think you shared this link with me actually that when you're a penetration tester and you want to spin up a windows environment with a domain controller maybe you want to mirror a customer environment you can use similar techniques to what i'm showing you today um to do that in in a couple different ways using the technologies i just described right so the the use case what i'm getting at long-winded way of saying the use case is i want to spin up a lab i'm able to customize some of the components but i want to just spin it up right i don't want to have to go through a lot of work to spin it up i want to run some commands and spin it up they get that right you can you can do this all with powershell using the adds deployment which will deploy in azure or you can have it deployed to a hyper-v via powershell you can do it with a vmware esxi i personally use vmware workstation if i'm building custom stuff full labs i use powershell and spend that to azure uh vagrant obviously detection lab red oak all of those are great options but this the docker version is one kind of missing piece to to having used all the technologies and they all have strong suits and different ways and use cases the terraform stuff works great with azure and aws not so good for local labs now there's a bunch of things for ansible but each of these has their strong suits so it's good to have all of them for individual use cases this is very fast very low resource and very diverse for for being able to do quick things yeah and my use case for it was i had this really cool lab that i spent a lot of time on that was based on docker and primarily linux containers and i wanted to throw windows in the mix and not lose what i you know had done and not recreate everything because uh i think one of the positive things on this is it's really easy to configure vulnerable containers and containers that run different types of software super easy like docker file even if you're just like learning how to build a container using a docker file it's pretty straightforward right if you know linux it's not that far away from being able to spin up containers in different configurations uh it's totally easy it's really not that hard i mean come on if you can do this linux windows and docker is no big deal nothing happens you don't let the magic smoke out it's not a problem right yeah well uh the kind of downside is the this lib vert um sub system i would call it is relatively new and it's like a little messy like you're gonna spend some time uh getting that to work it's a little a little dicey and it does heavily rely on some of the stuff on your host uh operating system so uh i constructed a go ahead lee as i say doesn't doesn't window get kind of upset if you don't have some sort of a head on it um so for the in when i spin up windows um it is not headless i can remote desktop into it oh okay so we virtualized that i can't i can't show you i'm having some issue on this system i spun up my other workstation just fine i sped up on this system i got some error about ipv6 or something so i gotta adjust the configuration but i can you can get it from the github repository and i'll show you all the configuration and walking through it um well actually i can't show it running it's basically a powershell module you can actually turn the gui on and off as well as deploy things like server core that doesn't have a gui but again that's a module with inside powershell to enable that which takes a little bit lower resources and things like domain controllers you can obviously do that for labs with the the core environment which makes it really nice but everything's managed through server manager anyway now so you do have the the option rdp still but the the server manager covers what what you usually interact with us and so the blog post gives me gives you the image if you go to that blog post that i link to in the show notes uh it gives you kind of the diagram right you got linux as your host operating system you've got docker uh in here you can see this docker file i'm building an ubuntu 20.04 uh container so that runs on top of docker now inside of this container you can see the packages i'm installing one of those is qmu kvm the other is lib vert which gives you a virtualization layer inside of your docker container which i was like uh like don't use this in production is what i'm saying great for a lab i would never deploy this uh in production in this lib vert stuff from what i've been reading is is kind of new um so you get your base packages installed then you pull down uh vagrant into the container and install it then you install the vagrant lib vert plug-in then the next line right here that we're looking at is line 22. this is where you add your box from uh vagrant now i want to pause on this one i probably should create my own vagrant box and control everything that goes inside of it that is like in probably another two-part technical segment to show you how to do that tyler have you done that if you create your own vagrant box like it's it's involved there's a lot to that and i highly recommend it if you're building larger scale or more complicated lab setups especially when you start to talk about domains you will probably have to do that anyway right but yeah in the ass so i'm pulling a a vagrant box uh from a user who has a pretty good repository i'm trusting him at this time again i'm not using this production i'm using a lab he could totally put malware or if it's accountcase compromise you can put malware in there um his repository is called peru and he has a windows 10 enterprise x64 evaluation vagrant box essentially like a virtual machine that i'm pulling down inside of my container um so i'm adding that again i just want to note like this is someone else's you know virtual machine file essentially so take that with a grain of salt i'm doing this kind of an exercise to show that it does work the next step i would recommend is again have your own box or you could also use some of the detection lab stuff they have some boxes available it's configured differently than what i'm showing you here uh right now um but i trust um is it chris long that does detection lab they get that right yeah i trust chris more than this other guy who i don't know um but you could you could adopt this and and pull in some of the boxes that um uh chris is building in in detection lab and he has a great job and he now supports uh vagrant in liber as well he calls it very highly experimental so just throwing that out there that means it'll break yeah he's like i'm not even answering like if you open up a git issue or send me an email it's like i don't want to hear it like this is experimental um and i don't blame them because it is so that means that only 50 companies will put it in production next week yeah probably i know that happens right so i'm doing my vagrant uh init which spins it up and i'm copying over my vagrant file and my startup so i did have to configure my own vagrant file which i don't think this might have been in the original one but i've added stuff uh into it and kind of balanced it so interestingly enough in the original post they were like oh you need to open up you know port 3389 for rdp but since the box i'm pulling down is built from his vagrant file he was already doing that so i got an error that said that port's already been forwarding so this vagrant file kind of is a second stage uh vagrant file if you will um so in here i'm sending the the box to that uh the box that is available in the vagrant cloud uh i am setting a specific ip address for this is the ip address within like qmu right so there's a network diagram in the blog post that i link to and uh you're going through multiple network layers here right so this windows box has a 192.168.121.10 address but that's in the virtualization the qmu that's running inside of docker inside of docker docker the docker container has a network interface on the 192.168 121 network and has an inter network interface on in my configuration 10.1.1.16 which then is a bridge to my host network all right to go if you go look at the diagram in the medium post it'll it'll make sense to you um so that so i uh in the original post he was not setting an ip address and that's one of the changes i made i said i want a static ip address so i can write firewall rules with that static ip address on the fly essentially it lets you do docker compose up and it'll all spin up and everything has static ips i'm also forwarding port 445 so i can attack it which is awesome and then i am that little snippet of powershell on line six right there will disable the firewall now you can get crazy with powershell if you want you can do in-line powershell as i'm showing you here on line six you can integrate if you've written a powershell script and saved it to a file you can have it execute as part of this vagrant file as well so you can whatever you can do in powershell you can do to this system uh as it's spinning up which is awesome i got a bunch of trial and error stuff here that i commented out all right so then once question no okay once the container has spun up right so in the dockerfile in the container right uh my entry point is startup.sh so now we're starting to execute this script inside the container uh you have to change permissions on uh this is actually your your this is mapped into the container i can show you that as well so your uh if it's a volume or a file in my docker compose uh it's a device so you're mapping the dev kvm device into your container yeah which gives it the ability to run qmu using that kvm device that it needs uh so you're changing the permissions on that uh then you're running lib vert d invert log d pretty simple then you're bringing up uh the vagrant uh box that you configured in your vagrant file and also in your dockerfile right so this actually starts up windows 10. then i had to go back and remember all my ip tables chains and uh tags and all kinds of routing and all that stuff right and so uh in i believe i don't know if it was the version i think it was the version of ubuntu that so i switched from his original post head ubuntu 1804 that wasn't working so i built this on ubuntu 20.04 and i believe it was that changed that changed all the firewall rules so all of the chains were named differently so his original ip tables rules did not work at all it was like iptable's chain not found and i'm like oh so to rewrite all of the ip tables rules to be compatible with how lib vert d is implemented in ubuntu 2004 it that it was not fun um but i did figure it out so basically you are port forwarding from the docker network interface into the virtual network interface that's running in qmu inside of the container that's all those rules do the last four disable some of the the other rules that would make this network traffic uh be blocked so i'm disabling that but all the rules above that lines 12 through 20 are essentially poor forwarding and natting uh between the docker virtual network and the qmu virtual network make sense that was pain in the ass yeah um and how long did this take you to go through when we were counting uh days and how much sanity so yes it's a lot of a lot of trial and error yes so with all that was it worth not just sucking it up and go to ubuntu 18. it didn't work for me to ubuntu 18. oh yeah so we had other errors and when you read his original post and you read the comments to that lee it was a bunch of other people going i tried this and it didn't work and i got this error i'm like yeah i got that error too oh my god okay fix that so i'm like well if i'm going to make it work i might as well make it work in ubuntu 2004 which is what i'm most familiar with and it's the more recent version um yeah so i got different errors right but i'm like these are my errors now and i'll fix them right so rather than trying to troubleshoot someone else's errors i'll troubleshoot my errors so you're either going to fix them or drink heavily or both right right it was got it definitely some drinking involved to get to this point then i had to convert his so he was not running it in docker compose uh he was just running a dock a run command so i had to translate his docker run command so all the switches uh that he added to his docker run command i had to translate into docker compose which sounds like it's like really hard smashing it you were more in your element there where i think you've done a lot with docker and you have that yeah that was that was one i was like i'm gonna have to translate this all the compose i'm like oh that wasn't that wasn't that bad like so basically um uh all the stuff i talked about before now he is running this as a privileged container this is one thing i haven't gone back and tested and i'm not sure if that's to give the container access to the devices uh dev kvm and devnet ton uh i'm not sure if the privilege container has to be running at a you know basically without restriction on linux to pull this off or not i haven't tried changing that what i also found was interesting is in addition to setting privilege to true he's adding some linux capabilities for netadmin and sysadmin and i don't remember by default i think this might even vary across docker versions if you set a container at a privileged i think that gives it all the capabilities or it might just be a larger subset of those capabilities that does not include net admin or sysadmin but again i haven't gone through the testing to go what if i make it not privileged and specifically allow the capabilities that it needs um or can i just set privilege to true and not add those capabilities in any case security wise you should never do this in any of your containers no probably bad i i can't help but wonder how much of that is related to trying to run basically a virtual machine infrastructure on top of that container inside the container it's going to do all kinds of stuff to memory and other drivers that you need admin for that's what i'm thinking right so yeah but again the reason you never do this in production is because it will enable people to do docker machine escapes um yep yeah which is which is bad which would be interesting if you got in windows could you do a virtual machine escape that got you into docker and then do a docker escape into the host it's very much inception uh and you have to map a volume too i didn't look into what i don't remember what the c group is but you got to map a volume uh as part of it again like you're running virtualization inside of a container on top of linux so there's stuff it needs access to as well as the devices so if you do this oh here's a link to the post right here too so if you do this it doesn't work on this machine unfortunately so i can't show it to you running because i don't know what something happened i need to fix but i was like i pulled down the uh master branch from the repo and i ran it on my other machine and it worked i was like okay good um so there's that um let's see let's go here yeah so a bed some hurry uh july 20th 2020 uh wrote this awesome post uh this is everything i just talked about right um i wanted to show you the network uh so yeah this is the inception diagram that i was telling you about right kvm qmu hypervisor windows 10 running on top of that linux container docker daemon ubuntu linux space os which i didn't think was possible but obviously it is um and i just wanted to show you the where is his networking diagram he had a network diagram on here somewhere i think i might have scrolled past it already maybe not maybe it's down here it does yeah right here yeah so you get your main os which has uh a virtual network interface on uh docker zero now his ip addresses are 172 17.00 that's the default i've changed that in docker to create my own network so i can statically assign ip addresses um to uh to the network adapters right so get your docker zero that container eth0 interface on the container maps to that docker bridge network with my host then there's another virtual network verbi one is my containers interface on the qmu virtual network and i'm statically mapping he was not i'm statically mapping 192.168.121.10 in there so my case my windows host your windows host that you would access in this diagram would be 172 17.0.2 that would take port 3389 with the ip tables rules and send that over to your vagrant instance which is running windows 10. pretty amazing yeah without this post it would have taken me a whole heck of a lot longer so even though like stuff didn't work in this post i give the author full credit because i would have gotten any of this to work if it had it not been for this post so thank you so you can pull that lever right right it's pretty it's pretty awesome this is pretty good i give you a lot of credit man for assembling all these pieces well done thank you so uh in my dev branch i have oh so the peru guy uh that publishes the vagrant boxes also has a windows 2016 domain controller box and i've got that pulled in so i can actually get windows 10 and windows 2016 domain controller and all of my linux docker containers with c2 linux targets kali all that stuff all running on the same network what i am uh when i get some time to come back to this working on now because i suck at windows in powershell i'm just learning powershell um what i need to do now is get the windows 10 box to join well i have to create right you got to create the domain first on the domain controller and then join the windows 10 box to that domain that can all be done though too with what that can all be done with code you can set your dhcp servers your dns scope zones uh promote set your fisma roles up and add add the domain and then add the box the domain so you first add the roles and add the domain and then set up the the box and dns and that's all done with powershell yeah yeah since i don't know powershell that well that's i'm challenged i'm gonna you have to spend some more time with me tyler and the other example and in the other example tyler sent me a lot of examples detection lab has an example it's like way more stuff than i needed like way more stuff than me i'm like i just need to create a domain and join one computer to it so i need like a subset of that code and not being great with powershell yet uh you know it challenging but all cool stuff to learn powershell is pretty amazing i'm actually kind of digging it now now i have a reason to really learn because i want to get this working so because i think now uh so you know once that's working now i've got this really kind of cool lab that i can show stuff right and webcasts and the shows right they've got a working windows environment got some linux targets they've got c2 infrastructure all kind of sitting on this flat network which is kind of convenient right where you can show different kinds of attacks and all kinds of stuff so how is it keeping track of what talks to what using what address ranges so all of the good question lee excuse me in my docker file in the docker compose script there is a network that i create called vulhub net each container i statically assign an ip address uh on that network so here's where i define the network it's a bridged network with my host adapter uh on my linux box um and i define the subnet in the gateway so my host adapter is always 10.1.1.1 um that's the subnet and then as long as i put in this this little block right here 182 through 184 lines of code i can statically assign ip addresses to all of the containers and then i modify my host file on my linux host on the host system to map these all out by name and i call these the this is a little out of date um but i have to update this because there's a couple more uh that's not in this list so in my etsy host file i map all of these out and i so i can reference them by name so within the containers i set a container name i set a hostname so this is always wind10 and then if i modify my host file i can reference it as wind10 so i don't get confused like wait what ip address is the windows box again like you can reference everything in the lab by name which is convenient because i forget i'm like wait which one was my vulnerable log for j oh right like which one is my driver c2 like it's all named ip addressed all the same every time which is convenient anybody do you think the overhead what do you think the overhead for the basically extra layer of virtualization in there is costing you i think it much it's a good question no it's i haven't like done any performance or load testing but also like this isn't the fastest laptop in the world um it it doesn't seem to struggle if i spin this whole thing up on it my other computer is just a bad test it's like a 24 core amd thread ripper so of course with like 256 gigs of ram yeah so that one has no problem spinning up the lab but i will tell you that in production i know people that run multiple levels of vmware yeah um you know i mean the fact that there's actually a an faq that says okay you can run vmware inside vmware but you have to have a dot release behind you know like like and they're running two or three levels because they're running a hypervisor inside of a hypervisor like yeah like i've seen people run kubernetes inside of a docker container right yeah it works you can run esxi on vmware workstation or fusion and that way you can emulate a vmware clustered environment um yeah but as far as what the overhead is i mean i wouldn't run this on like a slower system necessarily but again i haven't noticed like my machine completely in the bed because i spun this up and you could do some performance tuning uh actually in vagrant in your vagrant file there's a ton more options there if you go to the vagrant documentation um they pretty well document all the different options and so you can ty what can you specify in there ram cpu there's a bunch of stuff in your vagrant file right that you can size for this yeah you can customize that vagrant file very very heavily in fact you do that when you do a lot of the domain stuff because you're going to have you know something that your primary physical domain controller that has dns dhcp maybe you have some some fips or exchange or sql all those things need to be sized and you can do that with inside the config files it's really awesome uh i i feel like even if you bought like a pretty beefy machine it's still i think cheaper than paying you know a monthly cost uh or you know for your cloud i mean you'd only spin it up in your cloud when you're using it right but if you're paying on storage and compute in whatever cloud that you're in you know you're giving someone some money every time you want to spin up your lab i'm like that'd be kind of nice to just run this locally too also i want to point out these are highly vulnerable containers in instances like really like easy remote exploitation uh on these as well as like c2 and kali built in this is not something you just want to spin up willy-nilly uh inside the cloud i mean obviously you can put restrictions on that when you spin it up in the cloud but if you like accidentally expose this to the internet it could be a bad day of course you could just spin it down and and fix everything because it is just a lab but you know you do run that risk um i think it's awesome i think there's multiple labs out there right that use similar technologies so students can essentially spin all this up uh in an environment but i think it's also nice for you to be able to control and configure your uh environment as well i mean you could if it was your cloud you could do that as well but in this example right i can change any of these docker containers i can add new ones i can remove them so like when you know log4j isn't the shiny new hotness something else is i can swap that in if i need what was the vulnerability du jour uh cassandra and magento so i mean i could spin those up in there right and you could spin up a separate container for that or uh build it into some other container uh i know samba had a vulnerability uh recently we talked about as well so that you can also go ahead you can also modify the and and integrate these into your cloud lab or cloud solution using you know ansible or terraform these work these work well for spinning stuff up within sight of additional cloud environments and are very adaptable to the the other languages so yeah be awesome uh i'm most familiar with aws um i'm assuming i've not tried it but i'm assuming you could spin up a windows environment in aws and a container environment in ecs link them all together to be on the same network from what i understand from aw i've not tried that but i'm sure that's possible right yeah it's pretty straightforward and especially if you're using vagrant files or ansible yeah and then you use terraform as the orchestrator to spin it all up in the cloud yeah whatever your flavor is this weekend yeah there's multiple ways to do that even in aws right to spin all spin all this up yeah there's a lot of people doing that too and i think that's cool you know maybe you could adapt it to do that i mean we can just keep building on this because it's lots of fun um but you know the use case for me i think it's nice for me to have it local i can modify it spin it up and go hey like here's how you exploit this new exploit that's going to come out next month like here's how you exploit it and get c2 communications going yeah it's fun it's good stuff if i get really ambitious i'll start writing up labs for it so you can pull this down and spin it up and then follow along on a document and be like all right here's step by step how you use like this from you know step one to step 12. i've done that already for at least one of the segments that i did so keep building on it pretty cool man i got to give you credit that is actually pretty cool you've developed a way to put a whole lab on a work on a uh on a laptop or a decent desktop without using virtualbox which is becoming more and more not usable these days without you you know and and doing everything pretty straightforwardly yeah um thanks i have i have virtualbox uh running in a couple different instances to spin up labs and i did find it takes one it takes up a lot of resources and two like if it's you don't create it like within virtualbox four virtual box stuff goes horribly wrong right like it likes to be all native within uh within virtualbox but yeah it does it does suck up some uh resources big time yeah virtualbox is getting like i don't know my personal opinions my personal opinion is that virtualbox is getting behind the times yeah it is free it is free i mean for those just starting out right you've got to pay for vmware workstation or esxi and you would have to or pay for your cloud computing costs right well what is what is docker cost what is uh this is all free right it's all free how do you just build everybody a free lab my question for you folks is how does the windows licensing work i can spin this up in eval mode no harm no foul yeah microsoft provides 180 day trial for development and labs and that's part of their eula for that particular license and uh version so i'm not even violating the euler because i really am using this for our lab yep that's awesome that's awesome yeah so i think it's a really cheap way to do it yeah it really is honestly uh do you have are you documenting this how well is it documented that somebody who's not incredibly sophisticated could put this together the documentation is not great there is some documentation there but it is it is not it is not great because i'm like i gotta get this working and then show it on the show and then i never go back and flash up paul's looking for an intern to follow him around dictating yes [Music] they're gonna get intern dylan uh on this project and helping me document it yes that should be the next step i should have dylan spin this up and run into all the issues and then help with the documentation that would actually be pretty good and it could uh good learning you know like for example we had kevin on the on the webcast today right so uh secure ideas manages the samurai wtf distribution yeah and they they manage several different uh uh open source projects this would be a pretty nice open source project to come out of security weekly agreed agreed very yeah i hope people use it enjoy it and you know send me pull requests or if you want to write up some documentation like this is open and free right this is this is so people can learn right awesome with that uh we'll take a short break come back talk about security news for this week stay tuned

Channel: Security Weekly - A CRA Resource

Views: 26,135

Keywords: Josh Marpet, Lee Neely, Paul Asadoorian, Tyler Robinson, artificial intelligence, cloud security, endpoint detection, endpoint prevention, endpoint protection, firewall, ids, information security, intrusion detection, ips, machine learning, mssp, network security, quantum computing, security services, threat intelligence

Id: ZebsjUvJqwg

Length: 37min 59sec (2279 seconds)

Published: Thu Mar 10 2022