How the Department of Defense Moved to Kubernetes and Istio - Nicolas Chaillan

thanks everyone and thank you for coming to experimentation we're going to be talking about how the Department of Defense moved to cabanas and East Joe there's a little Eco here we go so we'll be sharing with you a little bit of the story of the last eighteen sixteen months now that I joined the Department of Defense and in case you don't understand my French accent we don't have subtitles so hopefully you're going to be able to follow on the screen what I'm saying but that tells you the lack of talent we have in the Department of Defense and we need Americans to help and join us so please come work for us so I'm the chief software officer of the FDA Air Force but I'm also the DoD and price deficit appreciative co-lead with the CIO of the DoD CIO so we're pushing deaf sack ups for the entire Department of Defense and that happened in 16 months so we'll show you a little bit of of what we've been doing but I'm gonna ask you a question first I guess I can walk here we go I'm gonna stay here a little question for show of hands who has put Q manatees on a fire of jet before okay who has path Q babies on the plane ah few more hands what about a ship exciting so I guess for the rest of you guys you're gonna see a little bit about what we've been doing on the final jet we're gonna show you today that we have put communities on f-16 and how that happened but first I'm gonna show you a little bit about the problem and what we were facing in DoD when I started which is really initially focused on waterfall right so we have teams using waterfall dod-wide not using agile with a software delivery every three to ten years that's obviously a very slow pace with the process too accurate it's software that can take anywhere from three months to a year so you have a very slow manual process to accredited software with manual testing manual security gates and that's obviously not what we want in a in a traditional ccd pipeline on top of that we have massive teams massive programs with a lot of people and our corners with a Dib which are the contractors and vandals are mostly using waterfowl don't really understand that scale or gel which is 19 years old let alone DevOps which is 10 years old so that's obviously a big impediment to moving faster and bringing modern software best practices so on top of that you had a lack of talent IT Enterprise Services cloud access and high-speed Internet even DNS can be a problem for us at scale so we really have to solve all the the common layers that you find in your companies that are brought to you to you as a ton key service to use we had to kind of build all these layers and we'll show you a little bit about the layers we built today but for us do you see a problem with this that's the defense acquisition lifecycle management so I wanted to show you every little box and we're going to go over every little box here I'm gonna spend a if you have a month we can go and talk about each box here but I don't think I'm gonna do that right now but just to show you kind of the current model which is absolutely not broken and what server but yet you must rapidly adapt to challenges and work as a team a loss team with various technologies and bring it with you right even to space with quite a few sensors also a lot of help and all while building software innovations easy right so that's what we created the Duty Enterprise def Cyclops initiative and what I'm excited about is that the entire stack is open source and publicly accessible anyone can go and consume our hardened containers can use our infrastructure as code and avoid vendor lock-in because we picked cube Indies and OC I complain containers mostly one for the abstraction two to avoid vendor lock-in at the infrastructure layer at the platform layer and really bring rapid prototyping to life as you can see we created the centralized artifact repo that's where we can put our hardened containers we are walking on 170 containers to re-check the supply chain make sure they are as secure by default as possible nothing on that hardening is really dirty secret sauce so any company can really use that it's pretty exciting because it's it's stepping up the game when it comes to baked insecurity and default settings that are already kind of following best practices on top of that so you see the two links here the the repo the sous cadre post you have full access to the source code and then you can also go to the decal repo which is the binary side and you can open an account and get access to the binaries as well which are signed so we can trust the containers as well what's pretty interesting as well is we have that baked in security 0co trust stack with East Joe so we picked these two as well to make sure we could have a zero trust onto the container layer and we also use K native so we have East you with n boy as well down to the function layer to be able to whitelist access of course blacklist by default denial and whitelist which container can't - which container very important for us to reduce the attack surface and be able to to mitigate threats on top of that you had kind of the the key native aspect to be able to do several s again we don't want to be locked into any club provider you have to remember for the DoD we have classified clouds we have environments that are disconnected and deployed at you know different scale so if you're putting committees on the jet you don't have time as an API or the Google API is right to be able to orchestrate the stack so you want to be able to bring everything with you and be in a disconnected mode which is much harder than anything else right because we're very used to updating using internet and having connectivity to Internet getting the dates directly from Internet right so for us we have to bring the entire stack with us and really do that in a than in a disconnected mode right with no internet we also use OPA or the open policy agents we have a few custom policies we also working on a on a custom admission controller with more baked in security particularly when it comes to East Joe and disabling East you as well so we want to make sure Easter is always running across the entire stack for us to be able to the GeoTrust enforcement so we created two teams that was a big win for the department the cloud one team and the platform one team so cloud one is really the infrastructure team bringing compute storage and baked in cyber and then you add the platform one team that can bring a CI CD pipeline the QA nice cluster and everything for us is container is running so the CI CD pipeline itself is containerized of course we had also to bring kind of that training at scale we have a hundred thousand people we have to train so you can imagine the complexity of training a hundred thousand people and we want to do it in a self paced environment self training model where they can go to to a tool be able to watch the the curriculums from commercial companies and have access to a cloud to put it into practice and not just learning a vacuum so a very big win for us to do a few pilots to get feedback on on a self-learning capability that's not too opinionated to a qHD stroke we can't to be locked into vmware OpenShift or any any product right we want to be able to have that code work on any CN CF certified cluster the next aspect is really showing you the layers and that's where you know we wanted to automate using infratry s code all these different aspects and be able to decouple the layers right so the cloud layer we can swap from a classified cloud to a on-premise environment to a to a jet right with virtualization and then you're gonna add on top of that a cube and each cluster that could be any any distribution that's compliant to to the CN CF requirements and then you're gonna add on that the the CIC D pipeline which will be using the Holland containers from the repo and then you had the service smash which really brings you that zero trust stack and then what's left for our teams is really the application layer makes that makes it much easier for them to move to micro services and taking mono-ethnic application turning it into smaller codebase and enabling reuse right you can imagine a lot of the code can be reused across teams right so if you take FCC f15 f-16 f-22 f-35 jets you can imagine that if we build things in a more modular way we can reuse a lot of the code across teams so that's just an example of a pipeline for us everything runs on humanity's everything is containerize using the hardened containers we don't believe in one-size-fits-all approach so we give freedom to the team to swap you know containers for us it's kind of Lego blocks to be able to swap between products so if someone likes you know sonarqube better than 45 or checkmarks they can pick same concept for you know programming languages we support 16 programming languages 23 databases so that reopens the door to picking the best tool to get the job done so you can see we cover every aspect of of the lifecycle to make sure we can take it all the way from planning to to continuous monitoring which is really the corner piece for us of that set ups a lot of people ask us why we use dev sack ups and our DevOps the sack for us it's not just we're gonna do some static dynamic crisis it's really the country's monitoring side with behavior detection zero trusts reducing the attack surface by default and really automating that so the development teams don't have to think about it so the platform team is really supposed to bring a few minutes plus so that's already secured right so implementing the see the cia's benchmark for communities and adding a lot of the hardening we add on top of that with East Joe and ensuring that people cannot disable it so when you look at that cycle continuous agree stack that's how we called it but there's a few things you know for the technical guys here that are not psycho I get it could be demonstrates could be other stuff but we call it the cycle to keep it simple for the people in the Pentagon so they can understand what we're talking about more or less so in this case we have multiple layers right we have the zero trust stack using East geode onto the container or the function layer with canid okay native and then we wanted to have you can think of the DoD having thousands of cluster right because we don't have a big cluster we have just a lot of small clusters right each weapon system each system is business system cyber offense defense system is going to have their own little cluster at different classification levels right so we need to be able to video to make the process so we using fresh address code for that at all the layers you can swap you know open ship for PKS or PKS hope and shift and of course we want to have centralized logging and telemetry so we mount a efk hence even though I wear a suit I have socks pretty cool socks trendy trendy socks so efk and then you add on top of that the continue scanning alerting and of course a behavioral detection for us it's not just scanning of CVS you find a lot of products today on the container scanning side that I'll focus on CBS which is great but that's really not enough we're pushing a lot on the behavioral detection side to detect what the container is running in time run time and of course we scan the registry and we kind of build as well but really we find a lot of value to them on the continuous monitoring side of the house and then on top of that a lot of people forget about inside of crap so for us it's very important to detect if a developer takes my container and is trying to remove the hardening we put into that container right and so here's uncle for that we create a custom police's to detect changes that would not be approved so we let teams customize containers right so they can take my openjdk Halden container and use a base image but then there's a point where I don't want them to remove my hardening right there's some stuff that they should not be allowed to change and for that we created custom policie that will detect changes in the docker files and prevent it from being built right so very important for us and then of course because with duty we have a fancy stick compliance who use openness gap to automate the stig aspects some more cool stuff now we're going to start showing you a little bit of the f16 stuff so first you know the the team built pretty cool I have to say simulators right so if you look at this it's kind of a giant earth looking bubble thing and yes earth is not flat I'm sorry so it's a bubble kind of thing right and if they have like projection things and the picture on the left even though I mean you can barely tell whether this is not a real view from the jet right this is actually all inside that simulator where we can have pilots sit and play with a software updates and all that that's pretty cool stuff but then I went to the team called Sony cube cube cube a nice sunny cube so Nick whatever I just want to make sure they understand so Sony cube is that team that I went to them and said you know can you put humanities in 45 days on the jet and so here here's what happened the team is interested in investigating coordinates on the jet primarily as a response to increasing warfighter capability delivery speed and a more modern development practice that matches industry we're able to deploy software much more rapidly and reliably it's the warfighter to help keep peace around the world and also just because it's applying a technology to a new platform it hasn't been applied to yeah some of the difficulties putting kubernetes on a jet so far have been mostly along the lines of learning and understanding the ecosystem all the other parts and pieces that are necessary to secure and provide visibility for what's happening have been a struggle but as we learn and continue to grow in knowledge those sort of concerns have become almost second nature and it's been really amazing to see the growth in capability that we're now able to just put together rapidly knowing the the building blocks that are part of that people say it can't be done and so we have to continually be persistent to try and prove that it can with those little increments of proving that then we're starting to see people come around and say yes this is a great idea let's continue to push forward some of the benefits of kubernetes that we're hoping to realize our high availability resiliency and the speed of delivery whenever there's a need to be able to produce a answer for that we can then more typically deploy containers onto the jet that contain new capabilities and we can quickly get them out to the pilot in the battlefield when he needs them has new threats arrived what this move represents for the Air Force is a step into the modern age and into the future it's very exciting I'm super excited to be a part of it all right let's give him a round of applause this is kind of what it looks like you can see the jet behind all the people in front of the Jets kind of blocking the view and fortunately but you can see that we we have a screen on the left side here and the screen is really a mural donut on the top screen so we can see better for the picture but really it's on the jet it's running East geo and humanity's on the jet and we did not have to change the legacy hardware right so this is all running on the legacy hardware in 45 days the point for the team was ready to demonstrate that one it could be done but to also benefit from the flexibility of running micro services and bringing new containers could be AI machine learning capabilities where you need to react to to some events and you also need to be able to be resilient in terms of you know software crashing or something going wrong so for us that that kind of flexibility was really critical and that's why we we've been pushing that abstraction layer it's also been exciting to see that we can test things on the cloud and automate a lot of the testing and we push it to the jet right and at that point you do a last set of testing but really o2 meeting as much as possible of the of the test stack before we get to the hardware so of course there's always a hard way in the loop test for us because we want to demonstrate that it's gonna behave the same way on the jet but it's pretty exciting to see that you can emulate and automate as much as possible both on the cyber scanning side and the testing side as well so if Cubana DS is good enough for duty weapon systems it's pretty good enough for your business so join us we need more tenants good people you guys are leading the pack when it comes to cue babies in the cloud Native community with a CNC F and we want to make sure that we all benefit from it so thank you for your time and I'll tech questions so that's the point for me to end early so if there's any questions please get to the mics any of the great presentation two questions one is synthesis in UD you Tod assume very high they clarified in the environment do you have to comply with a bedroom were socks in that way before you can deploy so that first question the second question is what's your support model do you live around the community or you work with any you know company just in case something goes wrong yeah so in term of FedRAMP FedRAMP is really necessary when we buy something as a SAS or passport lock that's why if companies have a cool cap ability if they give it to us as a container we can equate the container and run it inside our environment they can still sell the managed service and we pay per use and all that but as long as you give it to us as a container we can accredit it so you don't have to go through FedRAMP which can take a year and quite a bit of pain so FedRAMP is also live because we can go higher to unclassified clouds and for sure for us all these systems are going to classified environment so very important that we have it as a container and bring it with us so if you have a cool cuts open-source software that you're supporting as a company giving it to us as a container is the particular and so yes we work with a lot of companies we have about 25 companies on the repo commercial tools open-source tools like you know the gate labs the the OpenShift the PKS whether it's a it's a kaveri distro or whether it's a it's a container kind of product whether it's it's a source code repo whether it's a cyber scanning tool any kind of tool can be accredited by us and we cannot create Duty wide so that gives you access to the entire Department of Defense and many federal agencies also using the containers so you get access to DHS and DOJ and other agencies as well yeah great hey grant was replicated so you know kind of more on this topic you know I think be seeing kubernetes run in the jet is really quite a useful and interesting example but you know more to this like where else are you running kubernetes and data centers we're using it to power you know all the like sort of normal business activities that the air force is doing yeah that's been of course the large focus that's you know the jet is interesting but it's it's a tiny piece of the rest of the work we're doing we have a lot of business systems moving to cloud native environments moving to micro services being rebuilt right from the get-go so a lot of the committees cluster is of course for the backend side and AI machine learning capabilities as well great and that's all like high side so it's all sort of secure environment it's both you know we believe in the importance of being able to walk on the unclassified and push it up right and so being able to have that end class environment what you can bring stuff or maintain that and do your work it's critical for us and then you know pushing it up so we have every every layer great thank you very much thank you thank you for the talk Nicholas all right my question is specifically to why you chose Sto and how that's benefiting you in comparison to other service meshes such as link or D etc yeah I think you know we wanted to move fast we're going to partly supporting at least two or three different static meshes on the long run so we wanted to stop somewhere you know I think any boy is clearly a winner when it comes to the proxy side east geo until he's part of the CNC F or something like that is still debatable right and so for us we wanted to give it a shot and put it to tests and move fast so that's what's kind of the the step zero for us and now we're gonna also be looking at other options right there's a lot of commercial tools as well but you know that the key for us is really the proxy side all right can you you have this product that is deployed in a box in a disconnected environment can you talk a little bit about the development pipeline and what the developer experience is like is it connected disconnected and can you describe the CI CV pipeline yes so the the pipeline you know we're trying to do as much as we can on the unclassified side and bring it up and often you know what specify these settings or configuration settings so that usually is kind of a config file yeah Mel you know at the end of the pipeline that is brought you know so we have like fake data right and fake config settings on the end class site to do as much as we can on the in-class side but again some systems also the agar-agar is amol classified and that's a little bit harder so we're trying to bring the same CCD pipeline to all the classification levels but we're trying to do as much as we can on the N class side and that's why you know we use the containers to in the CD pipeline so there's no drift between environments right so you can think of and class having you know dev test staging prod but then you have the same thing on secret TS and so on right so that complexify the the if you had to maintain these CI CD pipeline without containers it would be a disaster of drifts and you know different tools with different versions so very important that we pre containers for that and you know the developer experience is just like you will see on inside any company on the N class side it gets a little bit more tricky on the classified environment because you have no internets you have to bring stuff with you from the N class but we're trying to make the push up right so it goes and class the secret to TS with no human in the loop ideally that's where we want to go next do you see yourself doing the singing air force after the jelly contract or do you thing you're going to keep doing these agnostic environments sorry can you say the last part again agnostic environment agnostic you said so so the point is are you so do you have plans to switch or to switch these current efforts after getting the jelly contract implemented or or what what what's the stake on that no I I think it's gonna be a case-by-case you know program but there's no plan on switching until until we get to the real scale you know I think it was still learning very fast a lot of things are changing and each team is allowed to make their own decision right so we're not Monday centrally these kind of decisions at the Air Force level is down like per per team level I'm wondering how do you handle distribution a distributed decision-making and redundancy while using kubernetes based flight control systems yeah so I mean that's that's a 20-hour discussion but I think on the on the on the simple side the key for us is to have systems where you know federated cluster is is tough for us because of classification you know and also because of age you know what have data if one cluster get hacked you want to be able to laterally move into other clusters you know so we're trying to really reduce the attack surface and so you know the HP's for us is like more hardware when it's disconnected it's gonna be more hardware so we can have redundancy on the hardware and more cluster or small nodes you know both on the control plane and the the nodes side and the rest is getting you know more and more complex per pair program so it's tough for me to answer the question so do you end up having like three independent covering these clusters on a jet can you said again do you run three concurrent kubernetes clusters on a jet yes thank you thanks for the presentation do you what did you find most challenging about the adoption of these tools that might inform improvements to kubernetes in this do I think a lot of the work done on the Coverity side is not very well designed for these connected environments you know I think a lot of it is supposed to be on cloud public clouds and using cloud API is it's not doing always a good job when it comes to being disconnected and being agnostic I feel the more we become opinionated in term of the stack and all these communities distributions now I think we have 90 now on the CNC F certified lists including quite a few that are from China which obviously we cannot use you can imagine the the disconnected side can be a problem for us if it's badly done and particularly when it comes to managing the cluster itself and updating the cluster self-sow you you talked about fit for use which is the release process of the release gating how did you take your DOD fit for use processes and fit that into a very fast pipeline here what's the one you sang Phipps sorry fit fit for you oh sorry yes that's that's the whole gated process where we really had to define steps and you know at the DoD level we just released a document that's actually accessible to anyone called the the DoD Enterprise def setups reference design that lists all the steps and the phases that software must go through to be able to be kind of DoD grade and it shows kind of all the stages of a pipeline and what's kind of a minimal requirement in terms of an MVP pipeline vectors like a real mature pipeline so it enables team to grow and so I think it's doing a pretty good job at listing all the steps both in terms of the basic static dynamic analysis side and the more complex you know Contino security behavior you know zero trust uh for pushing both at the endpoint to the all the way to the the cluster and really get making sure that no one is getting access to that cluster okay so your slide said like three years between releases what are the new time yeah it's it's continuous multiple times a day like you guys do on the commercial side hi um what specifically do you like about envoy and are there any features other than the disconnected environment support you'd like to see on the sto roadmap so what we like about envoy first you know it's it's open source and there's a pretty broad community and it's being adopted by most of the big players so that's always giving us reassurance that it's something that we can use when you start having companies vetting the entire stack on on envoi that's really a good sign that it's something that's going to be supported for a long time so envoi is also supporting a lot of features even though we want to kind of push the envelope when it comes to new capabilities like UDP support for us is important things like that so they still have gaps and it's still pretty emerging technology but can you repeat the second part of the question it's do a feature wish list other than the disconnected environments it's all your cutting like your cutting up your wish list for issue well you it's Jo I mean it's also somebody's doing and voi and and Easter but I think UDP support for us was a big one I think the more we're gonna have integral interaction between K native using e Co and the container of stack using East you having a single pane of glass that really doesn't have kind of two set of deployments and it's most seamless across Canada and and each two on the cluster will be critical and making sure that it's most seamless I don't know what else we would have to ask my team I'm sure they have a lot of ideas hi thanks for the presentation I see you guys rely heavily on open source which is good awesome and also you mentioned you had to make you make you customize some of the projects you work do you guys contribute back an app stream so we're walking on it you know we have a lot of partners that contribute back whether it's a VMware or Red Hat so often we're going to go through our partners because it's easier they already contribute back and it's an easier process and for the DoD but we were trying to do better that's why the repo is open people can come and do pull requests and really walk with us and we of course anytime we find something whether it's on East you or cube a days I'm actually goes back upstream because we're not gonna fork software so very critical for us so you're not trying to develop a new gaming platform right no okay the real question how do you bring this open source keeping like culture into it very closed as a culture yeah I think it's getting better you know first we have it Monday in duty to be able to publish 20% of our software open tools that's not classified which is already hard to do because it's not just about making it open source is sustaining it and making sure there is a community behind it as you know if it's just a stale project it doesn't bring any value to the Oh community so one is we want to do right by the community and have real teams behind the the projects that's why the the container stack for us and the hardening of containers is something where we're spending quite a bit of money to sustain and continuously update the containers and making sure we we keep track of that and really the cultural shift pretty obvious I think security by efficient is just not gonna work I think people start to understand that after you put a system into the hands of a hacker at DEFCON and they get in into 15 minutes or ten minutes or five minutes even though they knew nothing about it you know prior to accessing it that's usually good evidence that office keishon is not good enough and open source actually but a lot of eyes on code and making sure that we actually pay attention to the beauty of the code as well so very important to DoD today to to have that also stack that's now locked into one company hi thank you for the talk Jory Morrison for mitre I'm seeing a lot I know we all are seeing a lot of agile devops transformations across the DoD right now and so I'm wondering how you plan to make these service offerings available more broadly outside of small sections of the Air Force and across the entire DoD and also the second part of that question what is the point of convergence between this initiative and other initiatives we're going on right now like castle run that have a tremendous amount of momentum but maybe have slightly different approaches yeah how they do business and technologies so obviously Cass Aaron and and the entire force has different you know teams working on software cuz everyone is moving to the same stack than we are they started you know pretty early they learn a lot they brought a ton about you I think we would not have been able to do what we're doing now without castle run right they they kind of opened the the culture shift and they demonstrated that you can get it done but now we're moving to the enterprise right that's broader scale in fact the Air Force is supporting all of the Ori not just the Air Force right so the Navy the army as well and so cloud one and from one all kind of the two team will be able to bring a dev psych ops pipeline in a week with an ATO right and that's kind of the goal for us is to have that enterprise capability that that anyone can go and leverage without having to rebuild all these of these tacks so you know we have a full briefed on the on the plan one platform one on the CSA website I couldn't get into a lot of details here but that there's a lot of information online now yeah thank you I'm currently working on a program where we don't have access to a lot of these service offerings so I'd love to follow up shoot us an email we want to make sure you get access to it will do thank you awesome I with a lot of this cloud native tooling that you're bringing in and skills and people you'll be bringing in to support it what sort of process and people challenges have you had to adapt to such a closed environment what sort of what so you cut off with the idea in in a closed environment like this bringing in people who have got skills in these you know traditionally connected and open platforms what sort of process and people challenges have you had to adapt to that well you know these days obviously a lot of culture shift that has to happen for the teams to be able to even agree to using these kind of technologies so it's a it's a training in the value of the return on investment on the abstraction layer I think we get we get locked into a lot of tools and for us it's very important not to get locked in anymore and to have these open source tacks and be able to move to different products so I think that's been kind of the the carrot and also be able to accredit the software centrally and and do it at scale so each team doesn't have to do that that was the other way for us to do the the change right and I'm not sure i answering the question but I hope I did hello I have a question on the sidecar container stack um how you packaging it like all of those the login scan and all that is that packaged as a container or yes so we use containers of course for the day one we use helm and then we use operators for the day to stuff science allocation that doesn't run as a true sidecar or all pens of the tool you know some tools do right like flu Andy will run as a cycle and then of course the envoi will be a psycho as well and some are more like the among sets you know and stuff like that yeah I worked for one of those government agencies and there's no tutorial to set up a kubernetes stack on AWS gov cloud right I don't have the same AM eyes as some of the features don't exist do you guys have a way to share some of that knowledge to to help help us yeah on the right we have the the complete in fresh dress code to spin up an open ship for example to gov cloud with complete automation already Stig's already passing the the requirements we're gonna do better and improve it I think the documentation is lacking right now we're doing go to find that information the repo link was on the slide here and that's that link here the two links you see there yep thank you sure are we out of time okay sorry no just one more just one okay four sorry but enterprises are struggling with the adoption of all of open-source products and they have built in like very heavy processes around getting new software into regulated industries like financial health care and so on you are on the level you know up there insane kinda stuff so how heavy is the process of incorporating new products new software solutions into your environment and is there any it's kind of a little bit linked to the question before is there any program in place where you could actually probably contribute bad contribute back some of the tools that you have used as a distribution that was you know vetted by DoD so that enterprises can do okay this is good I can just take it right so the repo has all the source code other than containers so that's where it would find back all the tool that we secured so we - the open source tool we added configuration changes to make it do degrade so you'll see on that source code repo you're gonna find that back to the community right and then the process to equity the software the keys to use the same to so we have a CCD pipeline to automate the same process scanning and then the key for us is a country's monitoring side to really see what's going on in run time because you're not going to detect zero days right you can scan static crisis all day long with the number of updates of each open source stacks it's gonna take a long time to do that manually or even automatically so at some point you have to trust the computer monitoring site to detect so you can always tell those companies that if DoD can do it I guess any organization can do it as well so you say that you're subjecting all the open source tools that you're using to your internal scanning and otherwise that's correct thank you sorry we out of time thank you all right thank you again great talk [Applause]

Channel: CNCF [Cloud Native Computing Foundation]

Views: 28,585

Keywords:

Id: YjZ4AZ7hRM0

Length: 41min 13sec (2473 seconds)

Published: Fri Nov 22 2019

Reddit Comments

[u/sunk_cost_phallus]

Answer: they haven’t at all and this video is a fairytale that has no basis in reality. A couple of projects accidentally got done before the procurement mess caught up with it and now it’s back to captured by dinosaur dev shops who have CMMI and quals for the bad old ways.