How Angular manages community contributions and repositories | Joey Perrott | #AngularConnect

[Applause] [Music] hi everybody my name is Joey parrot I'm angular's developer infrastructure lean that's developer infrastructure for the angular team not for everyone else unfortunately though a lot of the same things do apply so today I wanted to look at a few pieces of our infrastructure and tooling and kind of what we do to manage such a large project on github before we actually look at what we do to manage things I want to talk about a bit about what dev infra means to me so first as we all know angular part of angular's goal is to enable developers to make great applications so dev infra enables angular to enable developers to make great applications so we do this through trying to make the people who make angular as efficient as possible with our tooling and processes so as I mentioned we're pretty busy on github every week we receive more than 120 issues 135 pull requests and we need to be as efficient as possible to get through all of these things so much of this efficiency I like to think at least comes from that devam for work so Deborah is typically a black box for people if it's working properly you don't think about it you don't hear about it and you shouldn't need to but today we're going to kind of look inside that black box so to speak three things that I kind of want to cover in active closed issues and kind of the way that we are now automatically locking those building angular itself building the framework as fast as we as we can and then also some of the ways that we consistently manage our repositories on github itself so first we'll talk about keeping conversations current so here's the problem over time one of the biggest sources of noise in our repositories is actually from people that are from comments that happen on issues or PRS have been closed and in and been updated for a super long time somebody come in and comes in and comments something like me too which isn't really helpful to the conversation plus it's closed or they comment and say I'm having this issue still but they can't but now we've actually fixed it so the context this is not there anymore and it just creates a bunch of notifications for people that result in us asking them to just open a new issue with new context so how can we actually minimize that noise so we minimize that noise by encouraging people to have the conversations in in the relevant locations so we can prevent reviving these dead conversations and instead encourage people to just create a new issue by locking and leaving a message saying hey if you're still having this just go ahead and do something so I actually I'm gonna walk through how we do this specifically in our in our repository so first we have to decide when we would lock an issue so for us it became to two pieces of criteria we only want to lock issues that are already closed we're not looking to just stop people from from talking to us or to say that their issue is an important this is when we've already resolved something and then past that we want to make sure that the conversation is actually ended so for us we decided on a over 30 days of time where nothing has happened on the issue so now that we know what our criteria are we actually have to be able to discover what those issues are so that we can automatically lock them so we looked around at what our options for doing this work were and there's a few different different examples of setups that have already been doing this nothing quite fit what we were looking for and also we've been looking for a reason or a way to test out github actions so we actually chose to use github actions for this so we're actually I'm going to walk through how our github action works so first we have to find the inactive closed issues so we do that using github API you're likely familiar with at least the syntax of what the queries look like is this the same thing that happens in the regular UI we're gonna check the actual repository we want to look in KACE angular angular we want to make sure we're only looking at closed issues we actually want to look at only unlocked issues because believe it or not if you try to lock a locked issue it does nothing and we want to look for things that have been updated 30 days or more ago so for instance for today that would mean looking at August 21st of this year and then the last thing is we want to make sure that we sort so that the that the we see the law the oldest issues first or the oldest updated issues first so that if we have more issues than we're ready to lock it once we start from the oldest and work our way back so now that we have this query we actually have to go and lock these discovered issues so it's relatively simple as well as a process we use we create a github client using github JavaScript API client which is called octo kit and then we use that query which we just made and search for the issues that we are interested in and then we loop through all of those issues and we lock them and it looks really easy all you do is just call lock issues but what lock issue actually does is we create a comment because we felt it was really important that we explain why we're locking issues so we have the same comment that shows up on all of our issues that are automatically locked and explains why we locked it where you can go for our policy as well as making sure to note that a bot has done this action and it's not it's not anything against people involved in the conversation it's just the place the criteria has been met and then finally we actually lock the issue so the github action is relatively simple we just run this for our repository I'm like a cron job and it will actually lock all the issues so we actually have to enable this for our repos though so to do so github actions are just configured using a yellow file so this is actually outside of the the commit message or the commit number I guess on this slide this is actually what we use for the angular angular repo so we sign it up to run every day at midnight and we are able to just reference though the lock closed action that we already created so we're actually are now seeing all these results so over the last three weeks since we turned this on we've lots about 60,000 closed inactive issues so for those of you who were subscribed to our github repo and I caused a lot of spam to go to you it's over you can turn your notifications back on and now we just run it once a day and we're just seeing a few day a few a day instead of this massive number and I really want to make sure that I note here Charles is actually the one who wrote the majority of this code and I just wanted to thank him for all of his work on it since I definitely I'm standing and talking about it but he's the one who came up with most of how we do it so next I want to talk about how we build the angular framework itself so it's getting more complex if we step backwards and look I'm sorry at the problem I guess step backwards Jets as angular's continues to grow it's getting more complex and bigger and that actually takes it makes it take longer to build and test all of the code so we need to find a way to not take it and make it take as long because that makes the our developer experience worse so if we look back now angularjs building was simple we had a bunch of files and we combined them together and we had our application but now with angular we've added this trend that's transpiling that happens with typescript so now we have to take the typescript files turn them into the JavaScript files and then combine them and that step of transpiling is or building is really where the time the time is growing so this is this is a very simple application of for files as an example but let's say we were to just update one of those files then once that one file is updated we would have to transpile every the types of file again and then combine them together but it doesn't actually make sense because three of them it's the exact same thing so we actually want to do this incrementally and only transpile the files that have been changed and I just reuse the results from before and so you've heard us talk a lot about how we use basil to do this but this is this incrementality concept is actually something that basil provides but is a general concept that we feel is extremely valuable for making builds as efficient as possible so as I said that's a really simple application but most applications that I've seen at least have more than four files you could have hundreds of thousands of files and pretty quickly we can become CPU bound because you can only be checking and only compiling one file at a time with typescript so let's say for my MacBook for instance it has two cores so with hyper-threading that's four at maximum I could be doing four at once and it's a really fast process but you still have to go through all them so we actually want to scale this out horizontally because all these cores could be used for something more useful or at least more entertaining for myself so instead of building these actions locally all in the machine we can actually build them using remote build execution and send all of that work to the cloud to some cloud executors to do this so now instead of being bound to just the four threads that I'm able to use we could use however many cloud executors we have and expand that out so now instead of building locally we're building these things in the cloud and getting back the results but if things are building on the cloud instead of building on our own machine we would lose that incrementality that we just talked about so to get that back we actually do remote caching so with remote caching the build is done remote and then saved and then any time we we attempt to build again we do that same caching or the Trant or incrementality concept but just doing it remotely but one of the big advantages here is that once we build it remotely it's available for anybody who is building remotely so by all using the same remote build executors in the same remote cache all of our developers as well as our CI can build the same objects and not have to and not have to rebuild them for each other so once I build angular core once if nothing has changed nobody else should have to build it again we can just use it over and over so over the last few months we actually have set up remote build execution remote caching for the angular repository we currently only take advantage of it in our CI but we've actually found that for instance on our unit tests we've seen a 33% or we are a 33% faster in running those so you can see how at the beginning of this graph we were adding more tests more complexity and it was growing and then around this point we enabled remote build execution and caching and our graph went down and you might be thinking Joey that's not centered that's cuz there's more to the graph I broke it then it fixed it and now it's it's flattening out again so we're good so brings us to our next topic we want to move fast and break nothing so this is actually about how we managed code moving into our repository so we need to bring in a lot of changes as I mentioned more than 135 pull requests each week and we want to merge them as quickly as possible but we need to not break anything for our users so we need to maintain quality all the time so to be able to achieve this we attempt to capture and mechanize as much as possible everything we can for deciding when a PR is actually ready to merge so we require code reviews for all PRS and we require that code under that code review to come from someone who owns the code and by doing this we ensure that the code actually is maintaining the quality that we expect as well as the conformance rules and then we also maintained that the code that's being committed is actually going where we want it to go we aren't going in a different direction because the code owner is responsible for the direction of that portion of the code we also do a lot of testing on circle CI so we have code and commit conformance contested conformance tests so we make sure things like linting and that the code that the code has changed has an owner and then we actually make sure that at any point in time we can create an NPM package with what is currently in our repo so we test actually packaging it making that NPM package we don't put it anywhere but we make sure that it builds that and is valid we also run unit tests and integration tests across multiple operating systems and multiple browsers and since our documentation site is store it alongside of our code we test that the documentation site still works as we expect for every build that happens so we've talked on stages in the past about how Google actually builds using angular master so to make sure that's possible we actually have to check that we don't break anything in Google for everything that we want to bring in to bring into our repository so luck and luckily for us Googlers are very good at using code what I've decided to call creatively and it leads to a very solid test suite because what seems to be any possible usage of the code is found somewhere in Google and Google store posit or E so we have hundreds of thousands of tests that are checked for each PR before we merge them into github and we actually are doing the same thing now for testing IV so we test View engine code and make sure that it passes a hundred percent and for IV we currently are at just under 98% so we're testing to make sure that we aren't regressing back so to maintain this as I mentioned we have to do this on all 135 PRS which is a lot of work and additionally we have more than 30 people who could theoretically go to github and press that merge button so this would be absolute chaos if everybody just merged whenever they felt things were ready so instead we came up with a new role on our team the caretaker so the caretaker is responsible for making sure that that our repost is in order so because of all the the checks that we just walked through if everything there has passed the anybody regardless of their understanding of the code itself that's being that's being merged can be confident that if if all of those checks have passed it's ready to be merged and so it takes away the responsibility of deciding when it's appropriate to merge it and it just makes it so that as soon as all of our tests all of our checks pass we can go ahead and merge it so this the caretaker is responsible for actually doing that merging they also monitor our CI categorize incoming issues so that they can be triaged they're also responsible for syncing those changes into Google and for releasing new versions of angular at the end of their rotation so it's a quite a task so we actually have an internal application that we use to just manage this process for ourselves it allows us to create some different sorting and different filtering around what we care about to determine what's ready and it also allows us to keep track of where things are going wrong what's kind what needs to happen for something to get into the right State so brings us to the last thing I want to talk about perfecting developer infrastructure I know what you're thinking Joey it's a lot of success that you just talked about you did it you perfected it no it's not perfect but we have very well shown that by dedicating time to making sure that our processes and tools get better we are making things better for the people that are making angular so I just wanted to talk about a few things that I see as places that we can find impacts next we often talk about how the angular angular repository works but the reality is that we actually manage a lot of different repos within the angular org and so if we can unify the processes that happen across those repos we can be even more efficient and reduce the overhead that it takes to manage all of those things so also with respect to the caretaker app with the help one of our interns this summer Alisa we've been working on a new version of that caretaker app that we hope to make available publicly that would give insight into the progress of each PR so that people can see what is being done on a PR or where a PR is at in the process and kind of demystify how things are happening through a repo and lastly I want to automate everything ideally we would automate writing the code but I don't think we're there yet so things like cherry-picking things automatically to patch branches rebasing PR is pushing new documentation and automatically following up to ask for reproductions or all things that I think that we could start looking to automate to make ourselves more efficient so thank you the slides are available there as well as you want to look at the actual code for our automatic locking it's available in our dev in for a repo thanks [Applause] [Music]

Channel: AngularConnect

Views: 1,052

Rating: 4.75 out of 5

Keywords:

Id: RaQDoDj3G8k

Length: 19min 18sec (1158 seconds)

Published: Fri Sep 27 2019