Samba for the 100,000 user enterprise: are we there yet?

okay we now have two talks about samba back to back the first by one of the Samba core developers and remodelers so I've been talking many many years about Samba at the at the scale of 100,000 user enterprise something a scale much larger than I ever expected to be say Brady to be PC nap but we really are making some good progress so I'm wondering are we there yet this is the team that I work with a catalyst catalyst has really supported this effort really well and we've got five seventeen members now our catalyst and I've got a team of eight including Alessandra my p.m. so we work out of Wellington and most of the work I'm talking about here is stuff that we've actually spearheaded as part of the same a team first then we'll talk about some of their other achievements I could talk a whole slide deck on moving to get laid but I've just got one slide on this so we've managed to move some of the same project to get lab we had tried to use github as a way of improving contributor engagement we hoped it would maybe have sis admins or our other users step up a little bit to doing a bit of working with Samba but turned out the Samba team members wouldn't use it and then it became the platform that people were told to use but God ignore it on that wasn't very good so get lab is open core so we use kilo lab calm as a hosted solution for that and we couldn't export the open source version if we want but we actually have up take the Samba team is using it which in particularly using it for get lab CI so samba school test which is run in parallel on the get lab CI infrastructure we've got get lab comm provides free CI services and then the Samba team run some services on rec space with a credit they generously provide that's the eye that allows our developers to help us produce software faster with better turnarounds and less put your patch in it failed the test or you know go back try again and that makes us fast with getting a development which helps us then spend more of our time trying to move on important things like scale but we still do all of our final build a merge on same draw card we because we're a project that started 92 we are still pretty old and conservative about how we do things another new thing that we've achieved in the last little while and sambar is moving to Python 3 so Samba 4.10 which is due out in March will support Python 3 by default the build system we'd be Python 3 the Baha'i some bindings generated or or python 3 the previous versions had partial support the number of the bindings could be generated but the whole build system and the complete actual running of actually director you could domain controller and path of 3 wasn't available so that's changed real light will apparently have Python 3 in if they ever get it out the door but rel 7 only as Python to see sees in this flagged a world that I just hate and Red Hat why did you do this to us so we're still going to build with Python 2 7 for a pure pile server we're not going to get rid of that very soon because we know that it's really hard to get modern software on all these enterprise pieces of equipment that people in system running modern same run anyway it's a requirement so we are going to keep that going but the and we're going to have a dual release the next release ember 14 will still work on python 2 and Python 3 but we're going to be getting rid of the Python 2 support as soon as I can get some consensus but here's what I was really here to talk about which is the goal of having 100,000 users 120 120 thousand users hundred thousand computers the whole scales we're talking about this so 300,000 user like objects in the domain but also we're looking for smaller scales that people send in quite casually deploy a same with the main on 50,000 users that are telling University when I've heard of this I thought you have no idea how badly that would have worked two years ago but while some we're working on other things scale and you know being ready for the hundred thousand user enterprise is far more than just does it work with this many users there are things that matter at that scale you really need decent audit logging you really need decent auditing of changes of that scale because you're not going to notice them sort of individually so we've done a number of things I've told this slide has been up for a few years in the as I've given this talk about how we've improved the connection processing in Samba so we've moved from a number of our components being locked into running in a single process to being setting up a forking model a bit like the old Apache forking model which I then also preformed model names are similarly stolen from from the Apache Price project for handling our LDAP server and now we're also managing the important net logon server which is used for until my authentication of which there's still too much on me networks as well as KDC which is the Kerberos key distribution center so we're improving the parts of Samba that we know are bottlenecks we're allowing them to run in multiple processes or there's better scaling across the CPUs on the domain controllers we've also introduced some process limits for the standard per connection football model try and provide some little bit of control over how well that explodes I'm moving fairly fast because I love slides and there's also a lot of things that we've actually moved through I would like some questions at the end or or ever as we go though we also done audit logging so as I say Auto logging starts to matter when you're at this scale and we've had people say that the best format for us to ingest those audit logs is not human readable bits of text that then have to be parsed out with the world's most horrible regular expressions but actually JSON I got trot we had some with one of our lovely Samba team members troll us on the mailing list and translate we are we moving to JavaScript what's all this - no it's just a really good well accepted representation for logs and we probably shouldn't have even bothered with the human readable form because no humans were actually reading it and the JSON actually was you know it could be could be actually parse fairly well and you know we doubled the amount of effort we had to put into the into the logging once I had fine-grained password policy without users to have different policies previous is one policy for the whole domain and now we can go and say administrators have these policies or this particular account has a lacks password policy perhaps it's a service account that's not really a service account things like that that need you know maybe you've got a desk a Welcome Desk account or something that needs to have a well known password so this one else asked to basically worked to a larger organization where it's not just one size fits all now um any of you do backups these backups these backups are part of your part of your job yeah so how many of you backup your database instances by just tearing the database server yeah so yeah not so not so good so we did have a shell script which did attempt to lock the database but as the davis is in four files it locked them one second third and so it was knocking and consistent snapshots so we've managing to make to take a consistent snapshot of the whole database by locking the whole database and then reading it let's have an online backup so near Samba when it's an active directory DC it has a replication protocol between their main controller so we use that protocol to do the backup that allows us to download the whole database in a form that's less likely to be corrupt because the replication protocol by the process of pulling it out of the records and doing it tends to get a nice of you of the database than just raw records so we have two different ways of back in the database up allow us to make sure that we get either a sort of sanitized copy or the forensic copy of what was on your disk which maybe will be very useful for trying to work out what went really wrong we also have a restore tool because anyone actually restore from backups so this we do recommend that if you do have a partial Samba domain that if you want to restore that you just go and say okay turn off the things they're broken start named domain controllers from the working once that's all good but if you're down to nothing then we have an authority restore it removes all the records the other domain controllers and says this is the only DC that has ever existed and I'm starting from here this allows you to then rebuild the network without oh it's looking for these dead things that were removed but not removed it makes it much easier to rebuild it in catastrophic restore scenario and also make sure that those other domain controllers that might not have actually been powered off don't suddenly resync into the network and cause total replication chaos now the other thing that's really important if you're going to be working these scholars have got to create a lab domain now how many of you are really good I probably do Liam can manage it but anyone else can create a totally isolated level to network that can still get to the to the mirrors for installing the packages and you know and won't talk at all to the rest of your production network I'm not seeing very many hands so basically we had advised customers to do this and they always just they merged again and it was a disaster so what we do is we rename the domain in the same processor where we restore it and change the important keys if I can't talk we rename them so they can coexist on the network this will actually create a lab domain with all the users from production possibly the passwords or not depending on which options you said but not the name of production so then you can then go and test your things do your upgrades and then try it again in production what's it been working on ad operation at scale so we've got traffic replay tool I've talked about this a number of conferences in the past we can pre create realistic databases we can simulate to to get the traffic and or even ramp up the traffic based on recording traffic seen on your network and then replaying it this also has is also able to operate against windows so we're actually able to benchmark Samba on Windows with the same traffic load which is really interesting because we gain there it's not not as fast as Windows but it's it's not too bad once I managed to get replication and working at scale it turned out that filling the database was enough proof that we could operate at 100,000 users you couldn't replicate a second replica so it fixed that little things but basically by stir by actually drawing and trying at this scale we are finding the bottlenecks and finding that they're actually quite fixable another issue at scale is into forest Trust's we want to be able to have multiple domains so this is an area that Stefan Mehta marker has been working on it's been possible for a little while to trust other forests it's still one domain perform a terribly much unless you're really versed in AD but these are limitations that sample works with but we can actually have decent trust between the forests that handle some group membership crossing them where were each step we make this work a little bit more we've also got replication Diagnostics because at this kind of scale you need to be able to know where things are failing with your replication we've got human readable text we've got real inspiration from the safe project from last year's talk about what tools saying how your project is working should look like and then JSON representation for ingesting into log tools we've improved Sambor tool so that we've got new sub commands for arcs for organizational unit management computer management and D and improve the DNS handling there's an as we found the number of group memberships and the distribution of groups like I have 110 thousand user group was a really important property for working out the scalability factor of a Samba in the large domain we now produce a tool that will tell us in a simple summary what your domain looks like because we were just guessing and our customers couldn't even tell us sort of what they'd look like in terms of the shape of that curve but here comes the biggest thing that we've done towards scalability is we've moved from a 32-bit database to a new 64-bit database we're concerned that for gear by database just would be filled too quickly they were all that we'd be seeing all the time repacking to get us back under to gig and have enough space for a transaction so none of our customers wanted to hit the hard limit of full gig and then start development so we chose emt-b it's used by open LDAP and pretty much did what it said on the team it taught us a lot more about our locking than actually problems in LM DB so we're actually really happy so you found and fixed a number of locking issues by introducing a new key value layer into samba previously we had the idea of a different back-end but it would take LDAP like structures so we ended up adding another layer in there that was key value structures and then said tell you basically value database lndhu is key value database and then put that extra layer in actually work pretty well with a lot of tests to lock down the semantics we actually got something which were pretty happy with until we hit locking even the prototypes found that we were missing locking now some of this is we fix quite a while ago for 747s now 18 months old but we hit more locking we had issues where we found out we were unlocking databases in the wrong order and so you could see the sequence number before you could see the data which then meant that you then told everyone about the new change the data wasn't visible yeah not good not good anyway we've found the fix those and then we hit issues around maximum key size we knew this one was coming LM DB has a key size limit of 511 bytes but we were putting our full distinguished name in now distinguished names most of them would never be that large but we know someone's going to try and stretch the limits so I riri work the database to instead use a good word for the key and then have a separate D into key lookup and then we looked at our indexes and said that's going to be awkward there can be very very long but actually why don't we just truncate them usually it'll match and the rest of time will just scan it seems to work well enough we've been measuring performance actually on an old AMD Athlon because bit like the free software mera group the way you get hardware at catalyst did you get the castoffs but also traffic replaying the cloud and adding users groups in my workstation so we went around the petrovic replay ah super simple form it's lost oh no but actually we found that basically we had an interception layer that was meant to be handling network traffic and also was handling all the right calls to the database and just looking up the link list to work out is this a socket that's a network socket was taking time anyway baby our 10% slower we don't think it's we think only to be a scale better but you know there's TDP it's actually done good database after all these years we use the traffic replay and they came out about even and then we went and written this benchmark which I've been running for a while which is just sort of film the database how long how many users can you fill in and with all the F syncs turned off in half the wood we don't need this what ok but actually FC was still turned on so a few patches later we made it also honor they don't f sync and we base a pudding 124,000 users in that benchmark sort of run so it's not really LMD bu is much much faster than this but actually what it's really showing us is that the common factor between it which is the module stack that sits above all is that still where all the slowness is so you know going to a lightning-fast database underneath doesn't give us you know ten times the performance it just makes things a little bit better so this is produced this scatter graph just to dry so the lines fan out because they go to what some of the users are in one two and three groups it's just but it's interesting it's a lot higher jumping in the latency if they'll only be on the right we don't really know what that's about it's presumably rebalancing the tree it's got a very different database structure the way TV works it's not so bad we address the customers need for scale we've set the database limit at six gig at the moment that's a compile time constant we don't have to it just changes how much virtual memory is being used people got really freaked out when they saw you know 100 gigabytes and more processors so I was trying to at the code in I can change a constant later with our people screaming but we open up new opportunities we can also start using Ellen DB sub databases which would be still into one giant lock which is actually what we need and would avoid our okay lock this lock these unlock them in the right order kind of game that we currently play and once the potentially L DB has a concept of an ordered walkthrough the keys and that could allow us to do range indexes which could be really useful saw some sharp edges LD be has different locking behavior there's no exclusive access so database operations can't we can't go and be put into maintenance mode you know we know that we're the only ones able to access it at all far as the sparse it's I don't really like these you know here's a file the maximum size it's ever going to be but it's not filled and so other things filling the disk and then make this really go badly TDB has always filled its files and administrators are used to that so I'm not quite sure how we're going to handle that I'm not sure whether it will really end up bothering our administrators but I've kind of like DF to represent how much time after we filled the database they're also not extended automatically which is they've gained different TTB so just we need some real-world experience from out from our deployments there's still stuff to do for support for a hundred thousand users is really tasks for the the September release of Samba we've got to improve subtree renames I mean one of our big clients is the French government and in my experience government organisations love reorganizing so I really like to be able to allow them to do that we're gonna change the pack format in the database to avoid reading data that won't be returned because that current pack format is a you know you actually end up have to read the whole amount of information about a user in order to actually read anything about the user and we're gonna improve the memory management so that we allocate less memory when we're returning data internally that may not when it's not required to be individually allocated so we're there yet probably I'm looking forward to the next two samba releases because we've got still got a good amount of our project to go with our customers are paying for this but real-world feedback could be really valuable so we really appreciate any feedback from any samba users no matter what the scale is because it just tells us things edge cases that we hadn't thought about finally please become an official Conservancy supporter the software freedom Conservancy keeps Samba with a legal home so they are really worth supporting and also this is catalyst these are some of the technologies and I know that even more because they might I told but in the last 10 seconds that I have are there any questions you take a couple of questions while Karen sits up high just a quick question we use CTD B for a load balancing is this gonna be transparent moving over to the new database or is it these changes are just across the the Active Directory side of the house thank you and thankfully doesn't the databases nearly these sighs more questions were you able to use any of the LLVM sanitizers else an and such to help you with the lock the locking issues or just different semantics and you couldn't we just we would diagnose them with mostly just picking back traces on the locks and chase them that way st. game figuring out which one was locked and then tracing all the processes there's some good guides on the web about some LS locks and things was very helpful but tell us who was waiting for a blocked lock and things so that's how we ended up doing it time for one more question okay thank you Andrew

Channel: linux.conf.au

Views: 5,609

Rating: 5 out of 5

Keywords: lca, lca2019, #linux.conf.au#linux#foss#opensource, AndrewBartlett

Id: KEyzrnr2UhU

Length: 21min 16sec (1276 seconds)

Published: Mon Jan 21 2019

Reddit Comments

[u/[deleted]]

If the title ends in a question mark I assume the answer is no.

[u/psycho_driver]

I still have nightmares about configuring samba. I found a working-in-all-my-use-cases configuration six or seven years ago and haven't touched it since.

[u/Sigg3net]

The funny thing is that many management types asking this question have 50-100 employees or less.

(We're 8 people in my regional office, and still the above and beyond want us to use SharePoint and MS Teams instead of a Linux file server and Skype..)

[u/tictatonifonical]

They have so many good and interesting topics!

https://www.youtube.com/user/linuxconfau2019/videos