[Music] the I know okay welcome back we will continue our session and the next is Thomas connect and he will be telling us how to manage 1,500 micro ticks with the single click hello everybody so this is the prime spot the second day after lunch so thank you very much for being here I expected this house empty room so this is much appreciated okay so yeah managing 1,500 routers with a single click this is a Network automation tail my name is Thomas some system architect some other stuff as well as I'm here with Dunamis which is a network backup and management social I have a stand down there so if you want to talk to me about it feel free to stop by so this presentation is actually an hour long originally and we have 45 minutes so a few disclaimers I speak fast and as you probably noticed I have a lovely Slovakian accent so I do apologize in advance if we don't understand some of my rambling also we won't have any time for any questions but if you have questions I would love to talk to you after the presentation you can find me in the exhibitor room so why are we talking about this network automation is definitely the way to go forward and there is sadly very little material around the internet on how to do it properly and what are the challenges and what are the various options and etcetera etc so that's why we do this talk I recently completed a project that deployed 1,600 my critique my critique is a nationwide network with full automation so this is going to be a tale about that project and how it went and what were the challenges what we did and how we did it etc so yeah let's talk about how automation can make a project like this or any other similar project or whatever you guys do on a daily basis how can it make your life better so a disclaimer number two the actual project and the entire social I did for for the 1,500 nationwide network is under MDA so I cannot show you sadly and the actual screenshots or the demo of the system but I did receive approval to talk about the system and to tell you the implementation specifics so I can tell you enough to explain what you need to do how it works why it works that way what are your options etc so a massive thanks to the people who cannot be named due to the NBA for for allowing me to even present as much as I can I promise it's not cut down in any way we will get very technical there there is lots of stuff it's just not no actual screenshots or the demo of the system okay so yeah let's let's talk about network architecture so let's start with the architecture right how are we going to do this what are we even doing in this particular example in this particular project so first of all the backend for us this project this particular network provided some private services to 1500 sites around the entire country so it was not public services we had a data center and mainly our data center this is of course just an example of how to do things on the backend this is not at all a like a you should do it this side this is just one example of course there is many different ways to to architect things but the thing that I would want to highlight here is you may or may not want VPN access concentrator in your back-end in your data center to terminate all of the clients you may or may not that depends on the next slide so that's one important thing internally of course down here you can have many multiple subnets DMZ management DMZ 1 2 3 etc you know this is the actual subnet the actual network there's all the servers and the services living which are offer to your customers and or clients and of course your servers and your sin and everything else that we see in here should be redundant and redundantly connected that's why Hiva we have such a spider map here right any services that you have to offer with an SLA that have to be highly available it's a drug you definitely want redundancy since we are talking about redundancy one really important thing is you really if we are talking about the high availability scenario with the slice you really want to have a failover site so a second failover site somewhere else different data center different if you are in the cloud in the ABS different availability zones etc don't rely on a single site to provide very high availability services there is you know I hope I don't have to explain why right okay so this is just an example how to architect stuff in your data center down here then you would have some one or multiple DMZ subnets where your actual services live as I said not as simple as this this is just an example so let's talk about transit now how do all of the clients from the different 1,500 sites around the entire country get to those services that their servers down here offer so that's that's the second thing basically here you have three options right first of all you can build your own transit network yeah so this is not going to be realistic unless you have a lot of time and a lot of money implementing your own national national size transit network it's not a very good idea usually so let's go to more realistic options so you can buy van services from a carrier on a provider from a national wide network that is already built up you purchase dedicated transit van services or the second realistic option is just use the Internet right so these days everybody is connected to the Internet there is a very high chance the site that you decide that you are connecting already have internet so but let's talk about these two options because we said first option not really so if we are talking about the nationwide network option two will provide us with the best ability with SLI and may leave it consistency right if you purchased a single service that nationwide from a single provider you know dealing with them is going to be quite easy because they will want to sell you that wholesale big service right so we can get consistent SFA's across the entire entire topology you over always deal with just one entity with just one network provider that is offering you those them services and the consistency is a big one especially since we are talking about an automated network about an automated infrastructure so when you get that consistency of knowing that every single site is connected through the same same mechanism whether that be pppoe or DHCP or how is it submitted do I get Publix do I get you know when you get that consistency which you get from buying the single service from a nationwide network it's really nice for automation of course the downside of this is that this will be quite a bit more expensive than just using the internet which everybody already has so let's talk about the option of using the internet so option free will be the cheapest one although the most problematic one so first of all you are no longer dealing with a single entity you are now dealing with a bunch of different providers across the entire country that all offer potentially different speeds different connectivity options different billing different SL A's different support its addressed so here the the choice of what you actually do depends of course on the project scope project you know the money that you have a lot available how much SLI is how much available available 'ti you need to provide etc so yes that's that's pretty much the options that you have the other two so what did we do we actually purchased vent services from a national white courier from a local incumbent company and yeah that's it we had nice consistency consistent SLA is dealing with a single entity all of that so we have the back end we have the servers the data center we have a way to connect other sites across the entire country to those services so let's talk about the the architecture poke part 3 the users decide the individual you know networks around the entire country that you are connecting through this transit network to your services so yeah here actually there is many many issues awaiting so how do you architecture this how do you architecture those 1500 different networks so this depends on the choice of transit method right because you would prepare and even architecture and you would think about this differently if you use the internet for transit and if you have been consistent then service across the country it also depends on if you are building a new network or if you are interconnected existing site which already exists across the country to the new service which will be provided by the Biden you know whatever service you terminate a datacenter that's also a big differentiator in how you how you think about this and the final problem is do you even have full control over the customer sites can you dictate policy to them can you dictate subletting can you dictate what devices but routers which is a travel they use because in certain projects you might not be able to do that in others you will be able to do that so all of this is you know it changes how you architecture the entire system so what about us what what about this this particular project that I did how did that work out so yeah we were we were interconnecting existing sites around the country that were already set up already had internet and they were set up somehow meaning this this is not good all sides already had internet from various providers with various cheapest brand routers with various speed various quality of service various things so this is our case right so how do we deal with this right we theoretically could have tried to force unified everything to get around submitting issues and duplicity and we terrifically could have tried forcing new out there so forcing router change on every side but let's be realistic so if we actually wanted to do this it would drive the timeline of the project and the cost of the project extremely high also suddenly we would become responsible for dealing with all the change that needs to be done to all the 1600 local networks and they had different things in their networks they had servers some had you know whatever so suddenly we would become responsible for doing with any issues that came out of that change that we were forcing on them and you can imagine yeah time equals money close project issues etc so this was our case so you can see how this is actually a very complex debate with many options where you need to choose the proper solution for the proper use case do the proper analysis before etc so yeah this this is what what we had and so what was the other solution to this problem so in our case the simplest cheapest and fastest solution was to add another mikrotik to every side this is not at all a perfect solution but if something is the simplest cheapest and fastest it's really hard to argue against so we actually added a second mikrotik to every site which we connected through our transit services provided by the national wide provider network this meant we didn't have to touch existing routers and existing networking at all which means we avoided having to do change on 1500 sites across the country and during it the consequences of that change which would have been huge and also this was not the scope and the part of the project there was no money in the project to overhaul 1600 existing sites as I'm saying this is not perfect it's sorry from perfect but in this case it was the valid solution so yes minimal change to existing networking minimal change to size minimal impact so yeah other problems so what are the problems with the solution there first of all what about routing right we are adding a second gateway connected to the local network that now devices need to use to access other services because other services are only available through this gateway not through the public internet so what about routing and luckily for us all the client pcs all the devices that needed to access these private services were in Active Directory meaning that the single GPO actually took care of all of this so what did the GPL do right on all the devices it deployed a route to the subnet which we terminated in the data center to go over this new router and the default route kept on through here so again this is not a perfect solution at all but the valid one for this use case so if this was not the case if we didn't have the option of deploying a GPO and influencing the routing at the end devices we would have probably had to architect the entire solution differently right so here you can see how you know proper proper tools for the proper for the proper job right so we were able to get away with this because of Active Directory and because of the GPS and again it's really hard when you're talk to project manager and the project owners if he offers the simplest fastest cheapest solution you know what are the options realistically right either we had to overhaul the entire 1,500 sites force addressing further altar changes etc or we did this so yes that's our problems one so other problems too what about subnet duplicity and because we said the existing sides are already set up somehow usually the dis which which is of course a problem if you want to get back from the data center to inside of the sides right so this this is a problem which nad map is the answer for so nut map creates a one-to-one static mapping of one set of IP addresses to another so we basically created a unique virtual subnet for each side and then we use that virtual submit with nap-nap map to assure addressing uniqueness this judgment that every site the virtual subnet if you connect it to IP address dot eleven on the virtual subnet it would go inside the network to the duplicity network but it would actually be unique from the point of view of other infrastructure does it make sense so again not the clean as they're the best solution that the valid one it works it's you know sometimes even if you architect things you might not like it but you have to make compromises if you know what are the other options right so if the project scope was like this you have to make it work and this is a valid social although not the best nor the cleanest one so a little tangent I love tangents various ipv6 when you need it so ipv6 would have made this whole project so much easier that it would have been designed completely differently none of them mapping and then the secondary gateway and all that would have been needed but sadly our oldish existing networks are far from ipv6 ready the various random internet providers at each sites are also far from ipv6 ready which means that some time in at some point in the future when ipv6 came along we would have to reconfigure the entire topology right because we would have to reassess and yeah there wasn't enough money in the project to allow a full ipv6 deployment excuses and more excuses what I want to get at here is that ipv6 that there is no avoiding it we all just need to do it let's start doing it please it would have been so much easier anyway tangent over so so mikrotik to the rescue yeah because it's the right choice so mikrotik is ideal for a project like this it has an also feature set everything that we need for this project and much more and the best price for performance ratio we will find imagine how happy the management will be when you tell them you know when you are deploying a network of 1500 routers the difference between deploying 1500 insert big their name routers and 1500 my critics you know it's 10 to 15 times cheaper to deploy the my critics so yes and my critics are actually great for automation we will get to that later on so let's talk about what we want to achieve here now that we've talked about the architecture about the problems of designing the entire solution and the entire system and interconnecting of 1500 sides across the nation let's set our goals to automate what we want to do so let's let's look at how we would do things without automation and how can we implement and design automation to help with the project like this to make other life easier so the traditional approach approach the deploy so if you want to deploy a network like this network admins would configure the routers somehow manually let's say five minutes per router times 1,500 routers see close to about 16 work days which is realistically a month since you know 24 days in the in amount so it would be a month just for a network admin to configure 1,500 routers and I know you can automate that you can scrape you can flash week but let's talk about a pure traditional approach without any automation without any scripting so then you know those routers slowly get configured they get put into storage and then the installers the techs slowly take out the routers go to sites and install them right so if there is a problem at site XL because you know problem outside why because of the you know something is wrong the site reported how they did their internal architecture wrong etc many things can happen all just the networking admins you know they had $1,500 the routers to configure its manual work there is an issue in the configuration it creates wait time and wait time is not good because the text that installs it has to create tickets has to get the hold of networking admit I have to debug somehow dream old access blah blah blah just straight time right and so that is problem one and the problem too is we are deploying 1500 new box is one box at every single site so you can be sure there will be required requests of you know can this new box also do think X for us for example a lot of the existing sites didn't have Wi-Fi so since these are already deploying a new box why not also give them Wi-Fi to their network since they didn't have it previously right the box can do it so why not do it it's a benefit to the user makes the user happier and you know happy users mean we have a happy life as well so yeah so if this happens again the tech or the installer needs to get a networking admin which need to connect somehow needs to configure Wi-Fi then the SSID and the pre-shared key needs to be distributed to the users etc etc etc okay so now that we've deployed our entire network you know network needs maintenance right so the deploy is just part one the second part you need to take care of the network so you need to upgrade router with an ultra boot so let's calculate you know two minutes per router upgrade times $1500 so it's about you know almost seven work days per single version upgrade rollout to the entire network again we are talking traditional approach no automation they are doing it manually and you guys know I hope everybody knows recently there was a big leak from a certain US government agencies think which one I I don't want to say their name because then I will get get it on the record that I told that about them and they will not let me back into the country but the wall so so yeah you need to roll out new versions let's just consider you need to roll out a new version every three months I think that's fairly realistic right so that's four version rollouts per year which means you have seven times four right let's do the math it's 28 workdays so that's more than a month of realistic time per year just on version upgrades and time equals money so you can imagine how much this actually cost in network maintenance okay so a breathing drought or as an altar boot yes sure but this is so okay you know not that often once every three months or even less okay we just upgrade our network once a year okay but what about deploying configuration changes this is something that you will have to do fairly often your policies change you're auditing shows that something needs to be reconfigured the networking topology changes whatever you know or you get a new new compliance requests that require you to change your configuration so here we assume one minute for a configuration change parameter which comes out to three work days per configuration change this is a killer to deploy one line of code across the entire infrastructure is three days and much of that you can imagine if first of all that's like the worst job to get imagine you as a network engineer you know here is a line of code they deploy at 1500 anchors you know the verse monotonous repeating job you can get ready nobody wants to do that right so yeah you can be sure that if this is done manually it there will be if just 1% mistake rate you know what's 1% in 1500 rotors right that's actually a fairly decent chunk of the network that's now getting inconsistent wrong configuration so yeah and again we are not talking on yes I know you can automate it somehow or fully we are talking about the full traditional approach and something specific at site X for example client side number 490 six months to change their Wi-Fi PSK push RT password right or they want to change the SSID or let's address addressed right now all of this is manual so it creates support center calls it creates tickets manual network admin work potentially different configuration at each side the consistency of the entire infrastructure just goes completely wrong it is a nightmare so yeah that's the traditional approach so let's look at automation let's look how we could be use and prepare automation to improve all of this so first the deploy so before we even start we put an automation system into place we prepare our automation system and then we feed data to a CMD be CMD is a configuration management database so it's a database which keeps data about your site about your networks and this is just run a whatever sequel script and the data entry team is responsible for this it's not our job and every node data entry is much cheaper than network engineering right so we prepare our automation system feed all of the data that is made into a database then techs go out our installers go out to every single site and install the hardware and after they do that they just click a button in a nice web interface to provision all the configuration to the router that's it no actual network administrator work involved and to solve the thing of can this box also do X or site-specific things we deploy a self-service portal where end-users can actually change various things for example enable disable Wi-Fi change the SSID change the pre shared key etc so the self-service portal allows our users to on demand without any support center calls tickets anything change certain various configurations on the router when they do that they you know they log into a nice urban interface they change their password they click Save and the automation system instantly automatically provisioned the new configuration on their side router much more friendly right you can imagine you know it's the users they you guys know how its dealing with with the end users right they don't want to call you if they call you them it's so you know I have to talk to the IT guys again and all of that knows it's we let them do whatever they want they have a system where they will not break anything we have chosen what we have allowed them to so it makes again every user as much appear happy users means we you know we have little issues and we all get paid and etc so at the deploy let's look at the maintenance so upgrade altruism the 1500 routers is actually one click two minutes that's it because we just click a button and the automation system does everything deploy a configuration change to 1500 routers and I'd like surely one click 30 seconds and it's done something specific get side x we want Wi-Fi well we don't have to deal with that anymore users have their self service portal then where they can do this right so yeah immediately automatically provisioned the users immediately see feedback see change they're much happier they can do things on demand it lessens the workload for us etc so there's a lot of benefits what we haven't talked even talked about is the consistency that we achieve with since this is done to automation right humans we we are not perfect nobodies so we make a bunch of errors so human error here is eliminated you know we makers machines well they may clusters so consistency is achieved across the entire network all the configuration everywhere is the same because it's automatically put into place client customer satisfaction is much improved the wait times and the ticket numbers we get on the support center calls we get are you know much less now so yeah welcome to the dream in the automation land and I know it's not as simple as I make it sound but so yes automation it's much cheaper in the long run if you consider like we said the router allows deploys six days per upgrade that's with automation that one click two minutes and you know it's done consistently it's done the same everywhere it works because the automation system ensures that the version is deployed it checks all of the things that are needed etc also think about you know is this 1500 sides which you don't have full control off like having this project you know at any given time five to ten percent of these sites might be a fine power outages you know somebody decided to unplug the router because they needed something else plugged into the you know the power sockets right so it has much benefit in the fact as well and it's automated you know the system actually just check back if ever after is unreachable it automatically checks back in ten minutes and you know every ten minutes from now on and we'll try to upgrade that they all turn to it succeed with us we would have to be they'd manually we'd have to manually somehow keep track of what is upgraded what is not we would have to come back to it it's just a mess the automation takes care of it for us a much healthier network by because of that consistency because of the assurance that I know for a fact that the certain configuration is deployed everywhere and there are no errors from manual deployment of configuration challenges etc yeah mass changes mess upgrades are now painless and you actually come to come to see the network in a in a different light because if your requirements change if you get a new compliance policy that you have to comply with or if you just don't like how something is configured now you don't have to think about the price of three days of recommended configuration you have to pay to make any change it's actually one click thirty seconds and the entire network of fifteen hundred dollars is now you know fixed for whatever issue ever trying to fix so yeah also this is actually a data-driven approach because analytics can now be ran on your CMDB data the analytics can show you various things across all of the sides across you know how did the configuration change when sighs our online offline blah blah and it goes also begin to monitoring but the point is is dull data-driven approach code driven approach rather than a bunch of manual stuff which somehow works so yes I am a vase or an ISP how does all of this apply to me so even as an ISP you will inevitably have cookie cutter conflicts so conflicts which are the same just very slightly different you know customer CPS and our customer routers brassey's or pppoe access concentrators aps and environments bridges which is tons of switches right etc it's if you think about it the configuration is not that different like what what do you guys do right you just have a template which you feel a bunch of things paste it into a router or you have some flash NIC scripts or something right it's not that different if you think about it so yeah you should at least consider automating configuration provisioning and management of all of these so this is what we are going to talk about now when should you alternate and when should you not then is it ver fit for you to actually will consider getting into all of all of these automation stuff so I would myself consider three base metrics for when to consider an automation system so metric one time to implement automation versus time to do the test manually metric to cost to implement automation where discuss to do the test manually and time equals money but actually these two metrics are separate we will get to why in a little bit and metric 3 is what other benefits do I get by alternating which I wouldn't have otherwise and do those matter to me or not so we are going to talk about each one of these in a bit more detail so time to empower an automation versus time to do the test manually do not forget to account for the initial deployment but also for all the maintenance that you need to do in the future so with any sizable Network 300 plus devices automation will become favorable really fast in this metric but it also depends on if you are automating an already existing network or automating a network which you are building from scratch because of course if you are automating an existing network the initial deploy is not there we already have some kind of a deploy but what you need to consider is what if you want to migrate their network to a different topology for example we want to start implementing MPLS VPN s or you want to migrate from yo IP to VPLS right in the complexity of this migration that you need to do is actually comparable to deploying the network from scratch again so if you have automation this can be easily achieved at 3:00 a.m. at night with one click and suddenly your entire network is migrated from yo IP to to MPLS VPS and I know I'm making that sound very easy I know it's not as easy as I just described it but the point is you can actually do that and minimize the customer impact such a migration would have if you have automation that with one click at 3 a.m. you migrate the entire network rather than doing it slowly over time manually reconfiguring etc and again I know it is not as simple as I'm making it sound but you know so yeah this metric one metric to cost to implement automation there's this manual cost and this can be hard to calculate because it depends on many factors first of all cost of developers versus cost of network admins in some areas developers are cheaper than networking engineers in other areas it's the other way around even availability of the right developers can you get developers or the cost of migrating your network engineers to net ops engineers so the cost and the time that required for you or your network networking guys to learn scripting and learn automation etc and another big big factor is the complexity of the automation system itself in a very unified homogeneous networks automation is easy to make in in networks which are very complex very diverse the automation system you will require to automate provisioning upgrades etc of all of those networks you can imagine is much more complex and the river will therefore require much more time to implement an automation system so yes with large-scale uniform networks this metric is very favorable to automation but in smaller scale or complex diverse metrics this metric is less favorable to automation and metric three other benefits of automation so the factors like elimination of human error factors like in a unification of configuration across your network okay how many of you guys have consistent firewalls across your entire network by this I mean that the address list for the administrator IPS is that named the same everywhere the basic structure of the firewall the chain that you use how many of you have all of that consistent across your entire network on their hands Wow that's actually very good so about 10 percent of you maybe more 15 percent of you raise their hands that's actually very good but for the rest of you you know consider that right because automation will make it that way since there is no more manual deployment and again I know it's not as easy as I make it sound because of course the firewalls will be different on subnet terminating routers and transit routers and PPP access concentrators I know it's not that simple but at least the basic structure of the firewall the changes that are there the input file the address list and how they exist and their names and their contents you know all of that can be made very easily consistent with automation and it can be very hard to maintain all of that manually so yeah one quick provisioning one quick configuration changes with your instant across the entire network substrate software upgrades on mass scale that now become non invasive operations because you can schedule them at 3 a.m. every night which has very little customer impact and they will be immediately applied across the entire network yeah improved network and service quality due to that consistency and all of that you can offer better SLA and I could go on the point here is that in all of these metrics automation is a big thing and it is something that should be considered if it the automation is worth doing or not for the benefits you can get of course together with the other metrics that we have here ok so of the conclusion I'm personally am a big brawl automation enthusiast I ultimate everything even even stupid things which should never be automated I automate them anyway because I hate manual repetitive tasks so I honestly think based on the facts and my previous projects and previous experience in the field that automation is like 95% of the time it's the right call of course you you have to know how to script you have to know how to automate or you have to pay people to do that I'm not saying it's easy but yeah and we can see this this is not just my opinion all other fields of IT are moving towards automation DevOps is all about automation infrastructure as a code there is frameworks like puppet chefs all stack that's all about automating service service deployment etc virtualization containerization and I know that's not just about automation because virtualization and containerization are about many things but it's also about automation right you deployed VM from templates you click a few buttons in a vs and you can have 10 virtual machines deployed immediately it's also about automation about the speed of deployment of services etc so yeah consider all of the metrics that we mentioned considered don't forget about the challenges and this is I'm not saying it's easy but decide for yourself if it verse investigate investing time and resources into I think in many cases you will find it is so yeah let's let's now stop with all the all the nonspecific non-technical things and let's go into the technical so automation basics what starts from the beginning what sorry what ok yeah oh yeah we have to hurry up so the anatomy of a network automation system component one CMDB configuration management database and self-service portal which allows users to change settings in the configuration database component 2 which is the network monitoring management system consists of a provisioning system monitoring system upgrade system and some way to synchronize all of the devices from CMDB to the NMS and component 3 is a backup system so we are of course not talking about the backup system for your servers which you should have anyway we are talking about the backup system for the configuration of your Motors and also some way to automatically get all of your routers from CMDB due to the backup system so here is a nice little diagram users are down here they interact with the self-service portal which changes things in CMDB which then the provisioning system takes and provisions onto the routers mainly have backup of our configurations so yeah right back up well first of all it's the last line of defense for you even if you have automation having backups of the actual proper configuration that you know was deployed on your devices and worked at some time is very useful if your automation system breaks for however many reasons and also it can be really good for disaster recovery so we can actually have backups off-site some where separate if your entire everything breaks last line of defense so yeah CMDB we said it keeps data about the routers unique things like IP addresses gateways Wi-Fi stuff etc and then the self-service portal which allows users to change these things one nice thing is now that you have now that you are teaching your users to use that self-service portal you can actually also include health of the entire system inside of the self-service portal are all the services up is there a known outage in the users region etc etc so we can actually teach users to instead of just you know the wheel is spinning and it doesn't work and Hajj doesn't work you can teach them to just check with the self-service portal first because you know users they don't like to call they don't like to make tickets they do this way you can actually teach them to have that instant gratification of nothing if there is an issue but there is an issue and you can even do things like you know we have already dispatched the text to change your router because it's fried or something similar so this goes into a business side of things but we don't have time to talk about that what's contain so what did we use in our project we use existing inventory system as the CMDB I'm not going to talk more about that because of time limitations and we wrote a simple web GUI as a self-service portal this is really easy for any good developer this is two days of writing code for an experienced developer just remember security security security so properly engineer eight HTTP you know username password blah blah blah so component to the NMS this is the heart of our system it makes sure all of our routers are on the right throughout your insertion handles upgrades mess upgrades later on if we need them then takes care of the provisioning of config the new routers or provisioning configuration changes to all of the routers and then we have full monitoring of our entire topology and as you guys know I am I am the monitoring guy if you have ever heard any of my presentations I can go on and as Greg has said I have gone on previously on other long runs about monitoring so yes proper monitoring is absolutely essential in fact it's so important that I have a separate man presentation just a monitoring so if you guys want to see that it's right here the difference between I think everything is okay and I know everything is ok for a fact is huge and my favorite statement which which which I have had some people argue with me about is that I honestly think monitoring is the most important thing in your network other than the network itself and I will be happy to argue with you on that point anyway yes so have a proper monitoring and so what did we use we use nedick's ms as the NMS it's an open source and i'm a system very flexible extensible see the previous link man presentation and we've drawn our wrote our own provisioning and backup scripts for the proof of concept system this was just some PLC and expect scripts and for the final solution it was Java VJ shell and expect yeah so we use an existing open source system and extended it to have provisioning and upgrade capabilities with some custom code on the mikrotik side router ed is really comfortable to automate around simply you know upload the the config file and run this command and suddently that's it your router is now provisioned with the new configuration so we need some way to get routers get all our devices from CMDB to the NMS for us this was a simple you know little application which synched routers into into nedick's ms the CMDB and the Dixons have api so it was Oracle application connected to the API on both sides seeing the state between those so backup system we talked about that so I will just skip that so so what did you use I have used Dunamis self-promotion warning our product but you have other options like rancid and oxidized or you can just write and clip your own if you would like to talk to me about this or why animals is better than the other options come and see me later on its so yes then of course you need some way for to automatically get all of your routers into the backup system so to synchronize all the devices between all of the systems so yeah for us only must already has a native nedick's MS connector so that was just a couple of clicks and that's it so you manipulate CMDB on me every casing gets synced between all the rest of the systems and users can manipulate what configuration gets deployed we made it so the demo this is usually where I would demo how the system works but I can't do to the MDA I would love to show you but I can so I will just tell you how it works so when adding a new router a new router is added to the CMDB by the data entry team so the data and you know there is some business processes which which are associated with bringing a new site up as a part of those processes the data entry team enters all of the information for that site into CMDB as we know that information gets synchronized to the provisioning system and the backup system and so a installer attack simply picks up an empty mic retic that has absolutely you know it's fresh out of the box not nobody have studied before as it came from the distributor picks it up goes to the place connected connects it up goes to the provisioning system quick provision done so yeah much faster simpler streamline much more process than the traditional approach of modulus errors interest consistency all the things that we have talked about when a client Val client wants to change something they simply go into the self-service portal quick buttons change what they want click Save and the system provisions immediately to the router which means that no pickles no networking administrator's ever touch it etc it just happens right data are updated in CMDB and provisioning happens automatically so yeah you can imagine how for the end users this is a much much friendlier process than having to create tickets support phone calls etc etc and the customers get that gratification that satisfaction that immediately think that they want to reconfigure they can do it it happens immediately security is not compromised because we chose what configuration options they can they can change etc so yeah when updating router has an entire topology how do we do that because software updates I mean it's a network admin simply walks into nedick's MS into the NMS selects a group of routers or the entire 1,500 router topology clicks upgrade and two minutes later the entire topology is upgraded because it's all automated and it just happens so of course this assumes that we tested everything previously in the web you don't just roll out a new version of software to 1,500 routers without testing it first please don't do that and yeah the automation effect again imagine how difficult this would be without automation in fact we calculated how difficult and long that would be about 20 slides ago so and again we have consistency attack surfaces are decreased we haven't even talked about security we don't have time etc so when we want to provision configuration changes to the entire topology we simply write a configuration change to the base template or template and click provision yeah and that's really it about one minute later the entire topology of 1,500 routers is run new config of course this again assumes that this has been testing previously in the lab there are change processes change auditing approval of change etc etc yeah this would be a nightmare without automation as we have talked about time errors etc etc the point is consistency is achieved the back surfaces are again again decreased because of the consistency we have achieved our network is much more defined since we know all the if there are attack surfaces they are the same across the entire network and if there some attack surface has been eliminated we know it has been consistently illuminated across the entire network and reaction time to change Rick's s is massively decreased you actually the fluidity of the network the how you know how elastic your network is and how fast you can apply changes becomes much easier much better so that's it yeah things to watch listen to and promise I'm done so yeah I have a bunch of other presentations from moms that I babe you can find it on YouTube more selfish promotion so you can find it on my youtube channel and yeah I am a part of the brothers which we are a bicyclic podcast hello everyone whoo yeah you can find us here give us a listen if you like and that's it we don't have time for any questions so if you don't understand anything out of this crazy stuff just come and talk to me afterwards thank you thank you so much and the next will be Justin Wilson and he will be talking about high availability
