Building a nationwide carrier for less than $1M: War stories & lessons learned

next up on the podium will be Johnny Martin and Tim Hoffman who used to work together at a small carrier New Zealand talking about how to build a network on a shoestring they'll both talk Johnny we'll start first in about slide 16 or 17 Tim will pick up from there and finish it off right thank you very much all righty thanks some Johnny Mountain I just noted we worked together at a company called FX Networks in New Zealand we had a short period of overlap and and then Tim continued working there while I moved on so we're gonna split the talk up approximately along those lines a little bit of background here the company was founded by a man called Roger de Salas he was tired of the over-engineered and overpriced options that we had in the market in New Zealand at the time and the way was holding back the country and its competitiveness and in terms of pricing you know this is back in the days when transit was on the order of a thousand dollars a Meg so expensive so we wanted to change price and performance by at least an order of magnitude and thankfully we did that and more the funding came from a family friend of Rogers who found civil works far more interesting than the network and things were funded as such so this is a man who used to take advice from his dogs when it came to investment decisions and he also enjoyed going out RVing with the the contracting crews late in the cable and actually watching the fibre go on the ground so back in 2005 there were two companies in New Zealand that really had in the city fibre they didn't really least to others it was very difficult to get anything approximating level one a layer one or a wave or add to capacity through the country if you could get it it was mighty expensive so over a period of five or six years we laid thousands of kilometers of fiber around the country and we'd build a nation I'd nationwide network with on the order of 50 pops we often started these fibre builds with only about a quarter of the funding that we needed so made for interesting times for estranja build the network and those that were ultimately having to pay the bills so here's that done not the best way of doing it but climbing often meant that fiber would get laid across easements through farmland luckily the the Sheep weren't too brutal the the five-point chef my answer obviously things like this would be a calculated risk if if customer retention and customer income was reliant on getting new infrastructure up I would go and do what we had to do so in this case it's lighting in the field other cases it would be cable faders up tied along the fence line down the bottom so if some of the some of the stories from pulling this off and some of the things we've learnt and I feel have really helped us in our careers over time in terms of understanding and being able to make some of the correct trade-offs when when you've got multiple options big disclaimer here that probably should have been bigger on the slide if X was acquired back in 2014 and the network's been refreshed they're networked out to there now which is owned by what I think must be the third largest carrier in in Australasia now is completely different so don't be reading into this and thinking that's what you're buying if you're buying from them now so back then we didn't have a lot of money to spend so we're in a country of less than five million people and it's about two-thirds the size of California so it's very low population density very large distances inhospitable inhospitable terrain that we had to cover we didn't have a lot of staff to do all this and wouldn't really have resident engineers or vendors falling over us to help us make this work and the last two points there we didn't put a lot of time or revenue at the start and they were kind of related because you know sales guys so ladies they'll often promise something to a customer sure we'll put 500 kilometers of fiber to join these two cities you know sign the contract I get get the initial initial revenue and leave it up to the engineers to fix it put the thing in so we couldn't got the the business off the ground with an idea work at the start wouldn't have the time or resources for it so we have to be very creative and really the only way you can make something like this work is when everyone involved has a deep knowledge about everything in the network so that I mean our systems person he also knew a lot about so we learnt a lot about WDM technologies physical running a fiber mechanics of fiber trays and splices and so on a power person got you know learned a lot more about the network and depth knowledge they're the people the crews that were actually laying fiber started to learn more about higher up the stack and has happened right across the small company and in fact I think at one point we had virtually 100% of the country had been out sorry 100% of the company had been out to visit a pop or do work at some point in time right so whether that's a road trip up the country delivering UPS's or installing chassis switches or just visiting local pops - to a system that's required so talk first a little bit about what we call our transmission network now at the top right there so you've gotta go there there's a little leprechaun unit and Telep a CH now this is really not designed for what we used it for but turned out to work great it's really designed to soft patching of different media types and our case what we cared about was 10 gig with Zr optics so that gives us 80 kilometres Founder jumps in a car starts driving stops around about the 70 kilometer mark and starts looking for someone to put a pop that's literally how it happens what are they one of the great pieces about this the founder was pretty impressed when his car bust but after half a million kilometres so with housing these pops a napkin or a layer to switch at these sites and it was just Ethernet segment up the ethernet segment after Ethernet segment a very cheap and effective way to to start out a network bearing in mind you're doing this without the luxury of time and money at that point in time the bottom two photos that we have here we go there this is in a railway control huts in the kapu coast on the bottom of North Island so on the right there we see we've got the filing cabinet mounting method which revolves around stacking neatly your equipment on top of the filing cabinet over time that moved to something approximating more of a normal install we've got a rack which came from I think it came from an auction somewhere and trailers of racks would turn up from time to time and given the instruction that these are the racks to use go make it happen so there we've upgraded to a bigger switch a lot more UPS since this is a site where we didn't control the generator we have to compensate with with UPS one of the interesting things about the the fiber here is the day before the Railway Company in New Zealand was privatized our illustrious founder managed to sign a deal and get access to two pairs of fiber along a couple of spans which the railway company had written off as useless because they couldn't work out how to use it and easement rights throughout the rest of the network so that was very very very lucky timing since the day after that I'm sure the privatized company would have realized that there was some value and the assets that were just signed up part of that was we had to migrate off of one of the five appears a to Meggie one circuit which has been used for train control so luckily you know we worked out how to do that TDM stuff over IP we gave them a replacement new one circuit and got that up and running let's get that one for a minute so as what the transmission network looks like I've got the north of the country on the left and the south of the country on the right this is actually a collapsed ring so day one we had a single fiber path so everything here is is hitting the same pop to the exact same path and the exact same cable so we have to be very careful when when you're trying to build redundancy with a single cable comes down to thinking about things like which tube in a fiber optic cable are you going to pick a fiber out of and we'd end up using three tubes one tube would pull out a pair of fibers for what we called our core another pair from the next tube which was as opposite and the cable that we could get it for access side and then another tube and another appear out of that for a management network which we get on to a little bit later and then we have a small number of peers smattered around the network which gave us the ability to bridge traffic between the sides and keep things running as best we could in the event of power or equipment failures obviously forget fiber cuts there's not much we could do about that the zwm spans we see two of them small ones there that would just point to point metro kit so cisco ons ons from back in the day right about 150 kilometers which was right on the other limits of what the gear could really do on the fiber we had without amplifiers in the middle took a little bit of finagling but we were able to get it working and then through the middle up here you can see we've got our point-to-point pieces with these app con layer one repeaters on our axis side this was a chain of Ethernet switches all the way up I think we got up to 11 hops everything it but again it gets you the get to the 10 gig channels you can start working you can start selling service and then you can improve over time as as resources become available through the building of this network optical knowledge and optical testing became very important so we had minimal optical test equipment at the start it's crazy to think that we had a fiber company that didn't own an OTDR all right you know it's just not something to do this in this day and age but that's where we were so we have to engage contractors to do the testing when it was critical and we personally carried around a Cisco 6500 with Dom optics that's our piece of test gear not the easiest thing to carry around that again gets the job done and that was the most important thing so that allowed us to measure our transmit power measure the received parrot fire and and and test the amount of headroom we had and performance of our optical links whatever a 80 kilometer spans turned out to eventually be 93 kilometers which made us all a bit nervous since the only the only backup plan for fixing this would be to drop a new pop in the middle of that span which we simply didn't have the time for even if we had the money so Kevin the story of this one is typical of most companies we get told you've got an 83 kilometer span coming is that going to be okay every year we can probably make do with that well I drove a bit more accurately it turned out to be 85 kilometers is that going to work well that's getting getting hard to see what we can do they lay the fiber they shoot the ocean of the hour they tell us actually it's 90 kilometres well this is the problem even more of a problem was the same contractors that laid the fibre later that day they cut the fibre and they were shooting an incomplete fibre span so that turned out to be 93 kilometers with this particular span we didn't run into any issues with with dispersion or anything on the link so with with straight up 10 gigs it our optics on new fibre that worked fine one of the important things we need to do on this span though was work out exactly what Headroom we had with our optics so it was really important to grade the optics we graded them took the most powerful transmitters and measured the received sensitivity which is as simple as introducing attenuation into the link and measuring the performance at IP over that link to see when it when it drops down so this allowed us to get the the best optics to put into production and just this important get the next best optics to include on-site aspheres since we couldn't just put any optics into that link and down the bottom right there we can see this is a technique known as the mandrel wrap or the mandrel Bend technique you take a chap stick and you wind your fiber around it and it introduces relatively controlled attenuation into into your fiber so we're using a duplex fiber and a light source and power meter on one side device on the test on the other you could wrap it very carefully until the point in time at which you start using packets and that gives you an indication of how much Headroom you've got so many stories like this as we went through which helped us to really understand the network everybody involved in the company got to got to learn some of these things and it proved to be very useful as as we were building the network power is a pretty important thing in any network so we we came to the eventual conclusion that would run with 1,500 and 3,000 V a commodity UPS's or stacks of them - to actually protect the network and the reason we made this choice is because it allowed not out Russians - to work on the network to upgrade it to change things out and one of the people on the team was a licensed electrician which really helped us go Lindsay druid which meant he could design power systems for us but more importantly he could install them neatly you could label things properly and document them so anybody can understand what's happening with the power one of the big keys here was from day one we made the decision to stick everything behind a transfer switch so these were high quality manual transfer switches it's basically just a big switch that that switches one power source to another and does it within 50 50 Hertz cycle so most equipment will be perfectly happy with then continue running so allowed us to upgrade things over time move things about without disrupting the network we're able to store up to 500 litres of diesel at any location New Zealand without licensing which turned out to be really helpful given how dodgy the power could be in parts of New Zealand and we can't rely on that in a few cases our our colocation facilities I wouldn't really call collocation we coined the term bro location these included such gems as the vault of an old bank that cafe didn't have a useful concrete bunkers that we just dropped on the side of the road and and hooked up rail wear control hats train stations the top of buildings pretty much anywhere where we could get an environment that we could make work for us and within you know 80 kilometers of the next nearest pub would pay people for space land easements whatever we needed 100 Meg internet connections in some cases complete farming communities would actually build a mini network for them to give them Internet connectivity and that greatly uses the the ability to to actually run our fiber we also made the decision in a lot of cases to run with events rather than air conditioning a lot of the scares just transmission gear and even in the bigger pops we weren't hosting servers we were trying to hardest to have our sales team not sell hosting to others so it didn't get too hot in these in these pops and passive air cooling or forced air cooling and a lot of cases was all that was required so it helps a lot with with power so you can actually design a power system that's robust and switches to a low-cost generator pre cheap way we chose to do it was with a lot of you probably know as a terminal server SNMP service on such with loads of dry contacts analog inputs analog outputs and so forth to allow us to monitor everything to actuate relays and such to to test generators for example and not only just test the generators that we could almost also create an actual power cut on the incoming mains and test that things work exactly as they should to live a life failure test and that's something that saw us pretty good over time we got to the point where at least once a month we were cutting power to a pop to be able to test that it works and we found that to be far more valuable than having a more advanced power system but one that you learn to rely on and never actually understand the intricacies of all know what's going to happen if you have a power failure and I think BT not BT and British yeah be a British Airways found this out in quite spectacular fashion recently another thing we chose to not do at the start was not by redundant power supply units due to cost so again this is where having transfer switches was incredibly important because it meant we could again service ups and do work in the future without him to interrupt power eventually we got to the point where we did have redundancy but not at start for some of the smaller kit which had dual power suppliers or you could do relatively cost-effectively we would use those and that allowed us to avoid the transfer switch as a point of failure so we could have switching from generated to UPS and rely on the fact that we've got two power supplies in the equipment to keep things going so down on the left here is is a stack of some event sensors I think century of site boss I think was the equipment we were using this is in fact one half of a Power Board so we'd have a main side which had all the mains there and then a second side which is a low-voltage side and that would have all that relays are sensing and actuation or the incoming environmental monitors and from the start we had really good monitoring but a few years later we had once we'd learned from some of the things we've done in the past we're at the point where we could monitor and control everything within a pop all relatively cheaply kinda like a poor-man's ladder logic PLC type thing except it's all scripting and SNMP boxes so our location we never really had problems with with heat or aircon we were able to control fans if we needed to proactively to cool things down or just rely on a local loop that's looking at temperature and activating them s required one thing we did find it's dangerous for higher up management to have master pop keys so that kind of hastened the shift to an electronic control system but the last thing you want in the middle of the night when you've got customers complaining is a founder drive around the countryside having a go at fixing it so master pop keys while it was necessary at start we got rid of them pretty quick here's an example of another pop in the Wellington railway station it's a back room in the railway station of a capital city it's an earthquake prone zone so they required ISO bases or isolating basis for the racks to separate the racks and their movement from that of the actual building structure we couldn't get any locally and from Portland would cost quite a bit and take a lot of time so ended up relying on some custom heavy steel plates that we had fabricated rolling on bearings with what was essentially elastic bands holding it together and there's another example where some of the folks on our contracting teams and the outside plant people were able to help us with that sort of design another site we had unreliable power and unreliable in the sense that we were given power we didn't control the generator or we saw as the power coming into the rack so we have to compensate with a lot of battery a lot of monitoring and detailed knowledge of the power drawer so we got into the habit of analyzing the power drain of every piece of equipment we had working out how long we could support a pop on new batteries and dr8 it for what would expect of batteries that are being used for some time and at this point we had a single PE feeding primary 9-1-1 call center not our choice we were the backup provider for them but when your backup link is 10 times faster and more reliable than your primary link your run with your backup and at one point in time 7 a colleague physically moved the p12 you up in Iraq without dropping a single packet or a single router which was quite impressive one of our best sites for power happened to be right here inside hydro Jam so this was yeah I think this is probably the biggest generator we had on the network exporting the site we have to get electrical certifications data to get entry to here to actually service things so there's a group of us up the top there after getting a green ticket doing our switchyard and generate a room training learning how to ground a vehicle without clipping to a plastic bumper and all that sort of good stuff downside of this so is it's it's pretty hard to get to and get into fast if you have equipment failure so again just comes down to planning in advance and knowing if you have a failure you've got to react quickly you've got very very small decision times but I yeah we never had a single problem with power and we were so happy with it and with talking with the the crews there that we didn't even bother the UPS's and the first incidence we gained there just to be a more of a a failure point than providing any sort of protection so here's we're having dual power supplies and a unit you know does flying right with a B power and see if you include the generator the got another one the bank vault says cafe bought a bank didn't know what to do with the bank vault it was in the right place for us so we put a pop there one of her favorites pops was the use of what we called the standard concrete bunker so these were prefabricated buildings which didn't take too long to to build and we could stick them on a truck turn up to a site drop them in hook them up to utility power and and the incoming and outgoing fibers and where we go one of these got dropped prior to obtaining formal permission unfortunately have to be on New Zealand Army land but not terribly forgiving about this sort of thing that was actually not far from from one of the training ranges of having my route and we got through that one by exhausting some personal relationships and smoothing things over with some people we knew in the Defence Force not exactly recommended but again as engineers we just have to deal with what we handed so in and here we can see about 20 percent of the the bunker is generated walled-off generator space at you know plenty of ventilation roller door coming down locking it all up the gas tank generator is all in there together and and then the rest of the room is there enough room for about five or six racks and nothing high-power drawer or generating too much heat this is one we chose to put a an air conditioning unit on at the start it gets pretty hot in summer and we deem that necessary and one of the other interesting pops we had was up in the sky tower which is the tallest structure in New Zealand right about 330 metres high it was previously the de facto interconnection point in New Zealand primary peering point in the Auckland appearing exchange was there and just about all transit we'll pass through those black bands up the top of the tower from the 48th and 65th floor there I blamed Joab early for this it's a terrible place for having a meaty room but it did have a single bonus which was the ability to run your own cross connects so you don't have to pay you didn't have to ask anyone you just jumped up a ladder drop the cable in your rack went to another one dropped it down there plug things in and everyone was happy until it came time to kind of trace things down work out what was going on but the power there was pretty unreliable if you wanted to run cables between floors that's when you need a licensed contractor because they have to jump in the core of the tower and run the cable without making it to the bottom the gear could reach up to 50 degrees Celsius and summer so pretty hot and fine for wireless folks because they get a great while a shot to virtually the entire Auckland region and a lot of ice please we you know used to look at the seasons by looking at the temperature of their equipment could actually see the the quadrille cycle I guess temperature throughout the year so it's a result of this we kept minimal edge equipment here says it wasn't a terribly great place to put all this but it was required since there were so many people that wanted to connect in a place like that so this point I'll hand over the temper is going to continue on the story and some of the things we learn thank you yeah so we the probably most notable pop we had was a banker on the roof of a television station in Christchurch this was pretty ideal because we partnered with a radio shop company if you could could tails anywhere around the city but it was challenging because you know you had to kind of walk up onto the roof and in the winter it could be a bit icy and there was kind of a generator up there and our pipe and a number other pots this is one of those ones we didn't control the ups or the generators so when the Christchurch earthquake happened and the building almost fell down no one actually knew how long we had on the generator before the pop would fail the pop actually survived even though the building was in pieces and the fibers were working we were all and we'll just kind of unsure as to how much try and there was to migrate services out so it's kind of a mad rush Jeff move it to the other pop with built-in Christchurch and actually after the earthquake and after their Billy had been knocked down they actually borrowed our office to do broadcasts from which is a little bit further out in Christchurch it was pretty fun and under the fibre we actually learn a lot about building fibre over time and Daniel Greg's did a great presentation on how we did this at PEC not seven as a link there but essentially we started by using diggers and multipliers but we kind of linearly just laid fibre straight into the ground and the challenge was if you ran into something you couldn't get past you couldn't just move on here to stop there and wait a couple of hours for a different piece of heavy equipment to come so you could plow through it or drill through it so we moved over time to using conduit then we blow the fiber through and that meant we could kind of run along with a you know digger or a plow and get all the easy parts in come along later with other parts and attach them all together and there's some quite cool equipment photos that are nerdy and awesome but one of the biggest challenges was fleeting the fiber so that's when you've got atoms hang and twist it as it comes off the coil and initially we actually did this by hand and then initially and over time we moved to and having our own fleeting machine that we got built and that made things a lot faster but a few kind of war stories from this is that at one point some of our contractors to put our directional drill through a lamp post and then the council decided that they wanted to aged out and remove the lamp post and in fact discovered that our fiber was punched through a hole in the middle it so we got this kind of a rate call from the council where they are politely asked us to remove our fiber from inside their lamp post another one that was interesting is we found it was really important to mark your fiber so another provider Telstra was very bad at this and there was a 300 kilometer section they didn't know where it was so we kept you know getting their contractors out to point out where the fiber was so we didn't drill into it and cause maybe five or six cuts on ER and got the point where they were saying if you cut it again we might actually not be able to use this due to the loss so we took great care on our fiber to make sure that we had very clear markings and Pole and posts and also we actually laid a plastic ribbon about 20 centimeters under the ground where the fibers of meter under the ground so that people would strike that first and see the warning sign and the other cool thing we did is that we actually but side the fiber we had a piece of copper wire so energy to the arrow is pretty good at telling you within a few hundred meters with your filters but you can use the copper wire which we popped up at every color meters or so into a pole to actually pinpoint exactly where the fault was because you could tell where with cast and then you can get with that 100 meters you can generally see whether there's a digger or something picking up your fiber and probably the other fun thing was when someone built a road or a fiber because they didn't check so we kind of did this fiber fault and we got there and there was been construction things and we realized that that actually managed to successfully seal over me to death it's buried fiber with a road for half a kilometer before they actually broke out further it's pretty impressive and finally the other sorry great fiber wall story was that we would get ran into problems with rats because rats like the gel inside of the fiber so what was really interesting is by selecting tubes on either side of the fiber often we would survive fiber cuts because the rats were unsure and far enough to get into the gel before you know before they got they got tired of it so we really tried to make sure we buried it well and had middle sheaths across the bridge but the same time we had to be very you know it was very good having you know those two tubes so we were at risk from that so the original network was actually built with just five PS and we chose 276 hundreds and then 65 hundreds of switches because at the time you could run the 7600 software unlicensed and unsupported on this the 7600 software on a 6500 so you get cheaper Spears and over time that's expanded to about 50 40 or 50 PS but what would actually do initially it's just you know long line customers from a switch to two different peas in two different directions and provide them to services so if one fell they could kind of failover that made a lot cheaper and the key here was actually really understanding how the network would fail so we didn't run spanning tree we ran everything lib free and the MPLS cool links were another VLAN on the transmission network you know well it would fail at times we knew how it would fail so if your plan for that and provide services around that and you know it's pretty simple because the main business were player two wholesale so we decided not to buy cars that could do shaping so it's figured if someone bought a five hundred mixer us we just gave them a one gig service it's actually cheaper than going and figuring out how to get a card the word work to do shaping to the kind of fine ground level as part of that we ran really old versions of code and a very small number of features so as we know in the more features you have the more bugs you have and we never really upgraded because of that so it's really stable simple dumb network and we over time bought a transit provider and what we decided to do is actually keep that separate from the main network we just had stacks and stacks of 7200 ones which were buying layer 2 services off the network even scaling pretty wide by having lots of different things plugged into lots of routers and just you know making sure you spread the customers out enough so everything we were doing was trying to make it cheap it was actually pretty easy to build a pretty good network with cheap stuff and in the most complex thing we actually did on the network was that in New Zealand thousand dollars and Meg transit it was much cheaper to buy domestic so we kind of separated that out initially under separate routers in long term under separate various we had no not no help desk just the five of us engineers and one manager where if you'd call up the company so a small number of customers you'd actually get an engineer was building the network and while it was a bit of a hassle at times could be focusing on something it's actually pretty good because it meant that your customers knew they weren't talking to someone who didn't know what they were doing but I'd be talking to someone who just log and it fix it and yeah we avoided all these complex services like ppl Sol through VPN she tried to keep it really simple and often keeping it simple would be would give you double the services you bought so that way if one went down you couldn't complain and radio was also our friend so as Johnny mentioned initially we had only one fiber up and down the country and it was really challenging when we had five accounts right because you know customers generally like their service to work so we bought a very low capacity radio path up and down the country and some quite expensive shaping cards they let us do hierarchical koalas and keep the customers we cared about happy during five accounts and you know we cut over time that's got harder and harder to manage as the traffic grew and we started with maybe a little bit more capacity on there quite a bit more capacity on the radio path available than we were using and then by the time we actually expanded the network to have multiple parts up and down the country we probably like had a quarter of the capacity we needed available in the radio path and customers weren't especially happy at that point we also use radio for accessing our customers so we had a lot of partners who would do a radio shot from our roof of a bunker to a customer on a farm somewhere or you know customers would sometimes bring their own radios and just install them on our roof and we give them an Ethernet port for power so kind of really yo did what we had within the constraints that actually built a pretty cool service some people at the time newzealand didn't really have a whole lot of Metro Fiber available so you had to really make do with what you had and the biggest thing here was learning to debug and reverse engineer so look a lot of people grow up and the grill with a resonant engineer or you know a good tax contact or they're spending a few million dollars with a vendor so the vendor will help them not so much for us we had to get pretty good at you know TC be dumping bgp session coming up to find that bug or be able to do IPF's around the network all sorts of different things so we would react Chua lee had an every pop server with four one gigabit NICs so we could plug in and run an I here for two whatever we needed to debug a customer problem it's pretty handy because you know for customer complain you could drop some IDF servers into their service and let them might be up against you and prove out where the problem was but the biggest challenge there was actually that most customers in new zealand's didn't really understand CSP windowing so it's been a lot of time teaching our customers how TCP window and worked and how to actually use this really fast service that got at more than 50 megabit and onto the outer band network one of the biggest challenges was you know that most of our fears run into is that you've got kind of an isp service you might buy for out-of-band and you've got to jump into a box and then try and log on to a console server we had 96 fibers so we built a totally separate out-of-band network on a different core again the rats struggled to get to it and initially this was just like a layer 2 on desktop switches that only bought in an auction and we've got to put them in the Pops and attach to IP phones and cameras and fees and you know Center these dry contacts to them and then over time we moved it to a routed network as we got more than just a linear up and down the country but what's cool is we were in a position where you never be making a change in a device other than the path that you were changing right so you'll be going totally out of band so if you screw it up you can roll it back pretty quickly which was pretty useful given that was a single path up and down the country and you screwing up could mean a pretty big impact for customers so at the end of the day making do with what we had we built redundancy it's in pretty unconventional ways and we made it better as we solved services so obviously the moment money is coming in the door you can do a bit more but we really had to have a detailed understanding of how it worked to be able to you know operate it responsibly and to be able to get to a position where customers were happy we became the net primary provider for a whole bunch of really important customers like the national power grid pure power plant police network government shared near book and or every ISP by the two telcos who ended up buying our fiber off us because they wanted our parts so we fundamentally changed the market taking the price of a 10 gig wave from you know multiple hundreds of thousands of dollars a month to around $10,000 a month and ultimately sold the company with 1 million dollars of network equipment for a hundred and 16 million dollars in 2014 so as much as you know this might have been pretty dodgy and start we actually achieved a really good outcome for New Zealand where users could get pretty fast and in it after we were done any questions ready although thank you very much thank you very much [Applause] Sorry Sorry I could take a hint Patrick from Berkeley this is gonna sound like a silly question but it's actually very serious you dumped essentially concrete bunkers in the middle of our meal and I've tried those kind of things in other places and there was a serious worry about them being blown up of anything like that yeah that's true for the fiber paths as well I mean army guys liked and gals like to do things that are not conducive to networking equipment or human beings luckily this film was about a hundred feet from the road and close to the actual the actual town there so we had a shared fight you could say yeah sorry thank you by the way so what's challenging though we're never presentation of this man oh thank you oh thank you thank you any else anyone else ready all right thank you

Channel: NANOG

Views: 2,363

Rating: 5 out of 5

Keywords: NANOG 70, Kaskadian

Id: r9DOAxTW96I

Length: 38min 16sec (2296 seconds)

Published: Wed Jun 07 2017