Open Undersea Cable Systems

all right so this slide is meant to quickly talk through why is Microsoft at Nano talking about open subsea cables so if you're running a Microsoft service you're going to run it on the Microsoft plan so we have our own backbone switches routers fiber-optic cable connecting data centers connecting regions throughout the world so our goal as close as possible to you the customer yeah more often than not a commercial customer will suck the bits onto the Microsoft plan and run them over our network we do it for a number of reasons cost a big reason but most important reason is availability we get end-to-end control over the network the supply chain etc and we feel we can deliver capacity quickly more quickly than going through third parties and in more controlled manner so we can we get that control clouds hot right now exponential organic demand I think throughout the week you probably heard that and it's real when I say organic demand it's actually you know cloud service adoption more and more people going to the cloud is causing exponential capacity demand that's hitting the when and that requires a very creative transoceanic strategy and that's what we're here to talk about so the azure blog published recently showed the scale of the Azure lamp and here it is again you know very high level it's not all owned majority of it is owned there's some leaves some partnerships one thing I noticed when I look at it a lot of subsea cable a lot of blue lines running through the water a lot of cables will talk about why there's so many later on what is a subsidy cable what are the building blocks when I look at a subsidy able I see essentially two pieces two things that we worry about a little bit separately a dry plant and a wet plant so the dry plant is the point of presence the data centers and shown here power feed equipment those sitting things called cable landing stations or datacenter buildings at the cable landing station we switch from dry plant to wet plant and that's where we go underwater when you go underwater you've got subsea cable shown here three different types of cable the thicker one on the left is double armored typically put in shallow water the armor is there to stop fishermen from hooking it and breaking it the once you go further under waters about 60 kilometres out you hit your first repeater so the undersea repeaters amplify the photon so there's actually amplifiers under the water and you keep going you can get branching units about every 60 kilometers more repeaters by the time you cross an ocean anywhere between 100 to 250 repeaters under the water and of course here it's shown up both laying it all laying the cable some pictures of what this looks like in real life top you see the subsea landing station the landing station a building building itself is not that big that's important to keep in mind when we talk about how to build a good optical and switching architecture so you don't have a lot of space in landing stations beside the landing station are in this picture two big generators if you're ultra paranoid about availability you have backup power so you're seeing two generators just about as big as the building to power all this there's a picture of a shore burial and also an undersea plow so when you're laying those components we just talked about the cable a double armored cable single armored or no armored cable and repeaters especially a plow at the bottom of the ocean that buries it and there's a guy in a boat driving it like a video game and burying the cable burying the cable burying the cable so you know pointing all this out to show it's a it's a feat of modern engineering very labor intensive it looks awful expensive right okay talk about the cable for a second and this stuff is it's really cool so around the outside of the cable you're seeing here steel tubes the steel tubes are they're not there for pressure they're there for armor in this case this is a double armored cable so you see two layers of steel tubes on the outside of the cable inside the two layers of steel tubes you get to kind of the meat of the cable that's the telecom network there is a power powering element it's copper there's a lot of science that goes into it how thick the copper is it directly relates to how much power you put on the cable which directly relates to how big your power feed needs to be which indirectly leads you to how much capacity you can put on one of these cables so you've got the power in there you've got an insulator and in the middle it's just very tiny little mill the optical fibers that run through typical cable can run between 8 and 12 v repairs put that in perspective typical terrestrial buried cable is somewhere near 100 some of them near 200 pairs of fiber-optic inside the cable so you get about an order of magnitude less fiber pairs in the subsea table so subsea cable about an order of magnitude lower capacity than the terrestrial cable the power feed equipment is really cool so you have this this long cable legend 12,000 kilometers across the ocean and all you have is stuff on the end that's what it's what you power with in this example I've shown a 14 kilovolt PA so what you do is you set your your PFE plus seven on one side minus seven on the other side and somewhere in the middle there's a virtual ground it hit zero and the 14 kilovolts runs across the copper you know I call it a few milliamps and it powers all of the active components of the cable ember every 60 kilometers to 80 kilometers we are amplifying the light so we send light on the end we amplify it we amplify it and we get the packets from one side to the other so the power feed equipment creates this virtual ground one of the most common failures of a subsea cable is a tear in the sheath so outside of the armor here there's a black tarry looking thing that keeps the water up and over time the cables chafes with current a little bit of water gets in and when the water touches that copper it immediately changes the ground right because the sea is it is a great ground so in this example you can see the bottom one if I'm a few hundred kilometers off the coast and a little bit of water gets into the cable the PFE actually changes and the ground moves and one side in this example goes up to twelve kilovolts the other side changes to minus two and i've got a real ground a sea ground that has just been created it's really important because fixing your cable requires calling a boat a boat comes out fishes it up takes it apart and put it back together if it's unscheduled it can take weeks sometimes a month for that boat to get out there so by having the power feed equipment have the capability to change the ground if the cable gets compromised is one of the reasons that these things are somewhat available there's some what you know they have a good uptime good availability now you can only handle one of these if two these happen you're toast so when they do happen there's all kinds of alarms and you get the boat out as quick as possible you know that's in a nutshell how the cable is powered so you want to design the cable to do what we call single and Phebe you want it's to have you know if it's too long you can't do this if your power feed equipment doesn't have enough voltage you can't do this if you put too many active components in the cable like repeaters you get better performance but you might not be able to do this so all these trade-offs the repeaters hear me say it a lot because I mean these are super cool as well there are these big titanium tubes and we brought one into our lab and many many hundreds of kilograms weight we had to get a special crane take Windows out to get it in so a subsea repeater is made to go down to you know five to eight kilometres of depth inside of it are just amplifiers erbium doped fiber amplifiers you see them in any terrestrial systems like hundreds so the technology inside is no different than what you see in a terrestrial system it's controlled in a little bit of a different way it's run in a little bit of a different way but fundamentally the same technology the cool part is they put it in this titanium tube that you can sink down to eight kilometers of depth and it can handle that pressure and run reliably for twenty five years now the size of these repeaters also eliminate or limits how much fiber you can put in the cable which limits how much capacity you can put in the cable you can imagine everything I just described it doesn't come for free so I have a note on the bottom your three dollar signs which means expensive these are very very expensive one of the most costly pieces of a subsea cable and of course we pay a lot for them so you get to put your name on them so if you have a cable that you own they like to scribe your name on the side so 25 years from now if someone finds one with our name on it reliability one of the biggest problems with relying on subsidy is reliability it's an interesting article from Mental Floss number two most interesting fact I don't remember the number one imagine it was good sharks are trying to eat the Internet this is real the the powering of that cable we just talked about imagine you're sending 14 kilovolts across the ocean it's a little bit of an electromagnetic field that happens these things are kind of like giant antennas and it makes the Sharks angry so sharks do tend to bite them it is a cause of cable cuts now the most common cause of cable cuts is no simple chasing from ocean currents you know you have rocks on the bottom the plow didn't bury it quite deep enough probably because we were trying to save some time and the ocean currents over 10 years tape that cable back and forth I take a shunt fault or even worse a clean cut if you take a clean cut in deep water in for example in North Atlantic it could be a month before someone goes out there and fixes it a month imagine losing a connection for a month so connecting to regions take us to Europe for example you don't just run a subsea cable and say you're done and of course a lot of people ask why don't you satellites and there's a lot of latency in satellites and you have to wait eight seconds for your SharePoint site to load you might be a little upset so we use the subsea cable so you run these subsea cables you can't just have your one if I want to get one bit from from here to Japan can't just have one it turns out you do the math and - case scenario you need three if you want to one one bit from here to Japan with any kind of reliability where you can stand in front of your customers and say we'll get you the bits there you need three cables so there's a tripling now if you build it wrong you're going to need more than three and these cables are very expensive very expensive hundreds of millions of dollars so it's it's expensive business so three or more tables to get a bit across which leads me to the problem statements we talked about cables are incredibly cool they do all kinds of cool stuff along with that is a price tag they're very expensive they have a very long lead time if I want to add a bit between here in Japan and there's no more subsea capacity or kick the project off in three to five years down the road the boats are laying the cable so they have a very long lead type a lot of the time is in surveys marine surveys permitting especially here making sure you don't make the orcas angry as you run through the Puget Sound so a lot of time goes into the permitting and planning so before you can even start to lay the cable you have to get all that in order it's a limited resource because they're expensive because there's only so many companies that make them there's a supply chain bottleneck and you only get about 20 terabytes per fiber pair 20 terabytes per fiber pair times 6 8 10 5 repairs it's not a lot of terabit in comparison to a terrestrial backbone address your backhoe bone can run something like 24 terabytes a fiber pair on 105 repairs the subsidy bottlenecks that into something like 8 so it's a very limited resource in terms of how many there are to choose from and if you need a lot of capacity you're out of luck here you might need more than those 3 cables we talked about and then they have a very low availability compared to a terrestrial asset your subsea cable you expect it to break you expect it to take a month to fix when it breaks so over over the lifetime of the cable somewhere between one to two nines of availability see a lot of redundancy to build the five nines cloud network so when you do invest in a subsea cable you have to be really smart about the choices you make it to be creative but where it goes if you're running a cloud network you don't want to run New York London like the stock traders have five cables - do you need more diversity you want to run somewhere different so diversity is the key not capacity you know not not necessarily topology its diversity and once you do invest in that asset you want to make sure you make the right choices around the asset to enable it to give you the absolute most efficiency we'll talk about s LTE so earlier we mentioned there's a wet plant and a dry plant so we talked about the cable got the cool power feed equipment incredibly cool repeaters very expensive in the cable landing station we have s LTE submarines line terminating equipment this is the optical transponders transceivers MUX condors use your favorite term that light the cable using modern equipment is the easiest way to get efficiency out of these cable I have a table here showing you know one of the common tricks use modulation format a modulation format and you can see in the table QPSK a comm sixteen qualm we don't have time to go into exactly what those means but they convert to capacity so if I can run QPSK I can do 12 terabytes a qualm 1836 teen qualm 24 terabytes and the fixed cost is the same so essentially is to hire my modulation format I've got this fixed cost of my asset I get more bits out of it so if I get more bits out of it by using this trick I essentially reduce my cost per bit which is a good thing a lot of technology advances in submarine cables have been here the cable itself fundamentally hasn't changed 15 to 20 years things got a little wider in terms of bandwidth the designs have gotten a little more cost effective but not they're not over engineered but the cables technology hasn't changed in a long time the terminal equipment the S LTE has changed really rapidly to get the most out of that cable a subsea cable is expected to last you 25 years so when you put it at the bottom of the ocean you've got 25 years of lifetime the SLT II you know follows Moore's law every three to five years what this can do double so you want to design your cable to enable Moore's law to do its work on the end and get more and more efficiency out of that cable as it ages out so you've got this 25-year static thing and every three to five years you're cycling but on the end to get the most out of it how to do that there's three different ways you can buy a cable you can do a closed system you can say I know I don't have the expertise to engineer this so I'm going to get a turnkey solution you're going to get the SL to ease the cable altogether upside is it's easy the downside is we'll get into that it's a bit ugly the middle is an upgradable system you start turnkey and then you do some surgery odds brain surgery but complex surgery to extract the turnkey es LTE and put somebody else's in down the road it's a little more flexible than the closed system but has its downsides - and then what our selves and the other cloud providers seem to be pushing and requiring our open cable system and very small differences between the three that have very large impact on the efficiency of the cable over 25 years so we'll ignore the turnkey the upgradable system starts his life as a turkey and then works its way into a quasi open system in an upgradable system you've bought it as a turnkey when you buy a turnkey system the measure of whether or not that cable is good so remember this this thing that costs us hundreds of millions of dollars and we've sunk it at the bottom of the ocean if you want to know how well it's performing in this model you ask the S LTE how fast it can go essentially how much capacity you can put on the cable modulation format that we talked about a few slides back if you think about it that's like it's like saying because my car can drive 70 miles an hour there's nothing wrong with it it's the wrong way to measure quality you're your abstract expensive thing behind the thing that you want to recycle so you lose visibility of what's actually happening on the cable and I've seen this firsthand on you know over 30 cases over the last 10 years where the cable itself has systemic problem it's degrading you know your tire pressures getting low your engine is failing but the car still goes 70 miles an hour right and if you're like me you've owned a car that had no business going 70 miles an hour but you did it anyways the same thing the SLT II was saying I can still hang on I can still hang on but something was getting worse so over time the cable has degraded and then you want to put the newest latest and greatest technology on it and you've let your hundred plus million dollar assets degrade because you did not have visibility of what was going on at the bottom of the ocean so how do we solve it how do we solve that I've got a picture of us doing a subsea cable turn up we purchased one of these closed systems we did two months of planning to surgically remove the turnkey SLT II and then we sent a rolling lab it's actually three crates over 500 kilograms to create a rolling lab out to site and we spent four people two weeks 16 hour days a lot of coffee and we extracted the current state of affairs from the cable we did a lot of measurements even that is not not the best way to do it but at least it's an order of magnitude better than just asking the SL te hey how fast can you go today so we send a rolling lab oh if you want to scale fast if you want to get capacity out there fast you can't you can't do this right you need a better approach you can't roll a lab and six experts for two weeks every time you want to add a terabit it's just not going to scale so what have we done we've opened the cables up so any any new cable that we purchase any new cable with a consortium any new cable we're released even we ask it to be an open cable the difference here you can see this clear separation between SL T and the cable and the gray boxes should have probably used a more interesting color because as these do really interesting things we've invested in technology to measure the cable with more precision so these gray boxes now can go query the cable span by span and ask that how it's doing today so we essentially created check engine lights and all the other fun lights or dashboards have one of the really cool advancements is line monitoring system or a coherent or correlated ode to the air so we send a signal from the shore and the amplifiers on the water have a have a return path built into them that's out of the band of where we send our bits and over time and a lot of math we can actually extract a span by span OTDR so in real time we run in service otdrs on the cable and we create this visibility every span i understand what's happening and then we stick an optical spectrum analyzer on the end and we measure physical properties of the cable like OSN are like power tilt so the actual figures of Merit for how well the cable is performing how healthy the cable is directly and then we alarm and alert on those you know it's no longer asking the SLP how fast it can go we actually have a completely new set of measures directly related to this asset at the bottom of the ocean and we alarm and learn on those directly simple change huge impact now I know in real time the health of that expensive asset a whole you know this manifests itself in some interesting ways improved availability is the most interesting if you have this you can in near real time see something degrading so you know we talked about the sheets of the cables degrading you probably get forwarding over time you build enough intelligence to see that process happening it's not primary it's analog not going to go into detail here this is the little detail on how you do it power versus lambda a picture I just wanted to point out you know the cable itself is analog delivering an analog signal so we've actually built the expertise on the end to evaluate that analog signal a little more expensive but compared to the price of the cable it's a good investment all right so we talked about the cable the S LTE what we do in the S II LTE to talk a bit about now we got to get us the data centers I generally don't build data centers on a beach a lot of good reasons tidal waves probably being one of them floods hurricanes etc certainly builds the data centers in land somewhere much more reliable the the old-school mentality is well I've got my cable I want to get the bits across as cheap as possible I'm going to put my transponders and switches in the data centers and just run everything like one path it turns out if you're you make a little change in your landing station you put a packet switch in here and this packet switch is a commodity layer 2 layer 3 switch you can buy from a number of vendors you know true blue packet switch and you stick it in your landing station and you have diverse back holes to your data centers so you run two fiber paths on land your availability improves significantly and this is the difference between needing three cables or for some cases the difference between needing two cables or three so if you can invest a little bit in your landing station and your the cost of these aren't very high you can save a cable and the cost of cable is very high so once you're in the landing station put a packet switch in and we call these packet switch s LTS our packet switch CLS so any new cable we like to see some sort of packet switching in the landing station and we go diverse to the data centers it's a very different eyetality than a true carrier a true carrier would probably run it what we call pop to pop where they would run this pack I switch to pack a switch in the end point and their end n Pass is susceptible to cuts on the terrestrial and subject so we're back in the data centers how do we run these things you know I think open line systems on the terrestrial side have been you know very well published talked about for the last five to ten years and we're no different we believe strongly the same reasons we just discussed in subsea we want to cycle the terminal equipment more often than the line all of our terrestrial systems going forward or open line systems with the Microsoft Sdn controller so we have our own network management system that we built and we want the subsea cable to plug into this why do we want the subsea cable to plug into it and subsea cables are incredibly cool you know from an optical guys perspective it doesn't get more cool than that but in terms of the number of bits you run over subsea cable they're limited resource they're expensive so you're going to tune all your software stacks all your replication patterns to not use it unless it absolutely has to so all of your data center replication you're going to try to keep it on a continent unless you absolutely cannot the majority of cloud traffic is this kind of replication storage replication so you see the Metro we have a lot of capacity long-haul we have a lot of capacity not nearly as much as the Metro the amount of subsea bits rerun and it's a lot in comparison I think to a traditional carrier but in comparison to what we run in the Metro and long haul it's very small so we architect our tooling for the majority use case the Metro and long haul and we'd like to fit subsea into that now of course there's custom bits but the building blocks are the same it's an optical system so we really focus on the scale the large applications and the open cable plugging into last slide here Mireya cable mariya cable is a well published a cable that we've partnered with Facebook on and we built it across the Atlantic I'm showing this you know not to brag although it is pretty neat this is the perfect example of why you might own versus just leasing you see the Maria cable it runs from Virginia Beach to build-out Spain Virginia is where data centers are it's the first cable to land that fire cells on the North American side a lot of the cables between US and Europe run as high north as possible New York to London to get that high high speed low latency path when you're running your cloud Network you don't necessarily need that you want your availability you want your five nines so we build a completely diverse path and then we made a completely open packet-switched landing stations so Maria is a great example of an ownership on the cloud so going forward when you're looking at you know what our selves and the other cloud providers are doing in subsea space you're going to see a lot of wacky landing points we're going to try and get these in diverse places where cables have never been and that's where I see the investment going Thanks [Applause] question great presentation thank you Matt peacock from Yahoo here so you took out the somewhat closed proprietary sltv system which wasn't giving you visibility into the a subsea fiber and you replaced it with your own cobble together homegrown system that replaces the SLT II and tells you what's going on with the fiber that's basically just kind of swapping out one smart box for a different smart box isn't it can you clarify why you felt that you couldn't just have the SLT be itself give you more visibility directly rather than replacing one box of another yeah yeah it's a good question the SLT II fundamentally abstracts the performance behind something like prefect bit error rate prefect that error rate is not a direct measure of the quality of the system there's a lot of things at work so you know for example and you could have a bad prefect that error rate because of high launch power and high launch power might be a sign of a perfectly working cable in fact okay but it's working better than you thought so the SLT II abstracts the performance behind this number that has no direct relation to how it's performing so when you're saying the SLT gave us what you needed the SLT give you prefect bit error rate and from that you had to guess whether or not the cable was the cause of the problem or something else was the cause of the problem so we still use the prefect bit error rate still use it it's still a valid measure it helps us understand the nonlinear performance of the cable but the linear performance of the cable the thing you're paying for the repeaters the fiber effective area things like that that is directed to O SNR optical signal noise ratio and tilt and power so what we're saying is we've added monitoring for OS n R tilt and power directly and we alert an alarm on that directly still have prefect better rates but now I know if the prefect better rate is bad I can look at this other thing and say it's the cable but another reason you do it I've got something not performing I want to fix it as soon as possible immediately I know who to call I know if it's the cable or something in time but you didn't go to the vendor and say he would like to have you add these capabilities in and exposed the results to us so that you didn't have to build a completely new box for it yeah yeah and then oh you're right so the the alternate approach is you go to the SLT vendor and you say do this for me which is valid if you want to vendors now you have two different solutions two different solutions integrated into your Sdn controller three years down the road the vendor you picked might suck you know be francs might have went a wrong direction for you you want to replace them now you've got the burden of doing this again what why not just put the monitoring in day one and run it for twenty five years and then use that same thing on every single subsea cable you own remember these subsidy cables eight five repairs about ten new ones going in a year you just just put a lot of these out there day one let them run for twenty five years and don't worry about it and that's our philosophy how do you onboard and turn these things over as quick as possible and I see letting the vendor do it is a barrier to you know doing this quickly eventually day one same thing five years down the road when I've got four different monitoring systems doing things a little differently I see problems that make sense all right thank you hi Eric kunti with dual communications my question is regarding a the use of a cable landing station as a terrestrial interconnection point a couple of years ago I did some work related to the AC II cable landing station in Freetown Sierra Leone in West Africa and the interesting thing I noticed about landing stations in West Africa and particularly all of their recently installed systems is that they are the most significant reliable used to telecom infrastructure anywhere in a given nation state so ISPs and network operators and mobile phone operators whoever is operator in Liberia or Sierra Leone or a location like that those cable annexations have become the de-facto interconnection point similar to a 60 Hudson or 201 wheelchair and what do you see the role of on the terrestrial side for interconnection between ISPs and colocation and hosting of a landing station in a developing nation environment considering that ordinarily people don't build kolos on the beach but that's what these have kind of become down yeah you're right I think I over generalize there are parts of the world and particularly developing countries where the terrestrial challenges are very very difficult you know for example Brazil we experienced a cable cut a day on the terrestrial side subsea much more stable so countries like that you're correct generally we can pull the switching closer to the ocean and still I don't think the right thing to do is put a data center potentially that close to the ocean you want power diversity you want a good fold segregation you just want an altar available building in an ultra available place so what we do in that case instead of just doing a simple packet-switched landing station we would probably do a full what we call a gateway and we would turn the landing station into a full gateway you need to know that ahead of time because that's a lot of space and power that's typically not there in landing stations but it is a good point going diverse how you do it it does change depending on what continent you're land on Thanks

Channel: NANOG

Views: 5,342

Rating: 4.7368422 out of 5

Keywords: NANOG 70, Kaskadian

Id: YQ8J7U2bO3Q

Length: 36min 15sec (2175 seconds)

Published: Wed Jun 07 2017