An Introduction To Software Defined Networking (SDN Tutorial)

all right good evening everybody thank you so much for coming tonight take a little bit time out of your evening to join me this is a seminar that's going to be on soft words of mine networking this is a brand new way to build networks if you know a little bit about how we do networks today that's going to serve you very well tonight if you don't know anything about how we do networks today don't sweat it we'll cover that we'll talk about why this brand new way of building networks is better quicker faster than the old way now this stuff is cutting-edge and what we're going to be talking about tonight is actually technology that is available in products that you could buy today but it's early on if you look around you're not going to find a lot of Sdn based networks out there you're going to find a lot of companies that are experimenting with STM networks they're poking around the corners they're taking a look and seeing if this is something that is going to have some value for them the most important thing is that you're going to see companies like AT&T which operates one of the largest networks out there issuing RFPs to their vendors saying this is the direction that we plan on going to tell us about the products that you have they're going to help us go in this direction remember 1880s America like a supertanker right it takes ten years to sort of turn the boat around and stuff like that so the fact that they're making those types of statements to their vendors today indicates that that's our plan for going forward tomorrow so this stuff has legs can i annual as we go through this you'll start to understand where the true power of SDN comes from and you also understand why companies like Cisco and juniper are so concerned about it all right now concerned means that they see it as a threat to their financial income concern means that they will be taking and are taking active steps to squash it 9 so there's an exciting battle going on in the world of computer networking today what used to be fairly cut and dried builder network who hit some Cisco routers and we're done has suddenly sort of been turned on its head and that's what we're going to be talking about tonight so as we go through this first off there's cookies in the room if you don't have cookies in front of you you'll fail you need to go get more cookies when you finish the cookies that you have go get more cookies because I do not want to be going home tonight with any cookies left on that plate if I if I do you failed me you don't want to do that because I'm very emotional there's we go through tonight's presentation if you have a question please ask it guy that I would like this to be as interactive as we possibly can I have information I'd like to share with you but if you have a question to ask I'm sure it'll benefit ever in the room that sound like a fair deal alright sdn this changes everything let's have a talk about what we're talking about ciao okay I have three goals that I'd like to accomplish during our time tonight the very first goal that I have is I would like to share with you what Sdn is maybe a little the history where it came from and why it's such a big deal the second thing I'd like to do is talk about Google Google has done some amazing things already with Sdn they are truly a pioneer in this field so like I will talk about what they've done and I'd like to wrap things up tonight by talking about what the future of Sdn is once again this is a new technology it's not established there are not a lot of books written about it there's not a lot of papers that have been presented on it yet it's a brand new stuff so the future is by no means mapped out but I hope I can give you a little bit of an insight into some of the different directions or some of the work is it sound like a fair deal three things that's why did you get four things with congratulations spectrally all right so let's talk about has dance so this is a little article that I clipped out of the Wall Street Journal it says what does it say ATT targets flexibility cost idioms with new network design new network design is software-defined networking guy what the article talks about is the fact that a team team a gigantic well-established telephone company that already has a built out network is actively looking at some for Defined Networking the question is well the reason is is this companies like AT&T end up having to rip out their network roughly every 18 months so cisco comes out with a new generation of roberts are quicker better faster fanciers off or whatever the story is so ATT has to reach into the network remove the gear that they put in there buy the fancy new gear and put it in and they spend literally billions of dollars every year keeping their network current high and that's sort of the way it's always been the smart bright folks at AT&T have been keeping their eyes open they're aware of software-defined networking they understand that it takes a really different way of building your network and then it holds the possibility of getting them out of this endless cycle of having to completely rebuild their network every year and a half to two years and that has them incredibly excited good evening you're in the right place sign in on my list grab one of those two pieces of paper I got a pile of cookies back here you need to grab some cookies before it's a damn thing around so hey GT is interested in this I'm telling you that I'm going to talk to you about what Google's doing these are names that you recognize these are big companies this trick this is what it is it's a stamp of a technology let us go back to networking 101 shall we go all the way back so no matter how much you know about networking don't worry about we're going to go back to the very basis so this is a very simple five node I've got five circles those circles or routers I could go out to Cisco today and buy a circle from Cisco I have five users represented by little circles labeled one two three four and five is all the way over here okay once say that Mister five is Netflix and let's say Netflix has a hot new copy of Avatar 2 everybody wants to see avatar 2 it's a great movie of sure whatever it is so user 1 2 3 & 4 say listen I need to connect to user number 5 so I can watch avatar - well that sounds pretty good so number 1 is going to hook it to router D which will connect to router e whose will then connect to router half and that's how you'll be able to talk the first number five which is Netflix and everybody else is going to do sort of same thing this happens a thousand times every day every time you log on every time you go to a website same sort of things happening so here we go the dash lines represent the connections so you can see there's a red line that connects rather be with user number four to avatar two over here at rather F the green line connects router a for our app and we say Purple Line they can x-ray our feet try to wrap so everybody has certain actions everybody sitting at home watching avatar - everybody's happy life is good and this is the way that networks work today unfortunately this is also the way networks work so the link between router e and router F has just failed undoubtedly there was a backhoe digging in somebody's neighborhood and it just nicely clipped that cable path so that's gone what's going to happen in the network router is going to inform D and C that it can no longer talk to F sorry can happen what's going to happen then this routers a B and D are going to go here well that didn't work out so well I have to rebuild my connection to user number five now remember there's this link here that they can use that link and then that link up there so there's three other links that they can potentially use in this network go give a chance to do that yeah everybody tries to use this link and the way it works in the network today is that's just too much there's not enough bandwidth on this link to support all three of these users watching avatar - so what's going to happen is whoever got there first in this case user number four wins they get to set up the connection these other users are informed not going to happen you cannot actually use that path to connect to user number five so they're going to have to go back and they're going to have to try again the good news in this network is that there's some other paths they can actually build an alternate route between them and Netflix so we had a problem in a leg failure we ran around a little bit like a chicken with her head cut off but basically we rebuilt all of our connections life is good again right or is it so let's think about this one so when we try to rebuild the connection around fat remember at least three of our users ran into a problem where their initials have to rebuild it failed right then they have to try again so time passed not a lot of time because we're not letting pewter some stuff like that but time did pass anybody here I'm a Netflix subscription at home if I watch Netflix movies online what happens when Netflix can't get the data that you need what do you see on the screen either stops or else you'll see a buffering or something like that so these guys are hanging out watching the buffering signal and how do you think that makes them feel not happy right you know how it is you know just an avatar to when they're well I won't give it away but let's just say it's okay now the other problem is now we rounded on these other paths what do we actually know about those paths we have some numbers which we're going to assume are like link bandwidth capacity okay that's good then apparently there's enough bandwidth for what we want to do are there other ways to describe other characteristics of connection besides bandwidth like latency right I can be an issue and ability to tolerate burstiness sort of another one right and you know during the action scenes in Avatar 2 when there's a lot of stuff going on back of you that could be a huge burst right can these connections handle that I have no idea and nor those users guy so yeah we rebuilt the connections but we really don't know if we did a good job of rebuilding the connections and that is the way things that works so life is okay but my life is by no means and it sure seems like things can be a little bit okay question please suggesting that it comprised of service provided yes I've simplified look at me okay there's a lot of other technologies a lot of other routing protocols a lot of other things that you can use at a very high level this is what occurs there's other sophisticated algorithms kind and there is additional data about the network that can be offered but this is the essence of what okay and the reason I say that is because we're talking about one Software Defined Networking ism so on the left is a Cisco Nexus 55 916 router you can go out today you can buy it from Cisco okay if you're going to buy that in your pocket you better have $27,000 it is a sweet piece of hardware kai and the reason it costs $27,000 is it it has custom hardware on it that hardware grabs packets as they coming off the wire as quickly as it possibly can it figures out where that package should be sent and then it shoves it out the other side of the wrap that's its reason for being now if it knows where it's going that is a very simple thing to do but it also runs a lot of very fancy software guy and that fancy software talks to other Cisco boxes it does management of itself it knows how to react when there's link failures except know it's got security built in cetera etc etc that's why it costs $27,000 okay in the world of Software Defined Networking you don't buy that box instead you buy this one from a company called out type of eight this is their p39 22 box it costs eleven thousand dollars less than half that extremely fancy Cisco box all right this one costs half as much good evening sign in and take two sheets and you're good to go you've got a pile of cookies back here grab some cookies people is the reason that cost half as much because it has virtually no software on it guy this over here so Windows eight bucks with a lot of Microsoft Office installed on this is a empty hard and that's why it cost so little okay so one of the core ideas behind software nepna Software Defined Networking is you pull the software off of the router you have build a network made up of dumb router to now mind you this is a completely 180 degrees away from how Cisco and juniper do things today they provide very sophisticated pieces of equipment high quality hardware with a lot of very complicated software on it and Sdn says no pull all that's opera off and put it on a centralized server in your network we'll talk a little bit more about that every understanding what we're doing here $27,000 for a Cisco router beautiful wonderful works perfectly $11,000 for an SDN router doesn't really do very much by itself we've got so let's talk about how computer networking is done today to serve a snapshot of how you would build a router okay at a very low level a router has specialized having to pack Authority hardware fantastic silica guy arguably you can now get generic something that does let's nose on the back in the past their own you can actually go to Taiwan and get mostly very good job with this let's get silicon your goal when you're building a router is you really if you could put the away that you would not want to involve any software at all because if you involve software you're slowing things down right you would prefer to process your packets 100% and hardware so the packets come in you read the header where are you going and then you shove it out the port that will get it closer to wherever they're trying to go to you've got a lookup table inside this says if I have a packet that comes in with a header that says 810 I should shove it out through port number 15 if I have something comes in with a header that says 512 I should shove it out through port number seven right and you try to do this as often as possible give me a favorite sign in and I also got two handouts there's the grapples and if you can grab those and do that in hardware then you have hit the jackpot because you can do that extremely quickly okay if you have to reference offers and still do it I do that as often because that's gonna slow it down sitting on top of this specialized hardware is an operating system cisco has a product called iOS be confused with the Apple phone but um that is their operating system juniper has an operating system called Juno's this is truly their crown jewels they have invested a great deal of money and energy tempt engineering talent developing these you've probably heard about some of the Cisco certification to get CC is DC as an OS or some of that effectively what they're doing in those training classes is they're teaching about the Cisco operating system ok different aspects of it and stuff like that on top of the operating system you'll discover that routers run applications I mean this is almost like Windows or Macintosh computer right but they're running very specialized applications generally these are routing applications okay BGP OSPF etc etc the list goes on it depends the technology depends the type of network you guys build and some of that but all the fancy stuff is up there all the security applications are out there so they're actually applications running on top of an operating system that runs the top of the specialized target any questions about how you build a router today it's relatively standard stuff right and cisco juniper have made incredible money this is another sort of shot of exactly the same sort of thing we once went out to the data plane once again we're operating the harbour level here we've got the transceivers that are receiving and sending actual patterns themselves we've got the switching a6 so remember this is the hardware logic that says if I see a header that's 810 I shove it out port number 15 okay there's a management plane which sort of matches the control plane to the data plane out of the control plane will have some flash code laughs the memory basically that's if there's any song for it to be done that's where we're going to be store you sort of a three level architecture for building a router today basically you know this is just in a nutshell sir how we built everything that runs on the Internet today so we've got applications like email and phone over the internet and stuff like that that's all built the top of a reliable or unreliable transport layer smth can be eccentric cetera best-effort global package delivery TCP UDP best-ever local package delivery now down to IP level as a local physical transfer of kids which is calm provider or radio - that's effectively how we're getting from one end to another remember it's all been layered on top of each other that's motherhood now by when it comes to understanding how the internet works so if we take that same concept and then we apply it to a router this is what a router looks like in the world of software-defined networking over on the left-hand side that's the data plane we've taken a lot of the things that were spread across the three planes in the traditional picture and we scrunched them all down and stuck in the data plane we have the transceivers we have the switching a6 we have a CPU we set up a little bit flash go to memory but there's not much there there's not much software running on an SDN router a little bit but not much what we've done is we've taken all the smarts and we moved it over here in this Sdn controller think of it as a high powered server go to jail by their fanciest server load it with as much memory as you possibly can and stick it in the middle ear Network and that is where you're going to run your Sdn controller this is an important piece of software because it's going to be controlling your entire network sir what was the rationale behind behind doing this yes there's several different reasons first off it reduces my network cost of your network dramatically okay so I do have to pay for this software what's interesting about the software is right now I can get this open source you're familiar with open source effectively free and available software I can buy this as the best open source now I survive from a company get that our support and stuff like that but now this is my $11,000 router remember my alternative is to buy the $27,000 if I may be how many routers do I have my network thousands maybe a million routers and I can cut my router cost in half or more I mean immediately immediately ever save money but here's where things get even crazier because remember we talked about this little bit at the beginning ATT has to replace their Cisco or juniper routers roughly every 18 months okay because they come out with the newer better faster quicker one right just after you get installed you have to pull it out okay in this case I'm putting Doug router metal there's no software here all the software's here to upgrade my network all after two get a new piece of software running here and all of a sudden whatever new functionality I want to introduce to my network is there and is available to all of my dumb routers so that you know year and a half to two year replacement cycle gone I don't have to do that and so there's a few other things but those that's the essence of it so esteana there's a completely different way to build a network but the financial drivers fort are huge the technology is pretty cool and technology I think hopefully we can all last we get through this presentation we can all go yeah yeah I can sort of see some advantages to that so you know what's interesting when it comes to making a decision as to whether or not company goes with it the chief financial officer probably gonna make the real decision all right hi we can spend you know the two billion dollars we spend every year on equipment and by the way next year spend two two and a half billion dollars or we can spend maybe one and a half billion dollars this year and spend maybe half a billion dollars next year well I'm going I'm saving so much money so a little while ago in the world of IT through outsourcing under right where jobs are taken from local jobs taken and loot like a foreign country where other people do it for like less money and stuff like that your whether or not you agree with it or not was an interesting question but didn't really matter because it made so much financial sense pretty much every company did it right you just didn't stop that were the same thing here this makes so much financial sense to implement a software-defined networking in network I believe you're going to see everybody doing ok and you're going to see a lot of vendor showing up so let's talk abou bit runs through this yeah let's take a look at another network so this is a very versatile now it's five notes those nodes are interconnected they're certainly not every node is not connected to every other node but if you have to get from one end of the network to the other network you could do it there is a path so we're going to add a little bit more information here okay so you can see that there's some routing outward that's running that are helping us do it what's interesting about the routing algorithm that's running in this particular network is what's called a distributed algorithm okay so this nodes going to run a little bit of it and it's looking to run a little bit less and so on and really nobody has a complete view of the network right everybody has a view of their portion of the network and they'll all communicate with each other trying to share information to help the other guys do a better job right so it's a computus Pacifica SPF which is shortest path first is a very very common algorithm that you'll find running in stuff like that you know this is the sort of a typical view over network today okay Sdn changes everything so SD inlays of layer of software on top of the network sort of a network operation and the network operating system will communicate with every single dumb router in that network guy what it's going to do is it's going to use that information to build a global network view so the operating system is going to have a map of the network the operating system is going to understand how each router in the network is in the interconnect okay once the network operating system has that information you can build control programs another control programs of the fancy sophisticated routing programs that are the security programs the management programs and instead of actually interacting with the routers in the network what they're going to interact with is the model the view the network that has been built by the network operating system so this is the exciting part about this the end what it means is you can write a program to program the network your program is going to modify the model once you modify the model that in the network operating system is actually going to reach out to the routers of the network and change their configuration so this is something we've never been able to do before it's the concept of network programming you have this model the network and your program modifies the model and then there's software that actually relates that modification back into the network that makes it happen and we do really understand what this means no we get excited about the idea wow that's really that's really cool do really know what it means ah not quite yet okay but we need is a really smart Russian missile scientist to show up and do something that we've never been able to do before and that were in that era and that will be half surf question a little poor being integrated with is helpful tour right which is the anonymous it would actually if not necessarily in the routing algorithm per se of the network tour works on enemies right or else it's a surfer application so is operating in a much higher level we're just talking about how you build your network here tor works on the ED endpoints of the network to analyze both users with receivers and some of that so it will be outside of it so really it won't have a role to play guys but eventually we have to it once again it's an application towards an application so there's no problem it's just it's not part of how you design that's still rather provided transport power David yeah so once you have it but the network will treat or just like anything else they'll treat it like a Netflix they'll treat it like Network backup it'll treat it like any other application running on non-special right well orchestration of the network based on the separation lovely data plane the control plane then you would prioritize it you decide anyone Network inspiration in the control plane of your control towards proton whether an application facilities now that's a very good point okay salute what do you say is that you can prioritize tour traffic over other things a hundred percent agreement on that one where that will become interesting is when you have a heavily loaded Network you have limited bandwidth you have to make decisions about who you give bandwidth to if you did that then as a tour user congratulations your stuffs still going through and maybe somebody else's isn't right so no question about that you are correct on that is significant you know I I don't believe tour applications are generally a bandwidth and test intensive applications I think there once again but yes you certainly could all right so here's another view okay so just another Sdn example user a does not want any of his packets to be routed through user B user B is Libyan I don't want any of my packets going through Libya just give you an example well in today's that work your God have won a pain in the butt right you're going to have to tell each one of these nodes don't let these packets go through me everyone's going to have to have that rule programmed into so when they receive information from a don't run it towards me in the uncertain chemical complex gifts can you make mistakes or we had nobody know it's all ice or stuff like that and things get complicated free quick the reason the SDN is so cool is because in the world of SDN the network sort of looks like this now you really have to do is worry about the end points so one's going to remember we're back up at that control plane right where we're doing the network programming stuff what we do is we say hey a does one of the packets are going to be pretty simple right we're going to worry about the network operating system telling all the routers what they actually have to do but at our control plane level we view the network as an abstract it's really pretty simple stuff package from there don't go there now and the story not terribly complicated to do right now hopefully you're also understanding there's some sophisticated software here right but it's running in the background from your point of view as a network programmer person or somebody who's developing a network application this is how you use the network it is that simple okay and this is a dramatic change from the way things are done today in a system network today you have to worry about every single node you have to figure out how everything is going there's a lot more that you have to take into account really can't write applications that deal with as a whole you can write an application that deals with box by box yes but nothing network as a whole now we're allowing you to deal with the network as a whole got it all right so now we're back to sort of the massive view of what's going on what I haven't mentioned is that there's another layer there's a virtualization layer so if I reach out like connect to each one of the routers in the network can build this sort of network view one of the challenges with that is that me our view is going to be relatively sophisticated I'm going to have a lot of information about each one of these routers which is great from a technical point of view but it's probably too much to present to the upper layers so that control plane doesn't need to know as much about the routers as the network os does so I'm going to put a virtualization layer in between and what the virtualization layer is going to do is it's going to sort of wrong way to say it but you'll get the gist of it dumb it down it's going to hide a lot of the complexity of that network it's going to present a much more simplistic or virtual view of the network to the control plane remember in the control plane that's where the really smart software developers are writing their programs that are going to manipulate the model of the network okay and what we want to do is we want to make that model as simple as possible for those guys we want to provide them enough information certainly so they can do their job and they can work whatever fantastic magic go to work but we don't want to burden them with too much information so we'll hide that information in the network os will use the virtualization layer to map between sort of the abstract view and the super detailed view so in the control program applications make a change to their virtual network it gets goes to the virtualization layer it goes to the network OS and then it actually goes out and we actually change the routers got it so it's software software software gettin this there's a lot of software here there's no question about that software-defined network and when the very first American word the title of software right but here's the thing you get this down you get this in place you build your network using this and it's done right you get to spend all of your time up at the control program level where you're dealing with this abstracted view of the entire network if you can see if your control program whatever it is can see the entire network and if it's not being crushed with too many details you can do some amazing things all right so this is a politics because this charts a little bit busy but let's sort of step through it or just reach over here there is a this is the cost of network equipment what you should see is that the cost of net load and I'm sorry right here this is cost the network you have it right here what you should see is that the cost of equipment is going up every year cisco gear and juniper gear it gets better quicker better faster more sophisticated but guess what they do with the price they raise it up right now here's how much the phone company's making from you and me so they're making more money every year for you making sorry your phone goes up your internet bill goes up your cable bill they all go up right but they can't raise it too much otherwise what are you gonna do right straight out through the blue Netflix right yeah Neil Neil Providence to make it to Atlanta the problem is is that the difference between this line and this line is actually getting smaller so in other words the cost to the network is actually rising at a greater rate than the telephone company is drawing money out of your my wallet and that makes me sad they go listen we can't them can't keep doing this we've got to change something which is why ATT Verizon Sprint are so interested in this the end see if I want to put s the end into the network this is how much it's gonna cost me notice that that's a lot less how much I traditionally pay for all that cisco juniper here remember 11,000 versus 27,000 okay now here's the one that really opens up my eyes this is potentially how much money I could make if I put Sdn into the network now it's a little bit of a of a made-up thing but the concept is via the software-defined network and I have that control plane and I can program my network man I can do an awful lot with software my network today I really can't I can do some management to take connections down well it arguably that sort of gift for what I can do if I have a network which I have this model and I can program the model and change things at will network why Wow you know the sky's the limit as far as the fancy sophisticated applications that I can come up with you know one example of course would be I gave you one phone number and wherever you have to be located I'm going to make sure that your phone rings if you're out of the house make sure your cell phone rings if you're at home make sure that your home phone rings got your laptop make sure that your laptop rings okay not there I'll grab the message I'll turn it into a text message I'll send it to you I'll turn it into an audio message and I'll send it to your calling on yourself up make sure you get that as you as an audiophile at one I can be all sorts of very sophisticated things like that I can only do that by program if I can figure out where you are yeah with my network is and somehow those things to work am I getting this expensive to own an operator Network it's getting worse if I don't do anything Sdn offers apparently a low-cost way to completely rebuild my network and there's a potential that I can make a truckload of money with Sdn applications so work if there is an entertainer if it's custom what size that work got and so that's a breakup so the question is yeah drunked the SDN kool-aid and I've decided great that's what I want to do my problem is I already have a network right how do I make that switch from today's old network to tomorrow's shiny new net market what does it make sense what size of the jacket you're exactly right now the simple answer to that is if I'm building a new portion of my network I do it at the end right that's a simple one problem where it gets a little bit more sophisticated what if I have an old network and I want to do a key do I stick with it just say yeah that's the way it is we're going by the bullet go ahead and convert it I think every company probably has a different answer for that but Google is going to provide us at least a partial answer so let's take a look at this so the reason that we're going to talk about Google is because well what do we know about Google search engine they make the money off of advertising so does Google have a little bit of money or a lot of money they've got a lot right now the interesting thing about Google is Google is very aware of what it takes to run their business Google has two plans or wide area networks that they use to run their business one is called the I think it's got the Iowa which is for Internet and that's the network that they own and they maintain that you and I use room access Gmail access Picasso we access Google Hangouts whenever we are Google search for that matter to write whenever we access a Google surface service it served off of there I glam they have another one called the gene when I actually I think on G scale G scale it's like the Google one that is an internal network that they use and they use that network to interconnect their data centers Google because they've been around for a while and because they are so large as data centers sprinkled all over the globe and they have a need for those data centers to exchange from a as part of our searches is part of their network backups it's part of the story involved the photographs and all the other sorts of things that we store in Google and some of that they're constantly moving information between those data centers in April of 2012 the vice-president for Google engineering went to a conference and he got up on the podium and he said hey about this Sdn stuff that everybody has been talking about I just like to let you know that Google has gone ahead at that point I'm even right so Sdn was an academic concept people were talking about it at universities they were debating you know some people's allow this grace things inside spread some people said that's never going to work with so many drawbacks to it you would never coexist in a world when we do networks today and stuff like that and no people are around maybe doing a pilot project in the university on a test network and he showed up he said guess what the network that we use to interconnect all of the Google data centers worldwide is now running on an SDN network we built it ourselves what made this even more impressive so this is roughly two years ago at this particular point in time there were no Sdn vendors you can't go by that's the enroute er then right so Google had to build had the spec and build their own routers and Google had to build write their own software so it's even more amazing they didn't but because they did it it answered a lot of the questions well I don't know if I can do this but yeah or you can I don't know if makeup they can do it yeah you can and to answer your question hey could I convert an existing network into an SDN Network yes you can't easy no absolutely not a tough dangerous dangerous thing to do right notice that Google did their network that interconnects all their data centers that's the less important Network the more important network is the one that connects all the Gmail allies sort of stuff going on right so them anything to manage their risk but let's talk about what Google did and how they did it so the reason that Google was willing to take this risk and actually go ahead and make these changes because they took a look and how they were spending money admittedly they've got a lot of money but hey even they are careful about how they spend money they took a look at the look we can basically break the cost of our network of our wide area network up into three groups Hardware of over-provisioning and basically human labor so hard work was routers transport gear and fiber that's where they were spending their money over provisioning because remember you have to you know if I need ten megabits I'm probably gonna have to provision 30 40 or even 50 megabits just in case something goes wrong yeah right shortest half browning slow convergence time need maintain service level agreements or SLA s new mentoring company failures you have the network and you know by the way they treated all their traffic the same and when your inter connecting data centers that's a huge deal because they don't connect engage that you're getting a lot of backups and backups are important but they're not nearly as important is like user traffic or something that you're doing in real time you can go back up you know at 3 a.m. and away from your parents right you can be doing a back up and allowing something or inform comes wrong to stop the backup take care of the more important and then start the back up again but in the Google Network that existed they all traffic is the same so they really couldn't tell the difference so and then I'm down here Google had a very sophisticated way of managing their servers remember we think nobody knows we think people easily has over a million servers to do all their searches oh I sort of stuff at that we think they probably have over a million servers in our network nobody knows for certain but Google has put in such sophisticated tools that they're a that adding more servers doesn't mean anything right you just drop them in it becomes part of the management they go on they manage their servers almost like a fabric is the way they've described right so if we had more services you just added to the fabric and it just becomes part of the collective consciousness the problem was their network was not that way they were managing their network out of box by box basis and it was killing them because their network was growing become a bigger mean obviously at your expense all right I seem to remember reading an article about Google about the same time span 12 that they were moving all their servers to ten gig interfaces and part of the reason I add versus mattering of some about them doing a new network design because of the costs of trying to hook those into and retrofitting all the routers up to ten cakes is so higher it cost the server tanking that's coming down and put 10 cake mixes still being very very high per port I didn't hear they were reaching out to somewhere Taiwan build their own router Bobby this time friend they're doing and you're exactly right they did build their own route and you know you bring up the next point there was probably multiple reasons for the making the changes is never just one thing but your exact correct they did end up going to time on they did end up spec'ing out exactly looking around or they won that and that's we also build it for the which of course if you had these Cisco salesmen not going to go to the store you're wondering why they're not returning your calls and all right all right so this is a situation that gave Google nightmares and again Google of nightmares for two reasons that we sort of skipped over last time but the first one is when this failure occurs you rebuild your connections it's going to take time right and remember you will have conflicts and got to redo a reset and stuff like that Google could not tell you how long it was going to take to recover from a link failure right there might be able to tell you the maximum amount of time but they couldn't tell you on a per incident basis how long it was going to take to rebuild those okay and then the other thing they couldn't tell you is that if you have a link Taylor I can't tell you what my networks going to look like after I rebuild them you know will the red link go down here I'll go there we'll go over there we'll the purple leaptv down here don't know remember it's whoever gets here first gets this path and depending on a whole bunch of very interesting things that are going on the network it could be anybody right so they had an on here security or 50 cent word for the evening they had a non-deterministic system so every type of link failure you never configuration changes but you don't know how it changes and also you don't know if it's an optimal solution by the trench and they hated that okay when you're running a very sophisticated network that is the core of your business you can't determine how long it's gonna take to fix it and you can't determine what the network's going to look like after you fix it you've got some major problems everyone else lived with it Google did something about it what they do is they basically implement Sdn and they implemented a centralized traffic engineering function which means that yellow box is now the smarts of the networks that yellow box reaches out to everybody when you have a failure the yellow box determines how the network should look after the failure remember the yellow box can see the entire network and end so it makes really good decisions it has all of the available information and then what it does is when the figures on the network or it goes handed updates each one of those routers with the way it wants that router to behave once it does that then the network works perfectly and all the right connections going so if those green each go down there ready to go there and purple see on top that is the optimal solution for a network when you've had a link failure you know that because the yellow box has all the information so it's not making its decision based on localized information it's making its decision on the entire network make sense that's a hell of a yellow box by the way right it's a very big fast server and it's a very sophisticated software okay it can be and in fact you'd probably want to write you wouldn't want to have a single yellow box in your network you probably want to have a couple of yellow boxes things get even weirder right because as my Network gets really really really busy I tried one I have a whole bunch yellow boxes but as the network calms down in the middle of the night I don't eat so many yellow boxes right so there's the concept of scaling up your yellow boxes and then also scaling it back down was just combining any other company that you happen to absorb to do a selective cluster our super cluster it is so much easier than what they're trying to keep try to implement two different it's a similar networks now so my marriage exactly right so if you have one belt with juniper or one built with Cisco and then those two companies come together yeah yeah good luck yeah you can have two separate management consoles this is what Google's backbone Network looks like interconnecting all their data centers basically they're all over the world just a short answer on that even down in South America and stuff like that so this was their internal backbone network this is the network that they decide to convert from a traditional smart router network to an SDN Network this is sort of a quick snapshot of how it's set up you got a bunch of routers connected into the data center you've got a lot of transport gear connects over a LAN to some more transport gear dumps it off into a router that delivers it to the other data center very traditional networks or the design this is a real quickshot of this is going back to the question of the gentleman in the back of the room answered how did they do it they had a fine network it was working perfectly now they decided to go turn it into an SDN Network how did they do that so they have their whole data center network here on the edge there's some cluster border routers which are designed to talk to other data centers it talks to basically the interconnection devices and that all goes up to the transport here that's just a really high-level view how at Google Data Center was set up what they did was they implemented this fancy little server here that was running six different pieces of software references stuff I don't know the names that Quogue is actually it's a suite of network routing algorithms is you got open shortest path first routing information protocol border gateway protocol and is is per unit system so basically all those sophisticated routing algorithms that you can get in a cisco box basically available in open source software on quad ok ofc is the open flow controller this is a piece of software that is responsible for talking to the Sdn routers you know those dumb routers you know I've told you over and over again it has very little software in it it does have very little software but one piece of software does have is something that supports with called the OpenFlow protocol open flow is a very simple relatively speaking protocol that how it allows a router to figure out how to route packets now it doesn't really do very much and basically their controller tells it everything it needs to do the way the controller talks to the remote SDM router is using the open flow controller analyses is glue allows an open flow controller to talk with the mod allocations of router glue blog that paxos is used to basically figure out who's in charge when you have multiple devices doing the same thing who's on top and paxos is just a particular piece of software that allows you to about different boxes that are all doing the same thing figure out who's in charge which turns out to be a really important question to answer all right do they virtualize it ok good question I don't actually have any information on that so let's think about this from home why would you virtualize you virtualize it if you want to be able to move it from box to box you virtualized it if you want to run other things on the box and neither one of those are germane to this situation I want to dedicate that entire box to doing software-defined networking stuff I want to free up all of its bandwidth just for that I'm not anticipating moving that functionality to another box right so they could have theirs there's absolutely problems to that whatsoever but there was probably no need to do that and actually the additional overhead that would take the support of these cycles it would take to support a virtual machine may have actually taken away some of the processing that would have normally been available to this so I can't actually say it but I don't think there's actually a motivation to do that does that make sense yes that is correct in the Davis areas that Lots good question it's a good question so inside the data center they have almost like a land a local area network right what we're really talking about here is though is the land the wide area network so the local area network will just talk inside the data center but there'll be a few boxes that talk to the land not all right and so we're really going to talk about the land portion so this is almost the standalone situation so you might guy would suspect you're probably lies every single server in that data center but this is almost a separate beach but you break with the point if you virtualized everything else maybe it just going to perch like this because you're just used to it second question the BGP going to be open all the other protocols or a lead-acid filter yes so therefore sophomore yeah so the question is so what's going on in violence equality is a whole bunch of network routing protocols and they are open you know they're an open standard okay you can pick up a book in any store you can order it online the specification for the protocol okay so somebody some group of folks implement it and this is open source this is freely available you go out today download it right and what you get is a suite of network routing protocols and I all stuck together incredibly bad great stuff thank you very much whoever did that it's very kind of you to do it but a whole bunch of them stuck in there and then once you install that you can use anyone those protocols with your network age so choose that make sense world of open source software it is fantastic all right this makes it have late sound sorry you have that was separate yeah yes your point but sir please so I want to make sure I answer the question the network way the tide that ties in controlling fire so in this particular case you see that there's a cluster board or router which is a very special device in the data center if you wish to communicate on the lam to like another data center send your information to the cluster or ground okay and then it'll shove it out the back door to another data center when data comes in from the other data center you will come into the cluster border router which will then pass it into the data centers land network and they'll go to wherever it's got a goat in the SN so stay with me because we're going to go ahead let's go in a little bit collect your question guide let's just see if we touch base again take care of our guests place one more question with the data centers you connect the wins in multiple data center and applications that are coming from the data center power how do you prioritize the application as we go across the let's come good question let's I have a partial answer for that so we're going to come back to that one okay and you tell me whether or not I answered so it's the longest living here so this is what the good folks engineers at Google did in this particular situation that took half of the existing routers out and they replaced them with Sdn routers so you have a high-bred environment here you've got your Cisco routers and then you've got your Sdn wrappers okay and basically they would throw traffic on one or the other but once it was there it would travel to the other data center on Sdn or would travel to the other data center on Cisco and the two would not intermix okay so it was a 50/50 solution and then they get their fingers crossed that they hadn't done anything wrong and it basically worked just fine after they got that taken care they replaced the rest of the routers right so the gentleman in the back of the room had asked the question how did they actually do this this is the approach and remember given that these might do it differently Google is a half and a half approach so they went in in the middle of the night and replaced half of their existing system wrappers with these Sdn ones they fired it up and made sort of a separate in two different pathways between data centers one on Sdn and one on Cisco kept their fingers crossed everything seemed to work out pretty good they went back in another night they replaced all the remaining Cisco routers with Sdn routers and turned it up and then at this point on the interconnection between all the data centers was completely completely based on Sdn then a group by the way congratulations a ton of work to get to the stage and what have you done the answer is nothing this network operated exactly like the Cisco Network okay you achieved absolutely nothing but good news you also can screw anything up right what they did then was then they introduced a traffic engineering control program and that's when the sophistication of Sdn gets working right that a way to rebuild links and ability to manage bandwidth the ability to a bunch of sophisticated things this is where they got the real payoff for the time energy and effort that it with it this just shows oh by the way in a typical data center you're going to have let's say 10 applications those 10 applications we don't want to talk to each other each one of them is going to have a different amount of bandwidth that it would like to establish for that guess what sum it all up and that's more bandwidth than you have available storable life right applications want to use too much panel not Miguel so this is the big picture for the Google Data Center so I'm hoping to answer your question the way it works is look these are all your traffic sources these are your applications right what happens is they talk to what's called the bandwidth request collection enforcement module this is how much family I would like to exchange with another data center it's as fantastic I appreciate that then it talks to a bandwidth broker bandwidth brokers like a cop so how much do you want what kind of application are you on the network backup application fantastic you'll get that bandwidth a couple hours you won't necessarily get it right now or I'm an hour backup application ok great I've been watching you and you said you were going to use five mega bandwidth but you've been surging to like seven I'm going to cut you off and give you two for a while right that's what the bandwidth broker does traffic engineering server that takes everything that it's told to do passes it over to the Gateway to the software-defined networks and then go ahead and configures the network to support whatever these guys want you to do so that's how Google configures their network that's how they manage the bandwidth all the different applications are using that's how they move from an existing network your brand code Sdn network any questions sir today after reboot oh well when they install them they were off to the empower and turnabout here's the interesting thing they don't have to change the routers and three select and drivers routers effectively have no software on there they have to change that big yellow box in the middle of the network yes because they'll upgrade that software I'm probably fairly frequently right the line bugs a lot new features that and actually Google's talked about that woman they've got some very sophisticated software application management techniques that they use to manage software in their network so I think they keep a hot version running it stalled cold version bringing up this whip over to Cole person we're going to take out purtytown they're very sophisticated because they use a fruity mail they use it for all the other service left-right open flow I need the spoon along a little bit here you see the controller which is in this case I apologize it through the orange box but it's really the big yellow button controllers that sophisticated piece of software the city at the heart of your network it sees the network it knows the network it's going to tell every single router order to do this is the router router has an open flow channel which is just simply designed so a good cognitive controller is group tables and flow tables these are the ways it picks through a packet that's coming in try to see if it's seen that packet before if it's seen the back and it knows where it's going to go if it's not seen the packet then it needs to make some decisions this is the fancy way I talked about how it makes the decision tables are used I have a packet comes in its header is eight hundred to 500 CE table want to go to I see a table in yes or no if it's no potentially I'm going to throw it away if it's yes I'm going to say hey what do I do when I receive packets with the header of 805 maybe it says make 2 copies and send it to two locations maybe says send it to this particular location it can be anything I can program the switch to do whatever it's with that this is just the sort of a decision matrix packet comes in do I have I already seen it before if not and I'm I all not do so drop the packet if I have seen it then I gotta update some counters because here's a trick there's almost an infinite number of packet header numbers one two three eight hundred five six thousand for a cadet packet headers from beginning right if I have a packet header that I saw an hour ago and I the other ones from there I never find out amount of space the same packet headers to remember 500 packet number 506 it's an outboard number for I haven't seen anything for 506 for a while and the counter hasn't been updated for a while it's time to clear it out I'm just noting getting traffic from there because you have to manage your limited amount of space okay go to table nsq action set so I can pick up a set of actions whenever you see that packet you should do the following four things that's an action set okay so this in a nutshell is what an SDM router does but there's not a lot of tables not a lot of decisions to make is really pretty simple stuff this is talking about controls remember the big yellow box is a controller there's three types of controllers you can have an SDM network first one is an equal controller which is sort of the bossman osmanthus control that's mine you can have a couple of equal controllers controlling the exact same Sdn wrap okay and they have to balance out with the good you can have something called a slave controller which is sort of a read-only controller you can see what's going in the router but it really can't tell the router what to do and your third choice is you have a master control masters in charge slightly publishes reading in the controller can make some changes but not as many changes as the master control master always wins so remember we talked about how the importance of a controller I mean it's really at the heart of the network it turns out that you can have multiple controllers and you can have different types of control okay the connection from a controller to router main connection very important with no question about that there's a lot of stuff going on in the network one connection isn't going to do it so you can establish a whole bunch of connections between your controller and your out but if you lose that main connection to whatever reason all the other ones go away just like that okay once again this is a bandwidth thing if I really need to have talks with that router I can set up a bunch of connections extensible future all right let's get through this one real quick this one bit so a lot of researchers deal with how do we move from where we are today with traditional networks took this grand glorious Sdn future one of the things that they come up with is what's called a dual stack and you know Google went through this for a little while right when they did their 50% replacement so part of my network is at the end and part of my network is legacy right they exist side-by-side package trouble here or they travel there and they don't really get interchanged right I could live with that basically when I build new stuff my network and you may get Sdn but I'm not going to rip out the old stuff the other one is called access edge basically what I do is down the edge of my network I go ahead and I put in Sdn and then I treat the center or the core the network built with legacy equipment and sort of a blob right so the the Sdn networks just use it as a one big honkin router it can be don't care Sdn view set as a single router and then it does all its sophisticated stuff at the edge of the network as one approaches Union and then there's this other operation called pen pen and open easy for you to say easy for you to say just on this one is they've done some interesting things that I've discovered that in a connection that has two end points if one of those end points is an SD endpoint and the other one is a legacy one you still get the benefits of an SDM network so in this particular case you can create a hybrid network that consists of both Cisco beer and also Sdn routers mix it all together and you still get the benefit of the network in this particular case we're talking about which we on an individual data set of networks what's interesting about data center networks remember we connect interconnect data center networks using actually really big pipe because there's potentially a great deal of data that's be exchanged between it turns out we do a lousy job of using those pipes efficiency on connections between data centers is somewhere between sixty percent which means we're not using sixty to forty percent of that connection that we are paying right so one of the really cool things about software-defined networking is it allows you to very carefully manage your bandwidth usage and Google has some amazing graphs that they show to show the bandwidth utilization between their data centers at just about 100 percent right and you know that once chief financial officers just zombies yelling if it's been a good because if he's fully utilizing it that means that he can basically scope it down to just what he needs he doesn't have to pay for extra in his family okay this is simply talking about reactive proactive if the router has already seen a particular packet before it knows what to do with it get an 805 packet it goes out port number five if it hasn't seen it before then it comes in goes into five I've never seen it before reaches out to the controller says hey I got an 805 packet what that was supposed to do with it the controller sees the entire network it knows exactly where at 8:05 packets supposed to go and it knows exactly where this guy is located in the network so he'll tell you what port that you go out so that would be the reactive part this is what so what do I do if I have a fault dentist and network by the fault I know I've had a fault I'm going to talk to the controller the controller sees the entire network the controller knows exactly what that route that I should build around the fault is it tells me I have to date my tables and I go on from there this is I think the last line I have my I think just bounced so this goes back to that question and this is a very hot topic this is a very research laptop because nobody has the answer to it we anticipate the misty in network they get busy right maybe it's during the middle of the day the night so you have an abacus coming along nicely then all of a sudden the load of the network starts to increase will you let that controller software is so very important you had better what scale it up as the network starts become important so you think you plan to make multiple copies of it right great so when do you grow it how do you hand off which routers that controllers handling those are sophisticated questions so you swell up and you grow you know spawn multiple copies of that control yeah you're handling your network everything is fine then all that traffic starts to go away well you've got all of these controllers you don't need them so now you have to start to shrink the number of controllers that you're using a network and as you shrink the Patrol you're going to take all the routers that it's responsible for and distribute it over the remaining controllers I can all be done but it is this ticket it's all right and we don't actually have an agreed-upon way between users yet so everybody's smart enough to acknowledge that this is something that has to be done oh we're not smart enough yet so you know there's a lot of open source software out there yeah that's great but when you start to go to sophisticated solutions to tricky problems like this this might be where you start to see some proprietary software but we're trying area additions to the open zone all right last slide in the deck so great SDM exists Google is given its stamp of approval we know that ATT is out shopping press T and network and network gear so clearly the time pressed yet if it hasn't arrived is almost here it is truly coming wouldn't have been great if somebody could have come to you in to hang out the internet thing is going to be really popular you don't love my dad about it or by AOL before it gets really expensive and those would be important so I am Telling You Sdn is coming I'm telling you Sdn is going to be a big deal there are three issues in the world of Sdn that are going to be occupying companies and researchers going for first off is interoperability we've got networks thank you very much now you've got the shiny new Sdn technology great how do I work the shiny new stuff in the old stuff that I already have great question no we don't have a complete answer right now next one scalability you know we're talking about a bunch of dump routers we're talking about this very important central piece of software how do I take this from a reasonable size network to a really big size network such as a TS network how the hell are they going to do that don't know nobody's really done it I mean Google has done a good job but remember Google did the same thing right they did their internal intern datacenter Network that's not the big scary No so this is a huge question that we don't have answer and then reliability you know when you have that centralized piece of software is it better not fail but of course we know it will come right this everything goes what are you going to do in advance what's your backup how do you switch over to something when they when it comes back up how do you switch back to it great questions we understand those important questions we do not have answers to those yet so if you're interested in anything like this now is the time to spend some time studying now is the time spent talking about these because you attended the seminar you now know more about Sdn the 98% maybe 99% of humanity so you use that information for good got a question up your circle is cisco responding okay so here's an interesting I work for Cisco so much good good array for application same infrastructure so all that he said you could look at it as the middle school separation of the data control plane fine but like everything else and I'll give you the best example to use there's all these protocols out there open standards OSPF PGP at a grip and a whole bunch since cause the protocol called vit RP and on every box that sold you can use whatever protocol you use 80% of people use AI GRP you made a point earlier that a lot of times open standards or certain protocol have the basic things you need but the things you really need usually when they are earlier on they are one provider that give you that nature so you end up using things that are not always so called over maybe on everything so when you look at Software Defined Networking where it's maturing the first issue is up extra duction of course but how do you train your people how do you do a forklift upgrade how much will it cost how do you use your existing infrastructure so it becomes a migration process and our approaches applications which included control that may have OpenFlow on it and multiple other controller protocols so you can do all the things he's talking about but the complete is how do you manage the applications though within your network because that's usually the most important piece when you look at all the proliferation of mobile devices within your network how do you manage that how do you deal with security which we haven't talked about yet so how do you address all these things scaling the data center multiple hypervisor acquisitions when you buy a company that might be using Microsoft hyper-v and you have VMware using something else how do you put them together so we're looking at it from a much higher level than just the basic network as a matter of fact when they did the first test at Stanford big forwards to do that so since this Co was mentioned so much we have our approach what we will ask for is to get some time to give our version that's your what's next so in the in the forum hardware offenders and there are well every winner by now has Nessie and story sisqó certainly does have there's an appraiser once again a Cisco employee is probably very familiar with theirs from a third person point of view looking at what the industry is saying Cisco's approach is time to harden guys the way they're implementing the rest yet does have relation to the hardware this is different from some other vendors or more Hardware agnostic an example of that is VMware folks who do the virtualization they have come on very strong in the Sdn space they have a very interesting solution which is quite obviously all software based ok juniper is also out there do refers a little confused right now they have stated what they're doing they've become a an active part of some open source efforts and some like it just switched out the CEO like that but every single company let's say has their own approach the SDM right and the market is not decided because people have made the purchase is horrible in the business of SaaS in the world of s10 not that I'm aware they don't think so because they'll be more of an application okay so they won't actually be providing network cisco juniper HP IBM those are all the names but now they came with a virtual machine open-source so I was guessing rolia they're getting into the SDM holes every day would in the portal of virtualization and once again the beauty of Sdn is that you basically virtualizing than that guy so you're stepping away from the hardware you're creating a software model of it you can manipulate that software model and as you manipulate that model then it gets related back to the physical network and the changes occur right which is an incredibly sophisticated way to do it and it is a great right question yesterday and that is being able big pipes they're sitting there not being and the networks of carriers can't get more of additional revenues space happen of the way star providing services to many other faces right horizon has actually been pretty open about some of the initial steps they've taken of STF one of the places this feels the right line is what you're saying about the importance of bandwidth and ability to manage it fries them to you seeing Sdn to determine you know they take a look at see what your or your end devices that you're watching a video on so laptop because the tablet is a cell phone and based on the type of device that you're using they using their Sdn network to set up a connection to you that will be appropriate for that device right gives them a chance to scale back the amount of bandwidth that they're using to support you but you're happy because you're watching on yourself you watching hockey and your cell phone right and it's good enough for you you're watching on your laptop it might in Turkey and dry ears are done but you're happy they're happening cuz I'm not using nearly as much bandwidth is it with other ones right and as Tim gives them the sophistication that they can the type of famine that yes now middlee you gotta get there and that's the challenge right all right I probably used up my time with you nice tonight are there any questions before we wrap things up tonight going once because this is cutting edge all right there certainly is something that actually and I'll tell you I should tell you exactly what's next is up at that control flight okay so this is what we talked about right now with nuts and bolts right I mean this side put it network together this is how to get routers to talk to each other this is this is plain vanilla stuff and the exciting stuff going forward is going to happen up that control plane where you've got some software folks going wait you're going to present to me a complete network a complete description of the network and I get through software you manipulate that any way I want one of the cool examples I saw that was if I'm going to take a server down to do maintenance on it I can through software say don't write anything to that server and it all just goes away magically then I just unplug it stick a new one in that go back and say start writing stuff to that server and then magically it just starts happening today I have to visit each router that was interconnected oh yeah no stylish is a cool example right can do time and a place and all sorts of something once again don't take it to each other folks thank you so much for coming out you have an evaluation form the evaluation form only has two questions on it I need you to complete the two questions leave it on the table by the door before you leave what it does is it allows me to make our next seminar that much better so if you'll take just a brief moment to do that what's our cookie supply looking guys that like to Congrats that you've done pretty good you're driving home all sorts of stuff could happen on the way home flat tires your battery and go bad you can get sleepy also - thanks Gavin take at least two cookies some food and you can potentially take four or more put five is your mouth but let's make sure you leave with cookies and I will be happy thank you so much for coming out guys a greatly appreciated here have a great evening safe travels guys you might be asking yourself where do I go from here the ability to effectively manage an IT department and to do it well is a skill that your company needs you to have it sure makes sense to invest the time and energy that it takes to become a great CIO a simple way to start doing this is to sign up for the free the accidental successful CIO newsletter this monthly publication is sent to your inbox and is overflowing with tips and techniques on how you can spend your time making the right IT decisions and really leading your departments IT professionals will cover how to evaluate new technologies how to foster relationships with other parts of the business how to hire the right staff how to work with the CEO so he understands just how critical IT is to his and the company's success and we might even cover a few CIO career tips or - hey the newsletter is free so you can't go wrong there in the description of this video you'll find a link that you can click on go ahead and click on it and you'll be taken to a web page that will let you sign up for the newsletter there's even a free gift in it for you when you do thanks for watching this video sign up for the newsletter and congratulations on starting your journey to accidentally becoming a successful

Channel: captain1morgan

Views: 95,476

Keywords: Software-defined Networking, SDN, CIsco, Juniper, Google, networking, router, switch, BGP, IS-IS, RIP, OpenFlow, tutorial, seminar, G-Scale, Gmail, network, network design, network architecture

Id: H_3Lk6XbWw0

Length: 82min 18sec (4938 seconds)

Published: Thu May 22 2014