Operating your own BGP Autonomous System on the Internet

[Applause] so this year I'm doing a talk on BGP again obviously I've entitled it operating your own BGP autonomous system on the Internet ake a BGP for fun and profit maybe in brackets on a budget I maybe I should have put more ghosts so Who am I so I'm currently a network architect at Damon defense systems and zero I'm an avid open source user kind of fanatic and also recently kind of a contributor as well I'm addicted to BGP even my house runs BGP I'm super obsessed with network monitoring and routing and I often write about that on my website bgp guru i'm also involved in several not-for-profits in winnipeg I'm on the board of the Manitoba internet exchange as well as the operations team and I'm also on the board of cold AK cold ak is a not-for-profit dedicated to furthering privacy security and freedom of speech and we run a handful of Tor relays and exits and that's actually been most of the reason for me to be to be doing BGP actually I was also recently selected as a CBC Manitoba 2017 future 40 finalists as of like this Thursday so what is BGP so BG b stands for border gateway protocol we're currently at version 4 you'll sometimes see that referred to as BGP for wikipedia defines it as a standardized exterior gateway protocol designed to exchange routing and reach ability information among autonomous systems on the internet it's open and extensible which is part of the reason why it's been so successful and there's a large list of rfcs that add features and modify B GPS behavior and this year I was actually recognized as a contributor nada not an author but a contributor to an RFC which was something that I was kind of I wanted to become involved in that and as a result of BGP being open and extensible there's other address families so layer 2 VPN a VPN and flowspec which is used for a DDoS mitigation so the other part of BGP is your autonomous system or autonomous system number and wikipedia defines that an AAS is a collection of connected Internet Protocol routing prefixes under the control of one or more network operators on behalf of a single administrative entity or domain that presents a common clearly defined routing protocol on the Internet policy sorry so that could be like an ISP sometimes ISPs when they get really large decided to get more than one because they have a different routing policy that's you know defined in the end their internet backbones as well operate autonomous systems et use enterprise networks and end users such as myself the autonomous refers to the the BGP routing policy and it's uh it's possible for a system to not be autonomous as well that's usually a problem but not always and yeses are assigned by AR AR s like Aaron in the North American region ripe in the European region and ap NIC and EMC a region yeah yeah age of Pacific region and then the other part I mentioned was routing policy and the routing policy is how the routes are chosen within your autonomous system so you could just be leaving it at defaults and just let BGP do its thing it does pretty good that might be determined by your needs and budget maybe you don't have one hundred and thousand dollars to to buy really expensive routers to hold Able's and so you want to work with defaults or defaults and you know a subset of routes you might have two links with unequal cost like monetary cost unequal performance unequal throughput rate you might have like a transit connection plus in there in that exchange connection or maybe a primary and a backup or perhaps a primary a backup and an internet exchange connection it's also determined by the number of routes that are installed so yeah if you need your full transit partial default route only and then the other part of routing policy is a special routing policies so things like like black hole injection and I'm going to cover that later on as well because that's pretty important so why would you even want to use bgp bgp is the best practice standards based way of doing dynamic routing with outside networks it gives you control of your routing policy so if you're like an eyeball network like a residential is be your mostly inbound so you can get some control over in known by having one or more links if you're an outbound network like a content or a content delivery network then you have an exceptional amount of control and enterprises are kind of a mix of both because there's often a mix of outbound hosted applications as well as users surfing within that ice B space as well too maybe you want to Ridge innate your own public IP addresses before v6 that's one of the reasons I do it another reason that you'd want to use BGP is automated DDoS mitigation which to be done with remote triggered black hole injection or with BGP flowspec if you have that capability that would not be on the budget side of the scale so why do I personally use BGP control of my routing policy was a big one I originated public IP addresses I have a slash 24 of ipv4 space which I applied for on the day that Aaron announced their waiting list that was my big push I also have a slash forty of ipv6 space that's approximately two hundred and fifty six sites worth of ipv6 space I also use BGP because I want to read the global BGP routing tables I'm interacting with BGP through software to distort and to to look at Manitoba in the routing also I want to experiment with BGP so I'm doing any cast on ipv6 as well as ipv4 within my own system on the experimenting with BGP I'm also building and using communities to to modify routing policy and working with my upstream to build communities that give additional functionality one of the other reasons I'm doing BGP is so I can offer BGP the six tunnels to people who want to get v6 and I have two routers so you may as well have two tunnels to me and do it in a dynamic way rather than the knots and then I'm also doing black hole injection for DDoS mitigation I've had several DDoS which like cover later on here so the process to get to do BGP so I actually started at one but process or step zero was to actually go to the company's office and register your company so I've registered hextech systems so that I can have a company so that I can interact with Erin as a company after you have your company then you can set up your your errand account and validate your organization so when you validate your organization that's where they they want to find out like is this company actually real so in Manitoba the new companies are only posted online for 90 days so you kind of want to get all your ducks in a row and and make all these things happen at once because otherwise then you're gonna have to pay for a $40 certificate as well you also need to find an upstream and make some tentative arrangements so that could be like a contract with with no start date I hear Jonathan laughing you also need to pick and/or require some hardware so in my case I got some micro tech hardware because it's pretty decently priced and it lets me do some some cool tunneling things you also need to get your v4 in these fix space I listed v6 first because in today's world there there is no more before just openly available if you want to get before you have to buy it from somebody else go on a waiting list or get v6 first and then you're eligible for 1/24 of v4 space to help your transition so you can dual stack servers in there you can have mats and stuff in there anything that you know that you're needing to do transition on you also need to get your ASM and once you've got all that happening then you can fire up BGP and it sounds really exciting and as it turns out you just you put in a couple commands and then get on the phone with your buddy and make sure that things come up and then traffic starts flowing just like that it's it's pretty it's pretty neat and yeah it's pretty exciting to happen and the profit question mark well that hasn't happened yet that's why put the question mark on it so what does BGP look like so if you've ever like interacted with a Cisco router it looks like this there's other platforms that have a syntax that looks pretty much identical to Cisco because most people are familiar with it quag is one of them Arista there's some others as well I can't remember off the top of my head what does BGP look like on mikrotik well mikrotik has a nice little Windows application which you can run through wine and so you can set up all your your peers and whatnot in a nice little interface so these are all the peers that I have I took the screenshot this morning I didn't intentionally make it unreadable but the information isn't actually all that important anyways the important thing is I have 25 peers and I think I have about 20 of them active right now and there's the largest peer is about six hundred and seventy thousand routes times two because I have two connections to my upstream and then this is the configuration for an individual peer which is completely unreadable I see basically it lists the name it lists the BGP instance that it's connected to lists the remote address list the remote a s number that's really important if you mess up the remote a s it will not connect there is timeout value specified in there there's filtering specified in there and then at the bottom there's a default originated value so it says never because I never want to send a default route to my upstream that would just be silly they would never want to send the whole Internet to me because it isn't behind me so types of networks so this is your your absolute simplest type of network this is your your d-link router at home the internal switch might not even be a switch it might just be several switch ports that are on your router there's usually no need for BGP in this type of network but it is possible to do BGP in a network like this you just need to have a lot of IPs behind so you it's pretty unlikely it would actually ever be used in this case the simple map network as compared to the simplest has an edge cpe device and this is usually used in an instance where maybe there was multiple IP addresses so the edge cpu device can either be your own router or it can be your your provider's router ya often the multiple IP addresses are used for hosting like services or perhaps a mail server or something like that where there's just need to have different traffic appearing as different IP addresses or different inbound on different IP addresses it is possible to do BGP in this type of situation it would depend on how many IP addresses are in use BGP doesn't necessarily have to be public to the whole world either if you had two connections to your single ISP you could do private BGP to your ISP and have redundancy and automatic failover in between two links and because your firewall is behind the routing device in this case it would never even need to know that anything had changed and things would just keep working this is a simplified enterprise network so this is slightly more complex this one has two edge routers instead of just one there's a redundant firewall and then there's some internal switching which might include much more switching than that and perhaps a bunch of security devices and whatever else as well that's why I put simplified in there I didn't want to have to explain or pretend to have full knowledge of enterprise networks in this case BGP could be used I have seen and worked on networks that look much like this yeah many networks look a lot like this this is quite common for BGP networks sometimes the firewalls are doing BGP but it's much better if there's a router outside that can do the the stateless routing and then let the firewalls worry about having their their connection to the the edge routing and not after a teep state on different interfaces ok good that did show up I wasn't sure how that would work so this is kind of a simplified isp network so in this case the ISP has obviously links to other ISPs where they're they're getting their bandwidth from this would be an ISP with no downstream ISPs because there's no customer bubbles there would be BGP definitely in this type of network along the along the right hand side where the the stack of four routers there those wouldn't say edge routing in between them I'm not sure how visible that is there would be BGP there as well as on the the inside layer there's two routers that say aggregation routing and those ones would also be participating in BGP as well BGP might even go further inside the network than that or there might be using some other routing protocol like OSPF or is is which is always interesting to Google these days so this is an example of a carrier Network where there's lots of BGP happening all over the place every single link in this case is BGP yeah so because it's a carrier network they have mostly other peers carriers don't tend to really buy from other carriers but in this case I could throw in upstream in a couple links because it's possible that it could happen this well - so my now work so my network is it's kind of a mix of all the networks that you saw here I have one upstream I have two routers both of those routers run BGP I have several BGP peers but they're mostly research type peers and not actual traffic and then I have some edge switching where I have my own VM servers and where all the tourist servers I that the not-for-profit then part of operates Odom's and then I have some virtual connections to my home firewall which also operates BGP because I have mmm probably about ten IP addresses or so at home and then there's my internal network which I've simplified down because it's almost embarrassing how complicated my home network is so I got my a s at the the beginning of April of 2016 so you can see this is the bandwidth usage of my whole system for the last two years it begins it's right near the beginning of April and the reason that the traffic graph looks so strange near the beginning way it grows and then shrinks is that tor is actually attempting to figure out how much bandwidth is available there and so they send some and then it's like well maybe let's back off on that a little bit or maybe let's give it a little bit more so that's why it's pretty uneven there you can see a couple sort of major dips those are where we've done hardware maintenance that lasted more than an hour so or where we maybe had a server problem or something like that you can't see any of the DDoS is on this level of graph but they're there has been several total traffic transferred here is 2.1 petabytes in and 2.07 petabytes outbound so abuse complaint so obviously because there is seven or eight IP addresses that are used for tour I get a fair number of abuse complaints but given the overall volume of traffic we're talking like 800 megabits at peak everyday I only see about 10 complaints a day right now and it's mostly for ssh stuff so I've seen 1394 complaints since August 2016 and I've seen 67 DDoS attacks in in 2017 alone which is when I installed my anti-ddos system it became necessary so having fun with BGP interacting with BGP with your own code the easiest way to do this is with exit BGP it's a Python package it's very simple you can write stuff in Python or because it has a very simple API language you can even write stuff in like bash scripts which is what I did through my previous besides talk I've well I get to that so reading global BGP routing tables into sequel so I rewrote this code from the last b-sides previously this was all done with shell and it was stored in coach DB and the problem I found was that coach DB didn't let me index on things that were deep in the data and he made it really difficult to search so I had to go and use a sequel database so I rewrote it using some Python stuff I'm using MySQL it actually has it's possible to use IP tape viable IP types within within MySQL but there's there's some functions to let you to work with that actually and I'm using that database data to actually maintain parts of my bgp guru site right now so I maintain a list of all the a s's in each province in Canada and if they're active or not and if they're advertising v4 or v6 and then some general stats on the number of a SNS that are active in kind of a how many have v4 how many have v6 how many r v4 only how many r b6 only etc it's all really easy to make all those stats once that stuff's in a database I have a lot of ideas on this but I've been kind of slow implementing them unfortunately something I have done is H a DNS resolver IPs so one of the most annoying things about operating systems is that you have two DNS servers configured in there and the operating system makes a choice of a which DNS server it's going to use and it sends out your query and some new operating systems are a little smarter about this but generally it waits for five seconds for a response now if it doesn't get that response it'll try the second one sometimes it's smart about remembering that the first one's failed and to use the second one first sometimes not so sometimes you end up with massive five second legs on absolutely everything you do because your first DNS server is down something that you can do is just make sure that that DNS server is always up or an instance of that DNS server is always up so I've been doing this by any casting internal resources within my own system there is a whole bunch of benefits to to this having all the IP addresses up that your clients have configured up solves the the five-second wait problem you can also load balance your front ends using ecmp maybe you shouldn't have just put one thumbs up I should have put a thumbs-up and I kind of like middle because there's some downsides with that too routing will also pick the local the closer geographic instance of your your DNS server then so if you have a large network that covers all in Manitoba and you have a local DNS server that answers your statically configured DNS queries in each community or town and that goes out well then you just use the next closest one health checks is the service down we'll just drop withdraw there out because there's there's obviously other with owns within the system I have I'm using this particular system in blog XE bgp health check script it's excellent it's very simple all you need to do is just specify a command line check that will return not zero if things are not good which pretty much every command does already like dig in colonel etc I'm using this from my own system and also for an ISP that I run ipv6 any casted DNS name server so I like playing with BGP and experimenting with anycast and unfortunately because before space is not readily available anymore I don't have a slash 24 of v4 space if I can like mess around with an advertise so that I can use one IP address out of that slash 24 or in any cast test because that would just be insanely wasteful so I'm using a place called dev capsule they offer BGP with customers they're pretty cheap they have locations in London Amsterdam and LA I think it was so I fired up some instances they were I think 2 pounds a month so like 4 dollars or 450 or something like that a month so it wasn't insanely expensive fired them all up got stuff all configured out had to send them a letter on my letterhead from my company because they they didn't use IRR values even though they asked for that in the the setups hmm I see Jonathan doing the face balm over there so I found out that these guys don't prefer the local ebgp route and that the only one they only learned one route network wide so what this means is that I don't get that you know you get the closest geographic instance feature anymore I just get there's three back ends and if one of them goes down the next one will respond and too bad if the active one is in Amsterdam and you're in North America or if the active one is in LA and you're in anywhere else other than North America basically and as a result I'm not using this for any like production stuff yet I'm just doing a testing sub-domain on it bgp for list distribution so this is something that i've played with a little bit I wish I'd play with this more I need to have an open bgp firewall again I guess so one of the cool things with open BSD and open bgp is that it's it's possible to interact between bgp and PF now PF is the the firewall on OpenBSD of course it's possible to actually have attributes like humanities get mapped into like a list of firewall addresses so if you got routes that were tagged with like the geographic source tags on them so so some networks say that are huge like global say you know I've learned this this route in North America or I've learned this route in in this particular city on their network so you could say if you had an active ute like that that I don't want to accept traffic or send any traffic to anywhere outside of like North America of course you'd find the whole bunch of stuff would break right away because you know there's there's resources that are global all over the world but if you had say a list of routes that were like signed as valid and then routes that were not signed you could say okay well I don't want to send you any traffic to these routes that are not signed or routes that are you know invalid in that case and you have a much shorter list of routes and because they're they're invalid in that sense you probably wouldn't have the same problems is just indiscriminately blocking traffic outside of like a geographic region it's also possible that if you get a BGP feed from like a nearby network so see if you're on MTS if you happen to get a BGP feed from an MTS BGP customer then you could get source a s and list of prefixes etc and you could build all these things into into firewall policies as well if you wanted if you had a whole bunch of distributed firewalls say around the world like you have you know 15 data centers that operate some website or something like that and they have open BSD firewalls involved you can run BGP between yourself not install any routes and just distribute SSH and millions blacklists so now when somebody you know tries to SSH into your network in Winnipeg or something like that and you have like four attempts it could communicate that into bgp with XE BGP and it could distribute that to everywhere around the world in seconds and so now your SSH annoyance is just blacklisted globally instantly you could also distribute Network white lists this way so that would prevent your you know if you white listed say sha IP space or something like that then you could say okay we will blacklist anybody whose SSH annoying us except for stuff from sha because we have users that are in that IP space and they tend to forget their passwords you can also working with the previous BGP example of a DNS server you could do round-robin load-balanced TCP port forwards or UDP port it's into a table so that could let you work with like multiple web servers or anything like that and your firewall policy would be or your firewall destinations would be controlled through BGP automatically and maybe even health checked so BGP for black hole injection so this became necessary after several DDoS attacks flooded my uplink it was a way of limiting collateral damage and I'm using an open source tool called fast nat Mullen Community Edition they recently split from just being a pure open source to a open source slash licensed model unfortunately but they're a really good tool and the the advanced one that I don't have apparently offers some really cool stuff with with flow spec which of course I have no support for going back to the limiting collateral damage so like I said there's there's seven or eight IP addresses within my slash 24 they're used for tour and so obviously people go online and do annoying things and people find out IP addresses and rent level DDoS attacks and try to flood people off the internet which it happens you know it's probably people on forums or Thor doesn't do anything UDP so it would have to be something TCP which rules out a lot of like interactive gaming plus nobody would want a game with one second of latency not unless you're playing Diablo or something maybe I used to have friends would play Diablo on satellite with like 900 millisecond ping it was crazy so the limiting collateral damage so basically when when one IP address becomes targeted it just gets dropped instantly it gets dropped within my own system gets dropped at my upstream and then my upstream also sends that on to their up streams and it gets blocked anywhere within their network so especially with hurricane electric they have they have a massive like 8,000 peers or something like that so everybody likes to send traffic for them so basically as soon as that advertisement goes out and hits Hurricane electric the the DDoS stops and so some some circuits that are established get killed off unfortunately but generally these DDoS attacks don't last very long the longest I've ever had was 15 minutes and that was before I had any sort of mitigation most of them lasts less than in two minutes now especially if you react right away if you react by just black holing things people get really bored and just go away it's kind of sad but it works out really well and you just drop that IP for two or I think I've moved it up to five minutes now just to be sure and I've never had like an attack last longer than five minutes now and so my up stream doesn't get annoyed because there is these massive ten or fifty or whatever gigabit attacks coming because they just get dropped before they get to large questions if there is time mark [Music] yeah so the question was is is running the Tor exit relays making me a source of DD or a target of DDoS attacks as a result of what users are doing and the answer is most definitely yes I've never had any DDoS attacks against any other IP addresses but the only other people that are on my IP addresses are myself and I'm not really i I don't game and you know like I do BGP stuff is fun no no nobody's like tried to attack any of my like web properties or anything like that no it's all attacks against for stuff so the question was about the number of peers that I have and if they're through tunnels and yeah there's mostly through tunnels so my own home internet runs through two tunnels because I control both ends of the tunnel then I can do QoS on both sides which is really nice all my Usenet traffic is like the scavenger class it's beautiful and then I offer bgp tunnels for some friends I have some some six in four tunnels and some other types of tunnels I offer transit over v6 to to somebody else in Germany who he's the the CTO of the the Dec IX German exchange and so I give his personal a s transit over a tunnel because I know it's a good relationship to have right he's the CTO of Dec IX well he's like got his a yes somewhere in Germany and so he doesn't really do a lot of traffic in fact I don't think I've ever seen any traffic on the tunnel no just oh just the tunnel yeah yeah he's got a local connection on his data center and then he advertises his v6 to them and hurricane electric and and then I think he has also some other peers that are some other people that have their own personal I asses as well but he gets transit too as well over some other tunnels I think so really I think he hadn't I think it a corporation register does that though didn't he huh maybe you don't but it was much easier to so that I've done multiple requests with Aaron now and and all of them get looked at differently it's really weird my own personal stuff gets scrutinized like you would not believe every time I have to go down to the company's office you know it's like oh get a new certificate prove you still exist where I do request for other places and they're like for instance I did the request for some of this this transitional IP space in the last week another than making the request in the complete wrong category the the request went through was zero questions asked so I think it helps if you have if you're doing it for a company that's like Google Obul or it doesn't just have a page that's coming soon actually if you if you're dealing with the like the business side of Shaw so not anything on the cable modems any circuit that that gets delivered you can at any point talk to them and say hey I want to do b2b on this and I did I have done that in the past before with both Shaw and MTS yep yeah and if you have IP space or you have a large enough piece of IP space from them and you have your own a s then you can advertise it so there's uh I did a blog post last year and I think it's September called for asns disappear anatomic aggregate it was actually seven but I didn't update the title because it would have updated URL and yeah so that was like seven asns locally that disappeared because MTS did an aggregate and and suppressed all their customer advertisements but there was seven customers that were that we're doing that so it's totally do and they've since fixed that yep yeah I'm not sure so the question was about whether where this type of stuff can be set up on an amazon web services account and I'm not sure about that I've I've never tried to do bgp with amazon on on the public side anyways yep all right I think that's that's oops I didn't scroll down to the end yep cool thank you [Applause]

Channel: The Long Con

Views: 1,362

Rating: 4.7647057 out of 5

Keywords: bsides winnipeg, bgp

Id: euGFNFVifhw

Length: 40min 49sec (2449 seconds)

Published: Tue Nov 07 2017