How NASCAR broadcast realtime telemetry updates with Ably

hello and welcome to this a webinar today we're speaking with Chad L about how NASCAR handles fan engagement at scale in particular we're learning about the technology behind the NASCAR drive platform which enables live racing fans around the world to choose and follow their favorite driver throughout the event and observe over-the-top real-time Telemetry data such as the Racer Speed red and how hard or bravely they're breaking into every turn fans they love it because it makes for a more Vivid engaging and conversation worthy experience delivering this type of refined second screen experience however is no small technological undertaking in a typical race you have more than 40 cars going around the track up to 200 m per hour each contributing to more than a terabyte of high frequency data per event this this is a webinar about how NASCAR reads said highfrequency data off the cars and then distributes it globally in real time to the user's hands with the highest degree of confidence and the lowest possible latency you're going to get a unique insight into NASCAR's realtime architecture and learn the story about how this project came to be the challenges and the impact on fan engagement so far I'm your host Alex Booker and I'm so happy you're here because this is a unique opportunity to hear from Chad and his decade of experience at NASCAR All right so Chad welcome to the show I was hoping we could kick things off by learning a bit more about your role at NASCAR and what your team does I'm the senior director of technical operations for NASCAR really my role encompasses three different areas of the company the Enterprise systems engineering media systems engineering and then really what we're talking about today and that's what has the most connection with a that's the NASCAR digital media systems engineering tell me a bit more about drive and where that fits within that organization so drive is an app that we created a few years ago and it's kind of morphed over the span of the time that we've been using it really the purpose of it is to give the uh consumers a way to be able to look at the incar cameras as the cars are moving around the track and then that kind of morphed into adding leaderboards on and then more recently adding in more of the realtime data that we use uh so really when you're when you're coming into the app it's really meant to give you a sense of what's going on in the vehicle just as you're the driver so you can see out the front window but then you can also see a dashboard that's telling them what's the driver doing with their hands what are they doing with their feet so you can kind of get a sense of what year are you in how much throttle are you giving it when are you hitting the break those types of things I feel like in NASCAR you're always looking for ways to deliver a great experience to the fans whether they're at the track or at home and this is a way they can either follow along alongside the television and the broadcast or they can even possibly sort of tune in on the go while they're away from home also yeah so it really gives uh consumers multiple different choices and really we're looking at a whole bunch of different groups of consumers we got those people that are on the go maybe they're at work and they can't tune into a race especially if there's been a rain delay uh that's moved it from one day to the next or it may Encompass people that are actually sitting at the track in venue that are consuming the race there but still want to have a better idea of what's going on so that kind of gives you that that option of sitting in the stands using your phone in order to be able to get a sense of what the driver is actually doing it adds so much more to the experience because in NASCAR you can see all the cars on the track which is great but you can't see what's happening in the cockpit unnecessarily tune in to how hard are they breaking into a corner or how eagerly are they pressing on the accelerator to make that overtake for example yeah I've gone to races before and I could actually start to watch one of the cars it's maybe falling a little bit behind and you can get a sense of why it is that they're falling behind if you look at one of the leaders you can see that they're holding the throttle a lot further through the turn but then one of the cars that's falling behind is actually letting off a little bit sooner and that's something that you can actually get a sense of when you're sitting there so instead of just going like I don't know why it is that they're falling behind if you actually look at the stats you can figure out what's the driver doing that maybe isn't getting them that far I'm actually super excited to share with people watching a little bit more about the architecture on the track and talk a little bit about how do you get this data off the cars and what does that look like once it's on a server we'll get into all of that I think and it's going to be super interesting so people can look forward to that I want to talk about the Inception of the project a little bit I was hoping maybe you could talk about the background and motivation that sparked the Inception of Drive really we're just looking to get consumers as much data as they want if they want to get more data if they want to know what's going on inside the vehicle they can have it if they just want to view what's going on on the screen or in the venue they can have it but we really want to be able to give them time to be able to spend on the website that they feel has been valuable to them they can get that data they can really dig in uh some are using that in order to go to various gambling sites and use that in order to kind of predict what they're going to do some are using it just to get a better idea of how's my favorite driver doing our fans are very much tied to a specific driver and they really want to know how well they're doing and this kind of gives them a better sense of what what they're up to in the vehicle so it's all about delivering a richer experience for the fans a more engaging experience for the fans on the website and in the app how do you measure the success of something like that what kind of metrics do you look at to determine engagement in this specific case we're not really looking at page views really we're looking for time spent on page so if somebody's coming to our Drive app uh and they're consuming five minutes we would consider that to not be so great whereas a lot of other sites might consider somebody spent five minutes on the page that's fantastic they spent a lot of time there a lot of our consumers are coming in they're spending 30 minutes an hour sometimes upwards of three hours on the page uh consuming the data so that's really where we're getting our success metrics from the the actual time that you're actually engaging with that data it's possible that someone watching doesn't know the ins and outs of NASCAR how long is a typical race and how many cars are on the track in a typical race so in a typical race you're looking at 40 cars on track and we could be looking at a race that's anywhere from about 3 to four hours long it really depends on what track it is how many Labs that they're doing so that's potentially 3 to four hours of someone on the app alongside the broadcast which kind of lends itself to this idea I think of a second screen experience which I have to admit is kind of new terminology to me but as I learn and read about it I realized that it's not exactly a new idea this has been around for a while yeah we've been talking about second screen experience now for years this is something that our website even before Drive was really doing all of the incar cameras we've always had a subset of cameras that come from the broadcast that have been in the app um it's just increasing so much more now that we have all of the cameras in every single vehicle that you can view in any Cup race that you're going to right now a lot of our consumers if they are at home and they want to be able to kind of get more information to kind of tack on to what they're actually seeing uh in the race they enjoy that I've seen some uh consumers that actually go through and they mute their TV and then they just can watch exactly what's going on for broadcast can consume their own data oh absolutely that kind of goes back to I can consume the data that the way that I want to consume the data it's not just us telling you this is the way that it has to happen yeah that's a very good point actually because when you watch sports often times you're kind of beholden to what the broadcaster chooses to focus on at a given moment whether that's the camera angle or what the narrator is talking about they might not be talking about your favorite driver in that moment but with the app you have the opportunity to follow V story and experience every curve of the racetrack alongside them yeah absolutely and and we see this very often where uh the broadcasters are focusing on maybe the top four vehicles that are in the race but with 40 Vehicles your favorite driver may be in position 23 and they may be fighting up to position you know 20 19 18 and as they're starting to make that climb you want to be able to follow along with what's going on not necessarily focusing on what's going on the top four drivers that's really interesting three to four hours on the truck potentially 40 cars on the track and they go upwards of 200 mil hour I want to understand a bit more and I think this is fascinating from a technological point of view how do you get this data of a car going around the track that fast and so many cars at that yeah it's tough to to think about it and really if you've not been out to a race it's hard to get a sense of the scale of what's actually going on at the track these tracks are massive and they cover a very large footprint and sometimes it really depends on which track you're at if you're at something like Tega um Daytona 500 you know you're out for that specific race it is a massive environment that you really have to cover uh so the radio signals is how we get all of the information off of the vehicles each vehicle has an ECU much like you might have in your your car at home but then we're also tracking other data that's coming off of it uh GPS we have different track sensors that are feeding all of this data and all of it's being filtered back across radio signals including the video is coming back through radio signals into a central location I would assume that central location has to be uh kind of close to the racetrack as well to minimize latency and things like that yeah that's a fair assumption we actually have mobile data centers that go from track to track every weekend so it's basically a traveling circus that we have uh semi-trailers that pack up and move everything every weekend we also do have data that goes to the cloud simultaneously but if we want to get the absolute lowest latency possible which our OEM and our teams want uh they're getting it from the data centers there at the track what would you say as some of the unique technical challenges broadcasting all that data from moving vehicles at such high speed some of it's just the radio transmissions in and of itself make sure that no one's got overlapping radio signals that we can get it over to the servers that firewalls aren't blocking data that it's flowing through quickly that it can get up to the cloud in a way that is expedient really a lot of these tracks that we're looking at they're not sitting in City centers for the most part they're usually further out so making sure that you've got a fiber line that's run to every single one of those and able to push data that much data that quickly up to the cloud is really important for us so that's the initial data Challenge and then you've got after that just being able to to consume it break it down into slices and then getting it out to Consumers yeah maybe this is a good opportunity to transition a little bit from like what's happening on the track and what's happening in the viewers hands in other words on the client side do you have a impression of how many kind of clients tune in for a given race our largest race of the year is the Daytona 500 so that's coming up soon um and that's something that we'll usually see in the order of tens of thousands of users that are connecting to that so we're not talking about you know 100 200 if we had something like that it probably wouldn't be that hard in order to push the data out but when you have tens of thousands of users and that many connections and and people are switching channels constantly throughout the race each each drivers in their own channel and if you want to view somebody different you would have to switch to that there's a lot of that that's going on at one time um so it's quite a lot to keep track of how crucial would you say low latency is for drive to meet its goals as an app so latency really depends on who's watching it from where if I am at home and I'm watching the broadcast latency is probably not as big of a deal um usually you're talking about a broadcast that's delayed by you know 15 to 30 seconds so we can have as much latency as that now if you're in venue and our and our users are in venue we have thousands of of users that are there when you've got all of those users there that want to see the data they're wanting to see it real time if a users If a driver is going around a corner and they want to see the throttle they need to see it right as soon as it happens and that's how quickly we are actually getting the data into the user's hands if the driver's hitting the brakes you're going to see it within milliseconds of that actually happening which is really cool to see very cool before we look at a is helping you take that data from the track and putting it in the hands of the users in real time it might be helpful for a viewer to understand where you're coming from as a technology organization what protocols and strategies did you previously use to broadcast Telemetry data from the track to users and what was your motivation to change so a lot of our data before was coming from a single provider that was providing that in a binary format that we had to really sit as a proxy in front of that so we would have to spent up multiple servers on any given weekend in order to be able to handle the load coming in from our clients our growth or our uh the amount of consumption that we get is really hockey stick so all throughout the week we have zero servers we have zero connections nobody's really doing anything and then as soon as the weekend hits we have just an absolute instantaneous volume of traffic coming in for clients looking to get the data so we would have to get servers prepared um before the weekend we'd have to spin up prior to the actual event happening and then we'd be prepared of that load that was actually happening so you've got the multiple different server and then as the uh the clients connect more during the race we'd have to be prepared to scale up or scale down in order to kind of be cost effective in those approaches the thing is when you describe these race weekends and what's happening it does sound really exciting just the energy around it and the technological challenges but it's also what you're describing it sounds like a lot of responsibility as well what are some of the challenges that kept you up at night I mean we're always concerned about are we going to have enough bandwidth are we going to have enough servers are we going to be able to scale quick enough uh if we have have a sudden influx of visitors from you know 40,000 to 880,000 up to 150,000 users coming in we have to make sure that we're ready to handle that load instantaneously users aren't typically too forgiving if they come on and they're not able to get the data that they're looking for or what they signed on to get they're going to leave immediately and most likely they're not going to come back and check later so we need to always be prepared to to provide them the data they're looking for the instant that they're looking for all right so it sounds like we've got a great introduction to the sports and what the weekends are like we've got a sense of the atmosphere and just how much data is Flowing on the track at any given time we've also got an impression of like what the client what the user experiences right we're talking about the Drive app and the experience within that there's a lot of kind of context floating around here I thought it would be super interesting if we could then look at a diagram which shows the architecture of how NASCAR are using a and you can talk through it a little bit and help us understand how these pieces all come together looking at the diagram here we've got over on the left hand side our vehicle sensors and our onrack sensors so we talked about that a little bit earlier we do have an ECU that is in every vehicle we also have some GPS tracking that's going on multiple different sensors that are kind of giving us an idea of what's going on is there been a high impact is the vehicle going to the left is it going to the right it gives us just a ton of information of what's actually going on they're also on track sensors they're giving us various information mostly what we're looking at is the data Loops as each vehicle is crossing over each one of these Loops it gives us an idea of where they are and as they move between Loop to Loop it gives us an idea their speed and where they are on the track at any given time as well all of that data is just compressed and put back together and then sent via radio signals back to our our main cluster uh this is using and that system in order to kind of keep track of all this data in order to be able to provide it to all of our different clients as they're connecting I think uh a lot of times we think of our clients just being our our web consumers or our invent new consumers that are using their mobile devices or maybe their desktops or laptops in order to keep track of it but you have to keep in mind that we're also serving out the entire ire industry and a lot of times that means that we're serving our oems our manufacturers of our vehicles as well as the teams that are at the track as well they need to be able to use this data to kind of get an idea of how it is that can better inform what they're doing during the race how to perform better or to do some Monday morning quarterbacking when it gets done to to kind of say hey this is where we could have done better this is where the vehicle could have performed better this is where the driver could have performed a little bit better so that n cluster is in two different places we have the mobile data center that we talked about earlier so that's at track and people can connect to that in order to get that information if they need the absolute lowest lency possible and then just a little bit slower behind it is the cloud but we'll really talk about in terms of milliseconds really is the difference between the on-prem and the cloud data center when it comes to that as far as digital media is concerned we have and we move over to the right hand side of the diagram there we have a client that is connecting over to that Nat's cluster and is connecting and grabbing all of the data as it comes through uh what you see on the web on a weekend is really just a slice of all of that data we're actually kind of taking that and breaking it up into multiple different pieces but it just connects over to the the N cluster grabs all of the data and then sends it over to a once we've done some some massaging of the data that data includes things like the throttle and Brake pressure we spoke about before you can figure out things like the speed and also the timings on the track based on the on track sensors it sounds like y so you get RPM you get your speed you get your throttle you're braking your steering angle uh we've played with fuel before fuel is something that kind of comes and goes and people wonder why um it's because it's a calculated number it's one that we kind of have to guess at a little bit people may be surprised to know that in NASCAR there is no fuel gauge inside of the vehicle it's actually considered part of the competition for you to be able to keep track of how much fuel you actually have and so there's it's kind of complicated it's not always accurate so whether we're displaying it or not any given weekend that's the reason why so there is this element of an ECU produces a ton of data some of it you probably don't need to relay so when we talk about data transformation in this area part of it is choosing what data to then broadcast is there another parts of data transformation that happens there yeah absolutely so when we're getting that data like you said there are there I think there's over a 100 different data points that we have um that we kind of toss out most of those when they're coming through most of them wouldn't make sense to a consumer but also the rate that the actual data is coming through is way too fast for anybody to really be able to do anything as far as looking at it on a screen um so we do SUB sample that down I think the current rate that we uh we're sending data is at about 120 updates per second that's what's coming off the Nats cluster and then we sample that down to about two updates per second so once every 500 milliseconds in order to send that to the consumer just for fun what would happen if you you know turned on the fire hose and sent all that data to a client device like a mobile phone if somebody was on their mobile phone it would probably get very warm very quickly your battery would die very very fast you'd probably be pretty upset and you probably would not want to use the app anymore you'd be pretty mad at us I don't know man it's pretty cold in the UK right now I might use it as like a little radiator while watching the race you might get 15 minutes worth of hand Waring before it died this is really interesting then because it sounds like that Nats cluster which the nod script is reading from to then broadcast data to the Drive app and all the client instances that's the same cluster basically that your partners and the teams are reading from at the track to inform their strategies and do analysis and stuff like that yeah and I think that's probably one of those things that our consumers really like to our fans that are really into the racing product and they have an understanding of how the teams operate they have an understanding how the oems operate they're able to get that same information that those teams and oems are able to get as well which really kind of puts you on another level that you don't really get in other sports yeah I think that's why I draw on it because I really like it it feels authentic and like you get access to the the real data you know not some like we've all done like progress bars on websites that say they're 100% but they stay at 100% for a minute so you know that wasn't you know there's all these kind of little ux tricks we see these days but it sounds like with NASCAR you're really seeing what's happening which is fantastic yeah absolutely none of it's made up you're getting the real authentic thing every time let's zoom in a little bit on that node script and how you're using a specifically can you talk to me a little bit about how you use a channels to broadcast this data from the track to the devices on the edge when we were first looking at how we were going to use a there was a little bit of confusion around do we put everything all in one channel do we break everything up like what does it really look like are we sending each Telemetry Point as in a channel in itself and really talking to the team we're able to figure out that the best way to send our data was to break everything down into a single Channel per driver the data feed that we look at in the Nats cluster really includes everything all in one Lop so as soon as any sort of data point is sent uh it doesn't matter for what driver or what Telemetry point it is it just sends it uh so we have our script that's actually looking at that compiling everything for a single driver uh putting a Tim stamp on it and then sending that over to a one channel for each driver so as you go to the Drive app you'll see that you have 40 different drivers listed and each one of them you can click on it and when it opens up that's actually connecting over to a in order to provide the data for that specific driver and that really kind of it goes back to what we were talking about earlier with sending consumers too much information wearing down their battery heating up their device that really keeps us from doing that as well I guess the kind of throttling is one part of it but I suppose another another way you could optimize that is if nothing has changed if the Delta hasn't changed then there's no point sending it over the network again yeah absolutely and we are when we send it from our node app over to a a lot of times we are sending duplicate data especially if you're looking at something like a red flag stop but a is going through and actually looking at that Delta and it's not sending that data as it doesn't change over to the client if there's no changes it's not sending that over the network which is is great for our consumers no one wants to be consuming data when they don't have to be consuming it I think everybody's a little bit conscious of how much are they using Wi-Fi how much data are they're using how much is you know their battery being used up as we talked about earlier so if there's not a reason to send the data why send it um and that's something that's really Challen alling to keep up with when you're talking about data that flows at the rate that that it does without per sending it how does a help you push data out to an ever expanding base of clients there's multiple different ways that they do that um besides the fact that we don't have to spin up servers on the weekends we don't have to worry about the maintenance during the offseason or even during the season during the week but also providing us with a set of tools that we can use in order to be able to integrate that data with our our different devices the sdks that they provided us give us an opportunity to push the data out to the web or out to Android or to iOS so once again it goes back to what we were talking about before we know that we have a lot of consumers that they want to make the decision on how they consume the our products and we want to be able to give them as many opportunities in many different ways as they can to do that so A has provided us with a lot of those tools the building blocks really in order to be able to do that if I understood well previously you were using a sort of solution that you hosted on a VM uh you had to pre-scale it before events and things like that compared to Dynamic scaling like you get with some hosted providers like a can you talk a bit more about what that looked like the prescaling was always very interesting uh we talked about that a little bit earlier where we have a hockey stick and traffic that we have that comes through so uh being able to know that you have the servers ready to go when the consumer is ready to consume the data is an important thing to do when you're scaling out servers there's there's definitely some lag time that takes in order for them to get ready to be in use that could be from seconds to minutes really kind of depending on what technology that you're using but not having to worry about that anymore takes our minds away from the activities around the servers themselves I'm more about making sure that we're able to get better data uh we're able to move data faster and then we can kind of concentrate on other products instead of just having to be concerned about maintaining a whole Fleet of servers I think whenever you talk about maintaining servers that invariably involves some like people allocation like you need someone to look after those servers do you still have to worry about that we have to worry about that so much less uh really we have one on call person that is for systems engineering team every weekend that's ready to to respond if something goes wrong and we have multiple other products that we have going on as well other than just our real time data so they're ready for that but they no longer have to keep an eye on all of these different servers that are just sending the uh the real-time data across which is a big win for us that's just a little bit less work uh it's a little bit less time you have to spend answering different on call rotations that are coming through we've not been so explicit about it I don't think but there is this obvious migration here where these are features you provided for while uh you've now adapted able to help deliver those updates to the edge these migrations are never easy but they're often worthwhile do you feel like you can move more quickly now and are you planning to add any new real-time features yeah we can definitely move a lot faster I think with the sdks moving away from the binary data that we had before into an SDK where the developers can have a better understanding of how it works has been big for them I think it makes them think a little bit more creatively um about what it is that they're actually trying to do there are so many times before we was just kind of like well that's too difficult to work with maybe we'll use it a little bit more here and there in the future and it never really happened whereas now there's like active discussions oh yeah those black boxes are dacing aren't they oh they're really bad to work with um so yeah we are looking at different ways of using more real-time data uh switching our leaderboards over to use uh to use a uh We've kicked around some ideas of using content as well in order to use real time data instead of things that are polling that's fantastic Chad thank you so much for your time I'm now really excited to switch over and take some questions from people watching thanks a lot Alex

Channel: Ably Realtime

Views: 125

Keywords: ably, realtime, websockets, mqtt, sse

Id: zzOY9NdTyI0

Length: 24min 35sec (1475 seconds)

Published: Wed Mar 06 2024