How to Burn Money in the Cloud // Avoid AWS, GCP, Azure Cost Disasters

when you decide to move to the cloud one of the first things you have to do is give them your credit card that's because they're giving you access to a massive network of computers worth billions upon billions of dollars when you write inefficient code on your own computer it's usually not a big deal the cpu fan might spin a little faster or things might run slower when you run out of ram but in the cloud it's a whole different ball game that recursive function you wrote can now scale infinitely around the globe blowing up cpu cores and melting solid state drives in the process and someone is going to have to pay for that mess in today's video we'll look at a real life story about how a startup just days away from launch racked up a 72 thousand dollar google cloud bill stand it's gone didn't do too well it's gone poof we'll find out how that's even possible but more importantly we'll look at ways to prevent something like that from happening to you if you're new here like and subscribe then let's get right into it imagine spending months pouring blood sweat and tears into your startup you're just days away from the big launch doing some final testing here and there your product is ready to go you just need to scrape some data from the web to enrich the user experience you throw together some ad-hoc code to get the job done dockerize it then deploy it to a serverless platform like google cloud run it only needs to run for a few hours so you deploy it then call it a night you wake up to a couple of interesting emails our firebase project was upgraded from the free plan to the pay-as-you-go plan we must have enabled an api that requires billing which automatically upgraded the plan not to worry we already have a budget alert in place oh no the next email tells us our seven dollar budget has been exceeded the third email tells us our credit card was declined i wonder why that would be let's open the billing panel in google cloud no god please no no a bill of over five thousand dollars in just a few hours it has to be a mistake let's refresh the page and pray no no five minutes later the bill's at fifteen thousand dollars after twenty minutes twenty five thousand dollars the final bill settled up after about two hours just short of seventy two thousand increasing your startups burn rate by about six hundred dollars per minute no if you're a bootstrap startup you might as well make it ten million dollars to make matters even worse this company had multiple products with multiple google cloud projects to manage them because things got out of control so quickly they had to disable all of their projects and that's the story of how one little recursive function took down an entire business i declare bankruptcy luckily the story doesn't end there the company immediately contacted google cloud to make them aware of the issue and let them know that they did not intend to purchase this much computing power after 10 days of deliberation google did the not evil thing and credited the entire 72 thousand dollar bill that must have been a huge relief for the company and was a big win for independent developers around the world but keep in mind there's no guarantee you'll get the same kind of credit when you rack up a massive bill the best strategy is to not rack up a massive bill in the first place however that's easier said than done i've actually racked up two massive bills myself one time i accidentally exposed my aws api key in a public git repo a few months later i had a 5 000 aws bill because a hacker scraped the key and then used it to spin up a bunch of compute instances luckily they gave me a credit for my bill as well it is very reassuring that these big cloud companies would rather keep you as a long-term customer than try to bankrupt you over an honest mistake but a bill of 72 000 is crazy i've never seen a bill get that high that quickly and we should all be thanking announced today for sharing that story with us check out their web app for a great example of angular and firebase in the wild then let's dive into the technical reasons for the massive bill and more importantly ways to avoid it yourself tip number one set up a budget alert even if you're on the free plan of something or in the free tier once you deploy code to the cloud there are different ways you might exceed the free tier without knowing it it's not common but it's still always a good practice to have a budget alert set on your account globally the company did this right by setting a budget for seven dollars the problem in this case though was that the cost increased so quickly an alert will only send you an email it doesn't actually shut down anything when the budget has been exceeded it's actually kind of crazy that there's not an easy way to shut down services when the budget has been exceeded that's been a major criticism of the cloud for as long as i can remember however there may be hope in october 2020 aws released a new feature called budget actions where you can define an action based on a budget threshold like shutting down a server once you've spent too much money in addition on google cloud you can listen to budget events with pub sub and that means you can write a little bit of code to create a kill switch that disables billing if the budget is exceeded now let's move on to the next tip do not create infinite loops in the cloud do not create infinite loops in the cloud do not create infinite loops in the cloud when it comes to serverless computing everything is event-based some kind of event happens in the cloud like an upload to a storage bucket which triggers a cloud function for additional back-end work it's really easy to shoot yourself in the foot here though if your function leads to the same event that triggered it in the first place for example a cloud function is triggered by a storage upload then you upload a file to the storage bucket again and the function thus triggering another function and again and again and forever until the end of time or until your bank account is empty that's what we call an infinite loop and it's probably the number one cause for runaway cost unexpectedly especially if your infrastructure is configured to handle huge workloads that brings me to tip number three don't configure things to scale when you're just messing around our startup used a service called cloud run which are serverless functions that support docker containers by default cloudrun will scale to a max of 1000 instances now if we take a look at the architecture for the web scraper you'll notice it starts with a manual request to the cloud run endpoint each run scrapes url and saves the results in firestore then it takes all the links it found in a scraped webpage and triggers another cloud run to process them if we look at this as pseudocode what we have here is a recursive function but a recursive function with no stopping point which leads to stack overflow or in this case cloud overflow which is a brand new word that i just coined because the infrastructure was set up to scale it resulted in a 116 billion reads to the database and 33 million writes along with 16 000 hours of cloud run compute time the moral of the story is to understand how different services scale and configure them to not scale if you don't need that functionality in the case of cloud run if the max instance option was set to 1 instead of 1 000 their bill would have been 72 dollars instead of 72 000 but in production you likely do want things to scale and sometimes you won't know how bad an issue is until your app does scale which brings me to tip number four consider how algorithm complexity affects your cloud computing usage a while back i made a video about how to not get a 30 000 firebase bill which was another case study about a company that went viral and a flawed data model caused their bill to accelerate exponentially allow me to explain they built a crowdfunding app and in the app they calculate the total amount donated to a crowdfunding cause they perform the calculation client side by reading all the donations from the database for that cause now let's imagine you have five people who donated to a cause and a hundred visitors to that page that would equal 500 document reads now let's imagine you have a hundred thousand visitors that'd be about 500 000 reads which still isn't too bad but now let's imagine you have 5 000 donations to a cause multiply that by 100k users and you now have 500 million document reads or in other words at the current traffic level each new donation adds an additional hundred thousand document reads to the total workload it just doesn't scale well a better option would have been to aggregate the data on the backend with a cloud function rather than calculate the total client side it could be done on the back end so that each visitor only has to read one document instead of 5 000 documents now i realize stories like this are pretty alarming but firestore is actually very inexpensive and because it's so easy to use you may not realize you're making a bad data modeling choice until it's too late using the right data model can go a long way and you can learn more about that in my data modeling course but yet another way you might experience runaway cost is by failing to secure your app and that leads me to tip number five make sure your app is secure most importantly make sure to protect any secret api keys like i mentioned earlier exposing my aws api key on github allowed a hacker to take full control over my account but there are many other ways you might fail to secure your infrastructure when it comes to firestore you need to define security rules to define who has access to what in the database and of course i have a full course on that as well leaving the database open means anybody could read or write to it not only does that leave you vulnerable to a data breach but also to hackers that just want to drive up your costs the same principle goes for any application that is available on the web when you have a service that's available on the internet it means someone can find that url and exploit it i'm going to go ahead and wrap things up there please make sure to like and subscribe and if you're building an app with firebase or google cloud consider becoming a pro member at fireship io thanks for watching and i will see you in the next one

Channel: Fireship

Views: 297,993

Rating: 4.943718 out of 5

Keywords: webdev, app development, lesson, tutorial, aws, azure, gcp, google cloud, amazon web services, cloud computing, the cloud, cloud, cloud pricing, cloud costs, backend

Id: N6lYcXjd4pg

Length: 9min 0sec (540 seconds)

Published: Mon Dec 14 2020