VSCode Tinder got DDoS'd

it's been one week since i launched my dating app of a cinder designed specifically for programmers and a lot has happened so let me catch you up when it first launched right out of the gate there was problems specifically with my lambda function so i had it to where it would generate images of code so you type in some text you'd hit the submit button and then the lambda function would handle the process of actually generating a screenshot or an image out of that and it failed like eighty percent of the time but the weird thing is it worked like 20 percent of time and unfortunately for me it always worked in development it wasn't until it scaled up and a lot of people were using it that it just started randomly failing and it was randomly failing for about i don't know the first six hours or so i was using puppeteer to control chrome and actually go to a website to generate the image so puppeteer was acting a little bit funky and it was randomly getting these areas where it couldn't connect for some reason and i tried a bunch of stuff and i had no idea what was going wrong until i finally stumbled on the answer and basically the way i was importing puppeteer and launching it and spawning it was i guess wrong for how i was using it but it kind of worked but it doesn't work when you ping it a bunch i guess i don't know it works now i just changed the import basically and we're good it was kind of awkward because you can't even actually use the app unless you successfully upload an image where like 80 of you were not able to in the beginning but luckily i was able to fix it like kind of fast-ish but that was not the end of my troubles with lambda so someone found the endpoint for the lambda function which is not that hard to do and they started spamming it with requests and at first i actually didn't even notice it because i wasn't like heavily monitoring my lambdas at all which was a mistake i should have been doing that as well but i did learn about it when someone messaged me and they're like hey ben carbon is down i was like what for those of you that don't know carbon is a website where you can paste code and i'll generate just like a nice looking screenshot of that and that's what i was using my lambda function would use puppeteer to visit carbon and then create the thing and then screenshot it and that's how the images were created so the website that my lambda functions rely on went down and i knew right away that this was definitely not a coincidence and that it was somehow related to the cinder and that it was causing the crash somehow and a few minutes later the head of because that is where carbon is hosted contacted me and said they received 2.3 million requests and three minutes and so they took down carbon and they were just letting me know at this point i'm apologizing to them i'm also like i'll turn it off asap and i just rush over to my aws console at this point i knew it was going to be bad but i didn't know how bad it was going to be i didn't know if i was going to be like one of those scrubs that just gets on their hands and knees and starts begging to bezos please please please waive my 2 000 land of function calls that cost me over a million dollars please when i got to the aws billing dashboard it hadn't updated yet so it just said zero and that i hadn't made any lambda requests today so i knew it was just behind and i just started sweating profusely and i rushed over to the lambda section to see what is going on over there as soon as i saw the graph for the number of invocations for lambda and i saw that giant peak i was scared of my mind and then i saw the y-axis read the label and relaxed instantly 10 000 is it's a lot but it's not a lot at all and then i'm thinking is this just normal traffic is the cinder just popping off right now either way i knew the next step was to stop sending requests to carbon because i didn't want them getting a huge bill from their service provider so i didn't exactly know where the off button was with lambda so i just decided to redeploy lambda and remove the part where it you know actually makes the request and i just throw an error and i was just like sorry guys i'm working on self-hosting carbon right now and come back in a few minutes so notice in the invocation graph like it dips to zero spikes dips to zero i'm pretty sure that dips to zero part are when i deployed my lambda function i can't remember what i deployed after the first deploy but i did something probably or maybe it's not the deploys i don't remember and then at the end i finally you know got my critical thinking skills back and i realized yeah this is not regular usage because you only need to call this lambda function like once when the user signs up and that's it that's all i'm using it for so it makes sense and i knew someone was just like they had a for loop or something and were like spamming it and then i found the throttle button first i didn't really realize how throttle worked so it went down to zero requests and then i realized you have to actually set an amount of what you want to throttle to based on the number of instances you allow lambda to spin up for serverless functions and then that's actually where i found out lambda caps you at one thousand so i just got lucky basically the ten thousand requests that were happening were coming from like i guess lambda's like one thousand invocations that it created for me and it stopped so more requests were probably coming in but lambda capped me which thank goodness because my bill would probably be way more if it didn't do that so i thralled it so it can only make up to a hundred instances and notice at the end where it kind of like flat lines that's where i started throttling with a hundred instances and then i throw that even harder it can only make 20 instances because why not i didn't want to just be wasting money for the spammer to send a bunch of requests the one part i haven't yet been able to figure out is carbon received 2.3 million requests but if we look at my lambda vocations it topped out at like 10 000 requests in a minute and i think overall it was like 60 000 requests if you total that all up if i remember correctly and so all my lambda function does is spawn puppeteer and visits the website so i don't know how those numbers don't line up i don't know if puppeteer has logic in there to retry the connection if it can't connect to the website and that's what caused like the number of requests because i don't have any retry logic in there and so i have no idea why those numbers don't line up but it most definitely was coming for me at least i feel like it was i don't know why i had way less implications though anyway the total cost from lambda turned out to be around 20 dollars and the only reason it was actually that much in the first place because you saw my invocations weren't actually that much but when i was looking at the lambda free tier the thing that was costing me was my lambda functions were running for a long time and testing i think i set the timeout to like 30 seconds because it took a little bit longer for puppeteers sometimes and i just wanted to make sure it worked and so what i think happened is puppeteer was trying to like connect to carbon but it couldn't write because there's so many requests and so it was just spinning and so the whole 30 seconds it was just sitting in the lambda function so i had a ton of lambda functions functions just sitting there for 30 seconds and that's where it cost 20 which can i really complain no also carbon is open source so i was able to just pretty quickly get clone the code and then sticking on netlify and then we were just up and running again smoothly oddly enough the bandwidth from netlify cost more than everything from aws but fortunately the attacker stopped shortly after i thralled it to 20 instances so if you're watching this thank you my wallet thanks you by the way that's the only thing that i do on aws the rest of my api is actually host on digitalocean they sent me like a 50 credit out of nowhere so i was like dope i'm going to use this for my next project so i spent up a 60 vps it's a big boy but i was like you know what why not i got the credit i have to use it within this month so i was like let's just see what happens and i wanted to make sure like the api didn't go down that that worked really smoothly and i'll show like the analytics from it so you can see the cpu in the memory and stuff and it did fine i could probably you know downgrade that maybe i will downgrade it at the end of the month and inside that vps is pretty much the entire back end and it's all managed by docu which is kind of like heroku but for self-hosting and so it has a nginx proxy sitting in front and reverse proxying and load balancing and doing all that fun stuff and then sitting behind that is the api which is a node.js server of course i use typescript and then that talks to a postgresql database so that's running inside of there too and then there's also redis as well which i'm using for rate limiting and maybe something else i think maybe just rate limiting but the code's on github you can go check it out and you can fact check me there and doku also has like a daily cron job that just backs up the database to s3 in case any bad stuff happens then the rest of the tech stack is react native for the ios and the android app written in typescript as well you know i can't go without typescript and the vs code extension also type script and it's also using svelt for the web views so that's pretty much the entire ui really and uh yeah you know expanding my horizons a little bit maybe we'll go react to the next extension but svel's been going pretty well i'm enjoying it and that's the entire tech stack actually but there's one other thing i'm kind of using google cloud because i'm using the google places api that's the thing like when you type in the location box and it auto fills like cities you can tap on that's really nice and convenient well it turns out that's actually super expensive it's been a week and it's i think like 60 bucks probably by the time i screenshot it and stick it here it's going to be even more money than that and it's not even that many requests have happened and probably by the end of the month it's going to be more expensive than the back end all of aws combined so i really need to find an alternative to that so i got to figure that out not sure what i'm going to do yet because it's also like such a small portion of the app and half the users aren't even setting their locations like why am i paying so much for this a lot of people ask for stats and the gender ratio so here you go about 14 000 people have signed up so far 81 of those are male 10 didn't make it through like the profile screen and set their gender eight percent are female and less than one percent are non-binary i can't really say i expect differently but there you go uh about seven thousand people set their goal as love and about six thousand people were in the friend category there's been thirteen thousand matches and sixty one thousand messages between users also i forgot to put it here but there's been forty thousand swipes right and like sixty 000 swipes left and yes you see that at the bottom there i'm sad to report it but there's been zero acquisition offers for microsoft i know you're sad i'm also sad you also wanted to know if there was a lot of problems with inappropriate ascii texts being used for the code snippets and that for their profiles and i'm happy to report it was not really a problem at all i got like maybe 10 5 or 10 reports of this happening they got shadow banned very quickly and they're gone and what i'm going to do is just keep track of these people and their github profiles and they will just be banned from future extensions as well and they're seeing a lifetime ban and so hopefully this will dwindle to like next time there's only two and the next time there's only one and then zero now the more prevalent problem was guys setting their gender as female because there wasn't that many females i guess and so they wanted to be on the other side and swipe with a bunch of people so i did have to ban a lot of people for doing that and that was way more of a problem also reports kind of got filled up because a lot of you wanted to say hello to me and a lot of you wanted to ban code because you didn't like the formatting or the style or the theme or thought it was ugly or you saw some angular code there's this poor girl that put angular in her description she was trolling you and every single one of you got baited to report her it was so many reports on her so i'm literally going to be adding an if statement in my code checking the report message to verify it doesn't have the words angular ugly code bad syntax hi hello and uh hopefully that should clean up my reports because i'm just scrolling through so many bad reports right now but anyway other than that i thought reports worked relatively well kind of ish time wise the vs co extension and the app took around two to three weeks to actually code everything but after that took a little bit longer to actually release it because getting through the apple store and the approval process was kind of a slog the rules that they require you to kind of comply with are okay you know i wish some things were different there but the thing that really is annoying or slows you down is it's a day between when they'll like check it which okay maybe a day is not a lot um but every time you make a change you gotta send it you gotta wait a day they tell you go change this and you wait a day you change that and there's another thing you wait another day and it just feels like it takes forever well it does take forever and so that was really annoying but it got through and it's out there and the other thing about the app store that's really annoying is costing things what am i talking about costing things charging things charging money a lot of people want the cinder gold for premium features and i'm not quite sure about it because charging in apps is super annoying if you don't know i'm gonna have to like implement something for apple and then they take a giant cut i'm gonna have to implement something different for google i think now because they have a thing or maybe they start doing that in january either way i'm probably gonna have to start doing that and then i have to implement it differently for the vs code extension so in like three different places just disgusting i might just make the features free so you can have some fun or maybe i'll have like some sponsored posts or some advertising so i don't know this is all to say that i'm undecided yet and i'm thinking about it lastly if you wanted to try out the cinder but you ran into some bugs day one or you just couldn't log in with android few people ran into that a lot of the bugs have been ironed out so give it another try there's definitely still some in there so hit me up github.com slash benawad slash of slash issues let me know you

Channel: Ben Awad

Views: 185,815

Rating: 4.9331565 out of 5

Keywords:

Id: thqX845NEgg

Length: 13min 55sec (835 seconds)

Published: Wed Dec 16 2020