Migration Lesson: Don't Use Prisma | Prime Reacts

we migrated to squeal our biggest learning don't use Prisma milky have you used much Prisma I have I have prism income stock uh in T3 and I've used Prisma a little bit here and there it was like the client's super nice super convenient to use okay anytime I've ever used an orm I eventually have regretted it yeah I don't know if this is if this is universal everyone even uh ter so says that the drizzle client is incredible uh but generally my my general take is orms are great until they're horrible like most technology there's a trade-off there's something to it so let's find out last week gration that switched our underlying database from mongodb to post degree no one actually nobody knows how to say this word it is the unpronounceable word because usually it's like post grease squeal or is it postgres queel like we don't know is it postgres or post gree I'm a postgres okay you're a gentleman I say postgres when I invite people to code with me okay I say post gree because I feel like Greece with me yeah because the S is a part of the squeal okay anyways we faced a lot of interesting challenges and the biggest of them was a two-time rewrite of the code base a back to back yeah you're doing something there's a lot of questions that need to be answered if you've Rewritten something twice like I get a rewrite like it does happen but but to be like you know what that last three right let's do it again we gotta try that again I do want to say I think it's a great decision they're going from mongodb to post Greece when I first read that I was already I was like this is this is a good article I'm going to continue reading this this is a good this hooked me in better than the don't use Prisma title so I was I was already invested at this point Yep this article is for developers considering using Prisma for the production workloads don't don't okay okay okay let's find out uh because is this a skill issue or is this like a prism issue sometimes it's just a skill issue that's you know you miss something we wrote our back end in Prisma first so I'm not sure what that means but let's go on one of the biggest mistakes we did was trusting Prisma marketing blindly look at this beautiful landing page damn this is like a this is a stopping Next Generation node.js and typescript orm I feel like you should have known right away that this was a bad idea because it doesn't say the word go or rust like I feel like I could just see it right away like oh you know where you screwed up I I honestly trust websites or Services where the landing page looks terrible like if it looks too good I'm like your marketing is way too invested I want something where it's like pure HTML a little bit of CSS that it's not good to look at but when I see the code or the use case I'm like I love this because if it looks too good I'm like oh this is this yes like browserify what's like the classic example of box or the marketing there's an image of a wizard some kids some person's a kid probably made they're like this is good we're good here we've done it this is the key Web 2.0 all right anyways you know you stock colors like they're just like yep you know what let's throw this on a weekend and let's keep going they still think that maybe it's only 600 pixels like that's all you're gonna have for the screen Ah that's me live by the way I was live by the way click here where my foot's pointing to uh by the way milky streams all right but that's pretty much the benefits of where the benefits end we rewrote our first version of the back end converting the full code base from Mongoose orm to Prisma uh it looked great by the way I've used Mongoose and that's uh I also have the exact same problem with Mongoose which is it's great until it's horrifying um all right problems with the uh deployment at code damn damn we used AWS lambdas for hosting our graph coil powered back end the first problem we saw with Prisma which nobody noticed earlier is that Prisma is shipping multiple 10 to 20 megabyte engines mm-hmm dang I've heard about this I've heard that they're they they have quite the load so I didn't know about this I didn't once you go deeper into this I like I was reading this and I was seeing the uh just on the side like the prison documents because I didn't hear about the engines or I really didn't pay attention to the engines until this kind of opened my eyes I'm like what like what what are they using the engine and then when you see how Prisma showcases like the breakdown and the layers of retraction and separation you're like huh I see and then just you go deeper into the hole so all right it was a shock finding out that Prisma uh needs almost a DB engine layer of its own read more about this okay we can look at this later if it's worth it our AWS Lambda deployments were failing because we exceeded 50 megabit uh megabyte limit size how do I mean it's still I I have okay I do have a question before this which is it's 10 to 20 okay so if it's multiple meaning that you you are actually taking up like 40 50 megabytes just by Prisma engines I get that but if it's just one that's 10 to 20 megabytes and you're exceeding the 30 what's the other 30 doing I got a lot of questions they didn't kind of quite specify uh we use es build to build the back end typescript and we had to write multiple patches in the build pipeline to remove extra engines extra Prisma files and node modules and bring the deployment size under 50 megabytes that is crazy this so this is the part where I remember I'm like you should have backed out here this is the point where I'm like you should have stopped using Prisma if you're doing all these patching getting in here to do some crazy stuff but they're like no we should we did it you know we we spent the two weeks patching Prisma let's keep going Lads this doesn't scare me yeah they're like oh that's the worst we're good now like we figured it out so what we have said running as part of our our build step it is it makes sense we remove things with said you're like no dude I'm telling you this is usually where you've gone wrong yeah I like how it's like the engines are like a core part of the Prisma and they're like let's just remove that like we don't like let's just remove a bit of that because we don't need it let's make it fit under our serverless hard limits and we're good guys like this is gonna be great I would like to invert it which is Howard okay multiple engines and you can just remove them and your thing works correctly I am like by like that alone my brain's like this I don't know what's happening like what kind of weird ass car are you driving where you can just like take out an engine you're like don't worry I got a spare one in the back yeah we don't need that one I don't need that one I can drive this thing with three wheels actually it's built for low performance okay okay our second mistake was that we turned on emulated foreign key relation mode in Prisma well I can tell you right away this already sounds like a bad idea because the database is not supposed to emulate it it's supposed to literally have a foreign key for any sort of relational database like I'm just like these are all red flags I'm just like hmm what is prison the right architecture for me right now I feel like there's some issue we've been solving that are kind of weird let's see initially we wanted to use Planet scale for our database wrong mistakes we used to or so but it turns out that it wasn't the best choice for us either but that's another blog post I'd like to read it Prisma generally and emulating foreign key relations in Prisma is a performance nightmare damn here are a few I assume it's because what ends up happening is it makes a squeal query it gets them all back it gets all these emulated foreign keys and then it makes a second follow-up query to the database to like do the relation because you can't quite do it is my guess I have no idea because I I just I don't know what they're doing here are a few things juicy okay I want to get Juiced uh here are a few things that we observed when we tried to test migrations of our data from staging mongodb to staging Planet scale via adapters written in Prisma you know I would just like to take a moment here this sentence is insane do you need abstraction always like at what point have you just abstracted we Prisma our planet scale no and Edge no database database you're like what the hell is it just rate sequels you got this I'm positive you could do this every new insert via Prisma opened a database level transaction uh an extremely weird design Choice by Prisma I mean is it I mean I like the idea of you always have transactional type of operations maybe you can chain them maybe there's some things we don't know about yet we often would exhaust our transaction pool uh on vitess on planet scale yeah yeah uh and and this error uh we will end up a 10 to 20 seconds in migration oh crazy yeah this one to me was like okay yeah that that I can see how that's bad but I was like I'm not too shocked at that you know that first one I'm like okay that's like an appetizer I believe yeah also you know if you have these hard limits any kind of client that's opening up and doing transactions and all that you should have some sort of pool in which you can you know or some sort of concurrent limit something in which you can control this if you needed to but if you're hitting a concurrent limit you've already like you a while ago right like you you like you're you need to back this up and go what have I done wrong here uh we were not dumping insanely wait hold on we were not dumping insanely in the squeal database I don't even know what the sentence means we would hit the above air doing maybe 50 to 100 parallel insert calls in DB some of our collections had records well over 20 million we couldn't wait for days to migrate since downtime was involved here I don't understand why we need days to migrate for downtime you know and you just said you're not dumping insanely into the SQL database and you're like but we couldn't wait days to migrate it's like what yeah also 20 million for me 20 million feels fairly insanely if you're like maybe not a Netflix or at uh twitch right those aren't insane numbers but you know we're also going to use Prisma to handle this stuff with that kind of traffic and that just sounds a little crazy okay there is no concept of squeal level joins in Prisma this is it so it is so it truly is built for something like not postgres because like I mean why would you ever use a back end that is a relational database if you cannot join Yep this this I I was floored when I read this paragraph here I was absolutely floored by it and the fact that they offer these like join methods under the Prisma client but really it's what you said it's like doing one query from one table following that up for the second query and then doing some additional logic from those engines I was like people would do anything but write SQL look at the extents people go to instead of just writing SQL in the code base it's really not hard to be like select ID and hairstyle from hairstyles joined with comb right like no no no no no no we need a client that I can do a JavaScript method that calls this logic that does it for me that's what we need is is squealed declarative I think it might be but I think not I need jsx to do my squeal yeah like what all right this was one of the most shocking revelations to us in some by the way this for me this is like a no-go if you cannot do SQL level joins in a in an orm which this seems kind of crazy this seems like the whole point of an orm is so that you don't have to write squeal but you learn a completely bespoke API this seems crazy that you can't do it okay this was one of those shocking revelations to us in some queries we inspected that supposedly should have used squeal joins or sub queries we discovered that at a low level Prisma was fetching the data from both tables then combining the results in its rust engine okay I don't know why you gotta scare quotes in yeah okay we know rust is fast what were they trying to prove here rust I think it's because everyone's like oh it's rust it's fast and they're just like not fast performance this actually seems it just crazy to me this was a path for absolute trash performance so one reason why this is crazy is that often especially on a client when or like a the middleware server right uh where you're just trying to like get all the data and then hand it back to the client you don't want to do a huge amount of computation there because your goal is to have like as much distributed requests as possible so if you're joining like thousands upon thousands of rows constantly with every request and you're manually doing it you're not even offloading to your database server and then Planet scale that's the point of Planet scales that you don't even do the computation like someone else does it Prime check this out I'm going to link you a Prisma docs to show you how far they are from the actual like database from the actual connection so I just linked you this is this is the architecture from Prisma and the the the blog uh links as well but look at the layers of extractions it's two layers away from your actual SQL database and it's just like that's where you're losing so much performance and getting all this trash like you know you're so far from it then of course when you're gonna hit like actual data you're just gonna be hitting so many bottlenecks because you're two letters away from just connecting to your database and making the sequel I'm sure there's a really good reason to use Prisma I'm sure someone can give me a good reason I just don't currently see it right this this does seem very Overkill considering most people's squeal queries are literally select star from this where IDE insert insert yeah they're like exceptionally simple stuff most people don't like this ain't Big Data you're not doing data analysis you're you're getting data or mutating data exactly wow okay okay okay let's this this we got to keep on going this is this is exciting this path wasn't at this this was a path for absolute trash performance are they saying something against trash death I think so I think it's a shout out for trash dive here yeah shout out I think it's like this is how we mimic his performance at his level a senior engineer by the way all right once we figure this out we were extremely doubtful about how Prisma actually emulates the foreign key constraints really now you start doubting now you're pressing x uh does this perform does it perform cascading select operations on all Associated tables on a uh code operation what's a cud operation I also didn't know that I just read it over I'm like I hope no one asked me that so I'm just like I could and I'm like all right let's see is there supposed to be an r in there like crud on all crud operations hey this is a Cuda operations it's just create update delete forget we don't need those R's here oh see seems like the LLC seems like the only way and since we saw that every insert used a transaction that has a limited pool size I'm surprised you can't turn that off there must be a way to turn it off we were sure it would be a non-negotiate negotiable performance penalty for us on production workloads I love the wording there non-negotiable as if you're talking to like prisma's agent or something it's like you're only gonna get this many pool types that's that's not part of the contract how about this one we'll we will drop squeal joins for a few more transaction pool items no non-negotiable you get what we give you that's it you can get something else you can get our uh we give you a pack you know like a t-shirt that says Prisma on it I I normally don't like on Technologies this much but I guess I never use Prisma so I'm a little bit shocked at exactly what Prisma is doing or what they're I I'm confused at what it's offering do you know what I mean I think it's also offering convenience like again like the client is super super convenient to use in like a JavaScript typescript ecosystem you just pull it in like it's like a hook and you basically just use it kind of anywhere in your code and it's super convenient you have these pre-built methods that allow you to kind of you know not only think too deep into uh you know the sequel itself and it works really well with other things like you know Zod or any other things that are part of the T3 stack but yeah I mean the performance that you'll get on something that at this scale and you know 20 mil like we can argue that's not that much data like I think you start seeing some bottlenecks and the convenience is you know not so much interesting okay yeah I guess I I don't have enough experience with it but like this article paints a very glib picture for its future all right on every insert Prisma opens a transaction where it inserts and then Returns the inserted record too even if I don't want it that's I found this one funny just FYI return yeah it's just like I get one I don't want this does postgres do that by default or does it not do I don't think it does it by default don't you have to call for it yeah you have to call from postgres yeah you have to do it really sure wheel light you have to say like you know insert whatever whatever and also return out the value which I like I mean I like when especially on uh primary Keys when you want to get the primary key like that if it's an auto incrementing index it's nice to be able to say hey give me automatically back the the ID but to get the entire record back just seems like a useless select yeah anyways uh Square results let's see squeal results patching inside its own engine was a deal breaker for us maybe it's somehow optimized it further too but there is no way you are outperforming the database in its own game even if you ship a 15 megabyte rust engine to do so again rustic rust is not going to beat it here further inspection we discover that your code never makes the actual DB call your Prisma code performs a graph wheel a network request uh to the Prisma rust oh whoa it actually does a it does a network request yeah because even if you're on the same machine this still is infinitely more expensive than not doing it like you could also just not do it and it's free or do it and it costs this which then translate your request into an actual DB calls really yep so those two layers of extractions all like it's like a rest API that you're basically calling this query engine you know the rust engine and then that you know makes the SQL equivalent to the actual database okay so it's not just a two litter separate it's like two like it's like a network request and then there's a further a further logic written in Rust and it's like well why do you need all this you know why not just at this point you could just probably go straight to the database you know there's probably a better alternative I think I understand what what's going on here there well I mean I don't understand why but I understand what they're doing the connection pool stuff here and I think they're attempting to optimize it in some sense as being the fastest possible execution which I totally get you want it as fast as possible uh and I think part of the reason why they came to that conclusion is that they're doing joins and so if you're doing programmatic joins doing that in JS would probably be an untenable operation even in the thousands uh versus in Rust you're getting the the speed you know speed speed but you're probably using like because you don't I don't I'm very curious about it because yeah I have so many I have so many Curiosities right now all right we're gonna go back let's keep on how why does it never go back to the thing I was just on all right dropping Planet scale now our planet scale database was able to pass our stress test of inserting and reading multiple records okay that's good it however failed when we used Prisma because of the limited transaction pool inside vitess which was getting exhausted very quickly and after that let's see and after that all further calls to plant scale DB will fail for a few minutes yeah that is just cray-cray however passing on planet scale was because of two reasons no foreign key support by the way Tercel just allows you to write squeal it's just sqlite that's all it is you just write squeal and you get done it and you can and you can foreign key you can build the tables how you want to there's just like none of this stuff all seems nuts to me that these providers don't just give you edge database these managed ideas because it makes it simpler for developers just seem like it's a pain in the ass it makes it super unsimple for developers yep yep I mean again it's it's like the instant convenience factor I think you know I think right away you get convenience because you don't have to worry about the setting up or the infra you just get like you know a string essentially plug that into your code base it all plays nice and and there you go you have things that can process data super quickly but then when you go into you know you know a day later you kind of realize wait a second a lot of stuff's going on here that isn't supposed to be going and I need to probably you know get more more closer to the details closer to the code to make it run how I want to I just made an edge database right there shall I create some tables like ter so super convenient and it can have replications and all that but you don't have to have all that extra stuff you just don't have to have all that other stuff okay anyways it's just like it's just funny because it's a weird Tech choice I don't understand yet uh this is something we overlooked initially but as we worked more and more on the migration and coming from relation data to manga or in mongodb we figured out that it is a huge safety net of not making data corrupt we had a lot of data corruption in mongodb just because we did not respect relations among collections yeah I mean again mongodb mongodbs again it's it's one of those conveniences you trade for pretty much everything else and weird pricing Planet scales prices uh Planet scale prices you on a row reads and row rights per month what is weird is that the row reads is something that nobody controls it is the squeal query planner which determines the query plan which results in how many rows you read or you are reading internally oh that's kind of weird a black box reading can be a little bit confusing like I totally get if it just executes and if you get a million rows it costs a million reads uh remember that row reads are not row returns you can write the wrong query that returns zero rows but you still uh do a full table scan of a million rows and Planet scale will charge you for it interesting see I've never used plan skill too much I don't know too much about it but that seems that has to be wrong this has to be wrong I can't imagine that that that is true if it is true then that's like a huge red flag but if it's not because yeah holy cow no VPC uh appearing mongodb Atlas uh supported VPC peering on dedicated clusters which avoided data flow over the internet Planet scale supported it only on Enterprise plans okay well that's just you gotta pay to play okay yeah yep all right Prisma plus Planet scale built for each other but not for production workloads not for the heavy loads yeah none of those here this is a seething line like this is as tenses I guess Planet scale is fine when we used independently but Prisma might lure you in with its DX I've been lorded by the D talk to the author the way they use the words here like non-negotiable it will lure you in Prisma devs if you're using Prisma don't like this is so great I love the way this is written it's clearly not chat gbt which I think just makes it refreshing additional features like accelerate and tooling okay if you love performance even one bit on heavy production systems avoid using Prisma Prisma PR in shambles right now hey Prisma devrel I'm sorry you know we're not we're not the ones saying this We're Just Don't Kill the Messenger okay prisma's website has a page called why Prisma it says three things raw squeal full control low productivity wait what foreign I see it I see it I see it I see it squeal query Builders High control medium productivity uh orms less control better productivity okay so I see okay okay so this is the old argument that we know better DX is more important than anything else Hardware is super fast you don't have to worry about this stuff which it sounds like you actually know you do have to worry about it here's the actual truth or here's the actual and true truth version of it Ross wheel full control low productivity insane performance query Builders High control medium productivity insane performance orms less control slightly better productivity and syntax Bonkers performance bonkers it's straight Bonkers uh okay Bonkers be good and like what if someone says oh I just built something in the performance with straight Bonkers I'd be like oh that's great that's amazing Bonkers tends to be the the good side so I just when I read that I'm like damn that's good performance yeah I'm like oh okay I want to I want to use Prisma now Bonkers performance and blazingly Bonkers yeah blazingly Bonkers our current stack after evaluating multiple options here's what we ended up with schema uh is written in Prisma using Prisma migration tooling for applying migrations on DB AWS Aurora serverless V2 postcrete AWS Lambda linked to AWS Aurora under private VPC no internet traffic flow uh kaisley keasley for squeal query Builder Okay easily is not a feature complete yet but it has Escape patches that still allow us to keep code Type safe even if we want to write Ross wheel uh at times okay so I'm going to make a quick judgment here again if you just use uh a different language a lot of these problems go away because in go you can't just be like as any right like you can you can't just do that you have to define the thing and so it is typesafe inherently you cannot make it untype safe like this and so same with rust you have to say what it is and when it isn't it returns you in error when you attempt to decode it and there's something about that that is really really nice versus this whole typescript thing which is like well if it's wrong yeah really what you're getting so it's not really type safety as much as it is really nice LSP integration to confidently tell you what the property is even if it's wrong and it'll give you some sort of runtime checks to ensure you're calling the right properties because you have some level of type safety to me this just seems like a whole problem we've created right here I think it's also worth noting here that the authors went off on Prisma but they are still using features of Prisma which we kind of looked over so the schema being written in Prisma is extremely extremely nice you get basically this one file that just defines all your screen uh your schema defines your models your relationships and that is you know completely separate from the actual engine or the client that prism provides and that is very convenient and it's great for migration tools it's great for you know kicking things over to the DB creating everything uh so there is you know besides the the wonderful use cases of Prisma outside of the the schema it the schema is a really good plus of Prisma I have to say Okay so this is obviously a good selling point which is which is migrations are a huge pain in the ass they've always been a pain in the ass there's never been a time where I've migrated a database and thought I'm going to enjoy this it's always terrifying it's it's just the worst experience ever and so this is good to see I like to hear that especially go from like a nosql database to a SQL database like a relation database like that's a nightmare like just going from one relational database to another one that's already like tough no SQL to relation that's like oh like I can't even imagine that yeah wow this is a great article milky great find milky we can find you on Twitch right and Twitter absolutely twitch.tv milky right is it milky Dev on Twitter and YouTube yes it's milky Dove on Twitter and YouTube I couldn't get I couldn't get the domains there I'm in a legal battle with the Milky who owns those domains but we're trying to get that milky we're trying to make the we're trying to migrate from milk it up to milky uh making it seamless is he milking the milk he is he is he drinks skim milk you know he does you know that throw straight skim milk Behavior right there all right uh the Milkman go check out milky right here milky oh sorry I'm reversed right there check him out thank you for joining us this was a great article really appreciate it thank you appreciate appreciate everyone have a great stream I love you guys peace bye

Channel: ThePrimeTime

Views: 143,335

Keywords: programming, computer, software, software engineer, software engineering, program, development, developing, developer, developers, web design, web developer, web development, programmer humor, humor, memes, software memes, engineer, engineering, Regex, regexs, regexes, netflix, vscode, vscode engineer, vscode plugins, Lenovo, customer service

Id: jqhHXe746Ns

Length: 29min 16sec (1756 seconds)

Published: Mon Jul 17 2023