We need to talk about this benchmark

if you're a big JavaScript nerd you've almost certainly seen this Benchmark floating around if you look at it closely you'll see next is really slow like really slow like why would anyone ever use this slow so obviously this went viral and everybody has strong opinions about it myself included what you might not have guessed is that Ryan carni and I are on the same side here this Benchmark doesn't necessarily mean much and when I the guy who pushes react really hard and Ryan the guy who made the fastest framework right now both agree and also perform poorly on this solution like neither of us have good numbers here but that's fine because the thing this is measuring doesn't really matter that much in fact the vanilla react example doesn't even spit out valid HTML so what are we actually benchmarking here great question let's dive straight in this originally was surfaced by Adam rakus because he saw Dax playing with it and the numbers that are here don't look great if you just read this for what it is react in the time frame which I think was 10 seconds was able to do 537 Ops per second sorry so it's 5,300 total so 537 Pages rendered per second but next can only do 41 so this is for when a request is made to the server how quickly the page can be generated by said server and sent to the user so this is obviously really important right like all NEX does is generate pages on the server shouldn't it be really fast at this eh I have a few things I want to emphasize here because this is this is an interesting problem I don't think this Benchmark matters a whole lot and I have a lot of things I can show to show why I should have probably scripted an order of events here but that takes way way too much effort and thought to do so instead I'm just going to show you guys some of the cool things I have found as I been testing this myself researching it getting changes made to The Benchmark Library this is all based on and we'll dig in stick around to the end if you want to see better benchmarks that more realistically represent production workloads cuz I've been working heavily on those too but first I want to look at what others have done as a result of this first off I got accused for dodging things somebody said uh you've been shown the code what now when I was literally at a conference so I just I found that hilarious that just because I've been shown the code that I'm immediately expected to respond and defend every single thing this is a nuanced thing that's going to take a long video look at the bottom to see how long this video is it's absurd because this is a complex problem serers side rendering pages on requests is difficult but importantly the speed of the rendering is not the thing slowing you down if we had a spectrum of the things that make your render performance slow we could say like this is vanilla react let's say react so reacts rendering time takes that long and then next JS rendering takes said 14 times longer right that long the harsh reality is that the actual time this takes to like do everything you would need to do like if you have database query and all these other things is going to look more like that and as such the amount of time you're spending on rendering just doesn't really matter a whole lot and that's the point I want to emphasize here is that while these numbers look bad if you look at them a little tiny any bit closer and you see that we're talking about Nan here it starts to matter a whole whole lot less but I'm not the only one saying this to be very clear I want to be sure that it's known that I'm not just here defending next because I'm sure if you look at the comments right now it's a bunch of people saying oh yeah versel pce the of course he's going to defend it in case anyone here somehow doesn't know this is Ryan carne he's the creator of solid JS which is the thing that wins all of these benchmarks almost always you might have noticed that solid actually isn't winning this Benchmark for once and if I go back to the Benchmark itself which you'll see here solid is not doing great react is eight times slower than Marco and solid is 10 times slower than Marco which makes it a decent bit slower than react but like for solid to be performing worse than react this should be a massive massive red flag for the usefulness of these benchmarks because we all know it's just common knowledge that solid is significantly faster than react so for solid to be this much slower than react and comically slower than Marco suggests that these benchmarks aren't super valid so I'm trying to emphasize here is that these benchmarks are valuable but not for individuals trying to decide what framework to use they're valuable for the people building these Frameworks and even then not that valuable because the percentage of your time in your compute spent doing the render on the server to generate HTML is so small that even 50 Xing it isn't that big of a deal and that's the thing I really want to emphasize is that this number is such a small percentage of your runtime cost that even the fastest framework doesn't optimize for it so let's hear what Ryan has to say because he is he is The Wizard of JS performance there is nobody I would trust more about this than him and he he put this much better than I possibly could have let's talk about benchmarking for a minute because I feel like it's a topic I it's funny enough my stream on benchmarking is probably my least popular stream to date he did touch on something there I want to Riff on a bit there's a couple of these topics that feel like everyone cares a lot about them because they bring them up on Twitter all the time those include things like lock in like uh unexpect really high bills and now benchmarks which is sad to see benchmarks fall into this space I frustrated because I don't feel like these things come up as real critiques even though all of them are real problems I feel like these things come up when somebody doesn't want to use something and they're looking for an excuse to not use it it's okay to just not use it you can say I just don't want to use next that's fine if you don't want to go use something else but when you keep making up these different things to sound like you're making a smart decision it's just weird and the reason I believe this is whenever I make a video covering these topics even if I am on the the mob side of the thing nobody watches them my videos about walk-in have all bombed my videos about the unexpected bill on netfi that won with the crazy bill which was entirely unacceptable by the way that video bombed too both Ryan and my videos about benchmarking have bombed as well nobody actually cares about this topic they just care as they can use it as a weapon to attack the things they don't like this is why I am frustrated and I believe why Ryan is frustrated too because it seems like everybody cares about this so we come out and explain how this actually works and then nobody watches it so if you're actually this far into the video and watching still let me know in the comments and also go check out Ryan's video about benchmarks it's even better than this one I'm want to go here because I wrote I wrote this this is such a good quote by the way benchmarking is easy unintentionally cheating benchmarks is easier creating useful benchmarks is hard having those benchmarks be fair is harder this is clearly a sub tweet reason I want to bring this up or or say this is because people Benchmark stuff a lot and I always hear it with Solid 2 like on the positive side where people are like oh I I took this react app and I or I took this solid example and solid's way faster it's like yeah I mean the thing's not doing anything you're seeing how fast you can update text obviously the thing that doesn't have to diff and literally just sets the text Noe is going to be faster than pretty much any other framework out there right no component render like even less than spel you know like you know this yes if if if I wrote a vanilla routine that just said set this text you know as fast as possible over and over again that'd be even faster than using the reactive system but what I I sometimes isn't always obvious um is that you can make assumptions that actually cheat the benchmarks so to speak right where you do things where like unintentionally you skew skew the results of your benchmark perect we're going to show some fun examples of how that applies here in particular I don't think you guys understand just how many hacks a framework can add in order to make those numbers look better usually they do it by just opting out of all of the features of the framework and having a happy path if you're doing literally nothing in order to make it so you're not even using the framework but the reason certain Frameworks are performing so well on this is they're basically not using the framework they're just rendering the HTML and then letting the client deal with the weight after the fact perfect example of this was some of those early tests uh sorry sorry Aiden um with million JS comparing the performance to react and preact and basically there's some overhead in The Benchmark that kind of skewed it this can also happen and I'm saying this as an investor in million. JS that is very excited about what they're working on the initial Ben marks did not really showcase the the reality of the performance difference there so I yeah I I'm calling out a company that I've thrown thousands of dollars too because I think benchmarks are that important and I think that representing things fairly is important too still love you Aiden you've gotten a lot better about this since when I I first entered the JS framework Benchmark with solid I wasn't doing some things that were required and I didn't realize why and I didn't think they would make a difference and guess what they actually did make a difference um when I implemented it initially um solid was like much faster and I was like thought you haha I knew this would be really fast but the reality of it was you know it wasn't nearly as fast as I thought it was because I would missed some details right it's very easy to make a mistake um and kind of cheat a benchmark I do highly recommend basically everyone watching this if you are interested go check out this part of the stream I'll be sure it's timestamped in the bottom because it's really good but there's a couple specific quotes I'm looking for here right now I'm T whe saying benchmarks are useless but on the other hand lots of benchmarks are useless woo Spitting Fire right benchmarks aren't useless lots of benchmarks are useless this one where it started actually hugely flawed because at least if it's the exact for for multiple reasons right first of all rendering a hello world well it depends on what you're testing let's see here if you're just testing the overhead of the framework then I guess this is fair because rendering hello world doesn't actually test how fast the framework renders right I'll granted this this original test is actually not the worst in that if you're just like how much to start to like if I if I had AWS and I was running AWS and I was like uh you know if I take my cold start plus framework startup on a request or whatever like what what am I dealing with yeah this is an important Point like this Benchmark as a way for framework authors to optimize like really happy paths where we're just spinning out hello world could potentially be useful to catch things like a regression what it is not is a way to make a decision about which framework you should use which is the thing I'm seeing that causes me to make this video now like to be Frank this is something that should stay purely in Ryan carne's world like this should not be leaking over to my channel because he's the framework guy I'm the app guy I'm the one who takes all the cool things he doesn't build really shitty slow web apps on top of it we shouldn't be agreeing this much but we're agreeing this much because everyone else is freaking the out for reasons they don't understand see this duplication thing where you notice all the ones that are times one let me blow this up a bit so you can see it are faster than all the ones that are times two it's because these are the ones that are actually sending data required for hydration yep another important part here is that like the things that go really fast aren't hydrated they're just spinning out HTML on the server the things that have the data duplication have enough data to continue rendering on the client because if I just send you an HTML page nothing can be updated but I send you the HTML page and some data about what was used to generate that page then you can do further rendering and that's just not accounted for here at all I like they added this to The Benchmark to make it clearer but also shows that like in order to make view even faster you have to make it so the client experience is fundamentally different from the server experience these apps actually don't send the data for hydration so you're basically doubling the payload basically either you choose to to use an example that's hydratable or you choose to use an example that's Nots some of these are streaming and some of these are not which also makes a difference too this is an important point I I see people saying that the Benchmark is good because people are making performance improvements off of it which like yeah cool there's per improvements but like how important are these improvements I think Ryan summarizes my frustration here weirdly perfectly you guys Shi JavaScript right you you you hydrate I some people saw this like oh it may view two times faster in this Benchmark well do you think Evan would have missed a 2X gain on something that was actually useful thank you so much Ryan do you think Evan U would have missed a two times Improvement on something that actually mattered let's be real here anybody who thinks this Benchmark is important is also saying evu and the view team are dumb and I don't think that the evu and the view team are some of the hardest working and smartest people in the entire industry so for them to find a 2X win here suggests that the thing that they multiplied by two was not an important number because they would have found it much earlier if it was so yeah this like perfectly encapsulated my frustration because people see these these two times wins and they're like oh my God benchmarks are so important they're making the web faster no they're letting us show off numbers that just don't matter and it makes us feel good about ourselves which is all this is people feeling good about the choices they've already made not actual good decision making or process so what's wrong with these benchmarks why am I so frustrated there's a couple things let's let's start breaking these down one at a time we should probably actually look at the Benchmark and what it does first so I'm on a forked version so we're going to read through this code first and then I'll go play with my Fork right after so in this version the codee's actually pretty simple and I I have to shout out the author of this repo EK and KC he's killing it I can this is as a framework nerd a phenomenal Benchmark of important inw working things that are fun for us to bicker about he did not expect this to blow up the way it did he built this for fun to be part of like the the framework meta conversation and if anything he seems a little bit frustrated that it got as big as it did and I'm sure it must be annoying having PO requests coming in from every obscure framework author trying to add their framework to The Benchmark and their opinions in it's stressful when something like this blows up the way it did so massive shout out and respect to the author this codebase has also been really Pleasant to work in I've been using it a ton over the last few days trying to better understand what's being benchmarked here so the big piece is that we have these handlers the handlers are this array of things being imported or used above that spit out a request response compatible Handler that you can send a request to and get a response from this allows you to take just the Handler and not testing any of the overhead of HTTP or stuff like that and just say hey here's the request that came from the like uh HTTP framework give me back what I would then send through the HTTP framework and it lets you just test that part so we're not testing almost anything else to do with this like runtime we're not testing how quickly the request is parsed how quickly the response is generated how quickly these things are handled by the HTP layer we're not testing anything to do with data anything to do with real async however there is a bit of async and here's where things start to get fun we have in our modules the test data call so this test data is used in all of the things being tested and it is returned as a promise that does ass set immediate so it does get thrown to the top of the or well I guess the bottom of the event Loop but it doesn't have to wait at all it just immediately resolves so everything else the framework does goes after immediately and then we get to the step but because this is async because this returns a promise it forces all of the handlers in all of the Frameworks into async mode if they support it so again welld designed this this does make sense for what it's trying to test which is what is the overhead of a very blank empty render in in these Frameworks so we look at the next examples to better understand how this code's actually being used we import test data we have the app default function that returns table where we actually await test data here and we have the table function here where we actually render the entries and here's the function for an entry the important piece here is we await the data up there and then we throw it in here and we return it pretty simple Benchmark so these numbers look awful I'm not going to pretend they don't I'm not going to pretend seeing next and saying 14.7% slower doesn't suck what I'm saying is it doesn't really matter I'll show you what I mean I've been playing with the Benchmark a bunch changing the formatting of things a little bit but mostly making one important change the one change I made is in the test data module I increased the amount of time it takes because by default it doesn't take any time I just bumped it to 100 milliseconds so now the data that your service is using takes 100 milliseconds to get that could be you're fetching data from a database that could be authenticating the user could be a lot of things and honestly 100 milliseconds for most Services is quite generous chances are it's going to take more than that even just to make one network requestor connect to a database it's not that it takes 100 milliseconds to just fetch data it's that you're doing multiple things in series such as you're authenticating the user and once they're authenticated you're then fetching the data and then you're sanitizing the data and then you're sending it to the user and if any part of that chain happens to be going through a serverless workflow that might be a cold start yeah this could quickly add up I've had services that take 3 to 500 milliseconds to generate the actual data that you need in order to render the page so now that we we bumped this up to 100 milliseconds we can take a look at the difference in The Benchmark performance and that is the only change here to be clear I had a random function that I was playing with to make the range different but I found that that made the Benchmark less reliable because different Frameworks could regularly have different like Min and Maxes to which made it kind of unfair so I chose to make this a fixed number for the sake of getting the most reliable data I could the other change I made I think this one's important too and we're going to be playing with this even further cuz changes were made to the library I added a bit of parallelism previously this Benchmark was testing request request request request in real load you're not getting just one request waiting till it's resolved and then the next request comes in right after you're getting multiple requests at a time and you're handling those some level of concurrency involved this Benchmark wasn't testing that at all and at the time the tiny bench Library it was using did not support parallelism if we go to Tiny bench though I actually cut an issue yesterday because previously it did support concurrency and then they added sequential as the default and then they added a PR that removed the parallel options entirely so I wanted to be able to test this with concurrency and of course Muhammad immediately filed a PR and got it added before I even started the stream Legend absolute Legend So what I was doing for concurrency before is I set this parallel limit of five and it wasn't actually parallel cuz like if you had five requests and the first one resolved the six would wouldn't come in it was in chunks of five and even that I saw pretty significantly different numbers I'll show you what those numbers look like quick and then we're going to change this to use the actual proper currency mode that they just added actually one other important change I made previously they locked this on time I found that that wasn't the most reliable thing especially once you get into the longer runs for some of these so I change it to 100 iterations so it runs these 100 times instead I had also previously commented out a bunch of the Frameworks that I just didn't think were as important for the Benchmark but I left them all in for this I did comment out the uh recently these were split into two sections there is the uh renderers and there is the Frameworks where a renderer something like viewer react is the actual JavaScript library that generates the HTML and the framework is the thing that hosts it on the server so like next or next so I separated those so that it was clearer when you broke them up but I just commented out the part where they run the um one for the Frameworks because I just didn't care yeah here The Benchmark handlers I just commented that out because I the renders is what I didn't care about so I commented that part out I care much more about the framework side cuz that's where the scary numbers were almost done cool so here we see just by adding 100 milliseconds of data fetching we went from a 14x Gap to 1.5x Gap from something that doesn't hydrate the client side I think this highlights just how not big of a deal this is but now I want to try out the new stuff that was added quick once again like huge shout out to everybody involved with this like none of the engineers who are involved in are contributing to this did anything wrong here at all they actually did something pretty dope what I'm upset about is that their work in benchmarking these like nitty-gritty details for fun has been mistaken as reasons to use or not use specific Solutions which it is not if anybody looks at these things and thinks that they're a reason to or not to use these Frameworks they fundamentally misunderstanding the Frameworks and disable all the things I wrote for parallelism because they're stupid comment that out rename this to run now it will behave how it's supposed to it should also be a lot faster I leave that on 100 iterations I might bump that up to 400 cool a run concurrently and this takes a number we'll give it five oh no is it running them all that was not quite what I expected for run concurrently I expected to do five steps at a time in one of the sections rip so close but so far I was hoping to try the new concurrent mode out but despite the great efforts by Muhammad there was a small bug so we might try that later we might not follow me on Twitter or even follow him on Twitter if you want updates on that regardless tiny Li has been awesome to work with and is a huge part of why this has been like viable again with just that 100 millisecond penalty though and the concurrency we're down to 1.5x slower it's significantly less bad than everybody seems to think uh the results are showing raw SSR performance without any data fetching and it doesn't account for data serialization which is why the react code seems faster than solid because vanilla react doesn't bother serializing the data that's handled by Frameworks this is another interesting thing I don't think people get is the idea of like how much a given tool is doing so if we have the like full stack react experience we'll call this this is full stack react the Spectrum has Parts part of the spectrum is going to be react and the other part it's going to be the framework so we could draw the line we'll just pretend for now that it's in the middle so I'll put this in the middle we say on one side is react and on the other side is the framework next remix Etc some Frameworks build more into the framework side so if we compare this to like solid you'll see that like solid does a lot more of the work solid starts like a relatively small addition on top compared to something like react where the react part of the full stack experience is actually quite small they Define the protocols and the expectations that react has but the vast majority of the work is things the framework does how do you get the data to react how do you respond to the request that the react page is making how do you stream in the pieces that the react server components use and utilize how do you handle server actions how do you bundle it all of these things are problems that the framework has to solve for and when you're just using react without that you're not getting any of the things that we now expect with full stack react if we look at the source code for the react example I think you'll understand what I'm saying here because it is very very dumb boring and simple here we import the test data we bind a ref for it they actually use suspense in it which is nice at least then they use the uh use hook which is the react way of handling a promise I've yet to get this working in almost anything specifically whenever I use use on client it bombs so why does it work here well we're not actually shipping anything to client here that's why we're rendering this on the server so function table returns T data and then we get the actual table rendering by the way mohhamad if you're still here I hope you're not stressing out like I I know how stressful it could be to have your work Showcase in front of thousands for the first time don't worry about it you're killing it and this is awesome stuff I'm like actually really pumped with this Library what the why can't it what broke what did I change that it's oh is it because everything expects an array now and it's not an array anymore oh yeah this is a mistake I made yeah the response zero I that up cool this is what happens when I write vanilla JS this is my fault entirely edit length error was a skill issue on my part and I want to try the concurrent thing again now too everybody loves to use the benchmarks that their tools enabled to complain about but nobody actually checks the benchmarks like the tools that make them possible oh look at that when I disabled the concurrency we're actually down to 1.28 times slower for next so like yeah hopefully you guys are starting to get the point here that these numbers don't actually mean too much oh I killed it I I wanted that open though God damn it that was a dumb mistake on my part I want to go to the react one guys this is going to be so awful to edit I feel so bad I I've posted the results once or twice since changing the Benchmark up and people were very mad at me so that's why I'm doing this video so I can at least have them give them something we point at this is some really good HTML that we just got back but if we go to here cool so here is the code we get back from the react Runner it has all that random data what you'll notice here is that there's almost no JavaScript here at all all so all there is here is like a very minimal react like data loading bit for the suspense and that's it there is no other JavaScript loading which is important because if data changes or any other things occur on this page it does not have the necessary context to do anything with it it doesn't even have like head there is nothing in here other than that one script tag that is actually useful for running on the client so it hasn't embedded any data it hasn't done anything to the data that we have above here it just spit out the HTML and it does that with a very simple reactdom server. render to pipeable stream that's it this is the framework in the vanilla react example there is no framework it's just the pipeable stream we don't even have the ability to run the JavaScript on the client because we're not shipping anything this is just the generation side whereas with next if we look at what the next code spits out we actually first off get an HTML page because it's an actual HTML document so huge win already the other page wasn't even HTML because it's just piping the Stream so now we have valid HTML first and foremost but if we go in here you'll see we have a lot of other useful stuff we have this script that includes a lot of things that are already here so we have this string if I select this I'll command f for it and you'll see that's an element here so why do we have that there as well as in the JS isn't that like unnecessary why do we have all the data twice because it means we can update the data it means we can get changes it means we can process changes it means we have the things to mount unmount and play with our website it's Dynamic and we have things we can do but importantly you see see all this crazy escaping and the server had to escape all of those things and put them in a format that they can be embedded this is important additional work because it lets you navigate and actually use the page it also enables things like pre-etching so if you have multiple routes you can go to one and the others will load in the background so when you go to it loads faster it also means that if you need data in order to switch between those routes and render the components without having to go all the way to the server for the whole HTML you have that it's a really important pattern that is used in most Frameworks now and I mean that like the only thing that doesn't do this that has good client and server behaviors arguably is HTM X but also there's crazy stuff going on in the quick world I have a whole video covering that if you're curious look for my quick two vid but you need something to allow the client to know what data exists and what it's going to use for those things Astro doesn't count because as is not a client side framework Aster doesn't run JavaScript but yes this is a huge part that's missing is twice the data is being included and it has to generate this in a format that actually is compatible with JavaScript and is serialized such that it's safe to over the wire these are not simple problems and they do a lot of work to make sure these things are good but yes if you're comparing this to a crappy HTML generator that isn't even valid HTML it's going to be slower absolutely it's going to be slower but that's why I don't think it's a fair comparison cool while that is running one other fun thing I learned while I was doing this these benchmarks went so viral that people were looking at Benchmark JS and they finally decided to Archive it it hasn't received an update for like over 6 years but just now just a few days ago they finally got killed so once again congrats SL I'm sorry to our friend Muhammad working on Tiny bench because you went from like a nice modern option to like the standard overnight congrats good luck I do not envy the issues you're going to be seeing popping up constantly over the next few months you build something awesome be proud of it cool look at that once we turn on concurrency the numbers are like hilariously close now like hilariously so I didn't expect him to get that much closer honestly I'm a bit surprised and also for um this is Marco right for for Marco's performance to go to once you're running things in parallel none of this is stuff I would have expected so yeah the more you make this resemble the real world the less these benchmarks matter is the thing I've noticed quickly like if you throw out the real world and you're just testing this one thing and that one thing for different Frameworks accounts for different stuff that you're not doing it's going to be an issue oh m mfng is an RSC thing oh my bad didn't know that I have no idea what that is I guess that if that's a new server component thing I'll definitely check that out in the future but it seems like it does not handle the concurrent stuff as well as I would have expected anyways I want to talk about the things that we've actually seen improved as a result we see in this jarky tweet framework users micro benchmarks are useless framework authors we fixed a real bug after looking at micro benchmarks yes there are things framework authors can find from this but the fact that a 2X win wasn't found until now shows how little these matter I don't know view well enough to go read the PRS and explain them but I do know next well enough to read the PRS and explain them so we're going to take a look at the first of what will likely be an arc of the next team getting nerd sniped into caring about this one it doesn't actually matter that much and here we see get server Reser HTML has been improved as well as create head insertion transformation stream this PR improves the server rendering performance a bit especially in the app router it has mainly two changes a rough benchmark test with a 300 kiloby lur Ipson page and SSR is about 18% faster on it as well as these two improvements this is the things you're going to notice here this is why I don't care the first step is that we we avoid an extra render to readable stream if we already know the content will be empty we avoided a stream await all ready for better parallelism with stream to string and we increase the progressive chunk size these are changes that arguably do not matter at all in 99.95% of apps they're literally adding opt out things so if we know this we skip this path like here's one real optimization we're not calling the path like generator function because it doesn't change within this Loop that's an optimization cool we've now saved oursel that check but also if it's the current one we just have an escape for the slash current page it just these are the types of like things they're doing where very specific use cases like testing the root path with really basic rendering is slightly faster now because that's what this Benchmark was checking for and we switch this uh render server insert HTML to just be the server inserted HTML object that we call above here and we skip the step now if we know the content is empty this is the the big change where I'm assuming most of that performance win came from is if you're not actually inserting HTML from the server to the client we skip this step the whole point of using App router is that you're inserting server HTML to the client so if you're using App router and you're not actually using App router here you go it's slightly faster cool but this is the point there's no need to wait for the stream to be ready like calling a wait stream all because stream to string will wait and decode the stream progressively with better parallelism Co another small but valuable win they just deleted this await stream. already because stream to string already handles that so by deleting an await they're stopping one another additional hosary check and then Jim Hopson another low hanging fruit there is that maybe we can just call render to string in some places where we know there won't be anything that needs to be streamed again they wrote all this code in a way that it is runtime agnostic and handles all of the crazy parallel server rendering things you can do in next but that means if you turn off or don't use all of those features the code that sends the response is still running that code so for them to optimize these things the solution is having escapes where if you're not using streaming or you're not using acing stuff or you're not even using server components that you can skip those parts but they wrote this in a way that is runtime agnostic and feature set agnostic which means if you're using none of next features yeah it's a little bit slower than if you're not using next but if you're not using next features don't use next like it's okay to not use next you don't have to use this framework if you're using none of the features that make the one not really meaningful Benchmark slightly slower don't use next if you can reasonably write your project in HTM X and you're already familiar with it go ahead no one's going to talk this isn't comparing that type of thing though this is the features that next enables the server component behaviors that next introduces that allow your performance to be significantly better than it was for your users very important that's what's so cool about this but these changes aren't really important and are going to take a lot of time as shooting says here this would require a large change but it's definitely worth it but it's currently blocked by their Edge react server not having rendered a string exported again if they want react to run in these other environments like the edge like Cloud flare workers these functions don't always work the same way on all of these platforms and what they built here is a solution that works on all of these platforms so could Shu go refactor a bunch of the code to make this Benchmark slightly faster and also keep it running on edge sure could they drop Edge support and immediately Slaughter this Benchmark possibly but they're not doing that because they're focused on building a framework that allows for people to build real quality production software I do want to talk more about what Jimmy's had to say though for those who don't know Jimmy's the engineering manager on nextjs great dude again versel is not paying me to say this at all we have not talked about this whatsoever this is my honest take on these things but I wanted to read his response to the drama because I think it's pretty good for those curious on why app router looks worse on those benchmarks I can offer a few technical reasons first a disclaimer benchmarks are good but they never paint a whole picture as I showed in funny enough you SI me as I showed in one of my tweets a simple slow fetch totally derails the numbers and makes every framework look pretty much just as slow this likely happens for 99% of your apps if you're really concerned with raw throughput then maybe look for a lower level backend framework I do think NEX has a lot more qualities to offer than ra performance yes if you actually think these benchmarks matter you should first off be making a lot of jokes about PHP because up until recently PHP was hilariously slow and it went from hilariously slow to almost usable but if you care a lot about how many times your server can generate an HTML page go right didn't go or rust maybe C or C++ but these numbers don't matter that much that's why we don't normally talk about them they just look bad in the screenshot and it makes the thing that we don't like easy to make fun of but that's the only reason it's went viral is we want to make fun of next sorry on the raw performance you can divide the time spent on a request roughly like this the routing part is done by the next router the rendering is done by the react server component server and the plumbing is done by next to make sure the response gets piped and streamed correctly to the user and if you don't think this part's necessary might I remind you when we last ran the Benchmark we don't get valid HTML back from the react so if you want react on the server to generate valid HTML it's a necessary step the next routing part is common to both pages and app router it is slower than it could be but it's not a big bottleneck we do have plans for a rewrite but app router stability in DX comes first another important point the amount of times that people myself included complain about instability in next in the overall nextjs experience is huge and it should be because when they make these huge changes and things break it is infuriating and it makes people bounce off the framework I've had my own frustrations I had one recently I'll even demo it quick so I have this parallel route open I have this text it's not even blue anymore I already Chang it back to White so I have to refresh to get that to persist now it shows it as white now if I change this to text blue save it it worked that time and that time it didn't because it's not in the bundle do you see the point I'm trying to make these are hard problems to solve and previously when I made these saves it would just kick you to the other view so it would be as though I refreshed here and even funnier when I was editing on this page which I can do now it would sometimes kick me back to the modal view there are so many of these weird bugs that have existed in next not because next is this evil bad thing that's full of terrible engineering rather because making a framework that does all the things next does while providing a good DX is a difficult problem and their priority is fixing these types of bugs so that we can benefit from the framework as much as possible and overall I think they've done a great job at that I just want to emphasize the types of bugs that they're focused on solving aren't things people are going to be outraged about on Twitter but they're much more important for the day-to-day experience using the framework like I didn't see anybody shouting at the mountain tops about how the parallel routing breaks hot module reloading specifically when you're using uh Das Das turbo which oh that's probably why it wasn't breaking as hard as I expected yeah you get the point fixing these painful small things might not get you a lot of cloud on Twitter but they're really important and that's been the focus for the team for a while not these unimportant benchmarks I will say this has bit them a few times like the dev experience using webpack with next was unacceptably slow when app router dropped it was beyond frustrating having like changes you were making not appear in the browser until seconds later like running out of memory was awful and the reason they didn't focus on that was admittedly similar to this where their focus was both making the best DX possible for app rouder as well as a bet on the long-term investment Into Turbo pack they would have made webpack faster if turbo pack was known to take as long as it was going to take but they thought it would come out soon so they focused on just making a good DX and let turbo pack handle the rest that's kind of what's Happening Here the difference is it matters even less so I'm a little concerned that the efforts of the next team making next the best experience possible are going to be sidetracked in a performance War around a number that doesn't actually matter that much yeah so more info on why it's slow for the server component rendering part this is where things get a bit interesting one reason why the performance is not one to one with a classic react SSR pass as shown in the Benchmark is that you can think of it as an additive separate step you render your server components and then you SSR that as well as the client code to HTML and you send it over today there's a bit of overhead in these two steps but I know that Josh has some plans to overhaul and improve this in react land another bottleneck in NEX perf is that we use the webstream implementation of the server components and SSR renderer mostly because it was easier to share code between node and Edge this is again why I'm so annoyed because they're making decisions to make next much more flexible and powerful and to be frank nice to use even outside of our cell the fact that you can run next on edge runtime means you can run next in web workers which means you can run next on something like cloudflare they could make this faster for node but by making it faster for node they would also be making it incompatible with Edge should they do that I would hope everyone here would say no there are a lot of stream operations during render in next and while some of the overhead is on us I believe that the node web streams are not fully optimized yet and there's some overhead when using them I also love the ownership while some of the overhead is on us node web streams are not fully optimized yes something I actually tried but I couldn't cuz getting all of the stuff set up really was annoying and I might do it for a future video so I wanted to actually try running next on edge runtime locally to see what the performance difference looked like if there was one at all just out of curiosity my gut is that node would be slightly better optimized still but I was really curious and didn't have a chance to test that someone else does let me know and I'll pin your comment one thing we're exploring is potentially refactoring the whole pipeline to use node streams instead which are more battle tested fair but again web streams are standard node streams aren't it kind of sucks that next's attempts to follow web standards are getting them here cuz if you guys remember next got so much for not following web standards and then they started and nobody nobody cared it's another one of those things that falls in the bucket of stuff people don't really care about like benchmarks or like lockin where they just say it so they have an excuse for not using the thing that they didn't want to use and again if you don't want to use next that's fine but they went really hard adding web standards cuz people were complaining about it nobody cared and now I'm scared the Michael really hard trying to fix the performance stuff here and no one will care but part of why they're having the performance issue in the first place is they're using web standards instead of node standards yeah so again they could make it so they're not following standards and it would be slightly faster similar to how the react example we just saw again I hate to keep doing this but I really want to emphasize serving a non HTML wrapped stream of TRS and TDs is not following web standards this is a dump of elements that aren't even part of a like HTML document this is not a meaningful comparison of anything and pretending it is a fundamental misunderstanding of web standards and the web as a whole so yes if you're just spitting out Mal formatted HTML you can make something faster than next and if you make something that doesn't follow web standards you can also make something faster than next but if you want to have a good experience both for the developers and for the users I still truly believe NEX is making incredible compromises and until I see other solutions that offer the same functionality both for devs and importantly for users it's hard for me to care like it just this isn't where your bottlenecks are if you built a service where your bottleneck primarily by the amount of time it takes for the server to generate the HTML rewrite it and go totally cool with that but none of us are actually bottlenecked on this otherwise EV and you wouldn't have found a 2X win nobody cares this isn't a real number that means much of anything beyond a thing for framework nerds to micro optimize for fun and like if this never escaped the circle of framework people if Evan just like quietly did this cuz he discovered that cool but the fact that every framework author is effectively being pressured into optimizing this number that doesn't matter that's incredibly frustrating to me and that's why I took the time to sit down play with these benchmarks and make this video I want to run this once or twice more before we wrap up because I'm just curious if I can consistently see the much better numbers now that I have the concurrency enabled I have it running 400 it should take a minute but not too long reun moving removing the delay I can do that waitting for that to run maybe shouldn't have had it run 400 times and that's pretty consistent yeah that's actually really consistent I was expecting the numbers to be a a bigger Gap there but it turns out when you run things concurrently which is how we actually deal with network requests that the performance on these things is a lot closer than it was before too so let's change uh per request the test data to not be 100 milliseconds um I should probably switch this back to the original code let me do that slow drop this guy instead have to rebuild this is why I want to have turbo pack because it'll make these steps a bit faster cool now we're not actually blocking we'll see how fast it is just with the concurrency of five down to just n .5 times slower with the concurrency enabled interesting that remix is so fast as soon as you can run multiple tasks at once like I did not expect remix to suddenly be like a champ what is the remix one even serving is it also doing the mouth format HTML thing no it's not it's properly formatted HTML oh remix is using defer so it's not blocking on render to string that makes sense yeah I did not expect these numbers to be so much better simply by running them concurrently in the end I hope you guys understand how little these benchmarks matter for most developers they are really cool for us to nerd out about on the framework side but sometimes our nerdiness spreads too far and people misread the stuff that we're talking about so if you're using these benchmarks to pick your framework you're using them wrong if you're using these benchmarks to make your framework perform better cool have fun with it you're using these Frameworks to shame others for their technical decisions you should feel bad about yourself because that is never the right way to go about these things otherwise again we'd all just be making fun of PHP because it has been the slowest Solution on the server for a long time anyways until next time peace nerds

Channel: Theo - t3․gg

Views: 64,740

Keywords: web development, full stack, typescript, javascript, react, programming, programmer, theo, t3 stack, t3, t3.gg, t3dotgg

Id: -cZP6CZdsBs

Length: 42min 25sec (2545 seconds)

Published: Thu Apr 18 2024