Refterm v2 - Resource usage, binary splat, glyph sizing, and more

hey everybody i hope you had fun watching the excuse parade roll by i wanted to show you some rough term v2 stuff what i tried to do was just take a look at all of what the biggest floats in the excuse parade were and just go ahead and fix those because you know obviously i don't really think there was much left of the imagination the previous demo but for people who had things that concerned them i went ahead and took care of them in v2 this weekend so if you take a look at the screen right now what you can see is there's windows terminal on the right and there's ref term v2 on the left and you'll notice some things we've got strikethrough and underline now and some more vt code parsing i also have a unicode thing that comes up on the original screen as well just for testing purposes i find it makes it a little easier if you're happy to play with the ref term source code so some of the things that i saw in the excuse parade were people were worried about the memory usage and the way that they were measuring the memory usage was by looking here at the little memory uh thing in the task manager so i thought i'd bring that up and demonstrate what the current memory usage actually is on ref term v2 so what you can see is windows terminal preview if you just start it it uses 23.4 megabytes of physical memory and rough term release clang uses 44.3 of physical memory now i don't think it's very useful to compare those numbers because you haven't done anything yet so i mean to me you have to actually run the things see how much memory the physical memory they're actually asking for you know in your system so let's go ahead and do that i'm going to start over here by just navigating to the ref term test directory in windows terminal and i'm just going to go ahead and have it use a splat of the gigabyte text file so here is it splatting the gigabyte text file and what you can see is that immediately upon trying to splat the gigabyte text file your memory spikes up to 280 megabytes in windows terminal preview and it will remain there for the entire duration of you know the the the splatting of the text file and you can also see how much resource usage uh is happening on the processing side so you can see that the cpu is at 22 and the gpu is at 1.6 on this machine now of course uh this is a multi-processor machine so you know when you see things like cpu usage what you're seeing is percentage of the total uh core capability it's not you know uh so so pegging a single core at 100 is going to show up as a lower percent in your cpu here and all that kind of stuff but just to give you a baseline this is something that i would assume everyone in the excuse parade accepts is an actual terminal because it's the terminal that microsoft is currently shipping as its most modern terminal so hopefully uh everyone agrees that 280 megabytes and 23 of the cpu for um 300 seconds or something uh is the kind of resource users you would expect if you're splatting a gigabyte text file with what you currently have now if we take ref term on the other hand and we go ahead and do that exact same thing so i go ahead and do a splat two and remember this is not using fast pipes this is just using there's no trickery here at all it's all going through conhost whatsoever although i did figure some things out which is why i'm using splat 2 that i'll talk about in a second but so we go ahead and use the same exact command on the other side and what you can see here is that rough term actually it is very hard to see because it's very brief uh but ref term does go ahead and use um you know it has a memory spike when you actually do that initial splat and one of the weird things that you'll notice if you use ref term or try it out is the next time i do it it doesn't so it does use cpu and gpu to like update the screen because of course it has to only again when it's actually doing anything because again the excuse parade had a problem with that so i just went ahead and instead of rendering at 7000 frames a second i just don't render a new frame unless something's changed so that that was the fix it's like you know one you know it's more than one line of code because you need some handles but you know it's message weight for multiple objects was that's that's the way you do that right so what you can see is there's a little bit of a weirdness so windows terminal spikes up to 300 megabytes or so when it's trying to do this uh and it does that always so it's going to be using up the 300 megabytes of your physical memory whenever it's doing anything and of course it'll use a tremendous amount of cpu and not very much gpu which i guess makes some sense uh given the the speed of it but what's interesting about ref term is ref term actually is completely flat memory usage pretty much all the time as you can see so it's it's pretty flat on the memory usage it doesn't really take much memory at all during actual processing in fact it really shouldn't change its memory usage whatsoever but the first time that you use it to actually splat something it does and it's you know kind of interesting question like what's going on there and i haven't had time to actually figure it out so i can't tell you for sure but i strongly suspect that what that spike is because i'm not actually allocating any memory all the memory is allocated once and then it never there's no allocations during the run of ref term so what i think is actually happening there is when you have not actually splatted anything yet uh you haven't cached any glyphs in ref term so i think what's actually happening there when you go and do that run is it's actually what you're measuring is the direct right memory spike because it has to call direct right to generate the glyphs so it may well be that not only is windows terminal the actual windows terminal code base might not be responsible for that huge memory spike right they may only be responsible for part of it it may actually be direct right that's responsible for a lot of that big windows terminal spike because ref term before it builds the cache it that first time it gets a little spike too not as big but it's there so i do think maybe there's some stuff with direct write happening where it's like caching glyphs it's doing it's allocating footprint for itself or something so i don't really know what's going on there'd be interesting if someone's got time to look into that see what's going on it's very interesting but point being ref turn will never do that again once it's got its cache built for what you happen to be using you won't be suffering windows terminal every time and also once you actually do one i don't know if this is scrollback buffer or just i i don't know because i haven't tried to profile their code base they actually go up to a very large they go up to about three times four times uh ref terms memory usage uh permanently so they're you know it's not doing anything right now but still occupying uh 150 megabytes of memory and if i run this again so if i say you know splatt2gig.text what you can see is it's right back up there so you know unlike ref term which which pretty much doesn't do that although for some reason i don't know why task manager turns us not into an app anymore at that point so it's hard to see i don't know if maybe it thinks it's in the background you know who knows why windows does what it does i have no idea i don't know if we can get it to show us that oh it kind of did that time for whatever reason maybe you just need to like coax it a little bit i'm not sure yeah i i don't know what maybe me here we go maybe we just do that and we ignore the fact that everybody else is taking a ton of memory and maybe we'll show up in there uh in a way we could actually see what the spike was if there was one i don't know so i don't i don't think there is one so let's try cpu gpu because we know we'll spike that right we'll see if it can list it um yeah so there you go there you go right you saw it really quickly there it's kind of hard to see but you can see it come up there 44 right so even during that we just don't spike again because once we've got the cache built we don't have to call windows and windows is the thing that's wasting all this memory it's not ref term because we don't allocate any memory during the actual processing so hopefully that takes care of resource usage and again those things are very controllable uh you can tune how big ref terms like caches are and also it's really not that useful to look at this because it you know you can also tune how big your gpu footprint is because these things have gpu focus as well for the back buffer and also for um like the glyph cache and that sort of thing and so those don't won't really even show up in here so it's kind of really not even a great way to compare them if you really wanted to go down the route of like let's analyze resource usages you wouldn't really do it this way but this is how people were complaining about it in the excuse parade so i'm just trying to show you that okay if you're measuring it with this i took care of any of the concerns that you had i don't allocate giant caches anymore which is what i was doing for testing i just allocate the size you would need in practice um and so the memory usage is quite small and a lot of that memory usage is things like the scroll back buffer which is 16 megabytes worth of scrollback in ref term and also uh all the stuff that like directx does that i don't have control over like there's there's just a whole bunch of things that i can't control because in order to put up a drawing service like that you know that's how much it allocates so you really don't have a control over a lot of those things uh now anyway uh as usual you know the performance is very good on ref term but it you know when i show the demo originally but it's it's even better now so one of the things i did in reference b2 is i hadn't optimized any code and in reference b1 but in ref term v2 i did just one block of optimization i wrote a little 20 line piece of code designed to make it so that the when data is flowing through ref term at a high speed it does not take a lot of time to pass that data through if it doesn't detect that there's anything in it so if it can tell that there's no like complex uh you know um if there's no things like vt codes that might move the cursor if there's no things like backslash ends uh if you know if if there's nothing detected if it's just text it'll skip over 16 bytes at a time in order to get out of the way of the ingress and then only when it actually sees that there might be a control code that it needs to pay attention to does it actually downshift to looking at the individual bytes and so what you can see is we've actually improved the total sync time for things like the gigabyte text file really dramatically you can see that it's at 1.3 gigabytes per second now um and that is way way better than it used to be and there's two reasons for that so number one is that optimization i did and number two is that through trial and error i was able to figure out that actually standard i o causes a lot of problems for conhost so if you remember the previous demo i talked about the fact that anything that you do in windows has this problem where if you're outputting the console you may accidentally uh be getting slowed down by the fact that conhost seems to introduce a lot of overhead and so i had that thing about fast pipes where if you turn on fast pipes you will get a bypass of conhost and you can get a lot faster and i use that and i showed like the difference between and the difference was pretty substantial i figured out after the fact that it's not just conhost it's con host when you write to conhost in a particular way and the way i figured this out was i tried switching from using standard io to just using straightforward windows write file calls and the reason that i wanted to do that was i was just like you know let me just see what's happening here because i want to test it i don't feel like i'm getting as much bang through as i could and also i wanted to test fast pipe that way and i wanted to make sure standard i o isn't swimming with fast pipe because standard io you know it does things like serial f translation and it's just it's inserting something in that pipeline so one of the things i did is i just switched to windows write file and what i found was actually switching to it as right file it got way faster without without fast pipes and i mentioned this to martins and jfhs on the myrocket discord and uh martin's went and looked at it as well and he said that he thought uh it was because of the 5k buffer size there's like a 5k buffer or something that happens inside the crt when you're doing crlf translation which it will by default uh if you don't set the mode to binary uh it'll go in there and and that causes the rights to be much smaller and so even though small writes don't seem to be much of a problem for windows kernel pipes small rights do seem to be a problem for conhost so conhost itself seems to have some problem transiting small rights through there and if you can make those rights bigger it seems to benefit so one of the really great things that came out of ref turn b2 is now we know if you just set your console mode to binary you actually don't have to worry about kano slowing you down and look even windows terminal gets faster because the con host part of it got faster too because remember that uh was a slower timing i won't run it again but that was the slower times like 340 or something last time we checked in on stuff like this right um and it to be fair i'd have to run it you know i don't know how sensitive is to windows size that's that but you could run splat and then splat two to see the difference but point being you can speed up your con host time just by doing that either switching to write file so you use windows output directly so you bypass standard io or you just set your standard i o mode to binary so it's not doing that checking and then do big writes and off you go so there's some actionable stuff there even if ref term is not what you're playing with there may be things you can do uh you know for other terminals uh like kitty a lot of kitty a lot of people have mentioned as a good uh alternative terminal um i know people mention alacrity but that just crashed when you tried to run turban it doesn't seem very good maybe it'll get better but kitty supposedly pretty good maybe things like that you could go look and you may get better speed out of your existing terminal just by taking a few steps to avoid that small right problem so something to think about now one of the things that uh i saw in the excuse parade is people you know wanted robust input parsing i i don't know why they think a rendering demo should have robust input parsing um but i did that too uh the only reason that it wasn't robust in the last version was there was a typo i had a while loop that should have been a do while loop so i fixed that as far as i know that was the only problem so uh here's a 12 gigabyte uh 12 gigabyte not one gigabyte 12 gigabyte binary file uh i'll go ahead and splat to that uh out to the console and actually you know what i shouldn't probably do that let me break that for a second because what i'm going to actually do is do splat 2 and i'm going to turn vt codes on in splat 2. the only reason i'm going to do this is because otherwise you know just to prove that i'm not doing anything weird i always actually fully parse vt codes that's why you can see stuff happens in here you get like strike through turned on and blink turns on and reverse video turns on and then it turns off and all that stuff it's actually parsing the entire thing when you send it down it's not cheating um so i'm going to let's tells flat 2 with the minus vt tag i'm just going to tell it hey at the end before you print the summary send a clear code because otherwise you know it could be all screwed up and we won't be able to see what the actual throughput is so let me go ahead and do uh splat vt uh minus vt large 12 bin and this is a 12 gigabyte binary file uh and so just for that part of the excuse parade that paraded by with that as its primary uh problem this uh hopefully would take care of that i'm happy to fix any other bugs that people find in the parsing but i don't think it's any worse than windows terminal is certainly um and so there you go and it gets pretty good speed 0.7 gigabytes per second it could be better uh why is it that slow i have no idea it could partially uh be just the fact that there's more random codes that happen in there and so there's more power for downshifting i don't know again this isn't really optimized code i put in one little thing to get rid of something i knew would be a bottleneck but i have not profiled this code still to this day it is unoptimized code for the most part pretty much throughout the entire pipeline um i can turn fast pipes on just so you can see the difference so you can see we had you know around point seven gigabytes a second with fast pipes off which is again great um so that's pretty cool and oh yeah i'm always going to forget that i should probably just make it do the vt code thing by default but i always forget um so 0.7 gigabytes per second with fast pipe uh off here's with fast pipe on um and we can just see what the difference is but i again once you fix that problem with the small writes conhost really kind of gets out of your way and typically i think i've seen like a 10 difference in speed um you know that's that's maybe a little more than a 10 difference um but again it's not the kind of major like 10x stuff we were seeing before so it's pretty cool right i think that's that's good stuff now again you know like i can't really run tests for you because windows terminal isn't going to be able to do this uh if you tell it to splat a 12 gigabyte binary file you know you'll you'd just be here all day um i i don't want to bother letting this run but you know obviously it's going to take a lot more than 13 seconds which you can see uh by the fact that if i let it run for 13 seconds and then kill it right it's much slower than ref term i don't even know how much of the file it's gotten through but not very much right um so in terms of other things uh again just in terms of parser robustness uh and things like that i i really again don't quite understand uh why people are focusing on that for a rendering prototype but even if they are focusing on that uh for a rendering prototype i think we've kind of got windows terminal into a mode where maybe the code page got set or something i don't know um i mean again this is a perfect example you know people in the excuse parade were claiming that ref term had a problem because it couldn't splat a you know a binary file this is what happens if you splat a binary file in windows terminal so you definitely weren't doing this right because this is not really usable you know i don't even know how to get it out of this state um it probably sets the code page or something because i don't know maybe there are control codes that aren't vt codes wouldn't have been turned on here um so i i don't actually know what that would have been parsing but they may be like old windows console control codes that i don't know about or something that allow you to change that i have no idea so that's what happened there let me restart windows terminal preview and again put it up here so in terms of again you know assessing whether things are good excuses or are not for things i just don't find many of these excuses compelling about parsing and things like that because if i actually run these tests on windows terminal it fails at almost all of them so for example another thing i did was i made a thing where you could just send one long line down so here's what happens if you just send one long line to windows terminal like it basically hangs it eventually kind of figures out how to start running again and then hangs again and stuff like that and this is just one line without a backslash n and it you know that's what it does right uh again uh we went through and made sure this was handled on ref term just to prove that there's nothing weird about ref term here's the long line it's all there you can scroll back up through as much of it as you want to uh and it's all dynamically wrapped right so we you know we're not doing any weird like cheating stuff uh you can dynamically wrap the line if you want to um and it will still you know keep it exactly as it should uh display right i mean that's even the correct end of the line like it's not even changing the position right so it's figuring out how to actually make sure that works so yeah i mean i am i would say i don't think there should be many excuses left for the excuse parade but it's always kind of hard uh to predict what kinds of things they will come up with uh the excuse parade seems to have floats for miles and you know even if you run two things side by side and demonstrate that the one that you're demoing is you know vastly more compliant with most user scenarios that you might want they're still going to go dig up some random thing that you didn't think of as the whole reason why none of it should be applicable or something right so i can't really do much about that but what i can say is also on rough term v2 i put in a couple other things if i can i don't know if i can actually well i don't know um whatever windows terminal is doing here is what it's doing hopefully at some point there it goes okay um so we also have some things like you know when you do something like a splat 2 of our simple test which just has some things in it right uh you can kind of see that you know i don't know i don't know what people think should happen on a terminal sometimes so that's one kind of thing where it's hard to say what should actually occur but there are some things we would say which probably shouldn't occur right um and some of those are things like this where like that's not really what that arabic should look like at all right and the reason that it's looking like that is because windows terminal doesn't handle arabic diacriticals properly and far be it for me to really talk about that because i'm not a unicode expert and i'm definitely not an arabic expert i can't even read it but i've had i've been nice it's been nice because people have helped me out um people who do speak arabic and can read it uh and are more familiar with it have have you know sort of guided me through like what's supposed to happen in certain cases so i tried to make sure that everything in ref term can handle it so this is not what this is supposed to look like so again like pointing to things that refter maybe does or doesn't do and suggesting that somehow these uh make the the enterprise illegitimate seems crazy because i can point to all these things that do not happen in windows terminal correctly and nobody is claiming that that delegitimizes that as a terminal so it's a bit weird but i'm also happy to say that although i have not had time to really focus on this i would like to maybe sometime in the future separate from this whole ref term nonsense but i just like to kind of know it better for my own self i did have time to fix uh the glyph presentation stuff that was where direct write was doing weird things so one of the things that happens with direct write is that it for whatever reason and i don't know what that reason is it seems to have this problem where it will um if if it doesn't have in the font that you've chosen the glyph that you ask for it goes to a fallback font which is which is what you typically would want but it renders it completely the wrong size and so one of the things that happened in the original reference demo as i'm pointing out i didn't know to do that i realized after the fact that because direct write runs through uh direct 2d it would be easy enough for me to just correct this by using direct2d's wonky stuff and so what i did is i tried to pick the least bad option because you don't really want this like what you want is a font that just has things that are properly sized to the terminal so that you get what the font designer requested um but what i did instead was just said okay look we'll try to analyze what it is that direct write does and i'll try to put it into something that does as little warping as possible but that still makes it fit the grid right so you don't get like all these weird things where stuff is off grid or whatever else is happening and that's what you can see here so hopefully you can notice that this is a really nice layout now compared to what it used to be nobody is clipped there's no clipping going on um arabic is combined correctly so not you're not getting this anymore you're getting that which is what that string is supposed to look like as far as i know i don't want to put that same approval on it because like i said i rely on other people to tell me uh if their languages look correct or not because i can't read them i'm terrible with languages so you can also see that windows does weird things with stuff like this i don't even know what's happening with those glyphs there i'm not sure why that's so tiny um it's it's kind of weird i i really don't understand but i i'm pretty sure our output is strictly better than theirs in pretty much all cases you can see that this is terrible to read here but it looks quite uh normal and natural here by comparison uh same with all of this stuff so i feel like we're just we're already way ahead of windows terminal on in terms of like producing glyphs that actually match what someone might think they should do in the terminal for languages that are more complicated than basic roman lettering now you'll also notice we do something else that windows terminal doesn't do you may wonder why these are dimmer than these are the reason is because we obey color even for emoji so if you have an emoji which is a color a glyph that has colors in it we still modulate by whatever you set the color to so if you want to you cannot you can turn this off right because when you output to the terminal you could instead just make those be full bright their color you know the vt code you could say i want these to be full bright but if you set like it is here a color that's dimmer it will dim the emoji as well so you can basically choose do you want things you know if you want things feel bright you can always say that you do but you can also change the color so you could also if you want to for example tint these green or do anything else you want because we support full color through the path as you normally would in a graphics application uh and we don't say that just because a font happens to have multiple colors in it you can't change uh the color using the terminal setting so i think all that's uh correct i don't know uh how good we've gotten on arabic so far because i need to sort of confer with people who can really read it well uh to help me understand uh all of that stuff but you can see an arabic test plate here that's getting better and i don't know if it's perfect yet uh because it probably isn't but you know we're that that's something that i think we can get pretty good and you can also see if i change the font size um you know you can see more clearly like what it's got for diacritical marks and things like that on there and i think it's getting pretty good so i think we're within with ref term anyway not with windows terminal but with ref term i think we're within a stone's throw of having some better support for languages so that people who use these terminals who you know i mean yes code is often written in ascii and stuff like that so you know ascii performance and using ascii is going to be the default no question but i mean people who speak other languages are going to want to do things like dump database stuff or whatever that has things like field names that have people's names in them or have addresses or whatever and those addresses and names and or just things that people write comments whatever are going to be written in languages that you know don't will come out weird like this and you can't read them so i think it's value valuable for terminals going forward to actually properly support gls and as you can see we just do that in ref term which is now a two weekend project as opposed to a weekend project but we just do it now so it's not that hard so people should start doing that like it's not very difficult to support these other languages some of the things are a little bit tricky largely because um it's very hard to find a good uh grapheme extended graphene cluster parser so right now i just call uniscribe and kind of use the output of it in a way that's probably not the best so it would be nice if maybe we had good libraries for that in the future or if someone built one into windows that would be great they've got two of them in there but both of them aren't very good um so i don't know but hopefully that's everything i don't know if there are any other particular demos that people want to see because i didn't see any i don't remember seeing any other excuses in the excuse parade obviously line wrapping and all that stuff still works even with uh arabic or any of those other sorts of things so there's no weirdness going on there and again this is all running through the same terminal all of this other than the fast pipe uh so the times that i did this the fast pipe thing um when it was on the stuff at the beginning was all with fast pipe off so you can see that now we don't even need fast pipes to do things uh like you know 0.7 second um splatting of a gigabyte file uh and you know and the like right um we can also do things like uh do a mini line dump uh just to show how fast it is when you're not reading from a file and you can see that there you know we get we get point eight with fast pipes on you still do get again a little bit of a speed up uh you can see that it went from point seven to point five three which is you know it's it's still something but it's just not as dramatic and the same would be true uh in the mini line test here you'd get a little bit of a of a bump right so you get something from fast pipes now but i really don't think it's that necessary so i think we can kind of solve that whole conhost problem for the most part uh if if you know we just start fixing the fact that things were getting written out in those small buffers so that's great news too that's everything i have to report for now um you know i'm sure the excuse parade will keep on marching because there's nothing you can do but uh hopefully that took care of some of the larger floats in it for anyone who was legitimately concerned um about those things i i don't think any of them really have uh much merit but there you go so uh that's it and uh yeah um hopefully i won't have to talk about ref term anymore maybe that's the main hope but yeah hopefully that would at least be the best case scenario is that's the last time i have to do a video on it

Channel: Molly Rocket

Views: 26,088

Rating: 4.9632063 out of 5

Keywords: Handmade Hero

Id: 99dKzubvpKE

Length: 29min 51sec (1791 seconds)

Published: Sun Jul 11 2021

Reddit Comments

[u/sievebrain]

Man, this guy is on a mission.

[u/allo37]

Honestly, mad respect for this guy. A lot of people like to walk around saying "It's so simple, I am very smart", but this dude is actually putting his money where his mouth is and proving it.

[u/PM5k]

This would have not happened if not for how arrogantly dismissive that one maintainer was, calling it a doctoral project. I think a lot of people piled on top when the conversation got diverted toward Casey’s comments being terse. I have read the original issue three times over and it feels like it’s people not even bothering to maturely handle the fact that he didn’t pepper his replies with tangents and emojis and the fact that they find that rude somehow (it isn’t, get over it.)

So that got to be a distraction to the whole point he was making: a bunch of very clever people could not accept the fact that a customer pointed out performance issues on their repo, they got precious about it and became dismissive and passive-aggressive instead of using all that typing time to investigate whether there is a quick win. When they outright lied (or maybe they just lacked experience to identify this quick win), Casey took it upon himself to prove a point.

I don’t think it’s a “crusade” like others are saying. He has the time, he wanted to show the maintainers and the rest of the community just how badly entrenched some of these developers are when it comes to accepting that they have made mistakes or released suboptimal code. The god complex is real and they leaned on it instead of listening to someone very clever and who actually had a point. Not only that but he even took time to write a proof for it.

In a month, not a lot of people will remember this, but I wish that someone in MS who’s got some semblance of power and a brain saw this and kicked the Terminal team a bit, if not for the fact that they could have made an effort, then just for them simply being atrociously dismissive and bullheaded about there being a fix that was in fact not at the scope of a PhD project.

[u/wisam910]

I'm really glad Casey did this demonstration.

I honestly had no idea it was possible to write software that performs this fast.

I mean, I knew software was slow and bloated, but I had no idea it was this bad.

I thought maybe terminals can get 5x faster than they currently are, but I also thought this would require some sort of herculean effort.

I didn't realize it is such a low hanging fruit.

This is a big teaching moment for the entire industry.

I've been listening to Jonathan Blow and Casey Muratori for a while now, but this demo still completely blew me away.

I really had no idea such a feat was possible.

This changes all my assumptions about everything in the web stack (my main "specialty").

I already know that most people do crazy stuff that don't need to be done (all the AWS and K8S bullshit).

But this pushes me to question even more fundamental points: the way we store/retrive/query data. To me it seems that SQLite is the "state of the art" in this regard, but maybe even that is 10x slower than it needs to be due to faulty assumptions.

[u/_Ashleigh]

This goes to show most optimization is in high level algorithms/design instead of low level hacks.

[u/Daell]

🍿 - https://github.com/microsoft/terminal/issues/10462

🍿🍿🍿 - https://github.com/microsoft/terminal/issues/10623

https://github.com/cmuratori/refterm

[u/jagt]

I agree with all casey's points. But there's only one thing that I'd like to point out is that enterprise software development is really different: there're just too many factors affecting the final product, and IMO it's a very real problem that is extremely difficult to solve.

[u/KuroSaru]

just watched his last twitter rant, 100% with this guy. frankly as a whole globally programming standards and quality of code has just gotten worse. AAA games are a perfect example of horrible code, sadly AAA game dev's have enough issues with the stupid deadlines they have to meet so I don't blame them for just getting X to work.

[u/flight212121]

How (fast) is the mac terminal compared to Windows’?