Preventing the Collapse of Civilization / Jonathan Blow (Thekla, Inc)

then when it came to giving a talk it's made me think about some things that transpired between boast of art both of our societies some decades ago for example in 1957 when the Soviet Union launched Sputnik the world's first man-made satellite that orbited the earth successfully back in the USA everybody freaked out about this they didn't like being behind in space and said oh my god we got to catch up we have to get a satellite into orbit in 90 days which seems crazy and it was our first attempt blew up on the launch pads but the second attempt succeeded that was January 31st 1958 it was almost a hundred and twenty days after that initial launch it was pretty fast by modern standards then in April 6 1961 Yuri Gagarin became the first human being to orbit the earth and again that made news all over the world everybody was very impressed by that and back in the United States people were upset because again we're behind we were behind in what was becoming clear was the space race and we didn't like that and what were we gonna do about it so in night May 1961 one month after that flight our president at the time Kennedy made a speech to Congress saying like look if we're gonna catch up and not be behind forever we have to do something big we have to commit a lot of money a lot of resources we're gonna go to the moon right and that was kind of a crazy idea and in 1962 he reiterated this in a famous public speech and he said look we're gonna go to the moon before the decade is over and that was crazy it was really crazy because we haven't done anything remotely like that but lo and behold eventually we did so Apollo 11 launched on July 16th 1969 before the decade was over and just this year there's a very good documentary that came out about this whole mission what it is is it's made of all original footage that NASA NASA took during the mission that's been sitting away in cupboards and closets and they restored the footage and they sort of made a recreation of what it was like to live through this mission and I'm gonna play a short excerpt from that documentary just to give you a sense of what the scale of this whole thing was like it's a lot and it's crazy we went from nothing to all that stuff in something like 12 years before Sputnik flew we didn't have much of a space program in the United States and in the end we had all that stuff and then of course after that we continued to do space things right we made this space shuttle it seemed like a really cool thing it's like a ship out of science fiction it could like take off and then land again that's so great right problem is actually most of it couldn't land like those tanks in the background there and therefore it was very expensive to fly and it was very unreliable people died on this on a couple of different missions and we decided to stop using it for all these reasons so after that if we wanted to put people in orbit we had to get a ride on the Soyuz and then from there the trajectory of our space program kept going downwards and so if you talk to somebody like me sometime around the year 2002 or 2005 we all had this attitude like isn't it a shame like the USA used to do all this cool stuff in space and now we like don't really do anything in the science fiction future that we visualized isn't really going to happen and we don't ever see that changing and but what can you do about it oh well shrug right that was just everybody's attitude but not quite everybody right at some point somebody came along who made a bunch of money on a website and said hey I want to do something about this despite having no rocket experience I'm gonna start a company to launch rockets and to do bigger stuff than we've ever done before and so here's an excerpt of a video about why he did that stuff then there's becoming a multipass species and spacefaring civilization this is not inevitable it's very important to appreciate this is not inevitable the sustainable energy future I think is largely inevitable but being space maximization it definitely not inevitable if you look at the at the progress and space in 1969 we were able to send somebody to the moon 1969 then we had the space shuttle that the space shuttle could only take people to low-earth orbit then a space shuttle retired and the United States could take no one to orbit but so that's the trend it's right it's like down to nothing this is not a mistake in what when they think that technology just automatically improves it does not automatically improve it only improves if a lot of people work very hard to make it better and actually it will I think it by itself degrade actually you look at great civilizations like ancient Egypt and they're able to make the permits and they forgot how to do that and the Romans they built these incredible aqueducts they forgot how to do it so his idea was pretty successful and today we're like landing rockets and we're seriously talking about doing another moon mission as soon as the Year 2024 we'll see if that actually happens but really stalking about it seriously and that's a pretty good thing given where we were not long ago so Elon talked about a few things from the past that were great achievements that have been lost and I wanted to go through a few more of those to reiterate his point that technology automatically degrades this thing here that you see is the like her gus' cup this was a relic found and dated back to the Roman Empire 380 and it's made of glass and this glass that it's made of is the world's earliest known nano material okay the color of the glass changes based on how you look at it like where the light source is so if you're looking at it standing in front of the glass and the light source is sort of over here with you so that you're seeing it with reflected light then the Goblet is green but if light is passing through it the Goblet is red they had this in 300 AD right and then the Roman Empire fell and that knowledge was lost until basically forever the way this worked was actually you know it got figured out around 1990 the glass is suffused with very small particles of silver and gold by very small I mean 50 to 70 nanometers which is so small you would not be able to see them with a physical microscope you would require an electron microscope to see these particles right but at some point the Roman Empire fell and they forgot how to do it a lot of craftsmanship went into this you can see you know how its hollowed out on the inside where the little guy's body is to give him more of a purple Sheen as opposed to a red in the background and if you hear people talk about this today or you read up on this they can have a dismissive attitude toward it like oh the stupid Romans didn't understand technology they probably didn't even know it was silver and gold that made this happen it was probably just an accident and they made like five of these right which is complete nonsense like anybody who actually builds things as opposed to just writing about them knows you do not get a result this good without a constant process of iteration and refinement you can imagine there will be some initial accident like maybe somebody wanted to make glass sparkly and they tried to put silver and gold in it and then they noticed a little bit of discoloration and they said like why is that there and maybe they pursued that like what happens when I change the proportions right what how thick should the glass be like engineering results this good takes a long time and what that means is that in Rome people were doing something that we would recognize today as material science and then that was lost other stuff happened like in the Byzantine Empire they had flamethrowers and not like little dinky things they had giant pressurized vessels in the bellies of ships that shot out a napalm like substance out of metal tubes that they would use to incinerate neighboring vessels it was napalm like in the sense that like water would not put this fire out right it was very serious weapon it was a state secret of the Byzantine Empire they used it to defend Constantinople over and over again for hundreds of years until one day they couldn't really do that anymore for whatever reason and this military secret just faded from knowledge nobody knows how to do it now right obviously we've reinvented flamethrowers but they're different but this is the Antikythera mechanism which is named after an island in Greece where this was found on a sunken ship it was just a corroded hunk of metal or a number of corroded hunks of metal but it was very clear when they were originally discovered that gears were involved and over time people analyzed this they realized it's a mechanical calendar that was used to say things like you know what what year is it what are the phases of the Moon where are the planets going to be right now when is the next Olympic Games right and people have run scans on on what is left of this and managed to deduce what all the gears were in this mechanism and it's very different from what I thought when I first heard news about this I thought like oh they must have had some cute little gear things in Greece that's surprising but let me just show you the scale of the generally agreed upon reconstruction of what this device actually was that seems like a lot of gears right but wait there's more so ancient Greece had that but that is not the picture that we have today of ancient Greece right and the thing to realize is you don't just get here from nothing it's not like one day there weren't any gears and then the next day some guy makes this right you need a whole process of science to create something that sophisticated and we don't know anything about that today right all of that was lost and I could go on and on with examples there's a whole bunch of things from history that are like this but we don't have time I'm just want to restate that right now we live in a very privileged time where technology has been in a good shape for a long time we see it getting better and so we imagine that the natural course of history is that technology always improves and that these moments in history are just like little blips or something that we heard about but they're not just little blips it's actually sort of the regular course of world history that great achievements in technology just get completely lost because the civilizations that made those achievements fell or you know have this sort of a soft fall where they fail to propagate the knowledge into the future right technology goes backward all the time and not just in ancient history also in the modern day right we lose knowledge all the time so I'm gonna read an excerpt from an interview with Bob Cole well who was the chief microprocessor architect at Intel for a while but this interview is from before that it was from the booming days of Silicon Valley when he worked at a start-up called multi flow they were trying to make a very large instruction word processor when that was a new experimental idea and they were having a lot of problems like when you try to design the chip you're using components from other manufacturers and he just couldn't get anything to work reliably and he was like what what the hell right so he says rich lesson and I made a pilgrimage down to Texas Instruments in Richardson Texas and we said as best as we can tell many of your chips don't work properly and does this come as a surprise to you I half expected them to say what you're out of your mind you've done something wrong come on you don't know what you're doing and go use somebody else's chips but no they said yeah we know let me see your list and they looked at the list and said well here's some more that you don't know about and by the way it wasn't just TI their parts were no worse than anybody else's Motorola's were no good fair childs were no good they all had this problem and so I asked TI how did the entire industry fall on its face at the same time we are killing ourselves trying to work around the shortcomings in your Silicon and the guy said the first generation of transistor logic was done by the old graybeard guys who really knew what they were doing the new generation was done by kids who are straight out of school who didn't know to ask what the change in packaging would do to inductive spikes right so when you change the voltage in places on a chip it generates a magnetic field because that's just what happens and when those fields interact across a chip bat it's bad right and you know the new people designing these chips didn't know to take that seriously and that's why technology degrades or it's at least one reason right it takes a lot of energy and effort to communicate from generation to generation these important things that you need to know in order to do a competent job making the technology and there are losses in that communication process almost inevitably and without this generational transfer of knowledge civilizations can die because of technology that those civilizations depend on degrades and fails so let's talk about a civilization that fell actually a whole group of civilizations the diagrams I'm going to show here are from a lecture you can find on youtube called 1177 BC the year civilization collapsed by Eric Cline and we're talking about the Late Bronze Age which was the time of a number of civilizations you've heard of probably like the Egyptians or the Mycenaean Greeks right or the Hittites the Babylonians and so this civilization or this network of civilizations was sort of spanning Mesopotamia and the Mediterranean Sea and they had developed quite a sophisticated network of trade so in this graph here each of these points is one of the civilizations and the lines are you know established communication and trade route between those civilizations and whereas not all of them were connected to all of the other ones they were interconnected enough that you could relatively efficiently efficiently route things from one place to another if you needed to and that was very important because bronze which the civilization depended on for things like defense was hard to make back then you had to do it by combining copper and tin and copper was relatively hard to find and was found in places like the island of Cyprus and tin was also really hard to find and was found very far away from those copper places like in Afghanistan and so you saw you somehow had to persistently ship these things around in order to make your bronze and the other things that your society depended on and nobody's sure exactly what happened in this collapse but people believed there was some kind of environmental stressor to kick it off like there was a huge drought possibly also some floods are theorized and this led to some people attacking some other people and you know maybe you need to start using your ships for defense instead of trading and basically you went from all these flourishing civilizations to a hundred years later none of them were left and by none of them were left I don't even mean that like the nation-states were gone like many of the cities were burned to the ground and the languages and cultures don't survive even though they wrote by you know pressing things into stone like nobody was able to translate those languages even today we still can't translate a lot of them so like so much knowledge was lost here in this collapse we'll get back to it later but so I want to bridge this to the modern day in some way and my thesis for the rest of this talk is that software is actually in decline right now it's in maybe a soft decline that just makes things really inconvenient for us but it could lead to a hard decline later on because our civilization depends on software we put it everywhere all our communication systems our software our vehicles our software so you know we now have airplanes that kill hundreds of people due to bad software and bad software only right there was no other problem with those airplanes now I don't think most people would believe me if I say software is in decline it sure seems like it's flourishing so I have to convince you at least that this is a plausible perspective and that's my goal for the rest of this talk and what I'll say about that is these collapses like we're talking about that Bronze Age collapse was massive like all these civilizations were destroyed but it took a hundred years so if you're at the beginning of that collapse in the first 20 years you might think well things aren't as good as they were 20 years ago but it's fine we're basically the same right but then you keep thinking that you keep thinking that every 20 years another couple cities get burned to the ground and then eventually there's like nothing right um fall of the Roman Empire was about 300 years so if you're in the middle of a very slow collapse like that would you recognize it would you know what it looked like from the inside so of course I expect the reply to what I'm saying to be you're crazy software is doing great look at all these internet companies that are making all this money and changing the way that we live you know and I would say yes that is all happening but what is really happening is that software has been free riding on hardware for the past many decades we've had amazing advances in hardware technology computers keep getting faster and faster it's really one of the greatest accomplishments in human history that we've somehow managed to do that and software gets better in air quotes because it has better hardware to run on that's the main reason software technology itself has not improved in quite a while I claim right and you can say but look at all these examples of cool stuff we can do even in the past couple of years so like alphago was an AI that'll beat human players it go and you can go on like Instagram or whatever app and like make your face look like somebody else's face that's crazy we didn't used to be able to do that and that's true but one most of these again are products of hardware being fast most of these cool things that we do now are due to machine learning algorithms and it you know those really are relying on quantity of computation right now to produce impressive results it's hard to imagine being able to train alphago 20 years ago on the computers we had at that time right so it's not a there are software technology improvements here right machine learning algorithms have legitimately gotten better but there's two things to say about that well the main thing to say about it I will say is just that it's a minority of actual software technology right so the of the volume of things that we run the thing that runs the machine learning algorithm that produces the actual impressive result is a very small piece of the program it's actually really simple once you understand the math and especially if you don't have to train it if you just have to use it right and so when you take an app on your phone like that that does something funny with your picture the part of it that does the thing that we think is cool and really value that piece of software is tremendously simple compared to all the stuff about like loading the bitmap for your face or responding to user input events right that part of the software is huge and complicated and is the part that's kind of falling apart so I would characterize software as having small local technological improvements like machine learning with overall inertia or degradation in the rest of the field and we're very impressed by the improvements though right and let me illustrate the degradation parts as best I can and it's to say that we simply don't expect software to work anymore and I'm not sure when this happens you know computers always had a reputation for being a little bit funny but it you know if you go back many decades ago it was generally due to like not being user friendly or hard to understand how to use it but today if you're using a program and it does something wrong you're just like yeah it's software restart it whatever and that didn't used to be and if our standards are shrinking over time how low can they shrink before it becomes unsustainable so I decided to say you know I want to quantify or illustrates how much I put up with this from day to day so from now on I'm just gonna take a screenshot every time any piece of software that I use has an obvious bug or you know unintuitive or incorrect piece of behavior and well at write it right when I decided that I was working on my compiler in the command line and the console that I use after a while just start saying attempt to index and you'll value in the prompt because it's written in Lua for some reason then I go to Emacs and I'm working on my code and Emacs is set to reload files that have been modified and that used to work fine but at some point they broke it so that it reloads the file too early and doesn't get the whole thing and half of it is cut off and I have to like manually reset that every time it happens then I go to Gmail and I'm going to send an email to the rest of the team about some graphics stuff making decisions about what to do and I copy a line of a previous email and paste it into the reply box and then I start typing my reply and it goes into like a three character wide column over here because somehow they've managed to reproduce all the kinds of stupid Microsoft Word formatting bugs that everyone was frustrated with in the 90s and 2000's now those are in Gmail and I don't know how to fix it like you fight with it for a while to get it to stop happening you have to like delete something invisible I don't know very annoying so then I say okay I'm gonna get some real programming done I go to visual studio and I say I'm going to type in my command-line arguments up there and as soon as I do that we get this box that says hey collection was modified enumeration option may not your operation may not execute why I don't exactly know why that's a problem like I'm just telling it a string we're not even trying to do anything with the string it's just like save this for later for when we want to run the program but apparently that's too hard right and this is far from the only problem with Visual Studio Visual Studio has many many bugs but this is the funniest one because it's so simple what I'm trying to do and it can't do it all the time this I don't know what percentage of the time this happens it's probably like 5% I don't know 4% so then I decide to break off blow off some steam and play some games so let me download a game on the door but we're unable to start the download for some reason so maybe I'll go to steam because that's a more reliable longer-lasting store and I'm able to actually download a game but then when I go to the install window it's just like a black window and I have to restart steam to play the game then I manage to play the game and then I all tap out for a second to check something and then now full screen is all messed up and the game's like up in the corner of a window right and then I have to restart the game to get full screen again and then I'm watching some counter-strike there was a really good match between cloud 9 and luminosity gaming about a month ago and but for the entire match there was a mysterious six player on the cloud 9 side called undefined up in the corner there let me zoom in on that map for you counter-strike fans is undefined is on the left a hundred thousand people were watching this match and it was there the whole time I was thinking about a game I like called Ultima 4 so I went to this website that had the map and the map was like screwed up because he was like wrapping into extra lines so I opened a different browser to see it correctly I needed to get a visa to come to the Russian Federation so I go to the visa site and I start typing my information and maybe I type out my phone number I put the plus one and it didn't like it or something so it says phone number is invalid over here but I couldn't fix the phone number no matter what I put in it wouldn't accept it because whatever the variable was for phone number is invalid would never get reset so I had to like stop the application close the website like clear my cookies go back and reapply in order to be able to and be very careful when I was typing my phone number there's just so many of these all of this was within a couple days like I didn't have to try hard to find these right I just had to stop collecting them but then I come here and as if to give me more examples in this talk so here in this hotel where I've been writing this talk for a couple days they have this software controlled heating and lighting system where it's like you kind of push the non button buttons and things happen and some percentage of the time not all the time when I turn the air conditioning on or off the phone rings it's not a full ring even it's just like a little blue bubble and then it stops but I know it's not intentional because it doesn't happen every time and I am NOT making this up this actually happens in my room right now okay and then for this talk two hours ago I was working at the last minute to make a diagram and I downloaded fully legitimate licensed Creative Cloud Photoshop to my machine the first thing I do is go file new document BAM the new document extension could not be loaded because of a program error right and so my whole point though is we are not surprised by any of this my other point is that it's getting worse over time so try this everyday yourself because we've gotten used to it I didn't even think it would be as much when I had the idea to record this I didn't think it would be as much as it was try counting for yourself just everyday just make a little list of all these things and I think you'll be surprised how many there are I don't know if anyone knows what this phrase means five nines I'm sure a lot of people don't this used to be a very common phrase in the 1990s and 2000s when people wanted to sell you software or a hardware system what it means is this system is up and working and available 99.999% of the time right four nines would be ninety-nine point nine nine whatever and we don't use this anymore I think in part because the number of nines would be going down and we can't make it go up again and nobody well certain parties don't seem to care so I was you know working on this speech for the about the past week and twice once when I was asleep on the airplane and once the other night in the room my laptop just rebooted while it was in sleep mode and just killed all my programs and stuff I guess cuz it was an update maybe it wasn't an update maybe it was just the operating system failing but I think it was an update so that automatically takes my laptop down to like three 9s or less less than three nines and if the laptop is less than three 9s nothing running on it could be three or four or five nines right so we've even lost the rhetoric of quality that we used to have right and so if you say this the software is buggy then people like web programmers or hacker news people or whatever will say yeah we know but the market won't pay for it right like we could make software better but that takes time and money to fix the bugs and all that stuff and our client won't pay for it or the market punishes that because you take longer to get to market and that's true to some extent I could definitely argue with some parts of it but here's the thing that I'm thinking today if you haven't seen an entire industry produce robust software for decades what makes you think they actually can right they're saying we could do it if we wanted to but we're just totally not but why would I believe that they actually can do it right because like we've said there's this generational transmission of knowledge factor that I don't think is being passed along right so I think the knowledge of how to make things less buggy is lost and even the knowledge of a technology company has changed and again this illustrates the difference between software and hardware hardware technology company used to be a place that makes advanced materials or defiant you know designs new radar or like does something but you didn't use to be able to do before right so now in Silicon Valley and as nearly as I could tell around the world a software quote tech company is just a company that does stuff with computers and and is then hoping to stumble into a market niche that it can exploit and the point is the market niche the point isn't the software and the point is especially not designing higher tech software that pushes the threshold of technology forward which is what hardware companies always used to do and so we've even corrupted than words tech company right ok so now I want to bring it a little closer to what we do there's been this sequence of abstraction that we've gone through as programmers over the decades right originally you had to program your computer in machine language then there was assembly language then we had the sequence of our level languages like Fortran and C or C++ and nowadays we have stuff like c-sharp or Haskell or JavaScript that are even further away from the machine and the justification for this is like look that we're working at a higher level of abstraction the higher your level of abstraction the more you work you get done because you don't have to worry about scheduling machine instructions and stuff so we're really being smart and we're saving effort and I think that's actually true like I don't think we want to program things in assembly language that's a waste of time but somewhere through this chain it becomes wrong and that that's how people are wrong a lot of the times right like you start out by being right and then you extrapolate it too far into wrong territory but the important thing to all of this is that we only see one side of it we see that we're being smart and saving effort and we don't see the flip side of all of these things which is that there's a corresponding loss of capability right because I don't program an assembly anymore I no longer am able to program an assembly right if I don't you know if I if I use languages that are too high level and have a little bit lazy as people often are I don't know where my variables live in memory or what they look like or even how remotely how big they are right I certainly don't know what the CPU is doing in response to the code that I've written I may be scared to use non-managed languages because the very idea of memory allocation just seems too hard and scary or even if I'm a person who programs in a non managed language maybe I'm afraid of pointers and start generating this cult of of being afraid of pointers and what to do about that like the modern C++ people do right and so the rhetoric that we have is I'm being smart I shouldn't have to do the low-level stuff right but part of the reality is the loss of capability that corresponds to those choices and both of those things can be true at the same time I'm not saying that we're not being smart by going up some well a little bit I mean there's a problem which is at the point of going up all these levels is supposed to be to make everybody more productive but programmers are not more productive now than they used to be in fact it looks to me like productivity per programmer is approaching zero and if that's true then where is the proof that going up this ladder of abstraction further and further is really helping so the way to at least you know get a feel for this is you look at a company like you know Twitter or Facebook it employs a lot of people and you look at their product and you say how much does that product change from year to year right how much functionality is added to Twitter year after year how much functionality is added to Facebook it's not that much right and then just divide by the number of engineers at the company right which is thousands or tens of thousands sometimes that's a very small number when you do that division right it's it's gonna be pretty close to zero so what's going on right and and to illustrate again the difference in productivity and that it's not just me that thinks this I'm gonna show an excerpt from an interview with Ken Thompson who is the original author of the UNIX operating system and he's talking about the time at Bell Laboratories when he first started making UNIX on a computer that by modern standards had like no software at all right some point I realized without knowing it up until that point that I was three weeks from an operating system with three programs one a week i netted run you didn't matter right I need an assembler to turn the code into language I could run and I needed a little kernel kind of overlay call it an operating system and luckily right at that moment my wife went on a three-week vacation to take my my one-year-old roughly to visit my in-laws who were in California disappeared all alone and one week one week one week and we had eunuchs yeah I think programmers aren't quite as productive these days yeah he says programmers aren't productive these days like that and everybody laughs but it's funny but it's not funny right it's really not funny when you consider like how much waste there must be in the difference between how productive people are and how productive they could be if everything wasn't so messed up right so I've made a case that robustness of software is declining productivity of programmers is declining so if you're gonna say that actual technology of software is somehow advancing it seems contrary to those two facts right so I think the argument that software is advancing is clearly false except again maybe in tiny local bubble-like areas so now why is it so bad why is it so hard to write programs why are we so miserable when we try to write programs today it's because we're adding too much complication to everything right and I have a way that I think about this called you can't just write where there's all kinds of things that you used to be able to do on a computer that you can't do today right so today you can't just copy a program from one computer to another and have it work right you need to have an installer or like a flat packs on Linux or like containers if you're a server hacker news guy right and so people think this is cool now we have containers that's an advantage or it's an advancement of software technology all containers are doing is get us back to the 1960s when we didn't have to do any of this stuff except it's actually not because it's adding all these steps that you have to do right and things you have to maintain so now let's think of it for a second like why do you need an installer to install software is it because of the CPU not really like imagine you have well you know imagines you have some x64 machine code and don't worry about how you got it into a computer's memory but you just got it there and you just jump to it you set the program counter to that code that code is going to do the same thing on a Windows PC as it does on a Mac as it does on a Linux machine it does on an Xbox as it does on a Playstation 4 all right because all of those systems use compatible CPUs so what's the installer for the Installer is to get around the incompatibilities that we added at the OS layer which is this immensely complex thing that we mostly don't want actually and so we tend to think about operating systems as adding capabilities to a system to the the system of the hardware in the software but they also remove capabilities like compatibility right and it's often very arbitrary and it it doesn't get any worse than I think it does for us today when it comes to shading languages anyone who ships 3d engines is going to know what I'm talking about so it used to be that if you wanted to compile a program for many platforms you could write it in some portable language like C or C++ and you might have to do some little if Def's to modify it for the different platforms but you could do that and it's mostly the same program today you can't do that because we've decided if you're running a shader it needs to be in a different programming language on every single platform even if the hardware is the same right so if you have an x86 CPU and an NVIDIA GPU then on one OS you need to write your shader and metal shading language and on another OS you need to write in an HLSL right and and they're different even though they're the same and so you either have to rewrite everything end times where n is large or you have to start using Auto translation systems to rewrite your shaders and those come with a lot of complexity and annoyance and bugs and why though a shader is a simpler program than the old programs that we used to write but why have we made it harder to build a simpler program it doesn't make any sense we don't care right so the list of things you can't just do you can't just copy a program you can't just statically link you can't just draw pixels to the screen oh my god the number of steps you have to do to draw a pixel today is crazy you can't just write a shader you can't just compile a program on Windows without a manifest and stuff and on these new closed platforms you can't just run an executable unless it's signed through this like whole process right and all of these things and many more that are not on this list add friction bugs time engineering time and headspace that keeps us from thinking about interesting things to actually do there a couple of examples of this that illustrate this isn't going to end anytime soon have entered my own life so one of my side projects is a compiler and two compiled programs you need to link them against libraries on people's machine like for example the Windows SDK and the C runtime library and now different versions of things install those in different places on the machine and so you have to like be able to find them to do the linking and rather than make this easy today Microsoft gives you a program called vs where which you can find on github and the job of vs where is just to tell you where these libraries are installed it is more than 7,000 lines of source code in 70 files ok and they didn't even try to bundle it as a library it's a standalone program so what they're thinking now is you can't just make a compiler that's a standalone program it's obviously going to be a suite of applications and once you have a suite of applications what's one more what's like a little vs we're hanging out in there right they're not even thinking that this would be bad it's crazy I made my own version of this based on some other people's work and got it down to like 500 lines of code which is still way too many to basically ask two questions that should be two lines of code right so multiplier of 250 and then in also in the programming language world there's this thing called language server protocol that is pretty much the worst thing that I've ever heard of and there are just proponents of this although they're building systems for this right now that are going to be living on your computer tomorrow or today already maybe and as far as I can tell it's basically a more complicated slower way to do libraries so say you've got an editor for some programming language and you want to be able to do stuff that we've been doing for decades already like look up the declaration of an identifier by clicking on it or tooltips that say like what type is this value right well they say the way you should do that is you know you have your editor and then it's it's a hassle to make plugins this is the made-up problem it's a hassle to make plugins for all these different things so in order to standardize you're going to run a server on your machine and then your editor talks over a socket to the server and the server talks back and gives you the answer right which has now turned your single program into a distributed system but the the flaw in this whole line of thinking that none of these people seems to actually like think about at all is that there's nothing special about like looking up the location of an identifier in your port that's just an API like we have all the time for everything so the obvious next step if you're saying that we should architect our api's like this is to do this for other tasks right so now your editors or whatever program is going to be talking to multiple of these things and now if you ever want to author anything for this you now have to author and debug components of a distributed system where state is not located in any central place and we all know how fun that is right but of course libraries are not that simple right libraries use other libraries so what happens at that point is you're running all these servers on your system and who don't you know some of them are going to like go down and like have to restart and people are synchronizing with each other no this is a disaster right and people are actively building this right now and meanwhile while we're spending all this time over complicating stuff that we used to be able to do in 1960 in the games industry we're not even able to do the things that we've needed to do forever so like today games can't run consistently in full-screen as you see from the screenshot and I don't wish to bag on that particular game because we all put a lot of engineering work into trying to make our game run in full-screen it's kind of embarrassing like why right also it's actually impossible on a PC right now to render at a smooth frame rate it is simply not possible no matter what you do Allen a lot of otso Croteam has a talk at GDC and a paper about what you actually would need to do this we just don't even have that capability which is insane right and yet we're spending all this effort on other things and so this complication that's introduced into all of our systems and not only makes our lives difficult in the present when we're trying to build something it accelerates the loss of knowledge over time right so first of all there's more to know when things are more complicated and so if you talk about a job spread among many people each individual person knows a smaller percent of what they need to do they have a less global view which makes it harder to do good work right and harder to transmit their knowledge on to people in the future another thing that happens is that deep knowledge becomes replaced by trivia so deep knowledge might be a general concept like here's how cache coherency works and that enables software to run fast on like different processors and stuff and trivia is something like well this sprite in unity doesn't display properly for some reason but we know we can fix it if you open this panel and toggle this boolean and that fixes it for a while but then some weeks later for random reasons the boolean mysteriously --until so just make sure to check that before you ship and it'll be fine right and the reason that's trivial is not only because it doesn't apply to anything else in the world but it's also going to be outdated in six months when the next unity comes out and it's just offensive that we're spending our brainpower on these things okay and the third thing that happens is good information is drowned by noise so if something is really hard to understand the percentage of people who put the effort into understanding it is going to be small and the harder it is the smaller that percentage and so if you ask people or you learn at a school or you search on the web your probability of getting a bad answer to the problem is much higher for more complicated things than it is for less complicated things and so the complication propagates and magnifies so let's get back to this collapse of civilization stuff right the more complexity we put in our system the less likely we are to survive a disaster right because we have to maintain all that complexity we're acting right now like we believe that the upper limit of what we can handle is infinity amount of complexity right but I don't think that makes any sense so what's the upper limit how would we decide how much complexity we can handle and that's different from what people today actually can handle so if you have an engineer who can hold a whole system in his head that's really complicated and work on it when that guy quits and needs to pass on his job to somebody new he's not necessarily going to be able to communicate all that right so the amount of complexity we can sustain over time is less than the amount of complexity that individuals can do today right so why am I talking about this at a games conference right like everybody knows that games aren't serious and whatever right but video games at least used to be about maximizing what the machine could do and like really impressing the people playing the game and maximizing the machine means you have to understand the Machine very well and that correlates with robust software because if you understand the machine well you're less likely to make the kind of bugs that come from misunderstanding there's anti correlations with robust software too but anyway now we're not really about that so much especially talking about independent developers people are shifting to unity and unreal and moss right not very many people write their own engines anymore so we have entire generations of programmers who have grown up learning to program by you know making little C sharp snippets that just plug into other parts of unity or something and they've never written something systemic and they've never written something low-level which on the one hand is fine like I'm not saying we shouldn't do that because there's a degree to which it's smart it reduces development time right it helps you ship your game sooner but like I said before there's a flip side that flip side is giving up the capability of doing the other thing giving up the knowledge of how to do the other thing so I don't think it's bad in isolation if a lot of people make games where they just put snippets into unity right but if everybody does that the nobody knows how to do anything but that and then after a while what's gonna happen because we're assuming that we'll just be able to use these engines forever but unity and unreal were created in an environment where there were lots of people at games companies making engines all the time right and that's where they hired people from and when there's no longer a natural way to learn how to make engines because nobody does it where our unity and unreal gonna hire employees from to maintain those engines that everybody's using right and you know to the extent that they can hire people is the quality of people gonna go down because they have less experience it just takes a long time to ramp up right so then maybe at some point well certainly at some point there's not enough people to make a new competing engine but maybe even at some point you can't really maintain the old ones and they just keep decaying over time that can happen and so the way I used to think about game developers is kind of like the foundation in the Asimov books where we kind of knew how to really program computers and you know also some other kinds of programmers like embedded systems people and high performance computing people all sort of knew what was going on with computers and after the rest of software just kind of decays and falls apart we still have the knowledge and we could bring it back and give it to people but I'm not really sure that that's gonna happen now because I just don't know I mean I don't know if there will be enough of us doing doing low-level work or even people doing high-level work who understand what's happening at the low-level while they do the high-level right so maybe there needs to be a second foundation spoiler alert for any one person read the book so back in the Bronze Age right one of the reasons those civilizations disappeared is that the way things were set up was that reading and writing was only done by a small elite class who went you know went to school for years and this was protected the public couldn't know how to do this they probably mostly didn't want to know and because those skills weren't widespread they were fragile so when the society was disrupted they weren't continued because not enough people could carry it forward day almost nobody knows what's happening on a CPU right that skill is not widespread so it's fragile and so do we think that this immensely complicated thing that we've built today is somehow more robust than what they had in the Bronze Age with just making bronze because that didn't survive if that didn't survive why do we think what we're doing now is gonna survive right and we might have some similar stressors we might have some climate change issues right or you might have some new stuff like what happens if there's so many cyber attacks that countries just start cutting each other off the internet right now lots of people in lots of countries can't get to Stack Overflow to figure out how to copy and paste their code so their code production is impacted right or what happens if China just says you know what we're just gonna keep all the CPUs now we don't want to sell you any what's gonna happen right none of these things in isolation I don't think will bring down civilization but it can certainly hit the system with a big shock and if the system is too complex it may not survive that shock very well and so I'm just trying to say like Elon Musk was saying the technology by itself will degrade and we need to as soon as we can start working against this right at every level that we have access to we have to simplify the hardware we're running on we have to simplify the operating systems we use the libraries we use the application code we write the communication systems we do this over like the internet we have to simplify how we compile debug and distribute software and we have to simplify how people interface with software and that sounds like really a lot of stuff to do but the good news is that all of these things are so ridiculously complicated right now that it's very easy to find things to improve simplifying any of these systems only requires the will to do it rather than a taste you have to have a taste to recognize how complicated things are and how they would be better if they weren't so complicated okay now a lot of people are probably like okay whatever software is complicated but I don't believe civilization is gonna collapse or anything and so you know maybe maybe but I would say if you're a programmer you should care about this anyway because even just your own personal future like programmers are not that happy today we're often very grumpy and the reason we're grumpy is because we're doing stupid things all the time instead of interesting things and that's not gonna get better if we keep doing things the way that we do them right so you personally will be happier if we change the way we do things and if we do things the way they are now maybe the future is deeply mediocre in the way that America's space future was going to be deeply mediocre now even if you just want to survive as just an individual game developer like you're thinking look I just want to get my game done I want to ship it I wanted to succeed financially even if you just want to have a very limited scope of concern like that removing complexity is still the right short-term play even if it doesn't seem like it I'm sure we all are very familiar with cases like well we're gonna ship in five months and we're having a lot of problems with this particular system it's really buggy you know it loses people's work all the time whatever but we just have to stick with it for five months and it'll be passed it'll be history and that's good because rewriting it would be a lot of effort it might delay shipping and so we're gonna stick with it we're gonna stick out the five months and that's always wrong because always what happens is it takes two years to ship instead of five months and so the amount that you suffered from the system is way worse than it otherwise would have been and maybe in fact that system was a large ingredient in why it took so long to ship so simplify and in simplifying your own code to solve your own local problems you're also building institutional knowledge about how to simplify which sounds really basic but I would claim we don't even really have that anymore here's some references of videos you can watch if you're interested in this kind of topic Casey meritorious video the 30 million line problem samo Boreas videos civilizations institutions knowledge in the future and the Eric Klein's video which I showed snapshots of earlier 1177 BC the year civilization collapsed and that's all I have to say for now thank you for your time Wow very impressive don't you think that the collapse will happen when we reaches the point of technological singularity because simplifying is I think it's some kind of way to prevent it you know once you start saying singularity it's too hard the point of singularity is you can't predict what's going to happen maybe good yeah I don't believe in singularity the way a lot of people do it doesn't seem realistic to me but as you get close to that kind of situation things move faster and if things are moving fast they break easily okay yeah it's what about foundation by the way right yeah well well in one of the objections because I'm always criticizing my own like what is the counterpoint to this and what if we just let software get really complicated and then just make an AI that understands it and that's fine and it's like okay maybe but you really want human beings to not be able to understand software it doesn't seem good okay we have about 5-10 minutes for questions so if you have one you can ask yeah come here thank you very much for this beautiful speech and beautiful mind welcome to Russia by the way thank you and I've been doing games in Jamaica for 17 years and we spend one year doing 3d indie maker and I was asking myself a question why we did this and actually now I know the answer and I have another thought about it Freed's ignition when he was already an old man he started to lose his sight and he couldn't write anymore couldn't think actually because he would think while writing and so he started using Hansen's writing ball and this was the first typewriter to use but his style changed and when I switched from game maker to unreal I'm as a game designer saw that my style my way of thinking changed don't you think that tools they somehow force us to think in the certain type of way and for you making a new language as a somehow to break through this and start thinking wider broader I I think I would agree with that and I would also say though you know because we think with tools often unnecessary complications or bugs in the tools interfere with the thought process right like you're in flow you're doing stuff and then something bad happens and you're like now I have to go fix this thing and you can't do what you were doing so I definitely think that's important you know when I when I'm making my new language I'm trying as best I can to get rid of all these complications that don't make sense but there's so many of them and some of them are baked into our assumptions because you know I learned to program by using these complicated systems so what I see as simpler may be very far from the actual simplicity that we could achieve because my thinking has been trained on those tools so we'll see we'll see how it comes out thank you thank you so let's say I'm an indie developer and I'm sold on your ideas I don't like unity either and I want to put pixels on screen with great ease but I'm not yet ready to write my own engine well you you would have to write your own operating system to put pixels on the screen it says so what's what these the set of currently existing tools I can use are like well I don't know because part of the problem is everything is this way so really what needs to happen is not about specific tools that you use it's about developing the aesthetics for things that are not a giant horrible mess and whatever tool looks like that to you just use that instead of whatever you're using and then maybe we could migrate everybody slowly over time yeah the problem is when I look for ways to like get to the lower things I find these like Visual Studio and C++ and it doesn't like helped very much it's super complex and it's breaks every time and yeah I don't know man thank you very much for your talk my question is in 1968 30 years after the concept of the computer was invented Edgar Dijkstra that that program where the mainframes were having 2 megabytes of memory even then a gap district said that the programming is just too hot by its concept to be done by human beings are you sure that simplification will help to any extent well it'll help whether it makes them completely understandable by us I don't know I mean I think I think you could I haven't read the Dijkstra piece that you're talking about the humble program oh ok I think I think you could rate it by what problem are you trying to solve right and how complicate like there's an inherent complexity to a problem first of all and so there may be problems that are so complicated it may be hard to understand what the software looks like to solve that but then there's also added complication because we're solving this with existing systems and those systems already prevent us from doing certain things and so there's a difference between ideal complexity and actual complexity and I just want to get closer to ideal complexity whether that's good enough I don't know you

Channel: DevGAMM

Views: 158,303

Keywords: devgamm, девгамм, game industry, video game industry, gaming industry, игровая индустрия, конференция, conference, конференция разработчиков, геймдев, разработка игр, gamedev, game development, game dev, devgamm 2019, девгамм 2019, indie game, инди игры, инди, indie games, indie gamedev, indie game dev, indie game development, indiegame, indiedev, Jonathan Blow, Thekla Inc, collapse of civilizations, software, how to make software, software development, programming, how to make games

Id: ZSRHeXYDLko

Length: 62min 41sec (3761 seconds)

Published: Wed Jul 10 2019

Reddit Comments

[u/CollapseBot]

The following submission statement was provided by /u/fleece19900:

---

Submission Statement:

In this talk a software engineer goes over ways that software engineering is degrading. Since that's his career that's his focus. However, I'd like to expand the scope of this idea to all of society. Information gets lost over time, there's no way to perfectly transmit it from one human being to another. And with that degradation, comes collapse.

Take as an example, a town. In that town they're doing the well-valued "infrastructure work". One of the old-timers stops some of the work and says "no that can't be done, there are important lines buried there". That information is on none of the documentation, or perhaps the documentation has been lost. Without that old-timer to raise his hand, the lines would be broken, costing the town tremendously.

Extrapolate that to everything - as the older workers retire or leave the workforce, they take their information or knowledge with them. Or, in the general course of time, documentation gets lost or destroyed. And the incoming generation is less capable, through no fault of their own (sorry Boomers who instinctually want to berate the young), and the result is that the system degrades, bringing it towards collapse.

---

Please reply to OP's comment here: https://old.reddit.com/r/collapse/comments/wg4gov/information_loss_an_underappreciated_factor_in/iixhan8/

[u/fleece19900]

Submission Statement:

In this talk a software engineer goes over ways that software engineering is degrading. Since that's his career that's his focus. However, I'd like to expand the scope of this idea to all of society. Information gets lost over time, there's no way to perfectly transmit it from one human being to another. And with that degradation, comes collapse.

Take as an example, a town. In that town they're doing the well-valued "infrastructure work". One of the old-timers stops some of the work and says "no that can't be done, there are important lines buried there". That information is on none of the documentation, or perhaps the documentation has been lost. Without that old-timer to raise his hand, the lines would be broken, costing the town tremendously.

Extrapolate that to everything - as the older workers retire or leave the workforce, they take their information or knowledge with them. Or, in the general course of time, documentation gets lost or destroyed. And the incoming generation is less capable, through no fault of their own (sorry Boomers who instinctually want to berate the young), and the result is that the system degrades, bringing it towards collapse.

[u/[deleted]]

Given all the dumb shit Blow has said over the years not sure I put much value in his opinion on this.

[u/funkinthetrunk]

Stewardship and institutional memory are vastly underrated in US corporate and government culture

[u/doge2dmoon]

A Taiwan war . ...