AT&T Archives: The UNIX Operating System

the usual way to get a large computer application developed involves a big team of people working in close coordination most of the time this works surprisingly well but it does have its problems and large projects tend to get done poorly they take a long time they consume an astonishing amount of money and in many cases the individual team members are dissatisfied so everybody in the computing business is constantly searching for ways to do a better job of developing computer applications there aren't likely to be any final answers both because the problems are hard and because as we find solutions we try even more ambitious objectives but there are some things that can be done to make life easier for everybody on a large programming project a good programming environment helps a lot and in the next few minutes we're going to show you some of the properties of the UNIX operating system that make it a good programming environment for many purposes in Bell Labs as in many industries almost everyone has some kind of involvement with software either they are actually producing software and that is their job or they are impacted by software or they use software fact that Bell Labs about 50% of the people are actually producing software and everyone else has some kind of involvement with it in fact that's one of our worse problems today there is a crying need for useful software to do effective jobs we just do not have enough people for write all that software keeping large amounts of software working and keeping it working in the face of change is a big job takes a lot of skilled people to do this now software is different from hardware when you build hardware and send it out you may have to fix it because it breaks but you don't demand for example that your radio suddenly turn into a television and you don't demand that a piece of hardware suddenly do a completely different function but people do that of software all the time there's a continual demand for changes enhancements new features that people find necessary once they get used to a system in other words we put the system out there people get used to it their jobs change they come back with more demands for different sorts of features in the system the result is there's no way to get perfect requirements in the first place and that means that we have to build the software to be very changed tolerant because we do not want to throw the software away the year after we wrote it there are a couple ways to do that one is to make the software fairly clear and easy to read and understand and change and you do that with some of the current popular structured programming techniques another way is to write many many small modules of code that way when you have a change perhaps you only throw out a few small modules or make changes in a few modules rather than in thousands and thousands of lines of code what we should be doing in the computing business is trying to raise the level at which we work so that a programmer can write a few lines of code that turn into many many instructions in the machine that way when changes need to be made one just changes a few lines of code rather than thousands and thousands of them in the mythical man-month fred brooks estimates that it took 5,000 staff years of effort to produce the operating system for IBM's 360 series computers clearly nobody is going to do that sort of thing very often certainly not for every new type of hardware or for every new class of application someone once said that software stands between the user and the machine and to me this conveys this picture of a great wall of software up there that you have to overcome to get anything done there's certainly a grain of truth in the remark anyway if you stop to look many many operating systems seem to spend a substantial fraction of their time and effort not in helping you but in impeding you in making your job difficult sort of providing obstacles to be overcome when Ken Thompson and Dennis Ritchie start the unit system 1969 they found a structure which simplified many aspects of the interactions between computers and people Thomson and Richie were aiming to keep their system simple and they found a collection of primitives that enable them to do a great deal with the very few primitives a UNIX system is made up sort of of three layers if you like the central layer the kernel is the thing that controls the resources of the machine then wrapped around that at least in conceptually is something called the shell which is the interface between most users and the kernel part it sits there and waits for you to type commands at it and then it interprets them and then around that sort of an yet another layer are useful programs things like editors and compilers for programming languages and document formatting programs and programs that you write yourself and what you can do is to think of these UNIX system programs basically as in some sense the building blocks with which you can create things and the thing that distinguishes UNIX system from any other system is the degree to which those building blocks can be glued together in a variety of different ways not just obvious ways but in many cases very unobvious ways to get different jobs done the system is very flexible in that respect I think the notion of pipelining is the fundamental contribution if the system is you can take a bunch of programs two or more programs and stick them together end to end so that the data simply flows from the one on the left to the one on the right and the system itself looks after all the connections all of the synchronization making sure that the data goes from the one into the other the program's themselves don't know anything about the connection as far as they're concerned they're just talking to the terminal let me give you an example of how this works in practice the system is as I mentioned is used a lot for document preparation kinds of things programs for helping you type letters or produce technical papers or write books in all of those things of course people when they're typing into machine make spelling mistakes so let's see how we could use some of these building block notions in practice to help you develop a program for finding spelling mistakes suppose I take a sentence this is a sentence which is in a paper that John Massey and I wrote some years ago now if you look at it carefully you'll see that there are actually a couple of spelling mistakes in there now suppose that we wanted to find the spelling mistakes using a machine how would we do that well basically what we would do the simplest thing I can think of is to split the sentence there in two words individual words and then compare the words against a dictionary and every time we find a word which is in that sentence but not in dictionary it's at least a plausible contender for being a spelling mistake now how do we do that what I want to show is that you can do that using just existing UNIX programs just gluing them together to get the job done suppose that we say first we will take a program called make words and we'll run that on the sentence now what that does is break the thing up into one word per line now I'll take the output and I'll pipe it into another program which will simply convert it into lowercase the reason I want to convert it into lowercase is that my dictionary doesn't have any capitalizations and so words like Bell and UNIX which are capitalized here would show up as spelling mistakes unless I did this now the next thing that I want to do my dictionary is in fact sorted in alphabetical order dictionaries are and so it's a lot easier for me to compare the words of my document to the words in the dictionary if they're started so I'm going to run them into sort and finally if you look at it carefully it doesn't show up very easily here but there are in fact there is a duplicate word their systems appears twice and in a real document words like TIG would show up many times so we'd like to get rid of duplicates so let's throw that through another program called unique so what we've got so far is we've got the words of my document in this case the sentence one word per line in lowercase neatly sorted and all of the duplicate words thrown away so there's only one word one instance of each different word then what I'm going to do is run it into one last program called mismatch which will simply print all of the words that came down this pipeline and print out the ones that were in the document that word in the dictionary what we have here is five separate programs cooperating to do this job and in one giant pipeline now if you look at the list that came out you'll see that indeed we got laboratories and provide which were our two spelling mistakes of course we've got two other words as well and this tells you not only was what what's good about the approach but also what's bad about it time sharing is not a spelling mistake but it's a perfectly fine example of technical jargon the sort of thing that means something to everybody in the computer business it means nothing whatsoever to people who are not computer types and the word UNIX is a fine example of something that's not going to be found in a normal dictionary so what do we do first we take the misspelled words and we go back to the original document and we correct them so we don't have any spelling mistakes secondly we take the words that like time-sharing and unix that are not spelling mistakes but which showed up here and we put them back into our dictionary so that the next time somebody has a document that contains the UNIX or time-sharing they don't show up as spelling mistakes so we've not only done our own job but we've improved the tool that we're using in the process so you notice that I did that whole job without writing any programs at all the whole thing is cobbled together out of programs that already exist it and all I did was to use the fact that the system provides this mechanism of the pipeline so that I can take programs and stick them together one after another to get my job done and I think this is one of the reasons why the system so productive that there's a large collection of things that people have already built that we use and as we build our new things then they become part of the repertoire of things that people subsequently can build on during the last decade we have discovered a number of new powerful pattern matching algorithms that are useful for locating patterns and texts many of these algorithms have been developed using insights obtained from theory obtained by studying automata and language theory as our knowledge of pattern matching algorithms increases we can very quickly take this knowledge and package it in the form of UNIX programs and we can spread these UNIX programs to the entire community very quickly the UNIX systems has many features which make it easier for the programmer to write programs these include form atlas files the hierarchical directory structure the ability to pipeline the output of one command as the input of another device independent i/o all of these things make programming considerably easier than on most other systems the heart of the system is really the file system the ability to store information for extended periods of time and the reason one of the reasons the system works as well as it does is that the file system is well designed and many systems you have to say an awful lot about a file before you can do anything with it you have to say where it is and how big it is and what kind of information it's going to that's going to be in it all kinds of things that are basically utterly completely irrelevant here you don't have to do any of that the file is as big as it is it doesn't matter where it is as long as you know what it's called and so you basically don't have to think of any of those complexities that you have in other systems when you want information in a file you put it there when you want it back you get it out again and you don't have to think about size or number of Records or number of fields or anything like that unless it's really germane to your program for most purposes it's utterly irrelevant a file is simply a sequence of bytes its main attribute is its size by contrast and more conventional systems file has dozen or so attributes to specify or create a file it takes endless amounts of chitchat if you want a UNIX system file you simply ask for a file and you can use it interchangeably wherever you want to file the UNIX system consists of a hierarchy of directories which a directory is simply a file that contains the names of either other directories or files and this whole thing goes on recursively when you log into a UNIX system you normally are sitting in a place that's called your home directory or users directory and I can say TWD which means print the name of my working directory and it'll tell me where I am it says at the moment that I mean user bwk that's where I start when I log in now I can go up a level in that I can change to parent level and now if I print my working directory I'm in slash user and I can go up one more level to the root of the whole file system let me go back down to BW k and I can list the direct the files that I have in that directory and I find there among other things a directory called TV and I can list the files that are there and I'll find among other things the sentence that we printed in a spelling mistake finding program they look at that and sure enough there it is so as you can see the file system hierarchy makes it possible for users to organize information into its natural grouping and to go up or down and find things quickly and easily the UNIX system interface for most people is through program called the shell or the command interpreter basically it's simply a program that watches what you type and treats it as requests to run particular programs now there's nothing magic about running programs the programs that you run are actually just the names of files in the file system the shell searches in the file system in a particular way to find a file whose name is the name of the program that you think you're running and it goes and executes it and in fact it's not possible for you as a user just by executing a program to tell how that particular program has been implemented for example it might have been written in a language like Fortran or C or it may have in fact been written as something like the spell program that we talked about earlier which is a combination of other programs stuck together with pipes or some similar thing and all put in a single file as a shell sequence or sequence of commands what Brian did earlier was he typed all the commands the the five program names for his spelling checker on one line using the pipeline facilities that's nice except that you may want to check documents often and you don't want to have to type that long sequence of commands so it's possible to put all of these commands in a file and tell the shell when I type the name of that file I want you to execute the commands that are inside that file let me show you an example of this we have a program called spline which fits curves to a set of data points and I've got a set of five data points that we're going to see what the curve looks like I'm going to run spline through a program that turns this into graphics called graph and I'm going to run that through a special program that turns the graphic language into something specific for this terminal I only need to type plot and data because inside the file plot is the string of commands and here is a result of plotting those five data points on this particular terminal the ability to put commands in files and only have to type the file name to get these commands executed makes the computing business a lot easier often you're doing things that are repetitive and you don't want to have to type long lists of things this makes our life much easier and allows us to tailor our environment for the way we want to work another nice feature of the UNIX programming environment is the concept of input output redirection normally when you type a command the output from it goes to your terminal and the input comes from your keyboard however the shell can be told by a simple notation that when you run a program you wish the output to be directed into a file or that the input be taken from a file for example to print the output of my spelling program on the line printer instead of putting it on my terminal all I have to do is say my spell sentence greater than device line printer rather than my spell sentence and the output goes into the file what looks like a file except that it's actually a file that causes the line printer to spring into action and print my three or four spelling mistakes on the line printer on many systems redirection of input and output is literally impossible because the programs have wired into them the notion that they have to read or write the user's terminal and there's simply no way to convince them otherwise they'd have to do that here that is not the case here any program can have its input or output redirected because the input and output redirection is handled not by the individual program but by the shell and so that way it applies to all programs without any exception at all and in fact this goes a little further than you might expect because not only are parts of the disk files as they are in other systems but in addition the i/o devices the peripheral devices connected to the computer are also files in the file system for example the line printer and the tape drive and even the thing that dials telephone numbers are all devices in the file system in the same program that will copy information from one disk file to another disk file will also copy information from a disk file to the line printer or from the magnetic tape drive to the printer the same program exactly a good operating system is easiest for programmer to use if the programming language fits with the style of the system so along the way in the course of the unit systems development Dennis Ritchie created the C language C is a very nice high-level language with many of the modern programming constructs in it the thing that's very important about it is that it lets you avoid the details of the machine when you want to but when you need to and sometimes when you're writing an operating system you really do need to you can get at the details of the machine and control everything but you're not forced to do that and that's important because that means you can write operating systems in this language and still have something that can be portable to other machines the UNIX system has been moved to many many different kinds of computers again that means that people can ignore the details of what a machine is underneath and get on with their job now so at that level C is by far the favorite language at the next level the shell programming language is very popular back on some machines people find that the shell meets all of their programming needs they are writing lots of procedures to help them manage their work they don't even have to go to a language at the level of see as it happens though because the system is such a pleasant programming environment programmers all over the world have imported or added their own languages so for instance you can find Fortran Algol Lisp basic fact almost any language you can think of exists on some UNIX system somewhere what's important about the unique system is not so much what Ricci and Thompson put into it as what they were able to leave out of it rather than produce a large number of primitives each one complex they were able to choose a small number of simple primitives which could be fitted naturally together to accomplish complex tasks this structure of the operating system makes it natural and easy for people who create applications to produce applications in that same style for example as the scale of integration of silicon circuits gets ever larger we find it necessary to have more and more sophisticated design aids to help people create large-scale integrated circuits our existing design aids are advanced and effective but advances in VLSI create a need for even better tools rather than produce those tools in the form of one humongous program designed to do everything the people here have been producing small packages each designed to do some individual function that's helpful in the design of integrated circuits then these individual packages can be combined using shell procedures to design a part of a circuit or a circuit and the parts of a circuit can be combined to make a whole circuit Steve Johnson is one of the people who has been involved in this effort he is currently working on a program called Elgin which takes boolean equations as input and produces logic circuit designs as output because tool building is such a way of life on the UNIX system over the years we've developed tools that actually help us make other tools these involve things like parser generators lexical analyser generators and other programs that help us organize and develop tools these tools have been used in the development of L gen and many other applications here we see the boolean equations for a simple matter down here we have the equations for the carryout and the sum in the middle we have some descriptions as to how we would like the cell to be laid out geometrically see we would like the two inputs on the left side the carry in on the bottom the carryout on the top and the output on the right side the input equations first have to be processed so that they can be more easily represented in silicon this process is very similar to recognizing common sub-expressions in the input of a compiler input language the boolean equations are read and processed by a program called yak which was originally developed to help us build compilers but has in fact been used in a large number of application programs as well yak is based on the theory of lalr 1 parsing represents it builds a small finite state machine which is able to control the actions of the program which reads the input detect errors accurately and structure the input in such a way that the program can then go ahead and perform its operations on it after these equations have been processed it's then necessary to worry about the geometric layout of the circuit this is done in the next 2 portions of el jem the first program worries about the ordering of these columns it uses a technique called graph partitioning to attempt to iteratively come up with a good solution to what is in fact an extremely difficult problem in theory after the columns have been ordered then the tracks where the signals run are laid out as well by another program and finally in some sense we now have the circuit designed and it's simply a question of realizing it with the particular rules for our fabrication process and that is done by a fourth program so once again we have an example of taking a very complex problem dividing it into pieces representing each piece with a separate program and then using the facilities of the UNIX system to glue the pieces together into a coherent whole again computing is going to be more and more inner woven with people's lives as the years go by so computer technology is going to have to evolve to be easier for people to use the unit system is not the end of the road in this regard but I think it's a good step along the way you you you

Channel: AT&T Tech Channel

Views: 1,256,192

Rating: 4.9639001 out of 5

Keywords: AT&T, Archive, Tech, Channel, Technology, Programming, Bell, Labs, Unix, Victor, Vyssotsky, John, R., Mashey, Brian, Kernighan, Alfred, Aho, Dennis, Ritchie, Ken, Thompson, Lorinda, Cherry, Language, Kernel, Shell, Fortran, Basic, Lisp, History

Id: tc4ROCJYbm0

Length: 27min 27sec (1647 seconds)

Published: Wed Feb 22 2012

Reddit Comments

[u/fx-9750gII]

If you enjoy this video, there’s an old book called “The Unix Programming Environment” you might like. It’s like an intro to Unix circa the 80s, with lots of explanations of how things work, and example commands (most of which still run). It’s a very easy read—unlike other technical material. And a great companion to “The C Programming Language” of similar vintage!

[u/blasto_tidesly]

Watched the whole thing. A treasure to behold.

[u/phobug]

https://web.mit.edu/~simsong/www/ugh.pdf

[u/calvers70]

He accurately describes software development in 2020 in the first minute 😂😂

[u/FredSchwartz]

Last year Kernighan had a reminiscence interview with Ken Thompson at the Vintage Computer Festival in NJ. One of the most amazing talks I’ve ever been to.

https://youtu.be/EY6q5dv_B-o

[u/Ramast]

Wish I could do today what they could do in the 70s

cat /etc/passwd > /dev/lp

[u/avaruuz]

The whole AT&T Archive on YouTube is fun to watch. Mostly educational and insightful videos, plus some really bizarre ones.

[u/pjdaemon]

Workspaces were so peaceful in those days, it looked like each one of them had a spearte room to work in. Today's so called agile workspaces are absolutely horrible, can't concentrate for more than 5minutes if I don't wear NC headphones

[u/BloodyShadow23]

I saw this video when it was posted to r/OldSchoolCool in the past. I think it is great that AT&T is able to put these videos online. It's amazing how we really don't use these exact programs anymore but the underlying architecture is still a solid platform.