GopherCon 2021: Linus Lee - When Toy Languages Grow Up

today i want to talk about programming languages and specifically toy programming languages so languages that you might not use that work to build things like databases and servers and your production grade services but things that you might build for a side project or to solve a very specific kind of niche problem i've spent the part of the last year and actually the last couple years building various little different toy languages and with go i want to share a little bit of both what the process was like what i've built on top of it and then also how you might go about implementing your own toy programming language building a language is is a little bit like this xkcd where you start out with a problem that you might want to solve maybe it's about building a database or maybe it's about solving a particular audio kind of engineering issue and you start with a problem and you say you know what this problem would really be great to solve if i had the language that's just perfect for it you start working on the language and of course the language isn't something that you build overnight over a weekend over a couple weeks it's a month-long year-long kind of project and so it consumes all of your time and that's a little bit what happens with me too but i do think that building a language is a really fascinating way to learn about different parts of a computer you might not come face to face with on a day-to-day basis and so i want to share what that process was like for me in a previous version of this talk that i gave earlier this year i talked a lot about a language called inc it was a language based on built with go it's a dynamically typed scripting language it looks a little bit like this what you see on screen and i spent it i spent the last two years using this language using the interpreter and the compiler that i built for it to build things like notes apps and twitter clients and various kinds of personal tools that me and a few other people use and ink was my first programming language that i built on top of go and so while it was useful it also contained a lot of little design mistakes and learnings that i had to to get over the last two years and so earlier this year i started working on a second version of sort of a successor to ink if you will called oak and so most of the rest of this talk will be about oak the language how it works breaking down exactly how i built it using go and then some things on top of it that i built using oak that might be interesting to talk about when you're building a toy language you're not really trying to replace something like go or rust or c plus plus because those are powerful general purpose languages that you're meant to be used that are meant to be used to build anything from lower level programs like databases all the way up to web apps or even more complex high-level applications with a toy language you're more targeting at a specific problem or maybe you're scoping it down just to be able to do the things that you care about and i would say those things are in the scope of little languages uh little languages are everywhere there are things like the bash scripting languages that you use in your shell the jsx templating language like configuration languages like yamo or tamil or even one of my favorites the google wefts language the wrangling untrusted file format safely language which is a fully featured language that you can use to write parsers that are memory safe that you can use to parse untrusted files like media files that you get on your phone or network payloads that you might get to your server my favorite example of little languages of all time is the awk programming language if you haven't heard of it there's a book called the aquarium language written by the authors of the language it's it's a pretty short read you can read it over an afternoon and learn the whole language because it's so small but it's one of those things where it's designed so carefully and so thoughtfully for a very particular domain in this case that's processing text files and streams of text files in a unix pipeline kind of context and all of these are examples of languages that are targeted at a specific kind of solution and it's sort of a language-based problem-solving approach rather than writing a program in go to say process files or or write configuration you're using another language to do it and i think little language is an interesting approach to solve problems and maybe you write a toy language little languages are an interesting way to think about what your language might be targeted at solving a problem about so why might when you build why might you want to build a toy language the biggest reason for me is fun not only is the fun in building an interpreter or compiler and being able to understand exactly how a programming language works but also once you have a language like inc or like oak you can build things on top of it the notes tab that i use day to day is written in oak the twitter client that i use to check my twitter is written in ink and once you not only is it the fun that you get from just building your own tools but once you're building it on top of your own language you're working with the syntax that you defined with the semantics that you defined with the tools that you built with the details into the language that you've baked in just because you wanted it there and you didn't really have to care about scalability or or availability or anything like that that you have to consider for production-grade languages and so building tools on top of your language can be really fun i also think building a toy language is a great way of learning about different parts of a computer that you might not be in touch with in your day-to-day work things like memory models things like the process model or asynchrony and concurrency how does your language deal with problems that happen when two threats try to write to the same memory location or what does what happens when you have to read a file and you want to do it concurrently these are details that programming languages that you use day-to-day today have worked out for you already because they have to think about it when they're designing a language so you don't have to think about it as much as you would if you're writing your own language and you have to define these behaviors specifically and exactly so you can then not only design it but implement it into your language to be correct so i think it's a great way of learning about these different types different areas of a computer and then sometimes very sometimes occasionally toy languages or personal project languages can become production grade with languages like clojure or like python having started out sort of personal projects that have now grown into production grade languages that large companies use to build their production products if you've taken a university course or if you've gone through another kind of tutorial on how to build a language before you might have come across material on how to build an interpreter or compiler and while that's what gets a lot of attention in the programming language world how an interpreter or compiler works how to do type checking how to do parsing i think there's a lot more to building a language especially a language that's practical that you want to build stuff with than just the interpreter and the compiler and so even if you've taken a university course or encountered how to build an interpreter elsewhere i hope to through this talk share a little bit about what else on top of it do you need to build to have a practical language things like for example if you have just the interpreter the go spec and the go compiler doesn't actually tell you how the go language interfaces with the rest of your computer how does it do things like writing files or processing network requests those you have to define separately in standard libraries that you have to write for your language or foreign function interfaces interfaces to the rest of the system and those are things you have to explicitly design as well another important part that gophers know a lot about is language tooling things like code formatting and profiling for memory and cpu usage how do you deploy your software these are things that typical programming language tutorials haven't covered that i have to sort of discover for myself and i want to share with you a little bit of how at least oak handles it so i want to dive a little bit into the interpreter because i think it's the most technically sort of interesting part of oak and share a little bit of how it works but before we look at the interpreter itself let's look at what actually what oak actually looks like to run so oak is a programming language written in go and that's a little bit weird but what that really means is that to run oak you just start type in oak enter and like python or like any other interpreted language you get a rebel and because it's an expression-oriented dynamic language programs can be as simple as just numbers so if i type in the program 100 it evaluates to 100. i can add numbers together i can add more numbers together i have strings in oak so you can say hi i can say hello gophercon you can concat these strings together also has floating point numbers so phi oak has composite types as well so it has lists of numbers and lists and objects in oak are actually heterogeneous so they don't have to be all the same type so you can have a list that contains numbers and also strings and also things like booleans and keywords and all these are just values that you can work with in oak there is no concept of pointers or slices or arrays everything is just lists and values and objects which are defined sort of the way that you would see it in javascript for example and lastly oak has first class functions so for example we could define a function called double that takes a number and returns two times that number so i can then say double the number 10 let me get 20. i can double it and then pipe it which is a little fun syntax to double it again that's how you would do it and then because okay first class functions you can use functions as arguments to other functions so oak has a standard library function called state range that just gives you a list of numbers up to that number so we can generate a list of 10 numbers or 100 numbers or a thousand numbers let's take a list of 10 numbers and then let's map it through the function level and all this is doing is taking the list and then for each element of the list doubling the number and so oak has first class functions it has numbers strings the typical sort of values that you would expect in a programming language but all of this is again a go program so oak is just by which it it's about a seven megabyte go binary and so when we look at how the oak interpreter works what we're really looking at is what is the go program that takes the source code the text strings that i'm typing into the structural and how does the go program understand it what it's supposed to do and then actually execute what the program is meant to be doing within the program oaks interpreter like most other interpreters or compilers works in a sort of a pass-based way where we start with initially a representation over source code which is the representation of a program that humans can read and process the most effectively and we progressively transform it down to other representations of a program that the computer can more easily understand so to a computer the source code of oak is just a string it's very opaque there's there's no structure to it and then we're going to go to token streams and syntax trees which i'll explain a little bit that contain more structure so that the computer can look at it and understand what the program is supposed to do so let's look at the token stream and the syntax tree and what that what that looks like in practice this is a little playground that i made recently to be able to look at type in any oak program and look at the token stream in the syntax tree representations of a program that correspond to the program that i'm typing in so let's start with a very basic trivial program with the number number 100 the number 100 is a program just evaluates that just evaluates the number 100 and the token stream is a representation of a program where every symbol in our source code has been singled out and the token stream is just a stream of these tokens each of which are a symbol that means something in our language and so for example in this oak program number 100 there's two tokens we have the first token at position one one which is line one column one it's a number literal and it has the value 100 and then there's a comma that's automatically added by the tokenizer in the same way that go has automatic insertion of semicolons at the ends of lines that's something that i borrowed from go but really there's only one meaningful token in our language in in this program here which is the number that are all 100. if we add to it number 200 now we have three different tokens that are meaningful we have the number little 100 we have the plus operator and we have another literal 200 and you'll see that the tokenizer one of the abstractions that the tokenizer peels back is the white space so no matter how many spaces i add here there's only three three tokens so as you'll observe that as we transform it down more to a machine understandable representation of a program some of these abstractions that we don't need like like white space will be removed by these progressive steps the other thing that's interesting about our token stream is that it's just a flat list there's still no sense of nesting or hierarchy so for example if we have uh one plus two plus three let's put that inside a list there is semantically there's nesting in this program now where we have a list value and we have numbers that are being added together in sort of nested expressions of course you add one and two and then you add three after it all that's inside a list but our token stream doesn't actually represent any of that nesting structure we're just looking at again a list a flat list of symbols we have a bracket we have a number literal we have operators other numbers and the right bracket to close it off and so the syntax tree is when the nesting structures come in to our program so let's look at the syntax tree together with our token stream representations of a program so now we start again with a number 100 again this is just a program that evaluates to the number 100 and we have a single token that represents the number literal but our syntax tree now adds additional information for the computer's people to understand or program so now we've gone all the way from just a binary blob of a string to okay we're looking at an integer and that integer has the value 100 if we add another number the token stream is still a flat list of just number plus operator number but somehow in the in the past between the token stream and the syntax tree we have to go from this flat list to a tree that says okay now this program is a binary expression it's adding something together this plus the first thing that we add is an integer with a value 100 and the second thing that we add is integer with a value 200 and so you can see we're adding a little bit more structure we're adding a little bit more nesting and we're getting closer to something that's structured enough for the computer to be able to understand our program if we add another number you can see there's even more nesting so now there's a binary expression nested inside another binary binary expression and let's look at a more complex example so we can use the print line function from the standard library to have a program that prints hello world and in this program there's a lot of structure there's enough structure here for say a program to be able to tell you exactly what this program does so at the top level of this program it's a single function call right so our syntax tree says okay here's a function call the function that we're calling is the result of a property access we get the identifier stood and we get the identifier print line and we access the print line property of stood and then we call that function with the argument string hello if we add other arguments our syntax tree will tell us yeah here are other arguments integers 2 3 and 4. and so again going from the program to the token stream to the syntax where we're adding structure we're removing arbitrary information we don't need like white space and the last important thing that the syntax tree step does or the parsing step does is it detects syntax errors so for example if i remove the parenthesis at the end here well now we don't know when our function call ends and so that's not evident in the stream because it's a flat structure and we're not really giving a lot of attention to the nested structure of our program text but in the syntax representation of a program we have to know when our function call ends to be able to tell you what the arguments of this function are and so when we can't find that parenthesis our parser is saying hey there's a syntax error there's an unexpected end of input or if we have a symbol that we don't expect it'll say there's an unexpected token so a lot of the rest of the interpreter work is going to be implementing the transformations to go from the source code to the token stream and then to the syntax stream so let's see how that works in practice to go from the source code to the token stream we'll get we're going to have a pass called a tokenizer a tokenizer in an oak is a single function defined on the struct called the tokenizer its job is to go from the source file representation of a program to a slice of tokens a list of tokens that we can work with and while it's doing that it also keeps track of each token's original position so if you look back at our tokenizing tokenization step or token stream you'll recall that each token has some position information about which line it's from and which column it's from and that's important because whenever there's a runtime error for say being able failing to look up a variable we have to be able to tell the user here's where your error occurred or a language will be undebugable so our tokenizer does that by keeping track of the file name the column and line for where we are while we're parsing our our source code and to parse resource code we also need a slice of runes uh and an index into that slice to go along our list of characters and and tokenize it in practice the way that we do it is a big switch case so this tokenized function that abbreviated here is going to go through our string which is a program and for every character it's going to say what is this character maybe if it's a plus sign it'll say the plus sign is a valid token by itself so we're just going to take that plus sign and say this is a token that's the plus operator and ship it off to our list of tokens if it's say a letter we'll say maybe this is the start of a variable name so we'll parse the rest of the variable name look at it if it's a keyword like if we'll consider it a keyword and ship it off as a token an if keyword token if it's a more general variable we'll just say this is an identifier if it's a string literal it'll start with a quote so if we see a quote we'll say okay the rest of this token must be a string that will almost parse it until the next quote will slice that string off from the rest of our source code and consider that our token and so it's a big switch case looking ahead a few characters to see what the next token is going to look like and aggregating all together into a single slice of tokens that represents our token stream which is what we see on the left side here the token stream within the oak interpreter is represented as a slice of tokens and each token is just a struct it keeps track of the type of the token it is so is it a comment it's a comma is it a parenthesis it's position which we saw earlier which is just a column and a line number um and this payload which is the string value of the token for the tokens for which that is relevant so things like string literals numbers comments those need to know not only what type of token it is but also what exactly is the value of that token what's the value of the string or what's the value of the number that this token represents and this is this is where that payload goes and on the left here you can see a shortened version of all the different kinds of tokens that has so commons comma dot all the parentheses and delimiters operators keywords as well as all the literal values that are language supports we need a token type for each of them once we have the token stream the next step is to go from this token stream to the syntax tree and that's the job of the parser if you've worked with parsers before you might understand that oaks parser is a simple recursive descent parser what that really means is that the parser looks a lot like a token a tokenizer in that it keeps a track of a slice of tokens which we got from the tokenizer and then we have also an index that points into that list of tokens to see exactly where in the token stream we are as we're parsing along and popping up uh syntax tree nodes of our tree the actual parse method on the parser is recursive descent and all that really means is that for every token that we see we're going to check okay what is this token and the what what syntax tree node might that be starting so let's say if it's a number then we're just looking at a number of syntax for you know like an integer 100 or something like that if it's a keyword like the function keyword that starts a function we might say okay actually we need to parse a function so we can invoke the parse function method that that's defined on parser and the parse function method is going to look at the next thing after the function keyword let's say maybe that's like an argument list and say okay now we need to parse an identifier and so we're going to call the parse identifier method and so on and so forth so it's parse methods recursively calling each other to be able to parse whatever comes next in our list and we keep parsing through our list of tokens until we get to the end and we have all of our syntax tree nodes in our program and what results is the syntax tree that has the structure that we saw earlier in our demo since the syntax tree in oak's interpreter is uh represented by a single type which is actually an interface type rather than a concrete struct type as before one thing that's interesting about this sort of series of transformation is that as we go along further in our inner steps we get more structured information and so when we were just looking at the token every token could look about the same because every token is really just a type and some extra string value whereas now we're at the syntax 3 stage and each node needs to know a lot more information about itself so the information that a string literal node needs to contain is very different than what a function literal node could contain a function looks very differently than a string and so each of these different types gets its own struct string that rule has just a payload of what the string literal contains as a string a function might have a name a list of arguments and some extra body an if expression contains multiple branches that it needs to know about and so all of these are represented by different struct types and they're all wrapped up in an interface type that we call ast node for abstract syntax screen node once we have a syntax tree now we have enough information to be able to run our program in a sense and so it's time for the evaluator the evaluator is the thing that actually runs our program it's a method defined on this thing called a context which i'll get to in a bit but all it does is it takes it looks at every syntax tree node every bit of our syntax that's now in a hierarchy and says okay what is this node telling your program to do for example if it's a string we just generate a value that represents that string and pass this off to the next thing if it's a function call we might look for that function and then call it if it's a branch of an if expression we might evaluate the condition and then call that branch if it's true or false and you'll see there's an extra bit of scope here and the scope is just the information that encodes what variables are in your current local scope and so the eval expert function is actually responsible for traversing through our tree and going through and actually running the program that the tree represents let's take a look at what all of that comes together to to be when an oak program is actually running so again oak the oak interpreter is just a go program and so in this go program that is running an oak program we have a big global interpreter state that keeps track of all the state that's relevant for the running oak program as a whole things like what are all the open files that our oak program has opened what are all the open network ports it also keeps track of a global lock which is just a mutex on the global interpreter state and this ensures that even if multiple concurrent functions are being called in oak only one bit of the oak program is running at any given moment so if multiple files are being opened and multiple requests are being served one convenient way that oak gets around the problem of data races is we just ensure that at any given moment only one part of the oak program is being evaluated only one eval expert function is actually running through our source tree or our syntax tree and that means that we don't really have to worry necessarily about multiple parts of our program writing to our variable or stack at the same time or complexities that arise from parallel execution so in the global interpreter state there's a lock as well as all the information that that the program globally needs to keep track of the internet program oh no program can import other oak files this is the way that oak sort of does module management in the way that you might do in lua and javascript and so within the global interpreter state there is multiple contexts or multiple files that we might have open so in this case we might start by we might have started out by running main.ok but main.org might have imported library.ok which might have in turn imported math.ok and in each of these files there's a separate chain of variable scopes because if you think about it a variable in one file shouldn't be able to access other variables from a scope in a different file and so each of these files need their own chain of variable scopes that the evaluator can look up variables from so let's look at what that looks like in practice in our source code the global state that we just talked about is represented by this engine type right here this is the mutex the interpreter lock that makes sure that all the at any given moment only one oak program on one bit of oak program is running we have some other sort of synchronous bookkeeping stuff we have a file map for keeping track of which file descriptors correspond to which open files that we've given to our program as well as some error reporting bookkeeping things that i won't cover to in too much detail here but within this global interpreter context every file gets this thing called uh a context it's a little bit confusing but this context type keeps track of okay for every file what are what's the local scope with the global scope that corresponds to that open file so if we start to define variables in our oak program then all those variables we got stuffed into this context data type it has a pointer to which global context that this context belongs to which engine it belongs to this root path thing which is bookkeeping for being able to do imports nicely and then it embeds a scope type which is the the go type that represents the local scope of variables and so in every scope there's a map that just maps variable names to the values that they represent so if you define a variable a that has the value of number two then it's going to be a map that has the string a and then it's going to map to the value that corresponds to the runtime representation of the value 2 interval program and each scope has a pointer to its parent scope because a local scope if the variable doesn't exist there looking up a variable in the local scope might involve looking it up actually in its parent scope so you can see how all that comes together to form a runtime state that looks like this where there's a global interpreter state and then each scope is sort of changed to its parent scope until you get to the global scope for each file and then all of that is wrapped up in a little context that corresponds to each file so now going from the source code to the token stream to the syntax tree which has enough structure for our interpreter evaluator to run in this runtime this is all we need to have an oak interpreter with all these parts coming together we can now take an oak program give it to the interpreter and have it do something that we want like adding numbers together or creating objects and running through objects or looping through lists all of that's possible with just this basic infrastructure but there's a lot more to oak than just in its interpreter for example with just the things that we've covered here oak can't write to your terminal yet it can't save a file all those things are outside of the realm of just the core language and the things that you have to build into the runtime separately as things that you can call out to how i'm going to highlight three things that are outside the interpreter itself outside the language that i think are really important for oak and for languages in general that i found interesting to implement within oak the first is concurrency how oak handles concurrency this is important because if you want to open files or respond to web requests one thing that i use oak for a lot is writing web servers and web applications and for that you need concurrency because otherwise you're going to be you're going to have to block on every web request which is unideal and normally when you're implementing your own toy programming language to do concurrency correctly to have multiple threads of execution running at the same time you would need to implement your own event loop your own little local state that keeps track of which requests are in flight which functions are paused or or unpaused but go makes this really easy for oak because for every request for every file that we need to open for every asynchronous task we can just spawn a new go routine and we can tell the go routine do this task read a file or write to this network network port and then once it's done run this bit of o code later so when we give oak a call back that says write this file and then call this call back when it's done we can implement that pretty literally within go and goes go routine and it's run time ago routine system in its runtime makes that pretty easy for us to do we don't have to implement any sort of asynchrony ourselves in the interpreter oak also has a growing standard library that i wrote from scratch for doing things like string manipulation formatting strings dealing with cryptographics secure randomness and dealing with setting up http servers and the like and the standard library is important one because it enables you to write actually useful programs like web servers and two because i think it's a good way to test your program if you're writing a toilet test your language if you're writing a toy language to make sure that it feels good to use under your hands and also set some conventions things like naming conventions things like how how complex are can functions be how does it feel to have a really long functions of relation versus really short functions and these things are things that are not built into the spec of the language or the interpreter there are things that you have to sort of conventionally defined as you use it and writing the standard library was a good way to exercise my muscle and actually feeling out how the language is supposed to feel to use and then lastly language tooling this is something that god does really well and i wanted oak to do this really well too and so i have a little more to say about language tooling specifically that oak has self-hosted language tooling what this means is that oak's code formatter it's bundler it's compiler javascript compiler it's testing and documentation tools all of it is written in oak itself so for example this is a screenshot of oak format okfimt which is directly inherited the name from gofund and it it formats oco you can give it a file program file source file and it'll format it and spit out a another oak program that's indented correctly and all this all the spaces are normalized and correct and it's written in oak go has something similar where gofund go doc go test all of these things are written and go itself right and the value of writing these tools in the language itself self-hosted language tooling is one that improving the language actually improves the tooling itself so if oak gets faster oak foam oak build all these tools are going to get faster alongside oak if go gets more secure or more memory safe all those benefits are going to be inherited by the tools too and i think that's a great benefit of of writing self-hosted language tooling not to mention you're exercising the language so you're making sure that it works correctly and there aren't any bugs the other underrated benefit of the way that specifically oak and go do it is that for both of those languages parts of the language tools the functions and the basic data types that you need to be able to use to write things like parsers and linters they're built into the standard library and for go that means there's libraries like go types go ast the go static analysis tools that you saw earlier in the conference those things are part of the go standard library or part of the more popular packages that come with go like xcode tools and what that means is that it enables sort of an ecosystem of tools that are compatible and it makes it really easy to write things like linters for your code or things like code formatters on top of that the initial foundation in the standard library so this tool that i showed you earlier for showing a token stream in a syntax tree that that results from an oak program is actually written in oak this is a web app written in oak itself both the front end and the back end it's compiled oak down to javascript and all of this output is generated by an oak program that is using oak's standard library support to be able to parse oak code itself which is a little bit kind of recursive in mind bending but what this means is that it's really easy to write oak programs that work with other oak programs and language tools are fairly straightforward to write because i don't have to write go code anymore to write these tools i can just write oak code that knows how to parse other code so within oak there's a few different tools that i've written so far for code formatting for auto indenting code a bundler that takes a bunch of different oak files that import each other and producing a single file sort of like webpack for the front end and a compiler that compiles out code to the web to be able to write tools like this because i like writing web applications and oak has similar enough semantics to javascript that compiling it to javascript is is not that difficult and then in testing and documentation tools oak test new docker not written yet but that's something that i'm hoping to stamp out in the near future all this time having written inc a couple years ago and then now working on oak i've come away with a few takeaways about writing interpreters and programming languages in go itself most of them positive some of them things that might not be might mean that go is not the best language for a job but overall go gives you all the basics it gives you good enough performance where you can if you design your interpreter correctly you can compete with slower languages like python and ruby it gives you easy cross-platform support because if you write your runtime in go then ghost cross-platform support is there to make sure that that language will work on all the platforms that go supports go also gives you a static binary executable when you build your language this is actually much more important for an interpreter than it is for maybe a web server because in a web server you can have multiple different files that you need to run your program and you can just put it all in a docker container ship it off to to your server but when you're writing a language like oak you want to be able to give a single binary to your user to an oak programmer and say this is all you need to write your own program just run it on your source code you don't want to give them 100 files that import each other and say okay set this up on your computer and then you'll be able to run oak so the fact that go compiles to a single binary is actually really important for how ok is practical to use as a language alk also gives you enough just enough control over the runtime in memory representation of oak values of things like how is an oak floating point number represented or how is an oak list represented in memory and it turns out that's important to be able to get any kinds of performance tweaking that you want to do in your language or to be able to do things like um make sure that your language doesn't consume too much memory when you're writing your programs and then my favorite part of using go for writing oak specifically for the language type of language that oak is has been inheriting ghost state of the art runtime so oak is a garbage collected language just like oh and that means that oak just inherits goes low latency garbage collector and because the state of the art oak also has that state-of-the-art garbage collector go also has great cpm memory profiling tools if you haven't checked out goes prop support for looking at what functions are costing you cpu time and what variables and what values are eating up your memory i suggest you check it out because it's been fantastic to be able to work with whenever i have a memory leak in oak it means that i don't have to write a memory profiler for oak i can just inherit ghost tooling and run it on the interpret run the go memory profiler on my interpreter and that'll tell me which kinds of values are eating up all my memory in my oak program the runtime also has a downside though which is that because it has all this great support for these tools and instrumentation inheriting goes complex runtime means that sometimes doing some things in oak can be impractical for example one thing that can't really do that well or at all at the moment is interface with a graphical interface so usually the way that you build a gui is by interfacing with a c library like sdl2 or an os specific library like coco on mac and to be able to do that your language has to call out to c functions and the because of the way that c go is and because of goes sort of thick runtime layer between the rest of the operating system and your go code it means that calling out from oak into a c function uh calling an oak function that calls a c function and passing values in between them is kind of tricky something that i haven't worked out yet and so sometimes go can't quite be low level enough so depending on what type of attraction you want to work out with your toy language goal might not work for you but i have found that for oak go has been good enough for me to get to build the types of apps that i've wanted to if you're interested in building interpreters or compilers in go or just in general how programming languages work after seeing this talk the first research that i recommend is first involves writing an interpreter and go writing a compiler and go i've heard a lot of great things about this book i haven't read the book yet but i've heard lots of podcasts and talks about it and you can find them in interpreter book and compilerbook.com and they both of these books will walk you through building a practical toy programming language called monkey in go and every every source code every bit of source code line by line and i think it's a great practical way to learn how to write your own language the way that i learned it is through crafting interpreters which is an online ebook there's also a hard copy but again written in java but walks you through the same sort of conceptual ideas on how an interpreter works there's a few other resources on the slide but the the three other ones i'll highlight are tiny go go for lua and auto all of these are go interpreters go based interpreters that are targeted at a real production language that people are using to build real products so these might be interesting to look at for looking at how a go interpreter might implement something a little more complicated and a little more real world a little more like the real world like a javascript interpreter my favorite part of working on a toy programming language and go has not necessarily been the writing the interpreter or the compiler or the parsers although that's that's educational i'm learning a lot my favorite part is actually in having built a language and then now being able to write side projects with it write things like my notes have with it or write things like a context manager with it ramsey nasser who works on the closure compiler for for the clr he says languages represent different ideas on how to capture human creativity on a machine and i think this is the thing that makes building languages side projects really special it means that when you're solving your own problem instead of using other what the rules that other people have defined to solve your programming problem you can define your own words you can lay out your own vocabulary for how you think about a problem what these like primitive values and rules that you work with are and you can define your own constraints by which you try to build things that you want to build or solve problems that you want to solve and i think that's really powerful part and the really motivating part of trying to build your own language and so if you're more interested in writing programming language and how oak has done it there's a blog post where i talk a little more about programming languages and writing them at linus pl i have a website at thecyphus.com where i talk more about software and also other things and then the oak project itself is open source it's just a go program after all at lightest headzone slash oak so if you want to go and check it out and fork it and build your own variations top evoke you can go there and read the source code it's not that complicated on the few thousand lines of go and i hope after this talk you'll be able to make sense of it and all the parts that go into it

Channel: Gopher Academy

Views: 176

Keywords:

Id: ALwmdcFiuGg

Length: 38min 11sec (2291 seconds)

Published: Fri Dec 17 2021