Welcome everyone, to my talk about Lua. My name is Andreas. You might have seen me on
the nickname of ComicSansMS which is a really silly nickname,
so it's usually not taken, except on Twitter, because
Twitter's full of crazy people, so I had to use an even
sillier nickname there. I'm one of the organizers of
the Munich C++ User Group, so yeah, if you ever happen
to be in our around Munich and want to get in touch
with the local community, give us a call, we'd be happy
to set up a meeting with you. And my day job is incredibly
working as a software architect for BMW on the autonomous
driving software platform. So today I'm going to talk about Lua. So can I maybe get a quick show of hands, how many people have used Lua
in like a serious project, like not a toy project but like, something production,
yeah, couple of hands. Okay. For those of you who have not used it, hopefully I can convince
you that it's a language that is worth investigating
even for serious projects. So, what is it that is
distinctive about Lua that sets it apart from other languages. I'd say there is two
features that are kind of unique to it, the first
one is that it's actually not designed to be a standalone language. It's an embeddable
language, how they call it. So usually you have like some application, like the most famous use
case for Lua is being games, that is written in some other programming language, often C++. And then you have Lua
in there to customize like some small parts. But, it's also being used for programming microcontrollers,
embedded microcontrollers, or also like serious,
serious desktop applications, like Adobe Lightroom. The other thing that is
kind of unique about Lua is that it's small. It's really small. So you're not supposed to be able to read this thing on the left, but that is actually the full EBNF syntax of the language, which
fits exactly on one A4 sheet of paper, the compiled
binary is really small. These 180 KB is actually
the full Lua package including the standard library if you strip out the libraries and the parser if you only
need to run compiled BiCode, you can make this significantly smaller. The complete reference
manual is only 82 pages is really the complete
documentation including the full CAPI, that is
required for integrating it with a enclosing program, it only has eight basic data types. As you can imagine, from these numbers, there's not a whole lot
of features in there. Sort of if you take like the Python approach of batteries included, Lua is sort of the opposite
of that conceptually. But, the cool thing about Lua is that, for them is actually not a limitation. Is actually a virtue. And not only because this
allows them to run on devices where other languages
wouldn't fit, it's also that from a language design perspective, they really embrace this
property of being small, and turn that into a
very elegant language. So, I always like to
say, the whole language fits into your head at the same time. Which is something that
I cannot say of most other programming languages. I certainly cannot say that of C++, unless maybe you're one of those guys, but like for me, if I use
C++, I have to constantly swap in and out of the
different features of my head, and with Lua I don't need to do that. Which is really nice. So, hopefully I will be
able to convince you, in the progress of this
talk, that although Lua is small it is quite a powerful
and interesting language. However, I have to say,
the problem with this talk, is like, um, I cannot assume that there are any Lua
experts in the audience, so on the one hand it's going
to be an introductory talk to Lua, but on the other
hand I also want to show interesting examples, right. And the problem with that
is that these interesting examples, they're sometimes
a little bit contrived, so and a little bit too
clever, so, what you see here might not be representative of what you would write in every day Lua. Like, typically it might be a lot simpler. In particular what I want to say is, if at any point during the
talk you have the impression that oh no, this is like,
really unnecessarily complex, and I don't get it and I don't really want to use this stuff, please blame that on the talk, and not on the language. That out of the way, this is Hello World. Notice that although my syntax highlighter likes to make the print
blue, it's actually not a keyword, it's just a function from the standard library,
which in Lua's case means it's just an ordinary function it does not get any special treatment whatsoever. So this is how we declare a function. It's a dynamic language, so we
don't have to give any types, we just say that we have three arguments and give them names, the
interesting thing about Lua, this is the first thing
where we actually see it, it being small and lightweight, is that functions are
actually not like this, this special kind of thing that they are in C, but all functions are actually lambdas, all functions are anonymous,
and this declaration that we see up there,
is just syntactic sugar for what we see down there,
which is the construction of an anonymous function,
which is then being assigned to a variable of type f. Right? So, f is just a variable like any other, and you just assign to it
a value of type function. And what that really
means is like functions are true first-class values. Function values are not
any different from strings or numbers or anything else. Okay? So that way we can reuse
all of the properties that we have for the other
values also for functions. That keeps it small, but that also enables some interesting things. So, for example, let's say I'm not happy with the print function here. I want to replace it by my own. I can just do that. It's just a value. I just assigned to it a
different function to it. No problem. Let's say I want to count
the number of print calls. That could be a use case for this, right? I just replace the vanilla print function by the new function, and
there's actually valid syntax, this is a very adept
function, which just increases the counter, and then calls the original print function within. Now, you notice that there's actually not any variable declarations in this code. I just assigned values
directly to variables that were not previously declared. This is something that
most scripting language actually will allow you to
do, so if you just assign to a new variable, you
don't need to declare it, it just automatically
creates a new global variable and puts the value into it. Which is not a behavior
that is always useful, but we will get to that later how we can. Maybe work around that
if we don't like it. What is really not nice about this code is that we now polluted
our global name space with a variable named count. So, if we only ever
count the print function, don't do much else in the
program, that might be okay, but really it doesn't look that nice. So what we would like
to do instead is keep this count function at a smaller scope, so, for example, have a
function enable counting, which like performs this
hooking of the print function, and have the count function
and the old print function as local variables in there. So here notice if I prefix
the variable with the local keyword that declares that
variable as a local variable, it works pretty much exactly the same as local variables in C++. Now the interesting thing is
if I now write the same code that I wrote before,
what will happen is that the local variables from
the end closing scope will be implicitly captured
in that new local function. So like in C++ if you wanted to do this, you would have to write
in the area brackets of lambda, the captures explicitly. So like the names of count and old print, you would have to specify
them, write them out. In Lua you don't need to do this. Just as lexical scoping, so it knows that it can pull in the
stuff from the outer scope. The interesting question is now. I just made the count
variable a local variable, so how can I access my count now. Like if I want to find out
of often print was called. I can just return a
function that gives me back the value of the counter. And this is interesting now, right. Because the count is an
integral value, so it has value semantics, and it
also has value semantics in Lua, just as it has in C++. So, now have captured the same value in two different lambdas,
in two different closures. And it still works. So this is actually where the full in full lexical scoping comes from. That allows you to do something like this. This is actually really
difficult to pull off in the implementation,
and Lua couldn't do this from the beginning, but
since version 5.0, I think, they support it, and it's
a really powerful feature. Notice that of course
since we capture stuff that means that our function
objects are actual objects, so whenever you read
function, interpret that as construction of a closure. So there's actually memory
potentially being allocated there and this needs to be garbage collected, potentially, at some point. Okay. Let's talk about tables. Tables are the most important
data structure in Lua. In fact it's the only complex
data structure in Lua. And, I actually don't
know of any other language that can get by with
just one data structure. Which is quite fascinating. So what is a table, like,
this is the simplest table, it's just constructing an
empty table with nothing in it. But a table, imagine it as
being an associative array. So, you can use it just as
you would use a C array, but you can also use it as a dictionary. Using a, using it to map keys to values. And the interesting thing about Lua tables is that they actually allow
you to map any type of value, to any other value, so for
example, you can use functions as keys, you can use other
tables as keys, if you want. This is again, very
much, the Lua philosophy. Right, like you only
provide very few features, but you make those as
flexible as possible. Notice that tables, unlike the
numbers that you saw before, have reference semantics. So, for example, what we do
here, is we basically built assembly link list, we
construct like two note tables, each of them holding a value, and then we link them together, through the next field of the first note. You don't usually need to
build link lists like this, because like if you just use plain tables, they usually powerful enough
for what you need to do, but it's useful to know
that you can do it. As with functions before, whenever you, read the curly braces, think
of it as table construction, so this means that you will
potentially allocate memory. Since tables are only complex data type, if we want to build like a
struct, or like a record type, you also have to do that with tables, which means like we simply create entries in our dictionary, in our table, for the different fields of the struct. And since the syntax that we
already know for accessing the tables is a bit
cumbersome for this use case, Lua provides syntactic sugar with the, for accessing the fields with a dot. So the last line is just syntactic sugar for the line buff there,
completely equivalent otherwise. So we want to add member
functions to our record now, we can just assign a function
to one of the fields, and then call it as member function. So notice Lua's tables
have reference semantics, so this function actually mutates, the value inside the table,
and the call down there, this is, think of it as
a member function calling in C whether to give
this pointer explicitly. Seems we don't want to pass it explicitly because it's kind of silly,
we also get syntax sugar for that, with the colon syntax. So, the thing about tables is that, we don't really have types right, so all of our complex numbers
have the same type table. So if we want to build
multiple values of this table, we usually have to supply
a constructive function like we see here. So let's say now we have
these complex numbers and we want to add to some. For this we would actually need
operator overloading, right? And Lua provides this through
a feature called Metatables. So everything in Lua is a table right? So Metatable, the idea here is basically, I provide a table which
I then attach to a value, and this table tells me
how to treat this type that is attached to, in the language. So, there's a couple of special fields, and they allow you to
specify how it treats the other authentic operators,
the comparison operators, how it treats the element access with the outright brackets and so on. So, I just changed my
construction function to add this Metatable, call that, assigns the Metatable
to my newly constructed complex number, and then I can just add them with the plus operator. So, let's say now I have
more complicated data type, where I actually have invariance
between the different, on the different fields. Might not want to allow the
user to access them directly, but rather encapsulate them. So, instead of exposing
your month and date of the date directly, I
actually want to provide getter and setter functions. And the way that we do this
is in the construction, we actually build a
local table that contains the actual data, then what goes
in the table that we return, is just to getter and setter functions, which again access the actual
data through the closure. So some of the similar trick that we saw with the counter before,
and that allows us to give, to implement encapsulation
without any explicit language support for it. Which is kind of cool. Of course, since a table is the end just an associative array, we also get a certain
reflection properties for free. Like, a table is a complex
number if it has these fields, real and imaginary. And of course I can also
inspect all of the fields by just iterating over the table which in Lua I do with this
syntax, so the pairs function just gives me back an iterator function, which with each invocation
returns the next field of the table, and then I
have this generic form, that processes iterators. So this allows me to perform
reflection on individual table that I have already in my hand, but what about global variables? Right? I might need to know, which objects are aligned around in my environment. So Lua has a solution for that. Global variables are actually
not global variables, they are entries in a table called _G. So, as I said, everything was a table, and global variables
are just syntactic sugar for accessing elements
in this global table _G. And this has a really
interesting implication. Cause like, as we saw
before, we don't need to be clear variables. So what people really
don't like about this, if I make a silly typo
like this, where I assign to fobar instead of foobar, the language will just silently create
a new global variable. Which is not what I wanted here. But, G is just a table, right? So this creates a new entry
fobar into this table. And I can just use
Metatables to forbid this. So, I can use, say, you're
not allowed to create any new entries into this table. And Lua actually provides
on their homepage module that does this in a little
more complicated way, and the idea there, is that
you can sort of restrict where you are allowed to
create global variables. So like only allow to
declare them in like certain parts of your script, or
only be allowed to declare them through declare function, which then bypasses the Metatable. And this is I think a pretty cool feature and it basically comes for free from the properties of the tables, and the fact that everything
in Lua is a table. So, this is all pretty cool,
but if you have ever tried to integrate scripting language with C++, you know that this is
usually not a lot of fun. So, how that does work with Lua. So as we said before, Lua
is an embedded language, that means the main functions
still belongs to C++. And in there, we use Lua
more or less as a library. So, we create a Lua State object, and then we can just run some Lua code. So who thinks that doing
something like this with a scripting language should be any more complicated than this? No one. Okay, that's what I thought. So, just running Lua, chunks of Lua code in isolation is not very interesting. We also want to communicate
with our C++ program. So in order to export
functions from C into Lua, so that Lua code can
call them, we actually have to wrap them in
functions of this signature. Like any function with this
signature can be directly injected into the Lua VM. And as you might notice
there, this actually does not take any function parameters and it also does not
provide any return values. It just takes the Lua State object. And that is by design. So the Lua API is all designed around this concept of a stack. And the idea of the stack is that whenever you want to get a value out of the VM or into the VM, you have
to put it on the stack, and then you can grab it from the stack. And this is, this might seem
a limited design at first, because like if you imagine
if you want to set a value inside a table, you have to
put the table on the stack, you have to put the key on the stack, and you have to put
the value on the stack. And then you have to make
the call that does the actual assignment, which is a little bit verbose. But there are two
fundamental advantages here. The first advantage is that
it keeps the API very small, because we have only
eight data types overall, and you only need two
functions per data type, one to put it on the stack and
one to get it from the stack. The second implication is
a little bit more subtle but I think it's even more important, is that it solves the ownership problem. Cause the problem with garbage
collector languages is, if I now grab like a table,
from inside the Lua VM, and expose it to the C++
code, I actually need to make sure that the
VM's aware of the fact that I am pointing to this table, so that the garbage collector doesn't pull it out from under me. And with Lua, I cannot access the table if it's not on the stack. But as soon as it's on the stack, the stack has ownership on it. This might not seem like a big deal, but if you're using this in everyday code, this is actually really powerful. So, pushing values on the stack, I'm just gonna use number and string as the example types here, some of the types are a little more complicated, but not much. So let's say I just
have two push functions, so two overloads for push functions, one that pushes a number, so Lua numbers are usually double, but if you compile for a platform where you
don't have double available you can actually change that very easily to like pure interval types. I would push a string. And then we can very easily
write a function like this in C++ 17, which uses the fold expression, to just push an arbitrary number
of arguments to the stack. Is everyone familiar with this syntax? I've seen it quite a
lot this week already. So if you don't have C++ 17, available, you can also use a library to do the same, so here's an example that uses boost hana. And if that also doesn't work for you there was actually a
talk, I think yesterday by Joe Fan-Cue, where he
have like some examples on how you can emulate this
feature with all the C++ versions, I just put the
code up for reference, you don't have to read it, you're certainly not
expected to understand it. Because it's horrible. But it's possible. So what about the other way around. How can I get values from the stack, so out of the VM into C++. So here the thing is that since
Lua is a dynamic language, I don't know what the type is
of, of the value on the stack. So I need to do the switch case here. But, the thing is that,
this is actually the only, the only point where I'm willing to stick to this factor where
I don't know the type. Because like, as soon as
this function returns, I'm actually again in the C++ type system, so I want this to be like, as
rigorous as possible again. So, what can we use to
as a return value here? So we could of course use a base class, like, solve the problem
with virtual inheritance, but that's just so 90's, right. So, in this example, you
have to define what like your unified interface of
all the different types is, in my case it's just a type function. So each value can tell
you what it's type is. And then you could just use a
C++ 17 variant, for example. So, like in a full
implementation you would have, there's only eight types in Lua. And you know them all beforehand. So variant is actually really good fit, and you can also implement it with very little size overhead. Like you might lose some
bytes when saving numbers or bullions in there,
but for tables, functions and the more complex types,
it's pretty much uniform for memory consumption. And then of course, this
is how you get stuff out of the variant with visitation. If you don't like variants and you would like to use a
different kind of type erasure, Louis Ti-Yong was giving a very nice talk yesterday about runtime polymorphism, where he explained all the
different options that you have if you don't want to use a variant. Okay, so let's call a
function now from C++. Let's call a function. So, we load the function on the stack, get it by its name, put
it on the Lua stack, and you push all of our
arguments onto the stack. And we call the function,
and then we inspect the stack to get the return values that were returned by the Lua function. And I noticed that in
Lua, a function can have an arbitrary number of return values, so I actually have to
check, after the Lua call, API call returns, I have
to inspect the stack to see like how many
values were put on there. This is also the reason
why here I have to return a vectoral value, because
I'm not sure how many arguments am I going to get. And then I can just call a Lua function, like here I'm calling a print function, from the Lua standard library
with a couple of arguments, and we just print them to the console. So of course this is the most general signature that you can give for such a core function. You could constrain it,
like if you know the number of return values and the
number of arguments beforehand, or maybe even know their types. Right? So, you probably want
to go to like the most restrictive signature
that applies for your case to make sure you get the maximum support from the C++ type system. Okay. So to wrap up. This is actually the
description that the Lua people give like how they
understand the language, their elevator pitch. It's a powerful efficient lightweight embeddable scripting language. And I hope through this talk
I could give you an idea of what this attributes
in the context of Lua. I compiled a small list of literature, the reference manual for Lua is actually really good,
there's a book by the authors of Lua called Programming
in Lua, which is, it's not a big book,
it's around 300 pages, if you're interested in the language I would really recommend
purchasing the book, not only because it's a great book, but also because it's
a great way to support the creators of Lua financially. Because it's created by a university, and like they don't get a
lot of funding otherwise. And that, I think we have
some time left for questions. Thank you. (clapping) - [Moderator] If you have a question, please go up to the microphone. Hello. Thanks for the talk. Yeah. You mentioned that you have an application that uses several libraries,
for example, dynamic libraries, and each of these library want to do some of its own stuff, and as
separate Lua interpreters. So, how do they get, do
they conflict with this or okay for them to have several interpreters in several application? You mean, I think,
several Lua interpreters, in one application? Yes, I mention several
modules are developed independently, and all of
them rely on their own Lua, something, so I want
those to be independent. Yeah, so what you can
always do, like we saw in the simple example
that you have to create this Lua State at one
point, and if you want the different modules
to be truly independent, you can just have each
module use its own Lua State. And then they can even run concurrently in multiple threads,
there's like zero overlap between the different
VMs, but then of course you have to take care that
if you do want communication from one Lua VM to the next,
than you need to model that, but if you're fine with
them being totally separate then that's actually a good way. The problem there is that
there's some memory overlap to those Lua States, so if
you're in a very constrained environment, you might
not have that memory. And what you can do there, is you can also restrict the environment. So this _G table, that we saw, that stores all the global variables,
you can also restrict that. So saying that I'm now
executing a chunk of Lua code under this environment. And sometimes that's
enough of a separation, but it's a little bit weaker than having truly separate States. Yeah great. And the second question is, it seems like to use Lua
functions conveniently from C++ coding, it's
some kind of boilerplate, but we demonstrate this, so I wonder, is there some standard solutions. Some what? Standard solutions, some libraries which contains such boilerplate. A library which takes care
of the bindings for you. Well that provides such functions, functions like Kol, like that is shown. So, there are libraries that
help with the integration, a pretty famous one is
I think called Luabind, you can use those, but
the interesting thing, is that the API is so
simple, and you want to roll your own, this is not an
unreasonable thing to do. Like you can probably get a decent library up and running within a weekend. So, there are libraries
available, but you don't have to use them, like it's easy enough to do it by hand, if you want. Thanks. Sure. [Audience Member 1] So
how do you debug Lua? Oh yeah, that's a good point. So the interesting thing is actually part of the standard library of Lua is actually a debug module, so you can actually write your debugger for Lua inside Lua. Which is pretty cool. [Audience Member 1]
You can separate points and things like that. Yeah. And you can inspect like local variables from closure and stuff like that. So. [Audience Member 1] And
then you write your own. Like if you want it to go to Standard Out, and have some sort of
interactive mode, you -- The thing is that this is just an API. This is not a ready-to-use debugger. So you would still need
to write a user interface in order to be able to actually debug. But since Lua's an embedded language, so you don't know what the
environment is going to be, they don't take care of that for you. So if you only use
vanilla Lua, you will have to write it for yourself, but it gives you all the tools that you
need to be able to this. Thank you. [Audience Member 2] Just
wanted to make a quick comment if you don't mind, on
the question of something to help generate binding? Sure. Sol2, that's S-O-L two, is
a really well-written modern C++ binding generator for Lua. Okay, cool. Thanks for the info. Okay? And I think we're through. Thank you very much. (clapping)
