- All right, welcome, everybody. My name is Craig Scott, and I'm here to talk to you about Deep CMake for Library Authors, and we seem to have a bit
of a feedback problem. All right. Just about me before I get underway, been a cross-platform C++
developer since about 2001. I'm currently one of the CMake maintainers in a volunteer capacity, and also the author of the
book Professional CMake: A Practical Guide. A lot of today's content in today's talk will come from that book. For those who are interested, I am also available for
consulting services. You can get in contact
with the details there or catch me this week, if
that's of interest to you All right, so today's talk
is going to be focusing on libraries, and
specifically shared libraries. I'll talk a little bit
about static libraries, but the focus is primarily
on shared libraries. And I'm also focusing specifically on cross-platform library projects. That's the main focus of the talk. Along the way, I'm
going to be highlighting CMake features that
help you with this task and also identifying some things that you would need to be aware of as cross-platform library authors to make sure that CMake works
for you, not against you. All right, so today I'm going
to essentially be focusing on these four questions, and as a library author,
these are four questions that I encourage you
to be asking yourself. So the first thing is the obvious one, what actually is your library? And this is about controlling your API. We often do think about
what is in our API, but maybe we don't think so much about what is not in our API. So we're going to talk
about that a bit today. Also going to be asking the
question of how do you evolve your library from one release to another? What are your changes in your API? How are you communicating that? How did that interact with
things in the operating system? And that sort of thing. Along the way, we'll be
talking about different ways that other projects might
consume your library, how they bring your
library into their build, and the different ways
that that can affect you. And we'll also be looking
at the different ways that your library can be packaged. May not just be you. There's lots of other different ways that your library might
find its way into a package. And we're going to be looking
at those different methods. All right, so API control, what do I mean by API control? We're usually pretty
good at saying, "Okay, my API is this." We're usually not so good at
saying, "My API is not that." And by not that, I mean
not exposing things that are not part of our API. Now, typically people are going to say, "Here's my documentation. That's my API. I've defined it there," or, "Here's my public header files. If it's in my public header files, that's my API." Problem with those is
that doesn't stop people from using things that
you don't want them to. So if you have parts of functions that are available in your library, but you don't want that
to be part of your API, you don't want people to
be calling them directly 'cause you might change them at any time, your documentation and your headers aren't going to stop
people from using them. If you need to stop
people from using things you don't want them to use, What we're talking about is
symbol visibility control. And by that, I mean in
the actual binary itself do we expose the functions
and the variables so that people can use them? If we don't want them to use
them, we shouldn't expose them. And I'm going to be focusing
on this particular aspect of API control today. I'm assuming that you're
pretty good at defining, "This is what I think my API is." We're going to focus on
this how we make sure you don't use the rest. All right, I'm going to use
this dinky little simple class to demonstrate symbol visibility for you. Now, if we had this as our header file and we had the corresponding
CPP implementation file, and we put that into our shared library, what would we get with
different compilers? Well, with Visual Studio it hides symbols by default. So if you did nothing,
if you just had this code and you use that directly, and you built that into
your shared library, you would not be able to use this class outside of your library. Constructors wouldn't be available. Destructors wouldn't be available. The nextValue() member
function implementation wouldn't be available. GCC and Clang, on the other hand, it's completely the opposite. Everything is public
and visible by default. So you could certainly use this class in the GCC and Clang case. Now we want to control our API. So we want things to be hidden by default and only exposed to things that I want. So Visual Studio gives
us the default behaviors that we want. So let's start there. So with Visual Studio, if you want to make your class visible, you would mark it with
a __declspec(dllexport) Now may look a bit sort of nasty and scary if you haven't seen these
sort of things before, but it's relatively straightforward, except there's always an except, when you're building with
Visual Studio, it wants you to put dllexport when you're
building your library, but if you're consuming the
library, it doesn't want that. Ideally it wants dll import. You need to differentiate
between I'm building my library versus I am consuming my library. Now I don't know about you,
but as a library author I don't want to have to
produce two different headers for the two different scenarios. So we do the usual programmer thing, how we add an extra level of interaction, and we replace that by our own symbol, and that symbol we could try
and define on the command line, but then so would
consumers of our library. And we don't want to make
consumers of our library have to do anything special. It should just work out
of the box for them. So rather than specifying as a
command line compiler define, the standard arrangement is you provide it via a dedicated header. The job of the header
is purely just to define that MYTGT_EXPORT symbol. Now, what does that header look like? Well, this is one example
of what it might look like. Now, might just feel
like we're just shifting the problem around a little bit here, but let's look a bit closely. When we're building the library, we define this symbol here, MyTgt_EXPORTS. That means that when we're building it, that symbol is defined, we get an export behavior. When people are using our library, they won't have that symbol defined and so they will get dllimport behavior out of the box without doing anything. So we do something when building it, consumers of our library do nothing and get the right behavior out of the box. All right, that's an
important aspect of this. So that's like controlling
simple visibility for Visual Studio. the GCC and Clang case, the first thing we have to
do is change their default. We wanna make things hidden by default. Now, GCC and Clang and any other compiler which aims to be compatible with those provide two compiler
options that control this. Why two? Well, in their wisdom, they have two separate options, one for inline code and
one for non-inline code. There is a good reason for it. I won't go into the reasons why today, but it's very important
that you have both. The -fvisibility=hidden That will control the non-inline case. The fvisibility-inlines-hidden, is there for the inline code. Now think carefully about
what that inline code means. All of your templates, all
your template instantiations, all your STL algorithms,
your STL containers, all the types mentioned in those, all of your own templates all of those would be exposed without the second compiler option. It's very important. It also has the added benefit
of drastically shrinking the size of your executable and
speeding up your load times. So win-win-win. All right, so you need
both of those options to get the right default behavior. GCC and Clang support
similar way of specifying that something should be
visible rather than hidden. And this is the way that it will recognize out of the box,
__attribute__((visibility("default"))) Default just means make it visible, the opposite to that would be hidden. Now GCC and Clang more recent versions do also understand the declspec that Visual Studio uses. Do not use that because
you have to turn it on with a compiler option, which means your consumers of your library have to have a compiler flag just to be able to use your header. And we don't want that. So don't use that compiler
option, please, please. GCC and Clang will recognize
this attribute out of the box. You don't have to do any
special handling for that. Now that looked a bit scary, and we can do exactly
the same transformation that we did before. Replace it with our own symbol, same name as what we did before. We'll define that by header, same name as what we had before. That's really actually kinda cool. Now I've got exactly the
same header for Visual Studio for Clang and GCC, and I've got my visibility control. My mytgt_export header is the
only thing that's different. And in this case for GCC and
Clang, it's relatively simple. They don't differentiate between,
"I'm building a library," versus, "I'm consuming the library." So it's a bit of a simpler
contents of this file. All right, so we've reduced
it down to this export header is the only thing that's different between the two families of compilers. How does CMake help us here? So this is all it would take to take care of simple visibility. The first thing we do is we say, "Regardless of which compiler I'm using, I want all of my symbols
to be hidden by default." And these two variables
that we're setting in CMake, they effectively correspond
to the two compiler options I showed you earlier for GCC and Clang. For Visual Studio, it will
essentially do nothing 'cause it doesn't need to do anything. You can set these things
on a per target basis using target properties, but chances are you're
probably going to be doing this across your whole project, and it's generally easier just
to set these two variables probably at the top level of your project, and that will apply to all
targets created after that. The other thing that we do is we generate that header, that mytgt_export.h header You don't have to write it yourself. Often people try with varying success. You don't need to. CMake can generate it for you It will generate one that's appropriate for the compiler that's being used. Now in the previous slides, I chose the naming of
things very carefully to match the defaults
of this command here. The generate_export_header command we give it the name of the CMake target that we want it to apply to. By default, it's going
to create a header file with the target name, converted to lowercase,
append "_export.h". The symbol that you place
between the class keyword and your class name, target name converted
to uppercase, _EXPORT. It will also take care of
adding the compiler define to the target for you so
that it does the right thing when you're building, and that does the right thing
when it's being consumed. Now, you can override all
those names if you want. It's very flexible, the generate_export_header command, and there are certain scenarios where you may want to do that. But for the vast majority of cases what you see on screen is
generally all you really need. And just to be clear, this also applies to
more than just classes. It also applies to free functions, and also to global variables if you really must use them. Just place that MYTGT_EXPORT out the front and you will also be able
to control the visibility of those things as well. All right. So that's how you define symbol
visibility for your library. What about how you evolve
your library, your API, from one release to another? You need to be able to communicate to a number of entities what's going on. People want to know what's
changed from the last release and packaging systems wanna
know, "How does this relate to the previous version that I have here?" What we're really talking about here is your versioning strategy. Now, I'm going to stand up
on a little pedestal here and make one piece of
advice, one piece of request. Please use a conventional
versioning strategy. You can invent your own if
you want to make life harder for yourself and for your users. But if you choose a conventional
versioning strategy, you will find your path is much smoother. Packaging systems will be
better able to support you, tooling support will be better, users will better understand
your version evolution. Now for today's talk, I'm just going to pick a common
one, semantic versioning, just to be able to demonstrate
how versioning interacts with the operating systems. If you don't have a
version of the strategy and you're thinking of one,
give this one a serious look. And the URL there gives
you all the details that you'll ever need. For the purposes of today's talk, I will just give you a very,
very brief version of it so the subsequent slides
will make some sense. So in semantic versioning, we have a major.minor.patch version number When you make a change to your library, which is purely just a bug fix, your API stays exactly the same. It's purely an internal
implementation change. What we do is we increment the patch part of the version number and a
major and minor stay the same. If you add something to your API, but it's still backwards compatible, so maybe you add a new class or you're adding a new
free function, for example, if someone had built
against the previous version they could drop in your updated library and their application
would continue to work. They wouldn't have to rebuild it. When you make that kind of a change, you increment the minor
part of the version number, and you reset patch to zero. If you make a breaking change, so something in your API has changed, if someone built against
the previous version, they'll have to rebuild
against your new version. Then we increment the major
part of the version number and we reset minor and patch to zero. Okay, so that's essentially
what semantic versioning boils down to. I've simplified it a little bit, but for all intents and purposes, that's what you need to know for today. So let's have a look at what happens on a Unix-based operating system, or something that's Unix-like. So when you build your library, you often see a set of files and symlinks that look
something like this. So my library name is
Example, and I'll freely admit that's probably the worst
name that I could have chosen for today's talk, but we'll go with that. Now, all the different
Unix-based operating systems will typically have
some variation of this. The version part might
come before the suffix, the suffix might be different, it might be .dylib rather than .so But essentially they all
have something like this, and they operate more
or less the same way. The part with the full version
number in it, this file, this is the real library. This is the one that will
contain the actual code. And this this is the meat, if you like. May surprised you, but
that that's actually meant mostly for humans and
for packaging systems. You'll see why in a few slides. The part that only has the major
part of the version number, that symlink is known as the soname, and that is meant for the runtime loader. So when the operating system
goes to run your application, this is the name that it
is going to go and look for for all the dependent libraries. So if you do on Linux something like ldd to list all the dependencies
of the particular binary or tool -L on Mac, it's the sonames that it's
going to be printing a list of. And the top one here with no
version information at all, it's called the name link, and that's for the linker at build time. So if on your linker command line, you have -l Example this is the file it's
going to be looking for with no version information at all. Now as a library author, it's the soname that you're
most interested in here 'cause this is what
determines your compatibility from one release to another. All right, so how does that
map to what CMake provides? So on the left is the CMake
code that would produce the set of files and symlinks
that you see on the right. We define our library and we basically just set a
couple of target properties on that library. The obvious one, the
VERSION target property, naturally that maps to our
full version that we see for our real library name. The SOVERSION is what
provides the major part of our version number for the soname. The name link, CMake is always
going to create that for you. If you provide no version
information at all, everything just collapses
down to the name link. If you provide a full
version of information, it will create the name
link separately anyway. Now, some interesting questions. What if we don't specify a soversion? What if we just say version 2.4.7? What happens? And you see this quite a bit in projects. Well, this is what you'll get, but it might not be what you think. If you omit the SOVERSION what CMake will do, will say, "Oh, you've given me a
VERSION, but no SOVERSION. I'm going to set the SOVERSION to the same as the VERSION." Now, the consequences of that is it now you're soname has
the full version number in it. So if I make a bug fix,
change nothing in my API, purely an internal implementation change, and say, "Okay, that's a patch. I bump my next release
number, will be 2.4.8." I've just broken any app that linked against my previous version 'cause my previous version would
have had a soname of 2.4.7. Yeah. So generally, if you omit
the SOVERSION target property it's probably wrong. It's probably not what you intended. What about if I specify a SOVERSION which has no relationship to
the version number at all? What do I get with that? Well, this is what I get. Is that valid? Yep, and there are packages
out there which do this. This is effectively highlighting
what I meant earlier about the real library is meant for humans and for package managers. Linkers don't care what
that version number is, and runtime loader doesn't
care what that full version is. They only care about the soname. So in this case, this is demonstrating
that full version number, you can almost think of
it as a marketing version, if you like in this scenario. Having a completely
separate SOVERSION of 9 rather than 2.4.7, it's perfectly legal. I'll take questions at the end. Yeah. What about on Windows? So I've focused just on Unix so far. What will this CMake code we have here produce for you on Windows? Well, you get this. You get no version information
at all in your file names. You get no help. But all is not lost. So the dll, that's your shared library. That's acting kind of like
what a soname does on Unix. On Windows, the runtime
loader is going to go looking for this guy. It's gonna go looking for that name, that dll name. So it's kinda similar to the soname. The import library, that's only used by the linker. So it serves a similar
purpose to the name link. So there's similar roles there, but you don't get the version information embedded in the file names. Now, in some cases, CMake may embed some version information. Let me clarify. If you're using the Makefile generator or the Ninja generator, CMake will embed just the
major and the minor part of your version number into the dll, and it'll embed it as the image version. If anybody can tell me what the
image version is useful for, please come and tell me after the talk because I went out looking for this. The only reference that
I could come across which had any reference
to the usefulness of this was to do with the installers. And even then, there
was very little detail. So please come and see me afterwards if you can shed light on that. it doesn't affect your ability
to run with that library. If you're using Visual Studio generator, you get nothing, nada. CMake will not embed any
version information at all. And that was only
something that was realized while I was preparing this talk. It's probably just never been implemented on the Visual Studio Generator. All right, so that's library version. Now, for many of you your library version is going
to match your package version. Quite possibly, your
library is the only thing in the package. So we need to talk about this. One way that people consuming your library will bring it into their build is via find_package in CMake, and this is going to go
looking for a pre-built version of your library that's
been installed somewhere. Now, when CMake sees a command like this, it's going to go looking
for a couple of files. Well, actually, two sets
of a couple of files. There are two naming conventions. Don't ask me why. It just is. You can pick one. They're entirely equivalent, either set. I'm just going to talk
about one set today, but all that I talk about
applies to the other set as well. So CMake is going to go
looking for these files. Now, in the case of this example the consuming project has said,
"Go and find me SomeProj." I'm assuming that's the
name of your project that contains your library. "Go and find me SomeProj. And I'm also going to make a
version requirement of 2.3. I require version 2.3." Now when CMake sees that it says, "Oh, I've got a version requirement. I need to find this
SomeProjConfigVersion.cmake file As a library author, if you have versioned your package and you do not provide this file, you effectively have a broken package because CMake will say, "I have a version requirement of 2.3. Here's this installation of a package with no config version file. I don't know what it provides. I'm going to reject it and move on to the next
directory search location." So if you do not provide
this config version file, your installation may never get used. So it's in your interest
to have one of these. Now, what is this file? You don't generally need to know. You can get, CMake to generate it for you. So the CMakePackageConfigHelpers module provides this command, the write_basic_package_version_file. And it's fairly straightforward to use. Give it the name of the
file you want to generate. That's prescribed. You know that. You provide the version of the package, again, your library version
and your package version will probably be the same, so your package version should
be relatively predictable and easy for you to know. And you also specify what
your compatibility strategy is or what's your versioning strategy. And remember before when I said, "Please choose a conventional
versioning strategy?" This is an example of why. So in this particular example, I've been using semantic versioning. CMake, provides a direct support for that by saying, "COMPATIBILITY
SameMajorVersion". So what CMake is going to
do is it's going to say, "Oh, I've had 2.3 requested. This one provides 2.4.7. Same major version: 2 Check." And the second part of the check will do is say, "2.3 has been requested. 2.4.7 is more recent than that. Great. Check. Okay, this package satisfies
that version requirement." You can use other
compatibility strategies. CMake provides AnyNewerVersion, which is basically like saying, "I promise to be backwards
compatible forever." I don't know about you, but I'd be pretty nervous
about making that promise. So I wouldn't generally
recommend saying AnyNewerVersion. You can say ExactVersion,
which is another way of saying, "I provide no
backward compatibility at all. Every release is a new one,
and I break everything." So there are your options. All right, so let's talk a little bit about how might your library be packaged. Now, obviously you'd be used to the idea of packaging your own library. Maybe you produce an SDK package. You're in full control of that. So that's pretty comfortable for you. Someone else might be
embedding your library in their package. They might be ripping your library up out of wherever they built against it, plonking it into their
package wherever they see fit, completely out of your control. It could be embedding it in their package any number of ways. A Linux distribution maintainer might be building your
package and installing it as part of the distribution. And they have their own conventions and policies that they need to adhere to. So they might want to install your library in some rather unusual
place, in your view, somewhere that you hadn't expected. There are packaging systems
that are not part of the OS, so something like a vcpkg,
or a conan, or a Homebrew. They might be packaging up your library, and it might be done by some volunteer who just wants to use your library and wants to move forward. So you might not have much of an idea of what they will do. Now, as a library author, you want to support as many of these different scenarios as you can. The better that you support these things, the more likely your
library is to be adopted and to be incorporated
into these various things. I'm assuming that's what you
want as a library author. All right, so let's see
how we do this with CMake. So here's where we'll start. So you'll often see library projects will install their library like this. We basically say
install(TARGETS Example ... the name of the library, DESTINATION - where you
wanna install it to. Now, if I'm going to give this a score, I'd give this about a two out of 10. And before we start looking
at what's wrong with this, let's have at least look
at what it does right. So the first thing we wanna know is it provides a relative
install location. So it means that if someone
wants to install this library, they can choose the base install location to suit themselves. We're only specifying where
this library gets installed below that base point. So it's relocatable. So that's good. And that's about where it stops. This particular example, it misses a lot of things. It doesn't account for a
lot of different scenarios of the way people package the library. Different platforms have
different conventions. Sometimes you want the library not in lib. Sometimes you want it in bin. Different Linux distributions
might wanna put it in different places. A Linux distribution maintainer has no way to change
where this gets installed. You've hard-coded lib with this example. If they need to put it somewhere else, they're going to have
to edit your sources, they're going to have
to patch your sources just to do what they want, and that's not very
friendly to maintainers. So he's an improved example. So in this case, we've
provided three different types of destinations to account
for the different places that different platforms
might wanna put things. And we've also made sure that
the destinations themselves are not hard-coded. Now, the RUNTIME destination, that's where Windows dll
files will be installed. Typically, if you're on Windows, you will say your dll will get installed to the same directory as your executable. So that can be found at runtime. So that's why it's RUNTIME destination. The LIBRARY destination
is for shared libraries on non-Windows platforms. And so we're talking here
about the real library, the soname, and the name link. All three of those will go
to the library destination. The ARCHIVE destination Now, if your library was being
built as a static library, then that's where your static
library would be installed. Similarly for Windows, that's where the import library would go. Now Windows is a really
interesting case here because this highlights that even just for one shared library, you need two different install locations because the dll would
typically go in one directory and your import library would
typically go in another. So that's why we've got two
separate controls for that. Now as for the locations themselves. These are variables, which means somebody can
override them if they want. Now the GNUInstallDirs
module is provided by CMake. Its name is probably no
longer all that accurate. It more reflects where it began its life. It began its life trying
to provide a standard set of install directory
structure for the GNU layout. These days, it does more than that. So it's probably about time we renamed it to something like StandardInstallDirs. But it's job is to define
variables like these. CMAKE_INSTALL_BINDIR, CMAKE_INSTALL_LIBDIR. Now, because it's going to
define them as cache variables, that means if those variables
already have a value by the time we include that module. It'll leave them alone. It won't change them. So if I'm a Linux distribution maintainer, that means I can set these cache variables on the CMake command line. I can override these choices. I don't have to patch the project. I can inject them in from outside, and now the project will honor those. So just by doing that, we've made life much easier for the Linux distribution maintainers, and maintainers of other
packaging systems for that matter. But GNUInstallDirs does
more than just provide a basic set of defaults. In some cases, it will go further and try and recognize what
type of system it's on and provide defaults, which
are specific to that system. Debian-based systems
are an example of this. So for Debian-based systems, CMake will look at that and say, "What system am I on? What's the convention for libraries?" Because even on Linux alone, there are all sorts of
different conventions for where libraries should go. They might go into /usr/lib They might go into /usr/lib64 On a multi-arch system, they
might go into /usr/lib/... some architecture-specific directory. As a library author, you don't want to deal with that, and neither should you. Your job is your library. Distribution maintainer's
job is their distribution. CMake will be the glue for you here. Where it recognizes a Debian-based system, it will provide a more conventional
default for that system. For a system it doesn't recognize, it'll just use the generic lib, which is still probably
what you would do anyway as a library author. So now this set of install destinations are so common and so widely used they became the default in CMake 3.14. So with CMake 3.14, you can now just say
install(TARGETS Example), and you'll get all those
destination behavior by default, which sounds absolutely awesome. That's nice and simple. I give that a five out of 10, and here's why. Once again, it doesn't
consider all the different ways that people want to use your library, and consume your library,
and package your library. Don't get me wrong, this is suitable for certain scenarios. That's why it just
passes at five out of 10. But in the general case, it's not enough. So here's an improvement on that. And what we've done this time is we've added install components. And why do we need install components? For those of you who
are familiar with Linux and their various packaging systems, whether that be RPM for Red Hat-based of Debian packages for Deb-based systems, you'll often see packages
split up into a runtime package and a devel package, Runtime package will have the libraries that are needed at runtime, devel package will have static libraries, headers, et cetera, and we want to support that case. So that's what we have here. RUNTIME destination. That's where your Windows dll will go for Windows platform. Fine, that's a runtime component. The ARCHIVE destination, static libraries, Windows import Libraries. Clearly a development set of artifacts, development component. For the LIBRARY destination, we actually have a mix. So the real library and the soname, those are runtime components. You need those at runtime. The name link, you only need
that during development. you don't need the name
link to run an application linked against your library. So we put that one off in a separate development component. Now, in full disclosure, the
name link component keyword, that was added in CMake 3.12. If you are using an earlier
version of CMake than that, you can still get the same end result. You just have to add a
second install command to handle the name link specifically. And in the interest of space
and time, I won't show that. All right, so that's
definitely an improvement. Still only give that eight out of 10. What are we missing? Well, component names that
we've chosen are pretty generic. Now, earlier I mentioned that find_package is one way that people
will bring your library into their project. Install a pre-built
version of your library then just pointing their build at that. But an increasingly popular way that people are bringing
libraries into projects is to incorporate it
directly into their build, meaning it's as though your
sources were mixed in with this. So people might use something
like git submodules. If you're using CMake 3.11 or later, FetchContent module
makes it trivially easy to bring in source code of
another project into your built. So essentially, instead
of doing a find_package() to bring your library into the build, it's going to do the equivalent
of an add_subdirectory(). So all your targets and everything else that's in your library build
becomes available to theirs. Now, if someone is going
to bring your library into their build that way, it's reasonable to expect that they're probably going to do that with someone else's library as well. And if someone else's library also called their install components runtime and development, it's going to be impossible
to tell the two of you apart. Now, if they want to make your library part of an optional component, and this other library part
of an optional component, they can't differentiate
between the two of them because you've used the same
install component names. Easy to fix. We just use a project specific
install component name. Typically, what I would recommend is you take the project name underscore whatever the install component that you've wanted to call it after that. And that ensures that your
install component names are now specific to your project. They shouldn't clash with anything else. So now you're being a really good citizen. So where are we at? Cross your fingers, cross your toes. All right, I'll give
that one a 10 out of 10 . A little asterisk up the top there. The little asterisk is because typically you will
have another couple of lines that you would add to this. That's to do with exporting targets. There's nothing platform
specific about exporting targets, so I'm not going to talk about that today. But if that is an area
of interest for you, at the back of this presentation, there are a few bonus
slides that do show you how to do that, and do you refer to what this
little asterisk is about. But in the interest of time, I
won't be covering that today. All right, so I'm going to
finish up with an example of a particular problem
that, as library authors, you need to be aware of. It's specific to Linux. Well, more accurately, it's
specific to a tool chain that's common on Linux, and it's something that is easy to fix, but you first need to
understand what the problem is. So in this example scenario, we're going to say we
have some application and it links to our library. Our library uses a few
other shared libraries as internal implementation details. So say for, example, our library might be some generic interface
to a bunch of hardware. And we might have a shared library that talks to each
specific hardware device. So we use that as our way
of communicating with it. Now, as far as the
application is concerned, it only knows about our library. It doesn't know about those
other implementation details, and nor should it. So that's our scenario. And so it goes like this. You check out your source
code with your library and you fire up your compiler, you run CMake, and you run your build. Great, everything builds successfully. We're happy. Build your test code as well, even better. So try and run your tests to
make sure everything passes. Fantastic. All our tests pass as well. Looking good. All right, so we've built successfully, tests all pass. Package up this thing. So you run your packager, you see what it's created, you check that it's put all the files that you thought it was going to put and put them in the right
place with the right name. Yep, everything that you
expected to be there is there. Packaging all good. You then go and take that package and you walk over to your
colleague at their desk and you say, "Hey, can you
go out and build your app against this package, please? Give it a try." So they dutifully go and
put that on their system. They go and fire up their application that builds against your
library, and fantastic, success. Build's great. And they try and run the app
that they built, and bang. It doesn't work. And they get an error that
looks something like this. Myapp error while
loading shared libraries, name of one of your internal
implementation libraries. And you scratch your head at this, and you say, "Let me look at your system." You go and look at their system, you see on their system,
there's your library libExample. There's the other library
that's being complained about right there beside it. You know it's there. It found libExample, but it didn't find this other one. Why? Let's rewind to see
what's actually going on through this example. So in the first case, when
you're building your libraries CMake is going to embed what's
called RPATH information into your library and
into your executables. What that is, that's the full
absolute path to dependencies. So in the case of your test executable, it'll be embedding the path
to your libExample library. When it's building your
libExample library, it's going to embed the full path to each of those shared
library implementation details. So when you go and run your test code, it just magically works. You've probably never thought about it. Your test code might be one directory, your main library in another directory. These additional vendor
libraries, somewhere else. You haven't had to specify
any environment variables. You haven't had to do any funny trickery. And yet the system was able
to find everything it needed. And this is because of
the RPATH information. And this is desirable as a developer. This is what you want. You want it to be easy
to run your test code, and to do that sort of thing. So this is good. And all major Unix-based
platforms will generally support RPATH in some form or another. Sadly, if you're a Windows developer, you're left out in the cold, sorry. I should point out that
there has been somebody who has a proof of concept
support for RPATH on Windows. And I think it was the BUILD2
build tool was supporting it. I don't know if it works. I've only come across it
in the last month or so, but apparently it's
theoretically possible. But it's not generally
supported on Windows. Anyway, that's an aside. When you go and package up
or install your library, CMake going to take the
binaries that were built, going to install them
into the staging area or wherever you've told
it to install it to, and it's going to remove
that RPATH information that it put there in the first place and replace it with a different set. It's going to replace
it with an install set. And that set is empty by default. And why does it do that? Well, you don't want your build paths embedded in a binary installed
on somebody else's machine. Those paths don't make any
sense on someone else's machine. They may leak private information that you don't want
exposed in that binary. So again, it's desirable. We don't want that build RPATH information in our install packages. But the fact that it's replacing it with an empty set is
where our problems begin. So that's where the source
of this difficulty lies. Our libraries are losing their connection. Our libExample is losing their connection to the internal implementation libraries. But let's press on. So when you gave your
package to your colleague, they built their package,
their application, and CMake did exactly the same for them as what it did for you. It's going to, when
building their application, embed the RPATH in that
application to your libExample. Now the linker at link time is
going to say, "Ah, application." You need libExample. I've got this RPATH
information that tells me I can look here. Oh, great. I found libExample. I can reuse that RPATH information to find any other
dependencies that are needed down the chain from that point as well." So the linker is happy. It still finds our extra dependencies because we install them to the same place as our main library. So that's why their build still works. But why things fail when they
go to run their application? And here's where it gets a bit messy. The runtime loader is going to look in a number of places for libraries and the first thing
it's going to do is look in the dynamic section of the binary and look for certain symbols. And it's going to look for
either DT_RUNPATH or DT_RPATH. Now, why do we have two? Well, many, many, many years ago we only had DT_RPATH, and this was fine. It worked until someone
realized, "Hang on a minute. The search order is a little bit funky." What I'm not showing here is LD_LIBRARY_PATH environment variable. The runtime loader search
order goes like this. It would look for DT_RPATH first. If it can't find what it's
looking for using that, it'll then look at the LD_LIBRARY_PATH
environment variable. And if it can't find anything from that, well, okay, bad luck. There are other places that we'll look, but we're not interested in those for now. Now, historically, you only had DT_RPATH. And the problem with
that is if the library is found via DT_RPATH, then the user has no way to
override that conveniently. LD_LIBRARY_PATH gets checked after RPATH. So if, as a developer, I wanted to say, "I wanna test this executable
against a newer version of a library over here," I can't do it. Well, I can, but I have to
do some pretty nasty things, like pre-loading libraries and so forth. Now that's not real great. You wanna give the user control. And so RUNPATH solves that problem. RUNPATH is only checked
after LD_LIBRARY_PATH. It's designed for a similar purpose to allow the runtime
loader to find libraries, but the search order is different. And if your binary contains
both DT_RPATH and DT RUNPATH, it will ignore the DT_ RPATH. RUNPATH takes precedence here. So this is good. Most modern linkers these days. In fact, it's been that
way for a number of years. When you link with RPATH information, it will embed it as
RUNPATH rather than RPATH, which means the end user
will be able to say, "I can set my LD_LIBRARY_PATH override," otherwise whatever the
library builder said, which sounds great. But there's one more difference
between RPATH in RUNPATH, and this is why the
problem falls apart for us. Runtime loaders man page
describes it like this. In essence, what it's saying is if I'm using RPATH, then I can use RPATH to find
my immediate dependency. So my application has an RPATH that allows me to find libExample. And just like the linker, I can use that RPATH to find
any child dependencies as well. For RUNPATH, I'm only
allowed to use RUNPATH to find my direct dependency. So, RUNPATH is embedded in the application and allows me to find libExample. I am not allowed to use that to find my dependencies below that. libExample must itself
have its own RUNPATH that allows me to find
those other dependencies. That's the critical difference, and that's why our scenario here breaks because when we installed our package we stripped out all the RPATH information and we're left with an empty set. So our RPATH or RUNPATH for
our library contains nothing. We've given the runtime
loader no indication of where to find our dependencies. So that's why the app fails to load. Now thankfully, this
is actually really easy to handle in CMake. It's quite trivial, and basically it's a one line thing. CMAKE_INSTALL_RPATH is a
variable that we can use to provide the set of
paths to embed in a binary when it's being installed. So instead of getting an
empty set, we get this. Now, we use a special
case thing called ORIGIN. This is not a CMake thing. This is, I guess you'd
call it a linker thing. It's more a platform thing. And what ORIGIN means, $ORIGIN, it means the location of the thing that I am currently looking
up dependencies for. So when the runtime
loader is trying to find the dependencies of libExample, $ORIGIN means the directory
where libExample is. It's a way of saying, "Here. Me. Right this location." Now, if I set CMAKE_INSTALL_RPATH before I create my library targets, that means that whenever I
install that library target that's the install
RPATH it's going to get. So just by that one line change, that CMAKE_INSTALL_RPATH $ORIGIN, I fixed this scenario. Now my application that
my colleague has built will load and run just fine. Now, there's this little
"if(NOT APPLE)" up here. Apple, in their typical way, they have something which
is different, but similar. So Apple's platforms
don't understand $ORIGIN. They have a different set of keywords. So they have things like
@loader_path, @executable_path. But in this case, the way
that things work on Apple is actually they do the right thing. They got it right the first time. So Apple platforms will always look for environment variables first. So we don't have this problem
with RPATH taking precedence over what the user might want to set for the environment variables. Apple will also allow
RPATH, just like on Linux, it will allow RPATH to be used not just for the immediate dependency, but also the child dependencies. Now I'll qualify this. When I say Apple got it right, I should really say Apple made it easy because it will work out of
the box generally for you. There will be cases where
you probably don't want that. You probably do want the RUNPATH behavior where only the immediate
dependency should be used. But, yeah. What I'm really saying is
for you, as a library author, it will work out of the
box for you, generally, unless you're doing
something a little unusual. All right, so that's the basic support for RPATH that CMake has. CMake does allow you more
control over RPATH than that. You can, for example, say, "I'd like to build with my
install RPATH right up front. don't even do this two
sets of RPATH thing." Now that may be attractive, for example, if you want your
build tree to be laid out in the same way that
your installation would. You might try and mirror the
same bin lib layout, et cetera. Keep in mind that if you do that, things like $ORIGIN will trip you up in some cases. Not all linkers, build time linkers will understand $ORIGIN. Some will, some won't. The default GNU linker typically won't. So you can do things like that, but you need to be aware
of the consequences. All right, so that's as
much as I've really got time to cover today. I'm quite happy to answer
any questions that you have about this stuff or
anything to do with CMake. I'm also at the Tool Time Labs tonight. Feel free to bring along your laptops and your curly questions. I'll do my best to not tear my hair out and to answer them as best I can. I'm here all week. Feel free to come up and
chat to me about your stuff anytime you like. And as I said earlier, if you'd like me to come
and work on your project, I am available for consulting. So you can get in touch
with the details there or catch me this week. Okay thank you very much. (audience applauding) - [Man In Blue] Hi. When you were talking
about semantic versioning, on Windows it's fairly
common for library authors to encode the major
version in the file name. Do you have a comment on that? - Sorry, I'm not sure if
the microphone is working. You're basically saying on Windows it's common to embed the major
version in the file name. Personally, I don't see
that all that often. It would be valid, but it has consequences because it changes the
base name of the library that things like CMake's
find_library might search for. So if we were to use the example I had here and I called
my dll Example.2.dll, if I was going to try and
find that library with CMake, I would have to go
find_library(Example.2), rather than find_library(Example). So there are consequences of putting that major version number as part of the file name on Windows because it's not part of the conventions that tools like CMake understand, and CMake wouldn't be alone in that. So yes, it would solve
the compatibility issue, but might trip up some tools. - [Man In Blue] I think
part of the reason it's done is to allow two major
versions to be installed in the same directory. Certainly, and Qt libraries
are a great example of that. You can have Qt4, Qt5, and whatever installed at the same time. But again, you've gotta handle
that at the tooling level if you want to continue
to make it convenient for consumers of your package. - [Man In Blue] Thank you. - [Man In Brown] You mentioned the component aspect of installation. I was curious if that
played nicely with CPack. And in the past we've seen some issues using the BundleUtilities module. Has that since been resolved? - So two parts there. So yes, it works
wonderfully well with CPack. I do this all the time at work. The install components, they are part of the install
stage with CPack users. And so that integrates perfectly. BundleUtilities. Hmm, problem child of CMake. So BundleUtilities has
not received a lot of love in a long time. It works for certain cases. I personally used to use it a fair bit, and these days I avoid it, probably for similar reasons
to what you've experienced. I generally find that
it's, in the long run, better to just work
out what are the things that I actually need to install, and just install them directly. BundleUtilities can be fragile. And I think somebody
in the last few months had been starting to look at
that again, some contributor. I'm not sure what the state of whether there's been
improvements there. Use it if you have to, but generally, I typically avoid it. Hey, Bowie. - [Bowie] Hey, Craig. First of all, well done, and thank you for coming such a long way. - Yay, Aussie. - [Bowie] So I get the
impression with CMake, it's constantly evolving to remove pain from the developers. But you had your example
of the five out of 10 slide where all the defaults
were set more nicely. I was curious whether you were aware of any of the future developments for CMake that are gonna remove or simplify some of the things that you've talked
about today, or whether- - Yeah. There are only certain
things that you can simplify. So install components is
a great example of this. It's really hard to do the
right thing by default for that because it inherently needs
to be project specific. The names of things have
to be project specific. The types of components you want to create may not always be runtime and develop. I've worked on projects
where we split up components into documentation, examples,
headers, and so forth. So CMake itself can't really always know what a project is gonna want to do. So what was added in the CMake 3.14 was it went as far as kinda could go in terms of doing more out
of the box for the user. But in that particular area, no, it couldn't really go that much further. CMake is always evolving. We are always trying to make things an easier experience for sure. I hesitate to mention this, in part because of the
person standing behind you. Hello. Packaging is an area where
dependency management is an area where I personally am very keen to see improvements in, and we definitely can
do better in that area. FetchContent helped
with that in CMake 3.11. In future versions, hopefully
we can take that further and make that a smoother experience. There are three major
feature releases a year, and so we're always trying to
catch up with what's changed, add improvements, so forth. Mathieu, be kind. - [Mathieu] I swear. I promise, I'm not gonna ask
about package management. No, actually I wanted to talk about something entirely different. You mentioned Windows and RPATH. And something I find
extremely painful with CMake is that out of the box,
unit testing dlls is hell because you create your test, you run it, and it says, "dll not found." - Correct. - [Mathieu] On Linux, RPATH does it. On Windows, every time I have
to write three lines of hacks for every test I ever write to be able to add the path. Is there any planned
feature to work this run? 'Cause it's always the
same thing every time. - Yeah, I'm not aware of any
plans next year that we have. But certainly, I share your pain. (indistinct) (Mathieu laughs) I do (indistinct) plans to deal with that, at least (indistinct) - [Mathieu] Thank you. - [Man In Black] Hello. My question is about warnings and maybe error messages from CMake. I've experienced that if you use it wrong, it doesn't tell you anything. It just doesn't work. As an example, I recently
put a, what do you call, an en dash on the command
line by copy pasting it from a website, from documentation, an en dash is a different character
than a hyphen, let's say. But it looks exactly
the same in some fonts. If you do this, it will
just completely be ignored. And are you- - CMake command line? - [Man In Black] Yes. And my question is, are you aware of is there any action being taken to provide
more feedback, let's say, to the user when you're using it wrong? Let's say the soname omission could be a warning. - We are always trying to
improve how we report errors and warnings and so forth. On the CMake command line
side of things, there is an effort to totally
refactor the argument handling. It's very fragile, the
way it's currently done, and it's not very maintainable. We are hoping to replace that with a proper argument
handling implementation. That would improve at least that side. A great example is the ctest command line has the same problems. You can quite easily put
something that's completely wrong and it'll just quite
happily just continue on. Okay. Everywhere else, where we
can, we try and improve our warning error messages, but it's an ongoing effort. - [Man In Black] Okay. So the command line
stuff is being worked on, but there's no other
initiative you're aware of to generate more warnings
or anything like that? - Not that I'm aware of. - [Man In Black] Okay. - Okay, time's expired. So I'll have to call it a day, but I'm happy to hang
around and answer questions for those who wish. Thank you.
(audience applauding)
