So far we’ve learned a lot about how to develop backend web services with gRPC. When it comes to deployment, one important thing we should consider is
load balancing. A large scale gRPC deployment typically has a number of identical backend servers, and
a number of clients. Load balancing is used for distributing the load from clients optimally
across available servers. There are 2 main options of gRPC load balancing: Server side and client side. Deciding which one to use is a primary architectural
choice. In server-side load balancing, the client issues RPCs to the a Load Balancer
or proxy, such as Nginx or Envoy. The load balancer distributes the RPC call to one of the available backend servers. It also keeps track of load on each server and implements algorithms for distributing
load fairly. The clients themselves do not know about the
backend servers. In Client side load balancing, the client is aware of multiple backend servers and chooses one to use for each RPC. Usually, the backend servers register themselves with a service discovery infrastructure, such as consul or etcd. Then the client communicates with that infrastructure to know the addresses of the servers. A thick client implements the load balancing
algorithms itself. For example, in a simple configuration, where server load is not considered, client can just round-robin between available
servers. Another approach is to use a look-aside load
balancer, where the load balancing smarts are implemented
in a special load-balancing server. Clients query the look-aside load balancer
to get the best server(s) to use. The heavy lifting of keeping server state,
service discovery, and implementation of load balancing algorithm is consolidated in the look-aside load balancer. Let’s look at some pros and cons of these
approaches. One pros of server-side load balancing is
simple client implementation. All client needs to know is the address of
the proxy, No more coding is needed. And this approach works even for untrusted
client, which means the gRPC service can be open for
everyone from the public internet. However, its cons is adding 1 more extra hop
to the call. All RPCs have to go through the proxy before
reaching the backend server. Thus causing higher latency. Therefore, this server side load balancing
is suitable for the cases where there are many clients, possibly untrusted
from the open internet who want to connect to our gRPC servers in
a data center. The client-side load balancing, on the other
hand, doesn’t add any extra hop to the call, and thus giving us higher performance in general. However, the client implementation now becomes
complex, especially for thick client approach. And it should only be used for trusted clients, Or we will need to use a look-aside load balancer to stand in the front of the trust boundary
network. Because of those attributes, client-side load balancing is often used in
a very high traffic system and microservices architecture. In this video we will learn how to setup server-side load balancing for our gRPC services
with NGINX. Since I’m gonna show you different nginx
configurations where TLS can be enabled or disabled on the
server and client, let’s update our code a bit to take in a
new command line argument. This tls argument is a boolean flag, which will tell us whether we want to enable TLS on our gRPC
server or not. The default value is false. Let’s also add it to this log so we can
see its value when the server starts. Then I will extract these interceptors to
a separate server options variable. We have to move the interceptor object up
here. Now we check the TLS flag. Only in case it’s enabled then we load the
TLS credentials, And append that credentials to the server
options slice. Finally we just pass the server options to
the grpc.NewServer() function call. And that’s it for the server. Now let’s do similar thing for the client. First we add the TLS flag to the command line
argument Update this log message to include the TLS
value Then we define a transport option variable with the default value grpc with insecure. Only when the TLS flag value is true, we load the TLS credentials from PEM files and change the transport option to grpc with that TLS transport credentials. Finally we pass the transport option to this
grpc connection And this one as well. Now the client is done. We will have to update the Makefile a little
bit to pass in the TLS flag. But first let’s try running the service
without TLS. For now if we run make server, We can see that the server is running with
TLS disabled. And if we run make client, It’s also running with no TLS, and all the RPC calls are successful. Now if we add this -tls flag to the server, And restart it, the TLS will be enabled. If we run make client now, the requests will
fail. We have to enable TLS on client side as well, by adding -tls flag to the make client command. And we can see the requests are successful
again. Alright, now the TLS flag is working as we
wanted. It’s time to do some load balancing on our
server. Let’s start with insecure gRPC first. So I’m gonna remove these -tls flags so that the “make server” and “make
client” commands will run without TLS. And I will add 2 more make command to run 2 instances of the server on different
ports. Let’s say the first server will run on port
50051 And the second sever will run on port 50052. The next thing we need to do is to install
nginx. Since I’m on a mac, I can simply use Homebrew So brew install nginx. After nginx is installed, we can go to this
folder to config it In the usr/local/etc/nginx folder, There’s a nginx.conf file. Let’s open it with visual studio code. This is the default configuration. There are several things that we don’t need
to care about in this tutorial, So I’m gonna remove this user config, Uncomment this error log, Remove the config for log levels and process
id. And let’s say we just need 10 worker connections
for now. One important thing we need to do is to config the right location to store the error log and access log files. In my case, Homebrew has already created a
log folder for nginx At /usr/local/var/log/nginx So I just go ahead to copy the full path of
this folder And paste it in this error log setting. This log format is also unnecessary for now. For the access log file, I will put it in the same location with the
error log file. Next, we don’t need all of these configs, so let’s get rid of them. In the server block, We have a listen command to listen to incoming requests from client
on port 8080. This is the default config for a normal HTTP
server. Since gRPC uses HTTP2, we should add HTTP2 at the end of this command. Let’s remove this server name and charset
since we don’t need them now. Similar for the access log because we’ve
already defined it above. Let’s delete this config for default root
HTML file. And everything after this block as we don’t
care about them for now. OK, now we want to load balance the incoming
requests to our 2 server instances. So we should define a upstream for them. I’m gonna call it upstream pcbook_services. And inside this block, we use the server keyword
to declare a server instance. The first one is running on localhost port
50051. And the second one is running on port 50052. To route all RPC calls to the upstream, In this location block, We use grpc_pass keyword, followed by the
grpc:// scheme and the name of the upstream, which is pcbook_services. And that’s it! The load balancing for our insecure gRPC server
is done. Let’s run nginx in the terminal to start
it. We can check if it’s running or not using
this ps and grep command. Let’s check out the log folder As you can see, 2 files are generated: access.log
and error.log And they’re empty at the moment Because we haven’t sent any requests yet. Now let’s run make server1. OK, it’s started on port 50051 with TLS
equals false. Then on another tab, run make server2. It’s running on port 50052, also with TLS
disabled. Now let’s run the client. Looks good. All RPC calls are successful. Let’s check the logs on our servers. The server2 receives 2 create laptop requests. And the server1 receives 1 login request and
1 create laptop request. Excellent! And after a while, there’s another login request comming to
this server That’s because our client is still running, And it periodically calls login to refresh
the token, I hope you still remember the codes that we’ve written in the gRPC interceptors
lecture. OK, now let’s look at the nginx access log
file. You can see first there’s a login call, then 3 create laptop calls, And finally another login call. So everything is working exactly as we expect. Next I’m gonna show you how to enable TLS
for nginx. In a typical deployment, the gRPC servers are already running inside
a trusted network, And only the load balancer (or nginx in this
case) is exposed to the public internet. So we can leave our gRPC servers running without
TLS as before, and only add the TLS to nginx. To do that, we will need to copy 3 pem files
to the nginx config folder: The server’s certificate, The server’s private key, And the certificate of the CA who signed client’s
certificate if we use mutual TLS. OK, so now I’m gonna cd to the usr/local/etc/nginx
folder And create a new cert folder. Then I will copy those 3 pem files from our
pcbook project to this folder. Alright, now all the certificate and key files
are ready. Let’s add one more client-tls command to
the Makefile to run the client with TLS. And go back to our nginx config file. To enable TLS, we first need to add ssl to this listen command. Then we use the ssl_certificate command to give nginx the location of the server’s
certificate file And we use the ssl_certificate_key command To give it the location of the server’s
private key file. As we’re using mutual TLS, We also need to use the ssl_client_certificate
command to tell nginx the location of the client CA’s
certificate file. And finally we set ssl_verify_client to on to tell nginx to verify the authenticity of
the certificate that client will send. And we’re done. Let’s restart nginx. We run nginx -s stop to stop it first. If we run ps command, nginx is no longer running. Then we start it with nginx command. And check it with ps command. 2 nginx processes are running: 1 is the worker and the other is the master
process. Our server is already running, So let’s run the client! If we just run make client, it will run without
TLS, So the request will fail, because nginx is
now running with TLS enabled. Now let’s run make client-tls This time the client is running with TLS, And all requests are successful. Keep in mind that our servers are still running
without TLS. So basically what happens is, Only the connection between client and nginx
is secure. And nginx is connecting to our backend servers
via another insecure connections. Once nginx receives the encrypted data from
client, It will decrypt the data before forwarding
it to the backend servers. Therefore, you should only use this approach If the nginx and backend servers stay in the
same trusted network. But what if they are not on the same trusted
network? Well in that case, we have no choice but to enable TLS on our
backend servers as well, and config nginx to work with it. Let’s add 2 more make commands to start
2 server instances with TLS. The server1-tls command will start TLS server
on port 50051 And the server2-tls command will start another
TLS server on port 50052. Alright, let’s stop the current server 1 And start server 1 with TLS. Similarly, I will stop the current server
2 And start server 2 with TLS. Now if we run make client-tls immediately, The request will fail. The reason is, Although the TLS handshake between client
and nginx succeeded, The TLS handshake between nginx and our backend
servers failed, Since the backend servers are now expecting
a secure TLS connection, While nginx is still using an insecure connection
when connecting to the servers. As you can see in the error log, the failure happened when nginx talked to
the upstream servers. To enable secure TLS connection between nginx
and upstream, in the nginx config file, we have to change this grpc scheme to grpcs. This should be enough if you just use server-side
TLS. However, in this case, we’re using mutual
TLS. So if we just restart nginx now, And rerun the client-tls, The request still fails, Because nginx is not configured to send its
certificate to the upstream servers yet. So we’ve got the bad certificate error as you
can see in the error log. Let’s see what happens if we go to the server
code and change this ClientAuth to NoClientCert, which means we will just use server-side TLS. Then restart server1-tls Restart server2-tls. And run make client-tls again. This time all requests are successful, Exactly what we expected. OK now what if we really want mutual TLS between
nginx and upstream? Let’s change this ClientAuth back to RequireAndVerifyClientCert. And restart the 2 TLS backend servers. Get back to our nginx config file, We must instruct nginx to do mutual TLS with
the backend servers by giving it the location of the certificate
and private key. We use the grpc_ssl_certificate for the certificate, And the grpc_ssl_certificate_key for the private
key. You can generate a different pair of certificate
and private key for nginx if you want, Here I simply use the same certificate and
private key of the servers. OK, let’s try it. First stop the current nginx process. Then start a new one. And run make client-tls again. This time all requests are successful. Perfect! There’s one more thing I want to show you
before we finish. As you’ve already seen, The login and create-laptop requests are now
evenly distributed between our 2 backend servers. But sometimes, we might want to separate the authentication service and the business logic service. For example, let’s say we want all login requests to
go to server 1, And all other requests to go to server 2. In that case, we can also tell nginx to route
the requests based on its path. Here I just copy the path of the Auth Service And paste it to this location. Then I change this upstream name to auth services. And it should only connect to the server 1
at port 50051. I’m gonna add another upstream for the laptop
services And make it connect to only server 2 at port
50052. Then duplicate this location block, Change the upstream name to laptop services, And update the path to techschool.pcbook.LaptopService. OK, let’s try this! We just need to restart nginx. And run make client-tls. Now we can see only login request goes to
server 1, And all other create laptop requests go to
server 2. Even if we run this make client-tls multiple
times. So it works! And that wraps up our lecture about load balancing
gRPC with nginx. I’m gonna push this nginx config file to
the pcbook repository So that you can download and play with it
if you like. Thanks a lot for watching and following the
course. Happy coding and see you in the next videos!
