Ingress exposes HTTP and HTTPS routes from
outside the cluster to services within the cluster. Traffic routing is controlled by rules
defined on the Ingress resource. An Ingress may be configured to give Services externally-reachable
URLs, load balance traffic, terminate SSL/TLS, and offer name-based virtual hosting. An Ingress
controller is responsible for fulfilling the Ingress, usually with a load balancer. An Ingress
does not expose arbitrary ports or protocols. You must have an Ingress controller to satisfy an
Ingress. Only creating an Ingress resource has no effect. There are a lot of different ingresses
available for Kubernetes. In this video, we will use the most widely used nginx ingress controller.
There are two different implementations. One from the Kubernetes community and another one from
Nginx inc. I've been using the community edition for years, so I'm going to stick with it. In this
video, we will create seven different ingresses. Since we're going to monitor our nginx ingress
controller with Prometheus and Grafana, we're going to create Ingresses for them
first. Then we will create a simple fanout example. A fanout configuration routes traffic
from a single IP address to more than one Service based on the HTTP URI being requested. An Ingress
allows you to keep the number of load balancers down to a minimum. The fourth example will use
name-based virtual hosting. Name-based virtual hosts support routing HTTP traffic to multiple
hostnames at the same IP address. The fifth example will demonstrate how to use a TLS
certificate and secure communication between a server and a client. You can secure an Ingress by
specifying a Secret that contains a TLS private key and certificate. In the following video, I'll
explain how to use certificates from letsencrypt and automate renewal using both HTTP-01 and
DNS-01 challenges with the cert-manager. Next is a little bit rare example when you
want to use ingress and route traffic to the services in different namespaces. By default,
ingress must be created in the same namespace where you have a service. The last example will
demonstrate how to use ingress and route TCP and UDP services. We will deploy
the Postgres database, create a TCP service for the nginx controller and expose it
on the same load balancer that Ingress uses. Let's get started. For you to follow along,
you need a vanilla Kubernetes cluster. I will be using AWS in this video, but those ingress
examples are pretty generic and should work in most public clouds. You can create EKS using the
config that you can find in the repository. I have to say that all the commands that you see I run
in the terminal and the source code are available in my GitHub repository; you can find a link in
the description. Let's check that we can connect to our Kubernetes cluster by running kubectl
get svc. To make this video self-contained, let me quickly create Prometheus and Grafana.
I'm not going to go over all the details. If you want to learn more about how to deploy
Prometheus and Grafana to Kubernetes and find some examples of how to monitor services, I can
suggest watching another video, How to Install Prometheus on Kubernetes Cluster? First, let's
create a Prometheus folder where we're going to place all related files. Since we will use
Prometheus Operator, we need to create Custom Resource Definitions. I'm going to fast forward;
you can always find all the files in the git. The last CRD is Thanos rules. Let's go to the terminal and apply all
of them. Let's see if we have them now. Alright, all of them were created. Now,
let's create a Prometheus Operator itself. It will monitor those custom resource
definitions. When we use them, Operator will create corresponding objects in the Kubernetes,
such as statefulset for the Prometheus server. We're going to use the monitoring
namespace for Prometheus and Grafana. Here is an important label, monitoring equal
to Prometheus. By itself, it's not going to do anything, but we will use this
label to instruct Proemtehsus Operator to watch any namespace that contains this
label. If it discovers the Service Monitor, it should update the Prometheus config. You'll
see later how it works. By default, Prometheus will select Service Monitors only in its own
namespace. I'm going to skip RBAC and deployment, and the last object is a service monitor for the
Operator itself. Now, let's go and deploy it. The last folder is for the Prometheus server. We
also need a service account and RBAC policies. Now, let's spend a little more time
on this object. We need to configure the Prometheus to watch specific namespaces; also,
I'm not going to attach any persistent volumes for this demo. That means if you restart this
Prometheus instance, you will lose all the data. We're going to deploy just one single replica; for
production environments, you may want to have two for highly available setup and have something like
Thanos to deduplicate data from those instances. Then we need to specify the service
account with RBAC policies attached to it. The service monitor selector will use a
label lesson equal to 082 to select service monitors. We will use this label for the ingress
controller later. Also, as I mentioned earlier, by default Prometheus Operator will only
select service monitors in its own namespace, in this case, monitoring. We can use label
selector, monitoring equal to prometheus to select different namespaces. Later, we would need to
update the ingress namespace to have this label; otherwise, Prometheus will ignore it.
That's all for now; let's deploy Prometheus. Let's check if our monitoring pods
are up. Looks good we can continue. Now, let's deploy the Nginx ingress
controller; I'll show you both ways YAML and Helm installation. Even if
you don't use helm in your environment, it's a good starting point to generate YAML
definitions using the open-source Helm Chart. Let's add the ingress-nginx
repo and run helm update. If you search for nginx, you'll get a helm
chart. I would advise you to use the same version and upgrade when you install
it successfully. There are a couple of ways to override default variables,
either to use helm cli flags or if you have a lot of changes, you may want
to create a values file; let's do that. Now, let's see want we can change; if you go to
the official helm chart, you'll find all default parameters. There is a bunch of stuff that you
can adjust. Let's start from the controller; we can add any arbitrary nginx configurations
in the config map; I'll attach a link for your reference in the README file. Let's
modify a couple of default nginx directives. First, let's add compute-full-forwarded-for
equal to true. It appends the remote address to the X-Forwarded-For header instead of replacing
it. Also, add use-forwarded-headers. If true, NGINX passes the incoming X-Forwarded-*
headers to the upstreams. And the last one, proxy-body-size equal to zero. It disables
the limit, which is 1 megabyte by default. Helpful if you allow users to
upload some images or files. The next ingress class. This name will
reference this particular nginx ingress; in case you have multiple ingresses, you can
use ingress class names to differentiate them. Very often, we have external and internal
ingresses in the same Kubernetes cluster. New Kubernetes api starting from 1.18 version, let
us create an ingress cluster object to reference, it replaces the old kubernetes.io/ingress.class
annotation on the Ingress. That annotation was never formally defined but was widely
supported by Ingress controllers. Optionally, you can mark the ingress
class as default if you want. Next is a standard pod anti-affinity rule that
deploys nginx ingress pods on different nodes; it's very helpful if you don't want to
disrupt your services during Kubernetes rolling upgrade. Try always to use it. Now,
replica count, I'll use 1, but you should use at least a couple; optionally, you can configure
autoscaling; this chart allows it. Optionally, we can deploy the admission webhook. It just
verifies the configuration before applying the Ingress. In case some Ingress objects
have a broken configuration, for example, a syntax error in the configuration-snippet
annotation, the generated configuration becomes invalid, does not reload, and hence
no more ingresses will be taken into account. Ingress is always deployed with some kind of load
balancer. You may use annotation supported by your cloud provider to configure it. For example,
in AWS, you can use aws-load-balancer-type to specify that you want a network load balancer
instead of the default classic lb. Also, if you want to have an internal load balancer with
only private IP that you can use within your VPC, you can use aws-load-balancer-internal annotation.
The same thing applies to most of the clouds. We also want to enable Prometheus metrics on
the controller. We will use a custom resource definition service monitor, which is provided
by the Prometheus Operator. Let's add the lesson equal to 082; this label must match the one on the
Prometheus object; otherwise, it will be ignored. First, before installing nginx Ingress,
let's use the helm template to generate yaml files. We still want to
provide the helm release name, helm chart name, version, values that we
want to override, and the output directory. Let's check them out. You have
an admission webhook with RBAC. Then you have Ingress itself. Here, for example, we have the ingress class
equal to external Ingress and the config map with our custom nginx directives. And a bunch
of other files, including ingress class. Now, if you don't want to use helm, you
can just use kubectl apply on this folder. We will use helm to install it. The result
will be the same. You just need to watch out few helm hooks for admission webhook and
clean them up manually, not a big deal. Also, you may face connectivity issues in some cases
with admission webhook since it needs access to the Kubernetes server. For example,
in the GCP private Kubernetes cluster, you would need to update your firewall.
Alright, let's use a similar command, but instead of template, let's use install and
instead output-dir, let's add create a namespace. It will create a namespace in case it didn't exist
before. This will be a problem later, and we will need to address it. It's deployed, and helm will
generate a couple of examples of how to use it. Let's list the helm charts; with helm three, you
also need to specify the namespace; in our case, we deployed nginx ingress in the ingress
namespace. Let's also see if those pods are up. Ingress always comes with the Service of a
type load balancer; let's see if Kubernetes successfully provisioned our network load
balancer. Sometimes it takes a minute or two since we have the public dns name, which
means it was successfully provisioned. Later, when we create ingresses, we always
will use the same public DNS name of this load balancer. For each new Ingress, we
will create a CNAME dns record. In GCP, for example, instead of dns, you will
get an IP address, and for Ingress, you would create A record that points to that IP.
Let's go to the AWS console and make sure that LB is ready. You can find your load balancer
in the EC2 section under load balancers. Here you can see that this load balancer
is internet-facing means it is public lb, and the type is a network and
not classic as it will give it by default without that label that we specified. Let's confirm that we can monitor the ingress
controller with Prometheus. We don't have the Ingress for the Prometheus yet, so let's get the
service name and use the port forward command to access Prometheus on the localhost. Right now,
we have two services, one for the Operator and one for Prometheus. Let's use prometheus-operated
Service and port 9090 in the monitoring namespace. Now, if we go to the Prometheus target section,
we should see two targets. One for Prometheus operator and the second one for nginx ingress
controller since we created two service monitor objects.; let's see. As I expected, Ingress is not
here since the ingress namespace does not have the appropriate label; let's fix it. In this video,
I'm going to be modifying some Kubernetes objects directly using the edit command. You should
probably have at least yaml definitions in the git under source control. Now, let's
edit the namespace to add monitoring equal to the prometheus label. Let's save, and
it should take a minute or so to update the Prometheus targets. A little tip, you can use
the KUBE_EDITOR environment variable to change the default editor from vim to visual studio
code, for example. Let's go back to Prometheus and refresh the page. Alright, we got ingress
target, but it's not ready yet. Now, it's up, which means Prometheus can scrape and save
those metrics; we will configure Grafana next. First, we need to provide the admin
user password for Grafana in base64 encoding format. You can use echo then
it's very important to use the -n flag to avoid a new line in the secret.
Then pipe it to the base64 tool. To decode, just pipe it back to base64 with the -d
flag. We're going to use that secret in the yaml. Let's create a folder for Grafana, again it's
going to be a very simple deployment only to demonstrate how to monitor Ingress.
Let's create a Kubernetes secret. We also need to provide admin user in
base64 format, so let's do that as well. Next is a simple deployment object for
Grafana without persistent storage. Then service object, that we
will use to configure Ingress. Standard port 3000 for Grafana. And let's
deploy it using the same kubectl apply command. Alright, the pod is ready. Next,
let's port forward Grafana to localhost and create a dashboard
for the nginx ingress controller. Localhost 3000. username
admin and password devops123. Before importing the dashboard, we need to create a data source that
points to the Prometheus server. The only one parameter that we need to
provide is the URL. Since prometheus is deployed in the same monitoring namespace as
Grafana, we just need to specify the service name prometheus-operated. If it would
be deployed in a different namespace, you need to use that namespace in the
URL as well after the dot. For example, prometheus-operated.monitoring:9090.
Monitoring is a namespace. Here we can omit it. Let's find a dashboard for nginx ingress. You can
simply google nginx ingress grafana dashboard. The id is 9614; let's copy it. To import
dashboard, go to manage and click import. Enter Id and press load. Select the default
Prometheus dashboard data source and click import. We already have some data in it; later, we will
come back to it when we have more ingresses. Now, let's create our first Ingress, which will be
for Prometheus. Often you would create Ingress by using internal Ingress only, but for this demo,
this Prometheus will be exposed to the internet, which is a bad idea in general. Let's
list services in the monitoring namespace. Create an example-1 folder for the first Ingress. It's pretty basic Ingress. We're going to call
it prometheus, and it must be created in the same namespace where you have the Service, in this case
monitoring namespace. Then let's use the ingress class that we created external-nginx. In case you
have multiple ingresses, here is a place where you can choose the one you want to use. Then specify
the DNS name prometheus.devopsbyexample.io. All our ingresses will point to the same load
balancer dns name. Then define the path, which is the root, type of prefix. That means
all the requests with all the paths will be simply routed to this Prometheus service. Later we
will do more advanced stuff with those paths. Then you just need to select the Kubernetes service
and the port. That's it; let's go and apply it. Now, let's get ingresses in the monitoring
namespace. Usually, it takes few minutes to update the address; even in some cases, if it
still is empty, try to create CNAME anyway. I had some issues previously when the ingress
controller wasn't able to publish its dns name. For the first Ingress, let's wait
anyway till we get the address. Alright, we have the public dns that we need to map using
CNAME with Prometheus. Just to show you, if you list services in the ingress namespace. You'll get
the load balancer with exactly the same dns name. Now we need to create a dns record.
I host my domain with google domains, but it does not matter; you just need to create
a CNAME that points to your load balancer public dns name. The host name will be Prometheus,
type CNAME. The value you can get from Ingress. Let's wait a few minutes till dns
is propagated. Okay, let's try to access prometheus.devopsbyexample.io. We got
our ingress prometheus working. As I said, it's okay with internal ingresses, but you don't
want to keep Prometheus publicly exposed. At least you want to set up a basic auth with username and
password and put nginx proxy at the front of it. Now, let's test the nginx admission webhook. There
are two ways to provide a custom nginx directive to ingress controller. You can use the nginx
ingress config map to apply your configuration globally, or you can target each ingress resource
separately. There are a lot of annotations supported by nginx ingress that you can use. If
your annotation is not supported yet, you can always provide a configuration snippet to the
Ingress. It accepts the raw nginx configuration block. That can be dangerous sometimes if you make
a mistake and your configuration becomes invalid. For example, we want to add additional headers to
this Ingress, not like Prometheus would care. You can use the more_set_headers nginx directive, but
if you make a typo admission webhook will reject it. The way it works, the nginx controller will
generate the full nginx config, including your custom directives, and run a test on it; if it is
invalid, the webhook will simply decline to accept it. Let's see how it works in practice. Let's
break that snippet and try to apply a new config. Now you can see that Ingress was rejected
and the error - unknown directive more_set_headers123. Let's fix it and
reapply. It went through. All ingress rules eventually transformed into plain
nginx config that you can always inspect if you suspect any strange behaviors. To
render nginx config, just use kubectl exec to the pod and provide the path to the main
nginx config, which is /etc/nginx/nginx.conf. Let's search for more_set_headers. You can see
that it is present here with the Foo bar header. In the next step, let's create Ingress for
Grafana. It's pretty much the same as with Prometheus. Find the Kubernetes service
that you want to use and map it with dns name in your Ingress. Here
is a grafana on port 3000. It's pretty identical, just a
different host, service name, and port. Let's apply and try to access it. As I said, you
don't have to wait for the address to show up; just create a CNAME to the same load balancer. It looks like it works; let's use
our credentials, admin devops123. Now, let's move to more advanced
ingresses. This one is simple fanout. I created a golang app for this demo.
Let's create an app folder and main.go. We're going to accept a couple of
input parameters. The first one is a service name and a port. Then let's
print out some metadata about the request, such as method, protocol, and headers.
And return to the client the service name and the path that was requested.
Finally, start that Service. Next, we need a go.mod file, which will be pretty
empty since we don't have any external libraries. And let's package our app as a docker image. Import golang image from docker hub
and use it for the first stage, call it build. Then you need to define the source
directory since we're using golang modules. Finally, build the binary. For the second stage,
let's use google's distroless images. In general, if your golang app does not depend on any
system libraries, you can use scratch or distroless images. Static
distroless image includes ca-certificates
A /etc/passwd entry for a root user A /tmp directory
tzdata Let's specify the user and copy the
binary from the previous docker stage. I already built it and uploaded it to
docker hub; it's a public image that you can use in your deployments as well. Now
let's create Kubernetes files and place them in the example-3 directory for this
Ingress. We need a new staging namespace. Then the deployment. It will take those two arguments; the service name
is foo and the random port. We need a service. And let's create similar objects for bar
deployment. The only difference is a name. Now we have two deployments, foo,
and bar. Let's create Ingress. Here you can see a new annotation
/$2. It will parse the URL and take the second half. I'll give you an example later.
We will use the same dns api.devopsbyexample.io for both services, and we will
use only different prefix paths. The first one is for foo service. On line 15,
you can see that we take the second argument provided by the nginx and append it to the path.
Otherwise, if you just use rewrite annotation, you will not be able to use anything except root
for your Service since nginx will replace it. $2 allows us to pass additional arguments.
The second path is pretty similar, just a different prefix. Let's
apply and see how it behaves. Let's add a new CNAME for api. Let's test foo service first. You can see
that rewrite target annotation will remove /foo from the request URL path and
provide the second part of the URL equal to $2. We don't want to pass the
entire URL, including /foo, to our Service, just remaining arguments. Same thing with a bar.
For example, if we omit rewrite-target annotation, it will pass the full URL to our Service. If
that's what you want, just remove the annotation. The following example will use host
names to differentiate between services. Here instead of different prefix paths, we
will use different host names api and a bar. Let's apply it. Now under hosts, we have multiple dns names;
we need to create CNAME for each of them. Let's try to access the foo service by its dns
name. You can see a response from the foo service with a requested path. If you access the bar,
that should route traffic to the bar service. Next is an ingress that uses a TLS certificate
to encrypt communication between a server and a client. In this video, we will create our
own CA certificate authority and a certificate for our domain. In the next videos, I'll
show you how you can get a certificate from letsencrypt in the Kubernetes using cert-manager.
We will use both HTTP-01 and DNS-01 challenges for that video. To generate self-sign
certificates, you can use either openssl, which is usually already installed, or my
favorite tool, cfssl. It is both a command-line tool and an HTTP API server for signing,
verifying, and bundling TLS certificates. You can download binary or use a package manager
such as homebrew to install it if you're on a mac. First, let's create a config
and a profile for the certs. Let's set default expiration to 3
years and a demo profile with 1-year certificate expiration. Keep in mind that
the certificates you want to use for the web cannot be generated with an expiration
longer than 13 months. The second file ca-csr is a certificate signing request for the
certificate authority. You don't have to generate CA and a cert; you can just go with one
single self-sign certificate for your domain. I just want to show you how to upload CA to
the keychain to validate your certificates. You'll see later. Here is the common name, the
algorithm, which is already default, but I decided to specify it here explicitly, and then some
names. C stands for Country Name, L for Locality, O for Organization, OU for OrganizationalUnit,
and then state. Now let's generate the CA. We got a CA certificate, certificate
signing request, and a private key. To decode, you can use openssl. Next, let's create a certificate signing
request for the foo-api subdomain. It's similar to CA, but here you must specify
hosts which translated to alternative names on the certificate. If you omit this field, your
certificate will not be valid in the browser. Now, let's generate it. Provide the
config and the profile that we defined earlier. You also need to specify
the CA here and its private key. If you decode it with openssl, you should see
the Subject alternative name with your dns name. To provide this certificate and a private key, we need to create a Kubernetes secret and
then encode those files in base64 format. So, here you need to provide a tls cert and a
key. Let's do it manually; first, a certificate. We can use exactly the same approach
that we used with Grafana credentials. Let's use echo and pipe this certificate
to the base64 tool to encode it. Now, let's copy the value
and use it in the secret. Same thing for the private key.
Encode to base64 and paste here. Now we can create Ingress
and reference this secret. Here is a new tls section; you need to specify
the host name and a secret to get a certificate and a private key. Rules will be pretty standard;
host and a path with reference to the Service. Now, let's apply and create CNAME as always we do. Let's try to access it. Alright, it works,
but since the certificate is signed by our own certificate authority, it
is not trusted by default. You can see the foo-api.devopsbyexample.io
certificate is not trusted. Let's fix it. You can add the certificate authority
to the keychain on mac. You still can do it for other platforms such as Windows or
Linux; you just need to find a way to add your CA and mark it as trusted. On a mac, open
a keychain and import CA that we generated. If you reload the page now, you get a lock. Now, the certificate is valid
and trusted. Keep in mind that it will be trusted only by your host.
In the next videos, I'll show you how to get a certificate from Let's encrypt to
make those certificates valid for anyone. If you want to test http2, you can use curl
with -i flag to get additional information. You can see that protocol is http2
since we secured our connection with the TLS certificate. Plain HTTP ingresses
still will be using the HTTP 1 protocol. Next is a little bit rare example,
when you want to use Ingress for services in different namespaces. By default,
you can only create Ingress, and the Service must be in the same namespace. Let's create foo
and bar namespaces and deploy services there. It's going to be example number 6. You can always
clone my repository to get all the source code. Alright, we have foo and bar
deployments in a different namespace; let's try to create Ingress now. Here we reference the foo service in the foo
namespace; it should work. But the second path references the bar service in a different
namespace. Let's see if it works, apply and test. Foo endpoint works as expected,
but what about bar endpoint. Nope, we got an error from the nginx;
the Service is temporarily unavailable. One of the workarounds is to create an external
service and place it in the same foo namespace. You can see the type is an external name that
references bar service in the bar namespace. Now we can just use this service
name bar external in the Ingress. You can try just to apply and see if it works. Sometimes you need to recreate Ingress.
You can see that it still does not work. Let me delete and apply it again. Since we already have CNAME, we can just
try to use curl right away. Alright, we got a response from the bar service that references
the Kubernetes service in a different namespace. Now, the last ingress example that demonstrates
how to add a TCP service. In some situations, you may want to reuse an existing load balancer.
By default, Ingress only supports HTTP and HTTPS services, but with nginx, you can add TCP
and UDP services as well. In this example, we will use the nginx controller to expose the
Postgres database. First, let's quickly create a namespace and Postgres deployment and use
the same devops123 admin password to access the database. I'm not going to spend a lot
of time on Postgres deployment right now. We're going to use a statefulset
and 8-gigabyte volume. Let's apply and make sure
that the database is running. Now let's create a configmap for the
nginx controller and not for the database. Here we're going to reference the Postgres
service in the database namespace. Also, I'm on purpose mapped it to a different port,
5444 on the Ingress. This config map should be created in the ingress namespace. By
the way, it's not a headless service; it's a regular cluster IP type
service, but it does not matter here. Now we need to update the nginx
controller deployment and Service. Here you can see that right now, only ports
80 and 443 are exposed on the load balancer. Now, let's directly edit the nginx deployment. We need to add additional flag TCP services
with reference to the configmap that we created. Now we need to edit the Service.
Let's add an additional port, 5444. When you save it, Kubernetes will also modify the
load balancer in AWS to add an additional listener and update the security group to allow that port.
We can see port 5444 in the Kubernetes service now. In AWS, under listeners, we got a new
5444 listener, and if we open the security group, we will find a new rule. When you're
using a network load balancer in the AWS, the security group that is attached to the node is
used to determine what traffic to allow and deny. Here you actually will not find port
5444 since it's routed to a different port on a Kubernetes node, in this
case, randomly generated port 30641. Let's create our last CNAME
for the Postgres database. Let's try to connect to the database. Port and default username postgres
with a password devops123. Alright, we successfully connected to the
Postgres using the nginx ingress and TCP service. The last thing, let's open
up the grafana dashboard; we should have more metrics now since
we created a bunch of ingresses. You can see ingress request volume
for each Service then the rate. Also, down below, we have p50 p90 and p99
latencies for each Service. P50 means that 50 percent of requests are completed in
a certain amount of time, same with p90, 90 percent of requests, and p99. For example, 99 percent of requests to grafana were completed
in under 9 milliseconds. There are more useful graphs that you can explore on your own.
Please do me a favor and like this video. In the next one, we will create an HTTPS
ingress using certificates from letsencrypt.
