Kubernetes Tutorial : An API Perspective on How Kubernetes Works

As beautiful as Kubernetes is, it often feels like voodoo-magic on how it does things. But for operations team planning to run Kubernetes in production, “it’s magical” isn’t a very confidence inspiring answer. As a layer immediately above the infrastructure, Kubernetes orchestrates several critical functions such as scheduling, deployment management, service discovery, service routing and much more. So, it is critical for operations teams to have a solid understanding of this framework and its inner workings.

Fortunately, Kubernetes is a quintessential modern system which relies heavily on API calls among its components. Observing and analyzing the API calls, we can learn a lot about systems such as Kubernetes, without requiring any code change or in-depth Ph.D. in the system. Netsil precisely employs this innovative technique of analyzing packets from service interactions. We will leverage Netsil to understand the inner workings of Kubernetes. In this blog post, we will understand the calls flows for namespace, pod and service creation.

An important broader point worth highlighting is that just like Kubernetes there are many 3rd party, OSS and external systems that are heavily used in modern cloud applications. Whether it is OSS such as Kafka, MySQL, Consul; or SaaS services such as AWS RDS, Stripe APIs, Google Auth APIs etc.; your cloud application is made up of components that are hard to instrument and monitor using existing techniques. A much more practical and effective approach is to leverage service interactions as the source of truth to monitor modern cloud applications (check out our previous blog for more information on this approach). With that digression out of the way, let’s first review our test environment setup and then understand how Kubernetes works.

Overview of Netsil Technique Used to Understand Kubernetes

We use a simple Kubernetes setup with one master and two worker nodes. The Netsil collector is installed as Docker container on Kubernetes master and as daemon set pods on workers (see figure 1). The Netsil collectors leverage pcap to capture and send copies of packets from API interactions between Kubernetes master and workers. These packet copies are low-level (i.e., L3 TCP/IP packets) which are reconstructed into application level protocols (i.e. L7) by Netsil. We use the Netsil Analytics Sandbox to query, analyze and understand the Kubernetes inner workings using the API data.

Setup for understanding the Kubernetes call flows using Netsil

Setup for understanding the Kubernetes call flows using Netsil

With that backdrop on API data collection, we use the following query to understand the Kubernetes API call flows.

  • We use http.request_reponse.count data; since we know these are primarily REST calls.
  • We will group the http data based on Client.hostname, Server.hostname. This will, naturally, help us understand who is involved in the request and response for the call.
  • We will also group by http.uri and http.request_method to identify the URI endpoint and the HTTP method type (GET, PUT, POST, PATCH, etc.)
  • And lastly, we use time window to focus the query on specific time when the Kubernetes calls are made.
Using Netsil HTTP Analytics to Capture and Analyze Kubernetes Inner Workings

Using Netsil HTTP Analytics to Capture and Analyze Kubernetes Inner Workings

Figure 2, shows how the query looks like in Netsil Analytics Sandbox. And now let’s dive right into the call flows for Kubernetes.

How does Kubernetes Work?

Create Namespace

>> date; kubectl create ns test-netsil-ns ; date

Using above command, we create namespace test-netsil-ns and use the date timestamps to analyze the API calls in Netsil AOC.

Key Observations:

    1. There are multiple GET calls from kubectl to Kubernetes API server before the namespace creation starts.
    2. Kubectl is wrapping up not just the authentication but also authorization, certificate checks, policy and rbac calls.
    3. For namespace creation, there were two POST calls. One to create the namespace and another to create a service-account. The service-account creation POST call is initiated and served by k8s-master. With every namespace, a service account is automatically created bearing the same name as that of the namespace.

Kubernetes Namespace Creation

Create Deployment and Pods

>> date; kubectl run nginx –image=nginx –port=80 –replicas=2 –namespace=test-netsil-ns; date

Key Observations:

  • One POST call for creation of Deployment. (client.host_name:null refers to Kubectl running on our laptop, where Netsil collector is not installed)
  • One POST call to create from k8s-master to k8s-master for creating ReplicaSet
  • Two POST calls to create 2 pods one for each pod creation also going from k8s-master to k8s-master. These will essentially just create entries in etcd. The actual pod creation happens later and is done by workers.
  • POST calls to bind the pods to nodes
  • An important step is where the kubelets on worker nodes read the pods assigned to them. This is done via a GET call to the API server as shown below. There is no POST call from k8s-master to k8s-worker for creating the pods.
  • Kubelets communicate each event of pod’s lifecycle (scheduled, image pull, container creation etc) to the master via POST calls. Kubectl describe pods gives greater description of these events:

  • Once the POD is running, the kubelets send the pod creation status to the master via the PUT call.

Kubernetes Pod Creation

Create Service and Nodeport

  • Create a file called nginx_svc.yaml with the following content:

  • Create service via following command: date; kubectl create -f nginx_svc.yaml; date
  • Verify that the service is working by opening http://<PUBLIC_IP_OF_ANY_NODE>:30719 this should result in nginx default page as shown below.

Welcome to ngix

Key Observations:

  • As expected, there is a POST request to create the service
  • In addition, there is another POST within the kubernetes master which creates endpoints for the service.

  • The workers then obtain the service and endpoint information using GET calls. This information is used to change the IP-tables (we are using Nodeport for this test.)
  • Below is an example of the changed iptables on one of the worker nodes.

sudo iptables-save | grep nginx  (nginx is the service we created)
-A KUBE-NODEPORTS -p tcp -m comment –comment “test-netsil-ns/nginx:80-80” -m tcp –dport 30719 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment –comment “test-netsil-ns/nginx:80-80” -m tcp –dport 30719 -j KUBE-SVC-5BY2RLIKAEGQ6ULL
-A KUBE-SEP-LW7P6RQEUTBPRU6Z -s -m comment –comment “test-netsil-ns/nginx:80-80” -j KUBE-MARK-MASQ
-A KUBE-SEP-LW7P6RQEUTBPRU6Z -p tcp -m comment –comment “test-netsil-ns/nginx:80-80” -m tcp -j DNAT –to-destination
-A KUBE-SEP-LW7P6RQEUTBPRU6Z -s -m comment –comment “test-netsil-ns/nginx:80-80” -j KUBE-MARK-MASQ
-A KUBE-SEP-LW7P6RQEUTBPRU6Z -p tcp -m comment –comment “test-netsil-ns/nginx:80-80” -m tcp -j DNAT –to-destination
-A KUBE-SERVICES -d -p tcp -m comment –comment “test-netsil-ns/nginx:80-80 cluster IP” -m tcp –dport 80 -j KUBE-SVC-5BY2RLIKAEGQ6ULL
-A KUBE-SVC-5BY2RLIKAEGQ6ULL -m comment –comment “test-netsil-ns/nginx:80-80” -m statistic –mode random –probability 0.50000000000 -j KUBE-SEP-LW7P6RQEUTBPRU6Z
-A KUBE-SVC-5BY2RLIKAEGQ6ULL -m comment –comment “test-netsil-ns/nginx:80-80” -j KUBE-SEP-LW7P6RQEUTBPRU6Z

Completing the Knowledge

We captured quite a lot of information on how Kubernetes works using Netsil and analyzing HTTP calls. Naturally, we didn’t learn much about what is going on inside the Kubernetes master or workers. This is where having deep knowledge (or as we call it Ph.D.) in the system comes in. There is no other better person to learn about internals of Kubernetes than Joe Beda, one of the founders of Kubernetes. And he recently wrote an amazing blog on “Jazz Improv” performed by Kubernetes. We have copied informative picture from his blog below and would highly recommend that you check out that blog.

Kubernetes Inner Workings (Credit: Joe Beda @ Heptio https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca)



In this tutorial, we explained the inner workings of Kubernetes API calls for some of the key functions. We used Netsil to capture and analyze HTTP calls among Kubernetes master and workers. We didn’t need to do any changes to Kubernetes code; and didn’t require in-depth knowledge of Kubernetes internals.

Considering the complexity of modern applications and the number of components they have, it is not practical for operations teams to have in-depth knowledge into all of them. As illustrated with Kubernetes, the Netsil approach of capturing and analyzing API calls is simple yet effective in understanding and monitoring these complex components and systems of modern microservices applications.

We would love to engage with you:



Get Netsil Updates

Get good content for good karma!

Copyright © 2015 - 2018 Netsil Inc. All Rights Reserved. | Privacy Policy

Share This
Netsil has been acquired by NutanixRead More