DevOps Bootcamp - Container Orchestration with Kubernetes

🚀Introduction to Kubernetes

                    What is Kubernetes?
                    Open source container orchestration tool
Developed by Google
Automates many processes involved in deploying, managing and scaling containerized applications
Most used container orchestration platform
Also known as "K8s" or "Kube"

                

The Need for Kubernetes

The trend from monolith to microservices has resulted in increased usage of containers. Containers are the perfect host for microservice applications, but manually managing hundreds or thousands of containers is a lot of effort.

Kubernetes automates many manual tasks and provides:

High Availability

No downtime - applications are always accessible

Automatic Scaling

Scale applications based on demand

Disaster Recovery

Backup and restore capabilities

Self-Healing

Automatically replace failed containers

🏗️Kubernetes Architecture

2 Types of Nodes

Worker Node

The containerized applications run on Worker Nodes
Each Node runs multiple Pods
Much more compute resources needed
Actual workload runs on them

Control Plane

Manages the Worker Nodes and Pods in the cluster
Much more important and needs to be replicated
In production, replicas run across multiple machines

Worker Node Components

Each worker node needs to have 3 processes installed:

1) Container Runtime

Software responsible for running containers (e.g., containerd, CRI-O, Docker)

2) Kubelet

Agent that makes sure containers are running in a Pod. Communicates with the container runtime and the underlying server

3) Kube-proxy

Network proxy with intelligent forwarding of requests to the Pods

Control Plane Components

1) API Server

The cluster gateway - single entrypoint to the cluster. Acts as gatekeeper for authentication and validation

2) Scheduler

Decides on which Node new Pods should be scheduled based on resource requirements and constraints

3) Controller Manager

Detects state changes and tries to recover the cluster state as soon as possible

4) etcd

Kubernetes' backing store for all cluster data. The cluster brain where every change gets saved

🧩Core Kubernetes Components

POD

Group of 1 or more containers
Smallest unit of K8s
An abstraction over container
Usually 1 application/container per Pod
Pods are ephemeral - new IP address assigned on re-creation

SERVICE

Permanent IP address attached to each Pod
Also serves as a load balancer
Lifecycles of Service and Pod are not connected
Internal Service (default) - not accessible from outside
External Service - accessible through browser

INGRESS

Entrypoint to your K8s cluster
Request goes to Ingress first, then forwards to Service
Better alternative to External Service in production
Provides friendly URLs instead of IP:Port

CONFIGMAP

Stores non-confidential data in key-value pairs
External configuration for your application
Can be consumed as environment variables or config files

SECRET

Similar to ConfigMap but for sensitive data
Stores passwords, tokens, etc.
Base64 encoded (not encrypted by default!)

VOLUME

Attaches physical storage to your Pod
Storage can be local or remote
Data persists even if Pod crashes
K8s doesn't manage data persistence

DEPLOYMENT

Blueprint for Pods
You work with Deployments, not Pods directly
Defines number of replicas
For stateless applications

STATEFULSET

For stateful applications like databases
Ensures database reads/writes are synchronized
Provides ordering and uniqueness guarantees
Each Pod has persistent identifier

⚠️ Important: Storing data in a Secret component doesn't automatically make it secure. Built-in security mechanisms are not enabled by default. Use third-party secret management tools for production.

📝YAML Configuration Examples

Pod Configuration

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: nginx
    tier: frontend
spec:
  containers:
  - name: nginx
    image: nginx:1.21
    ports:
    - containerPort: 80
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
    env:
    - name: ENVIRONMENT
      value: "production"
    livenessProbe:
      httpGet:
        path: /
        port: 80
      initialDelaySeconds: 30
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 5

Deployment Configuration

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp-deployment
  labels:
    app: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: myapp:v1.2.3
        ports:
        - containerPort: 8080
        env:
        - name: DB_HOST
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: database-host
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: db-password
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
      - name: config-volume
        configMap:
          name: app-config

Service Configuration

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: webapp-service
spec:
  type: LoadBalancer
  selector:
    app: webapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
    nodePort: 30080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

ConfigMap Example

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database-host: "mysql-service.default.svc.cluster.local"
  database-port: "3306"
  app.properties: |
    logging.level=INFO
    app.name=MyWebApp
    app.version=1.2.3
    feature.flags.newUI=true
  nginx.conf: |
    server {
      listen 80;
      server_name example.com;
      location / {
        proxy_pass http://webapp-service:8080;
      }
    }

Secret Example

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  # Values must be base64 encoded
  # echo -n 'mypassword' | base64
  db-password: bXlwYXNzd29yZA==
  api-key: YWJjZGVmZ2hpams=
stringData:
  # You can also use stringData for automatic encoding
  jwt-secret: "my-super-secret-jwt-key"

Ingress Configuration

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - webapp.example.com
    secretName: webapp-tls
  rules:
  - host: webapp.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8080
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-service
            port:
              number: 80

PersistentVolumeClaim Example

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: webapp-storage
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: fast-ssd
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: webapp-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  hostPath:
    path: /data/webapp
  persistentVolumeReclaimPolicy: Retain

StatefulSet Example

statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-headless
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: root-password
        volumeMounts:
        - name: mysql-storage
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: mysql-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
  - port: 3306

HorizontalPodAutoscaler Example

hpa.yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: webapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: webapp-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30

NetworkPolicy Example

networkpolicy.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: webapp-netpol
spec:
  podSelector:
    matchLabels:
      app: webapp
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: production
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: mysql
    ports:
    - protocol: TCP
      port: 3306
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 53  # DNS
    - protocol: UDP
      port: 53

🌐Kubernetes Services

Services provide an abstract way to expose applications running on a set of Pods.

Service Types

ClusterIP (Default)

Internal service, not accessible from outside
All Pods in cluster can communicate with this service
Used for internal communication between services

NodePort

Accessible directly from outside cluster
Exposes service on each Node's IP at a static port
Not secure - external traffic has access to Worker Nodes
URL format: NodeIP:NodePort

LoadBalancer

Exposes service externally using cloud provider's load balancer
More secure and efficient with single entry point
Automatically creates NodePort and ClusterIP services

Ingress vs External Service

Ingress is the better alternative in production!

More intelligent and flexible routing
Can expose multiple services under the same IP
Supports TLS termination
Path-based and host-based routing

💾Kubernetes Volumes

⚠️ Important: K8s offers no data persistence out of the box. When a container crashes, K8s restarts it with a clean state - your data is lost!

Storage Requirements

Storage that doesn't depend on pod lifecycle
Storage must be available on all Nodes
Storage needs to survive even if cluster crashes

The 3 Volume Resources

Persistent Volume (PV)

Storage in the cluster provisioned by an administrator or dynamically using Storage Classes

Persistent Volume Claim (PVC)

A request for storage by a user. Claims can request specific size and access modes

Storage Class (SC)

Provisions PV dynamically when PVC claims it. Defines the "class" of storage

Local vs Remote Volume Types

For database persistence, use remote storage!

Local volume types violate persistence requirements by being tied to one specific Node and not surviving cluster crashes.

🗄️StatefulSets

Stateless vs Stateful Applications

Stateless Applications

Don't depend on previous data
Deployed using Deployment
Managed by Kubernetes

Stateful Applications

Update data based on previous data
Query and depend on most up-to-date data/state
K8s can't automate the process natively

Deployment vs StatefulSet

Deployment Pods

Identical and interchangeable
Created in random order with random hashes
One Service load balances to any Pod

StatefulSet Pods

More difficult to manage
Can't be created/deleted at same time
Can't be randomly addressed
Each has persistent identifier: $(statefulset name)-$(ordinal)

                    Scaling Database Applications
                    Only 1 replica (master) can make changes
Replicas are not identical
Each replica has its own storage
Storages are constantly synchronized

                

⚓Helm - The Kubernetes Package Manager

                    What is Helm?
                    Package manager for Kubernetes (like apt/yum for Linux)
Simplifies deployment of complex applications
Manages Kubernetes YAML files as a single unit
Provides templating, versioning, and rollback capabilities

                

Core Concepts

Chart

A Helm package containing all resource definitions
Collection of files that describe Kubernetes resources
Can be stored locally or in a remote repository

Release

An instance of a chart running in a cluster
One chart can be installed multiple times
Each installation creates a new release

Repository

Place where charts can be collected and shared
Like Docker Hub for Docker images
Can be public or private

Chart Structure

Chart Directory Structure

mychart/
├── Chart.yaml          # Chart metadata
├── values.yaml         # Default configuration values
├── charts/            # Chart dependencies
├── templates/         # Template files
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── _helpers.tpl   # Template helpers
│   └── NOTES.txt      # Post-installation notes
└── README.md          # Chart documentation

Chart.yaml Example

Chart.yaml

apiVersion: v2
name: webapp
description: A Helm chart for my web application
type: application
version: 1.0.0
appVersion: "1.2.3"
dependencies:
  - name: mysql
    version: 9.3.4
    repository: https://charts.bitnami.com/bitnami
    condition: mysql.enabled
  - name: redis
    version: 17.3.7
    repository: https://charts.bitnami.com/bitnami
    condition: redis.enabled
maintainers:
  - name: DevOps Team
    email: devops@example.com
keywords:
  - webapp
  - microservices
home: https://example.com
sources:
  - https://github.com/example/webapp

values.yaml Example

values.yaml

# Default values for webapp
replicaCount: 3

image:
  repository: myregistry/webapp
  pullPolicy: IfNotPresent
  tag: "1.2.3"

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

service:
  type: ClusterIP
  port: 80
  targetPort: 8080

ingress:
  enabled: true
  className: "nginx"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
  hosts:
    - host: webapp.example.com
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls:
    - secretName: webapp-tls
      hosts:
        - webapp.example.com

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80

mysql:
  enabled: true
  auth:
    rootPassword: "changeme"
    database: "webapp"
  primary:
    persistence:
      enabled: true
      size: 10Gi

redis:
  enabled: false

Template Example with Helm Functions

templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "webapp.fullname" . }}
  labels:
    {{- include "webapp.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "webapp.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
      labels:
        {{- include "webapp.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: {{ .Values.service.targetPort }}
              protocol: TCP
          {{- if .Values.env }}
          env:
            {{- range $key, $value := .Values.env }}
            - name: {{ $key }}
              value: {{ $value | quote }}
            {{- end }}
          {{- end }}
          livenessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}

Common Helm Commands

Helm Commands

# Add a chart repository
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# Search for charts
helm search repo nginx
helm search hub wordpress

# Install a release
helm install my-release bitnami/nginx
helm install my-webapp ./mychart -f custom-values.yaml

# List releases
helm list
helm list --all-namespaces

# Upgrade a release
helm upgrade my-release bitnami/nginx
helm upgrade my-webapp ./mychart --set image.tag=1.2.4

# Rollback a release
helm rollback my-release 1
helm history my-release

# Uninstall a release
helm uninstall my-release

# Create a new chart
helm create mychart

# Package a chart
helm package mychart

# Lint a chart
helm lint mychart

# Dry run installation
helm install my-release ./mychart --dry-run --debug

# Get values
helm get values my-release
helm show values bitnami/nginx

# Template rendering
helm template my-release ./mychart
helm template my-release ./mychart --set replicaCount=5

Helm Hooks

pre-install-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "webapp.fullname" . }}-pre-install
  annotations:
    "helm.sh/hook": pre-install
    "helm.sh/hook-weight": "1"
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: pre-install
        image: busybox
        command: ['sh', '-c', 'echo "Running pre-install hook"']

🎯 Helm Best Practices

Use semantic versioning for both chart version and appVersion
Always define resource limits in your values.yaml
Use _helpers.tpl for common template functions
Document your values with comments in values.yaml
Test your charts with helm lint and dry-run before deployment
Use subchart dependencies instead of duplicating common components
Sign your charts for production environments

🔒Secure your Cluster

Authentication & Authorization

In K8s, you must be authenticated (logged in) before your request can be authorized (granted permission to access).

Authentication Strategies

Client Certificates

X.509 client certificates for authentication

Static Token File

CSV file with token, user name, user uid, and optional group names

3rd Party Identity Service

Integration with LDAP, Active Directory, etc.

Authorization with RBAC

Role-based access control (RBAC) regulates access based on user roles within the organization.

Role & ClusterRole

Contains rules representing permissions
Role: namespaced permissions
ClusterRole: cluster-wide permissions

RoleBinding & ClusterRoleBinding

Links Role/ClusterRole to Users or Groups
RoleBinding: namespace-scoped
ClusterRoleBinding: cluster-scoped

RBAC Examples

role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

🏋️Interactive Exercises

Exercise 1: Kubernetes Components Quiz

Which component is responsible for scheduling pods to nodes?

API Server Scheduler Controller Manager etcd

Exercise 2: YAML Validator

Try writing your own Kubernetes YAML configuration. The validator will check for common issues.

Validation results will appear here...

Exercise 3: Kubectl Commands Practice

Practice common kubectl commands in this interactive terminal simulator.

Welcome to Kubernetes Terminal Simulator!

Try commands like: kubectl get pods, kubectl describe pod nginx

$

Exercise 4: Service Type Selector

Match the scenario with the appropriate service type:

Scenario 1: Internal microservice communication

ClusterIP NodePort LoadBalancer

Scenario 2: Production web application with cloud provider

ClusterIP NodePort LoadBalancer

Exercise 5: Troubleshooting Challenge

Your pod is in CrashLoopBackOff state. What's the first command you should run?

kubectl delete pod [pod-name] kubectl logs [pod-name] kubectl restart pod [pod-name] kubectl scale deployment [name] --replicas=0

Exercise 6: Helm Commands Practice

What's the correct command to install a chart with custom values?

helm install my-release chart/ --values custom.yaml helm deploy my-release chart/ -f custom.yaml helm create my-release chart/ --config custom.yaml helm run my-release chart/ --values custom.yaml

⭐Production & Security Best Practices

🎯 Deployment Best Practices

Specify pinned versions on container images (avoid "latest" tag)
Configure liveness probes to let K8s know when to restart containers
Configure readiness probes to let K8s know when apps are ready for traffic
Set resource limits & requests to prevent containers from consuming all resources
Deploy multiple replicas for each application to ensure availability

🏗️ Infrastructure Best Practices

Use multiple Worker Nodes to avoid single points of failure
Don't use NodePort in production - use LoadBalancer or Ingress instead
Label all K8s resources for better organization and Service selection
Use namespaces to group resources and define access rights

🔐 Security Best Practices

Scan images for vulnerabilities - manually or in CI/CD pipeline
No root access for containers - use non-root users to limit damage
Keep K8s version up to date - latest versions include security patches
Use RBAC - implement least privilege principle
Secure secrets properly - consider third-party secret management

🚨 Critical Security Reminder: The built-in Secret component in Kubernetes does not encrypt data by default! For production environments, always use dedicated secret management tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.

📚Additional Topics Covered

Kubernetes Operators

Automated scripted operators for managing stateful applications with domain-specific knowledge

Service Mesh

Advanced networking layer providing observability, security, and traffic management (Istio, Linkerd)

GitOps

Declarative continuous delivery using Git as single source of truth (ArgoCD, Flux)

Multi-tenancy

Strategies for sharing clusters between teams while maintaining isolation and security