~/Rev_
← writeups

Jenkins on k3s With Persistent State

Helm said deployed, the pod said Init:CrashLoopBackOff. Reading the init container's logs found a volume-ownership issue; Jenkins now survives reboots.

2026-08-10#CyberDen
work: Aug 2026 · Done · DevSecOps, Kubernetes, k3s, Helm, Jenkins

Context

devops-k3s-01 is a single-node k3s cluster (Ubuntu 24.04, Traefik, local-path storage) and the base of CyberDen's DevSecOps platform. Gitea, its bundled PostgreSQL and Valkey were already running on bound PVCs. Jenkins was next, with its home directory on a dedicated 100 GB virtual disk backed by the T320's ZFS pool, so CI state outlives the pod.

The install

helm install jenkins jenkins/jenkins \
  --namespace devsecops \
  --values ~/cyberden/platform/jenkins-values.yaml
kubectl get pods -n devsecops -w

Helm reported STATUS: deployed. The pod reported:

jenkins-0   0/2   Init:CrashLoopBackOff
qm config of the k3s VM
The k3s node: tagged onto the cloud segment, with a second disk on the ZFS pool for persistent volumes.

Reading the right logs

deployed only means Kubernetes accepted the resources. The failure was in an init container, so the main container's logs were empty. The useful commands:

kubectl get pod jenkins-0 -n devsecops \
  -o jsonpath='{.status.initContainerStatuses[*].name}{"\n"}'
kubectl logs -n devsecops jenkins-0 -c init --previous
kubectl describe pod jenkins-0 -n devsecops | tail -80

Root cause

The Jenkins image runs as UID/GID 1000. The directory backing the persistent volume on the node was owned by root, so the init container couldn't write plugins or configuration into JENKINS_HOME. It crashed, restarted and crashed again.

Fix

  1. Changed ownership of the backing directory to 1000:1000
  2. Deleted only the pod. The StatefulSet recreated it, and the PVC and data were untouched:
kubectl delete pod jenkins-0 -n devsecops

The pod then walked through the lifecycle I was waiting for:

Init:0/2 → Init:1/2 → PodInitializing → 1/2 Running → 2/2 Running

2/2 is the Jenkins controller plus the config-reload sidecar.

Proving persistence

After a VM reboot: k3s starts → the StatefulSet restores Jenkins → the PVC mounts the same data → Jenkins comes up with its configuration intact instead of re-initializing. That's the property that matters for a CI server.

Lessons

  • Helm status is not health. Watch the pod lifecycle.
  • Init containers fail quietly. Use -c <init> --previous.
  • Know your image's UID before you hand it a hostPath or local-path volume.
  • Delete pods, not releases, when the StatefulSet already owns the fix.

Sources: Kubernetes Cluster for AI VM (2 Aug 2026), DevSecOps Kubernetes Plan (6 Aug 2026)