Skill

collecting-cockroachdb-operator-escalation-packet

Name: collecting-cockroachdb-operator-escalation-packet
Author: CockroachDB

collect CockroachDB operator escalation packets

Published by CockroachDB Updated Jul 12

Covers Monitoring Incident Response Database Kubernetes

Description

Collects a complete CockroachDB Operator escalation packet for TSC/TSE or operator-team handoff, including Helm state, Kubernetes resources, logs, operation-specific evidence, pprof goroutine dumps, metrics, and a customer action timeline. Use when general diagnosis cannot resolve an operator-managed CockroachDB Helm issue or before restarting a stuck operator.

SKILL.md

Collecting CockroachDB Operator Escalation Packet

Collects the artifacts needed for TSC/TSE or operator-team escalation. Use this after basic diagnosis identifies an unresolved operator, Kubernetes, migration, upgrade, scale, DNS, PVC, or certificate issue. Keep collection read-only unless the customer explicitly approves a mitigation.

When to Use This Skill

A customer needs to escalate an operator-managed CockroachDB Helm issue to TSC
The operator is running but not reconciling and logs/status do not explain why
A rollout, upgrade, migration, or scale operation is stuck
Operator logs do not explain the failure
TSC needs consistent artifacts before advising restart, rollback, manual decommission, or status repair

Safety Considerations

Collect this packet before restarting, scaling, or deleting the operator during active operations.
Do not delete CrdbNode, PVC, Secret, service, version checker job, or StatefulSet resources while collecting data.
Do not print private key contents. Collect only Secret names, keys, certificate metadata, expiry, issuer, subject, and SANs.
Do not manually edit CrdbCluster.status.
Do not run cockroach init on an existing cluster.
If the operator is paused or disabled, record that state before changing anything.

Execution Discipline

Execute one step at a time and inspect the output before moving on. Do not run the packet collection in parallel; earlier results determine which operation-specific sections are relevant.
Keep collection read-only unless the user explicitly approves a mutating action for the target cluster.
Do not run interactive kubectl exec shells, kubectl debug, port-forwards, pprof/metrics collection, or commands that use external images in production unless the user approves them. If impact or policy is unclear, involve TSE or the operator team first.
Do not patch, annotate, delete, restart, scale, drain, decommission, or run helm upgrade as part of packet collection.

Required Inputs

Operator namespace and Helm release
CockroachDB namespace and Helm release
Kubernetes context and cluster/provider
Current operation: install, upgrade, scale up/down, cert rotation, migration, routine maintenance, or recovery
Current and target CockroachDB image, if an upgrade is involved
Customer timeline: what changed, what commands were run, and what was already tried

Packet Layout

Ask the customer to gather outputs into a single directory:

mkdir -p crdb-operator-escalation

Use clear filenames, for example operator-logs.txt, crdbcluster.yaml, events.txt, goroutine_dump.txt, and metrics_dump.txt.

Step 1: Context and Version Inventory

kubectl config current-context > crdb-operator-escalation/kube-context.txt
kubectl version --short > crdb-operator-escalation/kubernetes-version.txt

helm -n <operator-namespace> status <operator-release> > crdb-operator-escalation/operator-helm-status.txt 2>&1 || true
helm -n <operator-namespace> history <operator-release> > crdb-operator-escalation/operator-helm-history.txt 2>&1 || true
helm -n <cockroachdb-namespace> status <cockroachdb-release> > crdb-operator-escalation/crdb-helm-status.txt 2>&1 || true
helm -n <cockroachdb-namespace> history <cockroachdb-release> > crdb-operator-escalation/crdb-helm-history.txt 2>&1 || true

kubectl -n <operator-namespace> get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' > crdb-operator-escalation/operator-image.txt
kubectl get crd crdbclusters.crdb.cockroachlabs.com crdbnodes.crdb.cockroachlabs.com -o wide > crdb-operator-escalation/crds.txt
kubectl get crd crdbclusters.crdb.cockroachlabs.com -o yaml > crdb-operator-escalation/crdbclusters-crd.yaml
kubectl get crd crdbnodes.crdb.cockroachlabs.com -o yaml > crdb-operator-escalation/crdbnodes-crd.yaml

Get the CockroachDB version from a Ready pod:

kubectl -n <cockroachdb-namespace> exec <ready-crdb-pod> -c cockroachdb -- \
  /cockroach/cockroach version > crdb-operator-escalation/crdb-version.txt 2>&1

Step 2: Resource Specifications and Status

kubectl -n <operator-namespace> get deploy,pod,svc,endpoints -o wide > crdb-operator-escalation/operator-resources.txt
kubectl -n <operator-namespace> describe deploy cockroach-operator > crdb-operator-escalation/operator-deploy-describe.txt
kubectl -n <operator-namespace> describe pods -l app=cockroach-operator > crdb-operator-escalation/operator-pods-describe.txt
kubectl -n <operator-namespace> get deploy cockroach-operator -o yaml > crdb-operator-escalation/operator-deploy.yaml

kubectl -n <cockroachdb-namespace> get crdbcluster <cockroachdb-release> -o yaml > crdb-operator-escalation/crdbcluster.yaml
kubectl -n <cockroachdb-namespace> describe crdbcluster <cockroachdb-release> > crdb-operator-escalation/crdbcluster-describe.txt
kubectl -n <cockroachdb-namespace> get crdbnodes -o yaml > crdb-operator-escalation/crdbnodes.yaml
kubectl -n <cockroachdb-namespace> describe crdbnodes > crdb-operator-escalation/crdbnodes-describe.txt
kubectl -n <cockroachdb-namespace> get pod,svc,endpoints,pvc,pdb -o wide > crdb-operator-escalation/crdb-resources-wide.txt
kubectl -n <cockroachdb-namespace> describe pods -l app.kubernetes.io/name=cockroachdb > crdb-operator-escalation/crdb-pods-describe.txt
kubectl -n <cockroachdb-namespace> describe pvc > crdb-operator-escalation/pvc-describe.txt
kubectl -n <cockroachdb-namespace> describe pdb > crdb-operator-escalation/pdb-describe.txt
kubectl -n <cockroachdb-namespace> get events --sort-by=.lastTimestamp > crdb-operator-escalation/events.txt

Summarize key cluster status:

kubectl -n <cockroachdb-namespace> get crdbcluster <cockroachdb-release> -o json | jq '{
  mode: .spec.mode,
  nodes: [.spec.regions[]?.nodes],
  regions: .spec.regions,
  image: .spec.image,
  generation: .metadata.generation,
  observedGeneration: .status.observedGeneration,
  statusImage: .status.image,
  actions: .status.actions,
  conditions: .status.conditions,
  labels: .metadata.labels,
  annotations: .metadata.annotations
}' > crdb-operator-escalation/crdbcluster-summary.json

kubectl -n <cockroachdb-namespace> get crdbnodes \
  -o custom-columns=NAME:.metadata.name,GENERATION:.metadata.generation,OBSERVED:.status.observedGeneration,PHASE:.status.phase,HASH:.metadata.annotations["crdb.cockroachlabs.com/hash-revision"],NODE_ID:.status.nodeID \
  > crdb-operator-escalation/crdbnodes-summary.txt

Step 3: Logs

Collect full recent logs, not filtered snippets:

kubectl -n <operator-namespace> logs -l app=cockroach-operator --tail=500 > crdb-operator-escalation/operator-logs.txt 2>&1
kubectl -n <operator-namespace> logs -l app=cockroach-operator --previous --tail=500 > crdb-operator-escalation/operator-previous-logs.txt 2>&1 || true

For each CockroachDB pod:

kubectl -n <cockroachdb-namespace> logs <crdb-pod> -c cockroachdb --tail=500 > crdb-operator-escalation/<crdb-pod>-cockroachdb.log 2>&1
kubectl -n <cockroachdb-namespace> logs <crdb-pod> -c cockroachdb --previous --tail=500 > crdb-operator-escalation/<crdb-pod>-cockroachdb-previous.log 2>&1 || true
kubectl -n <cockroachdb-namespace> logs <crdb-pod> -c cert-reloader --tail=100 > crdb-operator-escalation/<crdb-pod>-cert-reloader.log 2>&1 || true

Step 4: Operation-Specific Evidence

Upgrade

kubectl -n <cockroachdb-namespace> get crdbcluster <cockroachdb-release> -o json | jq '{
  specImage: .spec.image,
  statusImage: .status.image,
  actions: .status.actions,
  annotations: .metadata.annotations,
  conditions: [.status.conditions[]? | select(.type | test("Upgrade|Version|Validate"))]
}' > crdb-operator-escalation/upgrade-status.json

kubectl -n <cockroachdb-namespace> get jobs -o wide > crdb-operator-escalation/jobs.txt
kubectl -n <cockroachdb-namespace> get pods -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,PHASE:.status.phase > crdb-operator-escalation/pod-images.txt
kubectl -n <cockroachdb-namespace> describe job <version-checker-job> > crdb-operator-escalation/version-checker-job.txt 2>&1 || true
kubectl -n <cockroachdb-namespace> logs -l job-name=<version-checker-job> > crdb-operator-escalation/version-checker-logs.txt 2>&1 || true

Include current and target CRDB image and whether any version checker job or pod was deleted.

Scale Down or Decommission

kubectl -n <cockroachdb-namespace> exec <ready-crdb-pod> -c cockroachdb -- \
  /cockroach/cockroach node status --decommission > crdb-operator-escalation/node-decommission-status.txt 2>&1

kubectl -n <cockroachdb-namespace> get crdbnodes -o json | jq '[.items[] | select(.status.phase=="Decommissioning")] | {count: length, nodes: [.[].metadata.name]}' \
  > crdb-operator-escalation/decommissioning-crdbnodes.json

Include original and target node count, whether multiple nodes changed at once, and whether manual drain/decommission was attempted.

Scale Up

Capture whether machines, Kubernetes nodes, disks, storage classes, topology spread constraints, or node labels changed:

kubectl get nodes -L topology.kubernetes.io/region,topology.kubernetes.io/zone > crdb-operator-escalation/nodes-locality.txt
kubectl get storageclass > crdb-operator-escalation/storageclasses.txt
kubectl -n <cockroachdb-namespace> get pvc -o wide > crdb-operator-escalation/pvc-wide.txt

Certificate Rotation

kubectl -n <cockroachdb-namespace> get secret,configmap | grep -E 'ca|node|client|tls|cert' > crdb-operator-escalation/cert-resources.txt || true
kubectl -n <cockroachdb-namespace> get certificate,issuer,clusterissuer -o wide > crdb-operator-escalation/cert-manager-resources.txt 2>&1 || true

kubectl -n <cockroachdb-namespace> get secret <node-tls-secret> -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -dates -subject -issuer -ext subjectAltName > crdb-operator-escalation/node-cert-metadata.txt

Do not collect private key values.

Migration

Use debugging-cockroachdb-operator-migrations and attach its migration state output. Include source StatefulSet or v1alpha1 CrdbCluster, migration labels, migration phase, source ownerReferences, and migration controller logs.

Step 5: Operator pprof and Metrics

Collect these when the operator is running but not reconciling, logs/status do not explain why, or a worker may be blocked. In production or restricted environments, confirm with the customer, TSE, or the operator team before opening port-forwards.

In one terminal:

kubectl -n <operator-namespace> port-forward deployment/cockroach-operator 7080:7080

In another terminal:

curl -s 'http://localhost:7080/debug/pprof/goroutine?debug=2' > crdb-operator-escalation/goroutine_dump.txt
curl -s 'http://localhost:7080/debug/pprof/goroutine?debug=1' > crdb-operator-escalation/goroutine_summary.txt
curl -s 'http://localhost:7080/debug/pprof/heap' > crdb-operator-escalation/heap.prof
curl -s 'http://localhost:7080/debug/pprof/profile?seconds=30' > crdb-operator-escalation/cpu.prof
curl -s 'http://localhost:7080/debug/pprof/mutex' > crdb-operator-escalation/mutex.prof

Metrics:

kubectl -n <operator-namespace> port-forward deployment/cockroach-operator 8080:8080
curl -s http://localhost:8080/metrics > crdb-operator-escalation/metrics_dump.txt
grep 'controller_runtime_reconcile_total' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/reconcile-total.txt || true
grep 'controller_runtime_reconcile_errors_total' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/reconcile-errors.txt || true
grep 'workqueue_depth' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/workqueue-depth.txt || true
grep 'workqueue_longest_running_processor_seconds' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/workqueue-longest-running.txt || true
grep 'workqueue_unfinished_work_seconds' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/workqueue-unfinished.txt || true

What TSC should inspect:

processNextWorkItem followed by Reconcile and a long-running call in goroutine_dump.txt
kube.(*Ctl).Exec, cockroach init, HTTP calls, or mutex waits with long durations
increasing workqueue_longest_running_processor_seconds
growing workqueue_depth
no reconcile count movement for a controller that should be active

Step 6: Timeline

Create crdb-operator-escalation/timeline.md with:

When the issue started
What operation was in progress
What docs or runbooks were followed
Whether VMs, Kubernetes nodes, disks, storage classes, certs, network policies, or Helm values changed
Every manual delete, patch, edit, restart, drain, decommission, or Helm command, in order
What is currently serving traffic and what is unavailable
Customer restrictions, such as no cluster-admin, no debug containers, no external images, or private registry only

Step 7: Escalation Summary

Return a concise summary with:

Operation type and current impact
Current operator and CockroachDB versions
Current CrdbCluster mode, generation, observedGeneration, status image, and actions
Stuck resource or symptom
Any unsafe operations already performed
Missing artifacts, if any
Whether pprof/metrics suggest a blocked worker

References

More skills from the helm-charts repository

View all 6 skills

SkillTagsUpdated

More from CockroachDB

View publisher

SkillTagsUpdated