YourAI.tools
NVIDIA logo

Skill

k8s-launch-kit-troubleshoot

troubleshoot NVIDIA Network Operator deployments

Published by NVIDIA Updated Jul 14
Covers Operations Networking NVIDIA Kubernetes Debugging

Description

Use this skill when the user has problems with NVIDIA Network Operator on Kubernetes, or wants to analyze a sosreport diagnostic dump. Activate for: OFED driver crashes, SR-IOV pods failing, NicClusterPolicy errors, network operator pod issues, RDMA not working, NIC configuration failures, pods stuck in CrashLoopBackOff or ContainerCreating with network annotations, VF allocation issues, or when the user mentions 'troubleshoot', 'debug', 'sosreport', 'diagnose', or describes any NVIDIA networking failure -- even if they don't explicitly ask for troubleshooting.

SKILL.md

l8k: Troubleshooting

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and exit codes.

Debug NVIDIA Network Operator issues on Kubernetes, with or without a sosreport.

l8k Troubleshooting Commands

# Collect a diagnostic dump from the cluster
l8k sosreport [--kubeconfig <PATH>] --output-dir ./sosreport

l8k sosreport gathers cluster state, CRDs, operator logs, and per-node NIC info into a structured directory for offline analysis. Use the diagnostic commands and triage workflow below to interpret the dump.

Diagnostic Commands

# NicClusterPolicy status (check state: ready vs notReady)
kubectl get nicclusterpolicy -o yaml

# Network operator pods
kubectl get pods -n <operator-ns> -o wide

# SR-IOV node states (VF allocation)
kubectl get sriovnetworknodestates -A -o yaml

# OFED driver pod logs
kubectl logs -n <operator-ns> -l app=mofed-<os> --tail=100

# NIC configuration daemon logs
kubectl logs -n <operator-ns> -l app=nic-configuration-daemon --tail=100

# Check for pods stuck on network resources
kubectl get pods -A -o wide | grep -E 'ContainerCreating|Init'

Common Failure Patterns

SymptomLikely CauseFix
NicClusterPolicy state: notReadyOFED driver pods failingCheck mofed pod logs, verify kernel/driver compatibility
Pods stuck in ContainerCreatingVFs not allocated or SR-IOV policy not appliedCheck sriovnetworknodestates, verify device plugin pods
CrashLoopBackOff on mofed podsKernel module conflictCheck thirdPartyRDMAModules, enable unloadThirdPartyRDMAModules
No VFs on nodeSriovNetworkNodePolicy not matchingVerify nodeSelector labels match worker nodes
RDMA not workingMissing RDMA device plugin or wrong resource nameCheck rdma-shared-dp pods, verify resource annotations
l8k discover daemon pods stuck (ImagePullBackOff / Pending)Bad image tag, missing pull secret, or no feature.node.kubernetes.io/pci-15b3.present=true nodesRe-run with --keep-namespace then kubectl describe pod -n nvidia-k8s-launch-kit. Fix networkOperator.componentVersion / pass --image-pull-secrets / verify NFD is running.
l8k validate / deploy can't find Network Operator podsOperator namespace mismatchVerify --network-operator-namespace matches actual namespace (does NOT apply to l8k discover — it ignores the flag and uses its own nvidia-k8s-launch-kit namespace)
IPPool not allocatingNV-IPAM subnet exhausted or misconfiguredCheck ippools CR status, verify CIDR ranges
--for requires --node-selector--for was passed without --node-selectorAdd --node-selector key=val,…. The synthesized clusterConfig has no live worker-node list; the selector identifies target nodes at apply time.
--for and --discover-cluster-config are mutually exclusiveBoth flags passed simultaneouslyPick one: --for skips discovery, --discover-cluster-config runs it.
unknown preset "X"; available: …--for X doesn't match any directory under presets/Run l8k preset list and re-run with one of those names.
preset has no capabilities blockPreset YAML used by --for is missing capabilities.nodes.{sriov,rdma,ib}Add the block to the preset's topology.yaml. Discovery-time overlay does not require it; only --for does.
unknown field "productType" in YAMLHand-authored config still uses the old key nameRename productType: to gpuType: (the field was renamed).

For detailed triage workflow, read references/troubleshooting-guide.md.

sosreport Analysis

If the user has a pre-collected sosreport directory (from l8k sosreport or manual collection):

sosreport/
├── metadata/          # Cluster info, node list
├── crds/              # NicClusterPolicy, SriovNetworkNodePolicy, IPPool, etc.
├── operator/          # Network operator pod logs
├── nodes/             # Per-node device info
└── network/           # Interface config, routing tables

Triage Checklist

  1. Read metadata/diagnostic-summary.yaml for overview
  2. Check pod health in operator/pods.yaml
  3. Inspect CRDs in crds/ for status fields
  4. Read operator logs in operator/logs/ for errors
  5. Check per-node NIC state in nodes/<node>/

See Also

AboutPrivacy PolicyTerms and Conditions

© 2026 YourAI.tools. Every skill from an identity-verified publisher.

Independent catalog. Not affiliated with, endorsed by, or sponsored by Anthropic or any listed publisher. All trademarks belong to their respective owners.