[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nvcf-self-managed-installation":3,"mdc-mbn2ug-key":39,"related-repo-nvidia-nvcf-self-managed-installation":5638,"related-org-nvidia-nvcf-self-managed-installation":5670},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":34,"sourceUrl":37,"mdContent":38},"nvcf-self-managed-installation","install and operate NVCF self-hosted stacks","Install and operate NVCF self-hosted control-plane and separate compute-plane stacks. Covers Helmfile values and CLI profile installation flows, teardown, values overrides, pull secrets, and troubleshooting. Use for nvcf-self-managed-stack, nvcf-compute-plane-stack, split compute-plane installation, control-plane installation, CLI-generated control-plane profiles, Helmfile, self-managed, or self-hosted deployments. Do NOT use for local k3d environments; use the local k3d development workflow instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"CLI","cli","tag",{"name":17,"slug":18,"type":15},"Deployment","deployment",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Infrastructure","infrastructure",176,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvcf","2026-07-27T06:06:46.634385","Apache-2.0",23,[29,30,31,32,8,33],"cloud-functions","gpu","inference","kubernetes","serverless-gpu",{"repoUrl":24,"stars":23,"forks":27,"topics":35,"description":36},[29,30,31,32,8,33],"Platform for deploying and routing GPU-accelerated inference, streaming, and batch workloads at scale.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvcf\u002Ftree\u002FHEAD\u002Fai-tooling\u002Fuser\u002Fskills\u002Fnvcf-self-managed-installation","---\nname: nvcf-self-managed-installation\ndescription: >-\n  Install and operate NVCF self-hosted control-plane and separate compute-plane\n  stacks. Covers Helmfile values and CLI profile installation flows, teardown,\n  values overrides, pull secrets, and troubleshooting. Use for\n  nvcf-self-managed-stack, nvcf-compute-plane-stack, split compute-plane\n  installation, control-plane installation, CLI-generated control-plane\n  profiles, Helmfile, self-managed, or self-hosted deployments. Do NOT use for\n  local k3d environments; use the local k3d development workflow instead.\nlicense: Apache-2.0\ncompatibility: Requires helmfile >= 1.1.0 \u003C 1.2.0, helm >= 3.12, helm-diff plugin, kubectl matching cluster version\nauthor: \"nvcf-core-eng \u003Cnvcf-core-eng@exchange.nvidia.com>\"\nversion: \"1.0.0\"\ntags: [nvcf, self-managed, helmfile, self-hosted, control-plane, compute-plane, cli-profile, installation, deployment, pull-secrets]\ntools: [Shell, Read, Edit, Grep, Glob]\nmetadata:\n  internal: false\n  author: \"nvcf-core-eng \u003Cnvcf-core-eng@exchange.nvidia.com>\"\n  version: \"1.0\"\n  tags: [nvcf, self-managed, helmfile, self-hosted, control-plane, compute-plane, cli-profile, installation, deployment, pull-secrets]\n  languages: [bash, yaml]\n  frameworks: [helmfile, helm, kubectl]\n  domain: cloud-infrastructure\n---\n\n# NVCF Self-Managed Stack Operations\n\nOperational guide for matching NVCF control-plane and compute-plane Helmfile bundles.\n\n## Instructions\n\nUse this skill for install, upgrade, or teardown work in matching `nvcf-self-managed-stack` and `nvcf-compute-plane-stack` bundles; keep Helmfile values and CLI profile handoffs separate. For `functionType: \"LLM\"`, read [LLM Function Enablement](references\u002Fhelmfile-structure.md#llm-function-enablement).\n\n## Prerequisites\n\nAsk for the extracted `nvcf-self-managed-stack` path. For split installation,\nalso require the matching-version `nvcf-compute-plane-stack` path. Verify both:\n\n```bash\nls \u003Ccontrol-plane-stack>\u002Fhelmfile.d\u002F \u003Ccontrol-plane-stack>\u002Fenvironments\u002F \u003Ccontrol-plane-stack>\u002Fsecrets\u002F \u003Ccontrol-plane-stack>\u002Fglobal.yaml.gotmpl\nls \u003Ccompute-plane-stack>\u002Fhelmfile.d\u002F \u003Ccompute-plane-stack>\u002Fenvironments\u002F \u003Ccompute-plane-stack>\u002Fglobal.yaml.gotmpl\n```\n\nIf either directory is missing, download matching bundle versions from NGC:\n\n```bash\nngc registry resource download-version \u003Corg>\u002Fnvcf-self-managed-stack:\u003Cversion>\nngc registry resource download-version \u003Corg>\u002Fnvcf-compute-plane-stack:\u003Cversion>\n```\n\nControl-plane commands below assume the control-plane bundle root. Split-flow\ncommands and root substitution are in\n[Split Compute-Plane Installation](references\u002Fcompute-plane-installation.md).\n\n## Before You Start\n\nVerify tooling and context before any operation:\n\n```bash\nhelmfile --version   # Must be 1.1.x (1.2.0 removed sequential mode)\nhelm version         # Must be >= 3.12\nhelm plugin list     # Must include helm-diff >= 3.11\nkubectl version      # Client must be within 1 minor version of cluster\n```\n\nControl-plane commands run from the extracted `nvcf-self-managed-stack\u002F` root:\n\n```bash\ncd path\u002Fto\u002Fnvcf-self-managed-stack\nls helmfile.d\u002F environments\u002F secrets\u002F global.yaml.gotmpl\n```\n\nIdentify the environment from `environments\u002F\u003Cname>.yaml` and\n`secrets\u002F\u003Cname>-secrets.yaml`; for EKS, use the canonical\n[CSP End-to-End Example](https:\u002F\u002Fdocs.nvidia.com\u002Fnvcf\u002Fv0.6.0-rc\u002Fcsp-end-to-end-example).\n\n## How Values Flow\n\nUnderstanding value precedence prevents the most common configuration mistakes.\n\n```\nenvironments\u002Fbase.yaml          (defaults)\n    -> merged with\nenvironments\u002F\u003Cenv>.yaml         (your overrides)\n    -> consumed by\nglobal.yaml.gotmpl              (Go template, constructs per-chart values)\n    -> consumed by\nsecrets\u002F\u003Cenv>-secrets.yaml      (sensitive values)\n    -> overridden by\nrelease inline values: blocks   (highest precedence)\n```\n\nCritical: `global.yaml.gotmpl` only passes through specific keys to each chart (image registries, node selectors, storage, replica counts). Setting an arbitrary chart value in the environment file will be silently ignored if `global.yaml.gotmpl` does not propagate it.\n\nTo override arbitrary chart values, use a helmfile release `values:` block. See [Overriding Helm Chart Values](#overriding-helm-chart-values).\n\n## Clean Installation\n\n### 1. Create namespaces and image pull secrets (if using a private registry)\n\n```bash\nfor ns in cassandra-system nats-system nvcf api-keys ess sis vault-system cert-manager; do\n  kubectl create namespace \"$ns\" --dry-run=client -o yaml | kubectl apply -f -\ndone\n\n# Only if pulling from a private registry (e.g., NGC nvcr.io)\nexport NGC_API_KEY=\"\u003Cyour-key>\"\nfor ns in cassandra-system nats-system nvcf api-keys ess sis vault-system cert-manager; do\n  kubectl create secret docker-registry nvcr-creds \\\n    --docker-server=nvcr.io \\\n    --docker-username='$oauthtoken' \\\n    --docker-password=\"$NGC_API_KEY\" \\\n    --namespace=\"$ns\" \\\n    --dry-run=client -o yaml | kubectl apply -f -\ndone\n```\n\nIf using pull secrets, you must also configure each helmfile release to reference them. See [Image Pull Secrets](#image-pull-secrets).\n\n### 2. Authenticate helm to your chart registry\n\nBoth `docker login` AND `helm registry login` are required for NGC. Helmfile uses helm (not docker) to pull OCI charts, so the helm credential store must be authenticated separately.\n\n```bash\n# NGC -- both commands are required\ndocker login nvcr.io -u '$oauthtoken' -p \"$NGC_API_KEY\"\nhelm registry login nvcr.io -u '$oauthtoken' -p \"$NGC_API_KEY\"\n\n# ECR\naws ecr get-login-password --region \u003Cregion> | \\\n  helm registry login --username AWS --password-stdin \u003Caccount>.dkr.ecr.\u003Cregion>.amazonaws.com\n```\n\n> Note: `helm registry login` keeps only the last login per host (multi-org gotcha).\n> The stack and the caches span several NGC orgs that need different keys, but they all\n> live under the one host `nvcr.io`. A second `helm registry login nvcr.io` (or\n> `docker login`) with a different key silently overwrites the first, so pulls for the\n> first org then fail with a misleading `403 Access Denied` (not a 401: the auth\n> succeeded, it's the wrong-org key). If you pull charts\u002Fimages from more than one org,\n> either re-run the login with the correct key immediately before each pull, or pass\n> `--username '$oauthtoken' --password \u003Ckey>` per `helm pull` invocation instead of relying\n> on the stored login. (Pulling everything from a single registry mirror avoids this\n> entirely.)\n\n### 2b. (Recommended) Pull from a registry the cluster authenticates to automatically\n\nIf you mirror the NVCF images into a registry that the cluster can pull from without an\nexplicit image pull secret (for example, a registry where the node or workload identity is\ngranted pull access automatically), point the stack at it and skip the credential steps\nabove.\n\n1. Point the stack at the mirror in `environments\u002F\u003Cenv>.yaml`:\n\n   ```yaml\n   global:\n     image:\n       registry: \u003Cyour-registry>            # e.g. registry.example.com\n       repository: \u003Cmirror-sub-path>\n     helm:\n       sources:                              # only if you also mirrored the helm charts\n         registry: \u003Cyour-registry>\n         repository: \u003Cmirror-sub-path>\n   ```\n\n   > The manifest has a mix of OCI and non-OCI artifacts, so convert non-OCI charts before\n   > mirroring. Container images (`nvcr.io\u002F...`) are already OCI and mirror as-is. Some Helm\n   > charts, though, are published to an `https:\u002F\u002F` Helm repository (for example\n   > `helm.ngc.nvidia.com`). Those are not OCI and cannot be resolved by `global.helm.sources`\n   > (OCI) or mirrored directly. Before pointing the stack at your mirror, convert each\n   > `https:\u002F\u002F` chart to OCI: `helm repo add`\u002F`helm pull` the `.tgz`, then\n   > `helm push \u003Cchart>.tgz oci:\u002F\u002F\u003Cyour-registry>\u002F\u003Cmirror-sub-path>`. Only OCI charts can live\n   > under `global.helm.sources`.\n\n2. Skip the credential steps above. When the cluster authenticates to the registry\n   automatically (the node or workload identity has pull access), the pull needs no\n   Kubernetes secret, so you do not need: the per-namespace `nvcr-creds` from step 1\n   (still create the namespaces, just not the secret), the `global.imagePullSecrets` entry,\n   or the `docker login`\u002F`helm registry login nvcr.io` from step 2.\n\n   > One residual credential: nvcf-api still validates the user function image at\n   > `function create` via the account registry credential, so that credential must point at\n   > whichever registry holds the user image (set it to your mirror if the image is mirrored\n   > there).\n\n### 3. Set the route domain\n\n`global.domain` sets the hostname suffix for routes such as `api.\u003Cdomain>`. It does not configure the Gateway or its load balancer. Query the exact existing Gateway:\n\n```bash\nGATEWAY_ADDRESS=$(kubectl get gateway \u003Cgateway-name> -n \u003Cgateway-namespace> -o jsonpath='{.status.addresses[0].value}')\ntest -n \"$GATEWAY_ADDRESS\"\n```\n\nThe address may be a hostname or IP. Cloud provider and Gateway configuration determine the load balancer type and address format. The self-managed stack does not select either.\n\nFor temporary testing without DNS, use a stable reserved suffix such as `global.domain: \"nvcf.test\"`, connect to `GATEWAY_ADDRESS`, and send `Host: api.nvcf.test`. For production, use your base DNS domain and point its route records to `GATEWAY_ADDRESS`. If the endpoint later changes, keep `global.domain` stable and update only DNS or the client destination. See Example 4 in [examples.md](examples.md).\n\n### 4. Preview and deploy\n\n```bash\nHELMFILE_ENV=\u003Cenv-name> helmfile template   # Preview rendered manifests\nHELMFILE_ENV=\u003Cenv-name> helmfile sync       # Deploy\n```\n\n> Set BOTH Cassandra passwords in the secrets file. The secrets file documents\n> `DEFAULT_CASSANDRA_PASSWORD`, but the cassandra chart has a second credential,\n> `cassandra.serviceRolePassword`, that defaults to `ch@ng3m3`. If you set only\n> `DEFAULT_CASSANDRA_PASSWORD`, the service role keeps that default and every DB-backed\n> service crash-loops with `Bad credentials` once it tries to authenticate. Set\n> `cassandra.serviceRolePassword` to the same value as `DEFAULT_CASSANDRA_PASSWORD`.\n\n### 5. Deployment phases\n\nHelmfile deploys in order with dependencies:\n\n| Phase | Selector | Services | Wait time |\n|-------|----------|----------|-----------|\n| 1 | `release-group=dependencies` | NATS, OpenBao, Cassandra | 5-10 min |\n| 2 | `release-group=services` | API, SIS, ESS, invocation, grpc-proxy, notary, api-keys, optional LLM gateway\u002Frouter, optional Vanity Gateway, NVCF UI when the stack package includes the addon | 5-10 min |\n| 3 | `release-group=ingress` | Gateway routes | 1-2 min |\n| 4 | `release-group=observability` | Observability stack (if enabled) | 1-2 min |\n\nMonitor in a separate terminal:\n\n```bash\nkubectl get pods -A -w\n```\n\n### 6. Enable Vanity Gateway (optional)\n\nVanity Gateway is disabled by default and is only needed for customer-facing\nhostnames or path mappings in front of the standard NVCF API and invocation\nroutes. It is available only in stack packages that include the Vanity Gateway\naddon. If the extracted stack package does not contain a `vanity-gateway`\nrelease and `vanityGateway` route values, skip these commands and use a newer\nstack package that includes them.\n\nIf the stack package includes the addon, enable it in\n`environments\u002F\u003Cenv-name>.yaml`:\n\n```yaml\naddons:\n  vanityGateway:\n    enabled: true\n    mappingConfig: {}\n```\n\nThe default route host is `vanity.\u003Cdomain>` and the backend is\n`vanity-gateway.nvcf:8080`. Put deployment-specific host and path mappings under\n`addons.vanityGateway.mappingConfig`. If you need custom vanity hosts instead of\n`vanity.\u003Cdomain>`, use the route hostname overrides supported by the stack\npackage and create matching DNS records.\n\nAfter confirming the package includes the `vanity-gateway` release, preview and\napply the service and route:\n\n```bash\nHELMFILE_ENV=\u003Cenv-name> helmfile --selector name=vanity-gateway template\nHELMFILE_ENV=\u003Cenv-name> helmfile --selector name=vanity-gateway sync\nHELMFILE_ENV=\u003Cenv-name> helmfile --selector release-group=ingress sync\n```\n\nVerify the route only when the addon is present and enabled:\n\n```bash\nkubectl get deploy,svc -n nvcf -l app.kubernetes.io\u002Fname=vanity-gateway\nkubectl get httproute -A | grep -i vanity\ncurl -H \"Host: vanity.\u003Cdomain>\" \"http:\u002F\u002F\u003Cgateway-address>\u002Fhealth\"\n```\n\nDo not enable Vanity Gateway for standard API, API Keys, invocation, LLM\ninvocation, or gRPC traffic. Those routes are provided by the base gateway\nroutes.\n\n### 7. Enable NVCF UI (optional)\n\nNVCF UI is an optional, customer-facing admin-panel UI. It is disabled by\ndefault and available only in stack packages that include the NVCF UI addon. If\nyour extracted stack package does not contain a `nvcf-ui` release and `nvcfUi`\nroute values, skip this section. The NVCF UI admin panel is currently\nunauthenticated; do not expose it to the public internet.\n\nFor enablement, namespace and pull secret setup, apply, and verification steps,\nsee [NVCF UI Addon](references\u002Fnvcf-ui-addon.md).\n\n### 8. Install the separate compute-plane stack\n\nThe control-plane stack does not install compute-plane components. After it is healthy, choose exactly one compute-plane handoff:\n\n- Helmfile values flow: author separate environment files, then register and\n  install with the compute-plane Makefile.\n- CLI profile flow: use the profile generated by the CLI control-plane install,\n  then run the complete CLI register and install commands.\n\nDo not mix these handoffs. Follow [Split Compute-Plane Installation](references\u002Fcompute-plane-installation.md)\nfor commands, EKS routing, pull secrets, verification, and teardown order.\n\n## Clean Teardown\n\nDestroy every compute plane before the control plane, using the teardown for\nthe same handoff; see [Split Compute-Plane Installation](references\u002Fcompute-plane-installation.md#teardown).\n\nScope: Only destroy releases managed by this control-plane helmfile stack. The NVCF releases are: `nats`, `openbao-server`, `cassandra`, `api-keys`, `sis`, `api`, `invocation-service`, `grpc-proxy`, `ess-api`, `notary-service`, optional `vanity-gateway`, `nvcf-ui`, `admin-issuer-proxy`, and `ingress`; `cert-manager` is included only when `certManager.enabled: true`. The control-plane namespace inventory also includes `cert-manager`, but default namespace cleanup preserves it. Namespace preservation does not preserve the Helm release. External cert-manager must use `certManager.enabled: false`; otherwise Helmfile treats the release as stack-managed. Do not delete other helm releases or namespaces on the cluster.\n\nBefore control-plane destroy, identify whether cert-manager is external. For an external installation, require effective `certManager.enabled: false` and abort if it is true. For an intended stack-managed release with effective `certManager.enabled: true`, verify matching stack Helm release metadata and abort on missing or mismatched evidence. Preserve the namespace by default; only plan its removal after verifying namespace provenance and getting explicit confirmation.\n\n### Standard teardown\n\nRun from inside the `nvcf-self-managed-stack\u002F` directory:\n\n```bash\nHELMFILE_ENV=\u003Cenv-name> helmfile destroy\n```\n\n### Delete namespaces\n\n```bash\nfor ns in cassandra-system nats-system nvcf api-keys ess sis nvcf-ui vault-system; do\n  kubectl delete namespace \"$ns\" --ignore-not-found\ndone\n```\n\n### Verify clean\n\n```bash\nkubectl get ns | grep -E '(cassandra|nats|vault|nvcf|api-keys|ess|sis)'\n# Should return empty\n```\n\n## Overriding Helm Chart Values\n\n### Environment file (limited)\n\nWorks only for keys that `global.yaml.gotmpl` propagates (e.g., `cassandra.replicaCount`, `global.storageClass`, `global.image`).\n\n### Release values block (any chart value)\n\nEdit the release in `helmfile.d\u002F*.yaml.gotmpl` and add a `values:` block. In the examples below, `\u003Cprivate-values>` refers to the `secrets\u002F` directory at the helmfile stack root.\n\n```yaml\n- name: cassandra\n  version: 0.8.0\n  condition: cassandra.enabled\n  namespace: cassandra-system\n  \u003C\u003C: *dependency\n  values:\n    - ..\u002Fglobal.yaml.gotmpl\n    - ..\u002F\u003Cprivate-values>\u002F{{ requiredEnv \"HELMFILE_ENV\" }}-secrets.yaml\n    - cassandra:\n        resources:\n          limits:\n            cpu: \"8\"\n            memory: 8192Mi\n          requests:\n            cpu: \"2\"\n            memory: 4096Mi\n```\n\nYAML merge gotcha: When a release uses `\u003C\u003C: *dependency` or `inherit`, specifying `values:` replaces the template's values list. You must re-include `global.yaml.gotmpl` and the secrets file.\n\n### Preview and apply a single release\n\n```bash\nHELMFILE_ENV=\u003Cenv> helmfile --selector name=cassandra template  # Preview\nHELMFILE_ENV=\u003Cenv> helmfile --selector name=cassandra sync      # Apply\n```\n\n## Image Pull Secrets\n\nThere are three distinct credential types. Do not conflate them, and mind the\nformat rule below (two of them fail even with the correct key if the format is wrong):\n\n| | Control Plane Pull Secrets | API Bootstrap Registry Creds | Sidecar Image-Pull Secret |\n|---|---|---|---|\n| Purpose | K8s pulls NVCF service images | nvcf-api validates the user function image at `function create` | nvcf-api pulls the platform sidecars injected into every worker pod |\n| Where | K8s `docker-registry` Secrets + Kyverno ClusterPolicy | `\u003Cprivate-values>\u002F\u003Cenv>-secrets.yaml` (account-bootstrap) | `NVCF_API_SIDECARS_IMAGE_PULL_SECRET` in `\u003Cenv>-secrets.yaml` -> OpenBao `services\u002Fnvcf-api\u002Fkv\u002Fsidecars\u002Fimage-pull-secret` (field `secret`) |\n| Format | standard `.dockerconfigjson` (kubectl builds it for you) | `base64(\"$oauthtoken:\u003Ckey>\")` | `base64(\"$oauthtoken:\u003Ckey>\")` |\n\n> Note: the account-bootstrap cred and the sidecar secret are NOT a\n> dockerconfigjson. The secrets template ships the placeholder\n> `REPLACE_WITH_BASE64_DOCKER_CREDENTIAL`, which reads as \"base64 of a dockerconfigjson.\"\n> It isn't. Both values are consumed as a raw docker `auth` string: nvcf-api takes the\n> value verbatim and drops it into the `auth` field of the dockerconfig it builds. So the\n> value must be `base64(\"$oauthtoken:\u003Ckey>\")`, e.g. `printf '$oauthtoken:%s' \"$KEY\" | openssl base64 -A`,\n> not `base64(\u003Cdockerconfigjson>)`.\n>  - Wrong account-bootstrap value: `function create` returns\n>    `400 \"must be base64 encoded username:password format\"`.\n>  - Wrong sidecar value: every worker's sidecar pull gets a malformed credential and\n>    `403`s, regardless of which key is inside. The symptom looks like a bad or wrong-org\n>    key, but the key is fine and the encoding is the bug. To repair an already-installed\n>    stack without a full re-sync:\n>    ```bash\n>    AUTH=$(printf '$oauthtoken:%s' \"$KEY\" | openssl base64 -A)\n>    bao kv put services\u002Fnvcf-api\u002Fkv\u002Fsidecars\u002Fimage-pull-secret secret=\"$AUTH\"\n>    # if NVIDIA Cloud Tasks is enabled, repeat for services\u002Fnvct-api\u002Fkv\u002Fsidecars\u002Fimage-pull-secret\n>    kubectl rollout restart -n nvcf deploy\u002Fnvcf-api   # picks up the new sidecar cred\n>    ```\n\n> Note (multi-org): when pulling from NGC (not a mirror), the user function image and\n> the platform sidecars often live in different nvcr.io orgs, each needing a different\n> key, so the account-bootstrap cred (function image) and the sidecar secret (sidecars)\n> are frequently two different keys. Mirroring everything into a single registry the\n> cluster authenticates to automatically collapses this to one credential path.\n\n### Configuring with Kyverno (recommended)\n\nUse a Kyverno mutating admission policy to automatically inject `imagePullSecrets` into all pods in NVCF namespaces. This works uniformly for all charts -- no per-chart configuration or helmfile modifications needed.\n\n```bash\n# 1. Install Kyverno\nhelm repo add kyverno https:\u002F\u002Fkyverno.github.io\u002Fkyverno\u002F\nhelm repo update\nhelm install kyverno kyverno\u002Fkyverno -n kyverno --create-namespace\n\n# 2. Create pull secret in each namespace\nexport NGC_API_KEY=\"\u003Cyour-key>\"\nfor ns in cassandra-system nats-system nvcf api-keys ess sis vault-system cert-manager; do\n  kubectl create secret docker-registry nvcr-creds \\\n    --docker-server=nvcr.io \\\n    --docker-username='$oauthtoken' \\\n    --docker-password=\"$NGC_API_KEY\" \\\n    --namespace=\"$ns\" \\\n    --dry-run=client -o yaml | kubectl apply -f -\ndone\n\n# 3. Apply the Kyverno ClusterPolicy\nkubectl apply -f kyverno-imagepullsecret-policy.yaml\n```\n\nThe policy mutates every pod at admission time, adding `imagePullSecrets: [{name: nvcr-creds}]`. Verify with:\n\n```bash\nkubectl get pod -n \u003Cnamespace> \u003Cpod-name> -o jsonpath='{.spec.imagePullSecrets}'\n# Should show: [{\"name\":\"nvcr-creds\"}]\n```\n\nNot needed if using a CSP built-in credential helper (e.g., ECR with IAM node roles).\n\nFor the Kyverno policy YAML and pull secret creation script, see [references\u002Fpull-secrets.md](references\u002Fpull-secrets.md).\n\n## Debugging\n\n### Quick status check\n\n```bash\nkubectl get pods -A -o wide                        # All pods\nhelm list -A                                        # All helm releases\nkubectl get events -n \u003Cns> --sort-by='.lastTimestamp'  # Recent events\n```\n\n### Common failure patterns\n\n| Symptom | Cause | Fix |\n|---------|-------|-----|\n| `ImagePullBackOff` + `401 Unauthorized` | Missing or wrong pull secret | Check secret exists, check SA has imagePullSecrets |\n| `Init:0\u002F1` stuck on service pods | Vault-agent waiting for OpenBao | Check OpenBao pods + migration job status |\n| `Init:0\u002F1` + `vault-agent-init` shows `auth\u002Fjwt\u002Flogin` -> `400 ... no known key successfully validated the token signature` | OpenBao JWT auth configured with a static pubkey instead of the cluster's live JWKS -- the base default `openbao.migrations.issuerDiscovery.enabled: false`. Common on AKS \u002F any OIDC-issuer cluster | Set `openbao.migrations.issuerDiscovery.enabled: true`, delete job `openbao-server-migrations`, re-sync `openbao-server`. See [references\u002Fdebugging.md](references\u002Fdebugging.md) \"Init:0\u002F1 Stuck\" fix #4. |\n| `OOMKilled` on Cassandra | Default resources too small | Override `cassandra.resources` via values block |\n| DB-backed services crash-loop `Bad credentials` connecting to Cassandra | Secrets file set only `DEFAULT_CASSANDRA_PASSWORD`; `cassandra.serviceRolePassword` still defaults to `ch@ng3m3` | Set `cassandra.serviceRolePassword` in the secrets file equal to `DEFAULT_CASSANDRA_PASSWORD`, then re-sync cassandra + the DB-backed services |\n| `Pending` pods | Node selector mismatch or no storage class | `kubectl describe pod`, check labels and storage |\n| Helm release in `failed` state | First install failed partway | `helmfile destroy` the release, then `sync` again |\n| Account bootstrap timeout | Wrong base64 credentials in secrets file | Check `kubectl logs job\u002Fnvcf-api-account-bootstrap -n nvcf` |\n| function \u002F account calls return `404 Unknown client_id` | The account-bootstrap post-install hook never ran: the `api` release install failed, or was repaired with live patches instead of a clean re-sync, so the hook (and the account) was skipped | Verify and re-run the hook; see [Verify the account bootstrap ran](#verify-the-account-bootstrap-ran) |\n| Services fail to read vault secrets; `secrets.json` not found | Vault path hardcoded to `\u002Fhome\u002Fapp\u002Fvault\u002F` in `_helpers.tpl`; the runtime resolves the mounted path relative to the working directory and drops the leading `\u002F` | Override `podAnnotations` and set `JAVA_TOOL_OPTIONS: \"-Duser.dir=\u002F\"` in release values |\n| NATS connection fails at startup; placement tag mismatch | NATS server tags hardcoded to `dc:ncp`; app derives tag from `AWS_REGION` (e.g., `us-gov-west-1`) | Set `AWS_REGION=ncp` and `NVCF_AWS_REGION=ncp` in env config |\n\n### Verify the account bootstrap ran\n\nThe default account is created by a **`post-install` hook on the `api` release**\n(`job\u002Fnvcf-api-account-bootstrap` in `nvcf`). Because it is a post-install hook, it only\nfires on a successful `api` install. If that install failed and you repaired it with live\npatches (rather than a clean re-sync), the hook never runs, no account is created, and\nevery subsequent function\u002Faccount call returns a cryptic `404 Unknown client_id` with\nnothing obviously wrong in the running pods.\n\nVerify it ran after the first sync:\n\n```bash\nkubectl get job -n nvcf nvcf-api-account-bootstrap \\\n  -o jsonpath='{.status.succeeded}'   # expect 1\n```\n\nIf the job is absent or `0`, re-run it by re-syncing the `api` release so the hook fires\nagain (delete the prior job first if it lingers in a completed\u002Ffailed state and blocks the\nhook):\n\n```bash\nkubectl delete job -n nvcf nvcf-api-account-bootstrap --ignore-not-found\nHELMFILE_ENV=\u003Cenv> helmfile --selector name=api sync\nkubectl logs -n nvcf job\u002Fnvcf-api-account-bootstrap   # confirm the account was created\n```\n\n## Additional Resources\n\n- Installation: [examples.md](examples.md), the [CSP End-to-End Example](https:\u002F\u002Fdocs.nvidia.com\u002Fnvcf\u002Fv0.6.0-rc\u002Fcsp-end-to-end-example), and [split compute-plane workflows](references\u002Fcompute-plane-installation.md)\n- Operations: [Helmfile structure](references\u002Fhelmfile-structure.md), [pull secrets](references\u002Fpull-secrets.md), and [debugging](references\u002Fdebugging.md)\n\nAfter deployment, use the `nvcf-self-managed-cli` skill to create functions, create tasks, manage API keys, and invoke endpoints.\n",{"data":40,"body":71},{"name":4,"description":6,"license":26,"compatibility":41,"author":42,"version":43,"tags":44,"tools":54,"metadata":60},"Requires helmfile >= 1.1.0 \u003C 1.2.0, helm >= 3.12, helm-diff plugin, kubectl matching cluster version","nvcf-core-eng \u003Cnvcf-core-eng@exchange.nvidia.com>","1.0.0",[45,46,47,48,49,50,51,52,18,53],"nvcf","self-managed","helmfile","self-hosted","control-plane","compute-plane","cli-profile","installation","pull-secrets",[55,56,57,58,59],"Shell","Read","Edit","Grep","Glob",{"internal":61,"author":42,"version":62,"tags":63,"languages":64,"frameworks":67,"domain":70},false,"1.0",[45,46,47,48,49,50,51,52,18,53],[65,66],"bash","yaml",[47,68,69],"helm","kubectl","cloud-infrastructure",{"type":72,"children":73},"root",[74,83,89,96,135,141,160,342,347,476,488,494,499,581,594,642,672,678,683,693,713,733,739,746,1179,1191,1197,1218,1475,1538,1544,1549,1835,1841,1860,1986,1991,2039,2045,2122,2182,2188,2193,2337,2342,2375,2381,2402,2414,2480,2516,2528,2645,2650,2769,2774,2780,2801,2813,2819,2824,2838,2849,2855,2866,3002,3021,3027,3039,3074,3080,3183,3189,3247,3252,3258,3292,3298,3334,3605,3640,3646,3735,3740,3745,3908,4152,4160,4166,4179,4554,4567,4657,4662,4673,4679,4685,4805,4811,5308,5313,5367,5372,5442,5462,5566,5572,5619,5632],{"type":75,"tag":76,"props":77,"children":79},"element","h1",{"id":78},"nvcf-self-managed-stack-operations",[80],{"type":81,"value":82},"text","NVCF Self-Managed Stack Operations",{"type":75,"tag":84,"props":85,"children":86},"p",{},[87],{"type":81,"value":88},"Operational guide for matching NVCF control-plane and compute-plane Helmfile bundles.",{"type":75,"tag":90,"props":91,"children":93},"h2",{"id":92},"instructions",[94],{"type":81,"value":95},"Instructions",{"type":75,"tag":84,"props":97,"children":98},{},[99,101,108,110,116,118,124,126,133],{"type":81,"value":100},"Use this skill for install, upgrade, or teardown work in matching ",{"type":75,"tag":102,"props":103,"children":105},"code",{"className":104},[],[106],{"type":81,"value":107},"nvcf-self-managed-stack",{"type":81,"value":109}," and ",{"type":75,"tag":102,"props":111,"children":113},{"className":112},[],[114],{"type":81,"value":115},"nvcf-compute-plane-stack",{"type":81,"value":117}," bundles; keep Helmfile values and CLI profile handoffs separate. For ",{"type":75,"tag":102,"props":119,"children":121},{"className":120},[],[122],{"type":81,"value":123},"functionType: \"LLM\"",{"type":81,"value":125},", read ",{"type":75,"tag":127,"props":128,"children":130},"a",{"href":129},"references\u002Fhelmfile-structure.md#llm-function-enablement",[131],{"type":81,"value":132},"LLM Function Enablement",{"type":81,"value":134},".",{"type":75,"tag":90,"props":136,"children":138},{"id":137},"prerequisites",[139],{"type":81,"value":140},"Prerequisites",{"type":75,"tag":84,"props":142,"children":143},{},[144,146,151,153,158],{"type":81,"value":145},"Ask for the extracted ",{"type":75,"tag":102,"props":147,"children":149},{"className":148},[],[150],{"type":81,"value":107},{"type":81,"value":152}," path. For split installation,\nalso require the matching-version ",{"type":75,"tag":102,"props":154,"children":156},{"className":155},[],[157],{"type":81,"value":115},{"type":81,"value":159}," path. Verify both:",{"type":75,"tag":161,"props":162,"children":166},"pre",{"className":163,"code":164,"language":65,"meta":165,"style":165},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","ls \u003Ccontrol-plane-stack>\u002Fhelmfile.d\u002F \u003Ccontrol-plane-stack>\u002Fenvironments\u002F \u003Ccontrol-plane-stack>\u002Fsecrets\u002F \u003Ccontrol-plane-stack>\u002Fglobal.yaml.gotmpl\nls \u003Ccompute-plane-stack>\u002Fhelmfile.d\u002F \u003Ccompute-plane-stack>\u002Fenvironments\u002F \u003Ccompute-plane-stack>\u002Fglobal.yaml.gotmpl\n","",[167],{"type":75,"tag":102,"props":168,"children":169},{"__ignoreMap":165},[170,273],{"type":75,"tag":171,"props":172,"children":175},"span",{"class":173,"line":174},"line",1,[176,182,188,194,200,205,210,214,218,222,226,231,235,239,243,247,252,256,260,264,268],{"type":75,"tag":171,"props":177,"children":179},{"style":178},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[180],{"type":81,"value":181},"ls",{"type":75,"tag":171,"props":183,"children":185},{"style":184},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[186],{"type":81,"value":187}," \u003C",{"type":75,"tag":171,"props":189,"children":191},{"style":190},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[192],{"type":81,"value":193},"control-plane-stac",{"type":75,"tag":171,"props":195,"children":197},{"style":196},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[198],{"type":81,"value":199},"k",{"type":75,"tag":171,"props":201,"children":202},{"style":184},[203],{"type":81,"value":204},">",{"type":75,"tag":171,"props":206,"children":207},{"style":190},[208],{"type":81,"value":209},"\u002Fhelmfile.d\u002F",{"type":75,"tag":171,"props":211,"children":212},{"style":184},[213],{"type":81,"value":187},{"type":75,"tag":171,"props":215,"children":216},{"style":190},[217],{"type":81,"value":193},{"type":75,"tag":171,"props":219,"children":220},{"style":196},[221],{"type":81,"value":199},{"type":75,"tag":171,"props":223,"children":224},{"style":184},[225],{"type":81,"value":204},{"type":75,"tag":171,"props":227,"children":228},{"style":190},[229],{"type":81,"value":230},"\u002Fenvironments\u002F",{"type":75,"tag":171,"props":232,"children":233},{"style":184},[234],{"type":81,"value":187},{"type":75,"tag":171,"props":236,"children":237},{"style":190},[238],{"type":81,"value":193},{"type":75,"tag":171,"props":240,"children":241},{"style":196},[242],{"type":81,"value":199},{"type":75,"tag":171,"props":244,"children":245},{"style":184},[246],{"type":81,"value":204},{"type":75,"tag":171,"props":248,"children":249},{"style":190},[250],{"type":81,"value":251},"\u002Fsecrets\u002F",{"type":75,"tag":171,"props":253,"children":254},{"style":184},[255],{"type":81,"value":187},{"type":75,"tag":171,"props":257,"children":258},{"style":190},[259],{"type":81,"value":193},{"type":75,"tag":171,"props":261,"children":262},{"style":196},[263],{"type":81,"value":199},{"type":75,"tag":171,"props":265,"children":266},{"style":184},[267],{"type":81,"value":204},{"type":75,"tag":171,"props":269,"children":270},{"style":190},[271],{"type":81,"value":272},"\u002Fglobal.yaml.gotmpl\n",{"type":75,"tag":171,"props":274,"children":276},{"class":173,"line":275},2,[277,281,285,290,294,298,302,306,310,314,318,322,326,330,334,338],{"type":75,"tag":171,"props":278,"children":279},{"style":178},[280],{"type":81,"value":181},{"type":75,"tag":171,"props":282,"children":283},{"style":184},[284],{"type":81,"value":187},{"type":75,"tag":171,"props":286,"children":287},{"style":190},[288],{"type":81,"value":289},"compute-plane-stac",{"type":75,"tag":171,"props":291,"children":292},{"style":196},[293],{"type":81,"value":199},{"type":75,"tag":171,"props":295,"children":296},{"style":184},[297],{"type":81,"value":204},{"type":75,"tag":171,"props":299,"children":300},{"style":190},[301],{"type":81,"value":209},{"type":75,"tag":171,"props":303,"children":304},{"style":184},[305],{"type":81,"value":187},{"type":75,"tag":171,"props":307,"children":308},{"style":190},[309],{"type":81,"value":289},{"type":75,"tag":171,"props":311,"children":312},{"style":196},[313],{"type":81,"value":199},{"type":75,"tag":171,"props":315,"children":316},{"style":184},[317],{"type":81,"value":204},{"type":75,"tag":171,"props":319,"children":320},{"style":190},[321],{"type":81,"value":230},{"type":75,"tag":171,"props":323,"children":324},{"style":184},[325],{"type":81,"value":187},{"type":75,"tag":171,"props":327,"children":328},{"style":190},[329],{"type":81,"value":289},{"type":75,"tag":171,"props":331,"children":332},{"style":196},[333],{"type":81,"value":199},{"type":75,"tag":171,"props":335,"children":336},{"style":184},[337],{"type":81,"value":204},{"type":75,"tag":171,"props":339,"children":340},{"style":190},[341],{"type":81,"value":272},{"type":75,"tag":84,"props":343,"children":344},{},[345],{"type":81,"value":346},"If either directory is missing, download matching bundle versions from NGC:",{"type":75,"tag":161,"props":348,"children":350},{"className":163,"code":349,"language":65,"meta":165,"style":165},"ngc registry resource download-version \u003Corg>\u002Fnvcf-self-managed-stack:\u003Cversion>\nngc registry resource download-version \u003Corg>\u002Fnvcf-compute-plane-stack:\u003Cversion>\n",[351],{"type":75,"tag":102,"props":352,"children":353},{"__ignoreMap":165},[354,420],{"type":75,"tag":171,"props":355,"children":356},{"class":173,"line":174},[357,362,367,372,377,381,386,391,395,400,405,410,415],{"type":75,"tag":171,"props":358,"children":359},{"style":178},[360],{"type":81,"value":361},"ngc",{"type":75,"tag":171,"props":363,"children":364},{"style":190},[365],{"type":81,"value":366}," registry",{"type":75,"tag":171,"props":368,"children":369},{"style":190},[370],{"type":81,"value":371}," resource",{"type":75,"tag":171,"props":373,"children":374},{"style":190},[375],{"type":81,"value":376}," download-version",{"type":75,"tag":171,"props":378,"children":379},{"style":184},[380],{"type":81,"value":187},{"type":75,"tag":171,"props":382,"children":383},{"style":190},[384],{"type":81,"value":385},"or",{"type":75,"tag":171,"props":387,"children":388},{"style":196},[389],{"type":81,"value":390},"g",{"type":75,"tag":171,"props":392,"children":393},{"style":184},[394],{"type":81,"value":204},{"type":75,"tag":171,"props":396,"children":397},{"style":190},[398],{"type":81,"value":399},"\u002Fnvcf-self-managed-stack:",{"type":75,"tag":171,"props":401,"children":402},{"style":184},[403],{"type":81,"value":404},"\u003C",{"type":75,"tag":171,"props":406,"children":407},{"style":190},[408],{"type":81,"value":409},"versio",{"type":75,"tag":171,"props":411,"children":412},{"style":196},[413],{"type":81,"value":414},"n",{"type":75,"tag":171,"props":416,"children":417},{"style":184},[418],{"type":81,"value":419},">\n",{"type":75,"tag":171,"props":421,"children":422},{"class":173,"line":275},[423,427,431,435,439,443,447,451,455,460,464,468,472],{"type":75,"tag":171,"props":424,"children":425},{"style":178},[426],{"type":81,"value":361},{"type":75,"tag":171,"props":428,"children":429},{"style":190},[430],{"type":81,"value":366},{"type":75,"tag":171,"props":432,"children":433},{"style":190},[434],{"type":81,"value":371},{"type":75,"tag":171,"props":436,"children":437},{"style":190},[438],{"type":81,"value":376},{"type":75,"tag":171,"props":440,"children":441},{"style":184},[442],{"type":81,"value":187},{"type":75,"tag":171,"props":444,"children":445},{"style":190},[446],{"type":81,"value":385},{"type":75,"tag":171,"props":448,"children":449},{"style":196},[450],{"type":81,"value":390},{"type":75,"tag":171,"props":452,"children":453},{"style":184},[454],{"type":81,"value":204},{"type":75,"tag":171,"props":456,"children":457},{"style":190},[458],{"type":81,"value":459},"\u002Fnvcf-compute-plane-stack:",{"type":75,"tag":171,"props":461,"children":462},{"style":184},[463],{"type":81,"value":404},{"type":75,"tag":171,"props":465,"children":466},{"style":190},[467],{"type":81,"value":409},{"type":75,"tag":171,"props":469,"children":470},{"style":196},[471],{"type":81,"value":414},{"type":75,"tag":171,"props":473,"children":474},{"style":184},[475],{"type":81,"value":419},{"type":75,"tag":84,"props":477,"children":478},{},[479,481,487],{"type":81,"value":480},"Control-plane commands below assume the control-plane bundle root. Split-flow\ncommands and root substitution are in\n",{"type":75,"tag":127,"props":482,"children":484},{"href":483},"references\u002Fcompute-plane-installation.md",[485],{"type":81,"value":486},"Split Compute-Plane Installation",{"type":81,"value":134},{"type":75,"tag":90,"props":489,"children":491},{"id":490},"before-you-start",[492],{"type":81,"value":493},"Before You Start",{"type":75,"tag":84,"props":495,"children":496},{},[497],{"type":81,"value":498},"Verify tooling and context before any operation:",{"type":75,"tag":161,"props":500,"children":502},{"className":163,"code":501,"language":65,"meta":165,"style":165},"helmfile --version   # Must be 1.1.x (1.2.0 removed sequential mode)\nhelm version         # Must be >= 3.12\nhelm plugin list     # Must include helm-diff >= 3.11\nkubectl version      # Client must be within 1 minor version of cluster\n",[503],{"type":75,"tag":102,"props":504,"children":505},{"__ignoreMap":165},[506,524,541,564],{"type":75,"tag":171,"props":507,"children":508},{"class":173,"line":174},[509,513,518],{"type":75,"tag":171,"props":510,"children":511},{"style":178},[512],{"type":81,"value":47},{"type":75,"tag":171,"props":514,"children":515},{"style":190},[516],{"type":81,"value":517}," --version",{"type":75,"tag":171,"props":519,"children":521},{"style":520},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[522],{"type":81,"value":523},"   # Must be 1.1.x (1.2.0 removed sequential mode)\n",{"type":75,"tag":171,"props":525,"children":526},{"class":173,"line":275},[527,531,536],{"type":75,"tag":171,"props":528,"children":529},{"style":178},[530],{"type":81,"value":68},{"type":75,"tag":171,"props":532,"children":533},{"style":190},[534],{"type":81,"value":535}," version",{"type":75,"tag":171,"props":537,"children":538},{"style":520},[539],{"type":81,"value":540},"         # Must be >= 3.12\n",{"type":75,"tag":171,"props":542,"children":544},{"class":173,"line":543},3,[545,549,554,559],{"type":75,"tag":171,"props":546,"children":547},{"style":178},[548],{"type":81,"value":68},{"type":75,"tag":171,"props":550,"children":551},{"style":190},[552],{"type":81,"value":553}," plugin",{"type":75,"tag":171,"props":555,"children":556},{"style":190},[557],{"type":81,"value":558}," list",{"type":75,"tag":171,"props":560,"children":561},{"style":520},[562],{"type":81,"value":563},"     # Must include helm-diff >= 3.11\n",{"type":75,"tag":171,"props":565,"children":567},{"class":173,"line":566},4,[568,572,576],{"type":75,"tag":171,"props":569,"children":570},{"style":178},[571],{"type":81,"value":69},{"type":75,"tag":171,"props":573,"children":574},{"style":190},[575],{"type":81,"value":535},{"type":75,"tag":171,"props":577,"children":578},{"style":520},[579],{"type":81,"value":580},"      # Client must be within 1 minor version of cluster\n",{"type":75,"tag":84,"props":582,"children":583},{},[584,586,592],{"type":81,"value":585},"Control-plane commands run from the extracted ",{"type":75,"tag":102,"props":587,"children":589},{"className":588},[],[590],{"type":81,"value":591},"nvcf-self-managed-stack\u002F",{"type":81,"value":593}," root:",{"type":75,"tag":161,"props":595,"children":597},{"className":163,"code":596,"language":65,"meta":165,"style":165},"cd path\u002Fto\u002Fnvcf-self-managed-stack\nls helmfile.d\u002F environments\u002F secrets\u002F global.yaml.gotmpl\n",[598],{"type":75,"tag":102,"props":599,"children":600},{"__ignoreMap":165},[601,615],{"type":75,"tag":171,"props":602,"children":603},{"class":173,"line":174},[604,610],{"type":75,"tag":171,"props":605,"children":607},{"style":606},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[608],{"type":81,"value":609},"cd",{"type":75,"tag":171,"props":611,"children":612},{"style":190},[613],{"type":81,"value":614}," path\u002Fto\u002Fnvcf-self-managed-stack\n",{"type":75,"tag":171,"props":616,"children":617},{"class":173,"line":275},[618,622,627,632,637],{"type":75,"tag":171,"props":619,"children":620},{"style":178},[621],{"type":81,"value":181},{"type":75,"tag":171,"props":623,"children":624},{"style":190},[625],{"type":81,"value":626}," helmfile.d\u002F",{"type":75,"tag":171,"props":628,"children":629},{"style":190},[630],{"type":81,"value":631}," environments\u002F",{"type":75,"tag":171,"props":633,"children":634},{"style":190},[635],{"type":81,"value":636}," secrets\u002F",{"type":75,"tag":171,"props":638,"children":639},{"style":190},[640],{"type":81,"value":641}," global.yaml.gotmpl\n",{"type":75,"tag":84,"props":643,"children":644},{},[645,647,653,655,661,663,671],{"type":81,"value":646},"Identify the environment from ",{"type":75,"tag":102,"props":648,"children":650},{"className":649},[],[651],{"type":81,"value":652},"environments\u002F\u003Cname>.yaml",{"type":81,"value":654}," and\n",{"type":75,"tag":102,"props":656,"children":658},{"className":657},[],[659],{"type":81,"value":660},"secrets\u002F\u003Cname>-secrets.yaml",{"type":81,"value":662},"; for EKS, use the canonical\n",{"type":75,"tag":127,"props":664,"children":668},{"href":665,"rel":666},"https:\u002F\u002Fdocs.nvidia.com\u002Fnvcf\u002Fv0.6.0-rc\u002Fcsp-end-to-end-example",[667],"nofollow",[669],{"type":81,"value":670},"CSP End-to-End Example",{"type":81,"value":134},{"type":75,"tag":90,"props":673,"children":675},{"id":674},"how-values-flow",[676],{"type":81,"value":677},"How Values Flow",{"type":75,"tag":84,"props":679,"children":680},{},[681],{"type":81,"value":682},"Understanding value precedence prevents the most common configuration mistakes.",{"type":75,"tag":161,"props":684,"children":688},{"className":685,"code":687,"language":81},[686],"language-text","environments\u002Fbase.yaml          (defaults)\n    -> merged with\nenvironments\u002F\u003Cenv>.yaml         (your overrides)\n    -> consumed by\nglobal.yaml.gotmpl              (Go template, constructs per-chart values)\n    -> consumed by\nsecrets\u002F\u003Cenv>-secrets.yaml      (sensitive values)\n    -> overridden by\nrelease inline values: blocks   (highest precedence)\n",[689],{"type":75,"tag":102,"props":690,"children":691},{"__ignoreMap":165},[692],{"type":81,"value":687},{"type":75,"tag":84,"props":694,"children":695},{},[696,698,704,706,711],{"type":81,"value":697},"Critical: ",{"type":75,"tag":102,"props":699,"children":701},{"className":700},[],[702],{"type":81,"value":703},"global.yaml.gotmpl",{"type":81,"value":705}," only passes through specific keys to each chart (image registries, node selectors, storage, replica counts). Setting an arbitrary chart value in the environment file will be silently ignored if ",{"type":75,"tag":102,"props":707,"children":709},{"className":708},[],[710],{"type":81,"value":703},{"type":81,"value":712}," does not propagate it.",{"type":75,"tag":84,"props":714,"children":715},{},[716,718,724,726,732],{"type":81,"value":717},"To override arbitrary chart values, use a helmfile release ",{"type":75,"tag":102,"props":719,"children":721},{"className":720},[],[722],{"type":81,"value":723},"values:",{"type":81,"value":725}," block. See ",{"type":75,"tag":127,"props":727,"children":729},{"href":728},"#overriding-helm-chart-values",[730],{"type":81,"value":731},"Overriding Helm Chart Values",{"type":81,"value":134},{"type":75,"tag":90,"props":734,"children":736},{"id":735},"clean-installation",[737],{"type":81,"value":738},"Clean Installation",{"type":75,"tag":740,"props":741,"children":743},"h3",{"id":742},"_1-create-namespaces-and-image-pull-secrets-if-using-a-private-registry",[744],{"type":81,"value":745},"1. Create namespaces and image pull secrets (if using a private registry)",{"type":75,"tag":161,"props":747,"children":749},{"className":163,"code":748,"language":65,"meta":165,"style":165},"for ns in cassandra-system nats-system nvcf api-keys ess sis vault-system cert-manager; do\n  kubectl create namespace \"$ns\" --dry-run=client -o yaml | kubectl apply -f -\ndone\n\n# Only if pulling from a private registry (e.g., NGC nvcr.io)\nexport NGC_API_KEY=\"\u003Cyour-key>\"\nfor ns in cassandra-system nats-system nvcf api-keys ess sis vault-system cert-manager; do\n  kubectl create secret docker-registry nvcr-creds \\\n    --docker-server=nvcr.io \\\n    --docker-username='$oauthtoken' \\\n    --docker-password=\"$NGC_API_KEY\" \\\n    --namespace=\"$ns\" \\\n    --dry-run=client -o yaml | kubectl apply -f -\ndone\n",[750],{"type":75,"tag":102,"props":751,"children":752},{"__ignoreMap":165},[753,822,895,903,912,921,955,1011,1043,1056,1083,1109,1134,1171],{"type":75,"tag":171,"props":754,"children":755},{"class":173,"line":174},[756,762,767,772,777,782,787,792,797,802,807,812,817],{"type":75,"tag":171,"props":757,"children":759},{"style":758},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[760],{"type":81,"value":761},"for",{"type":75,"tag":171,"props":763,"children":764},{"style":196},[765],{"type":81,"value":766}," ns ",{"type":75,"tag":171,"props":768,"children":769},{"style":758},[770],{"type":81,"value":771},"in",{"type":75,"tag":171,"props":773,"children":774},{"style":190},[775],{"type":81,"value":776}," cassandra-system",{"type":75,"tag":171,"props":778,"children":779},{"style":190},[780],{"type":81,"value":781}," nats-system",{"type":75,"tag":171,"props":783,"children":784},{"style":190},[785],{"type":81,"value":786}," nvcf",{"type":75,"tag":171,"props":788,"children":789},{"style":190},[790],{"type":81,"value":791}," api-keys",{"type":75,"tag":171,"props":793,"children":794},{"style":190},[795],{"type":81,"value":796}," ess",{"type":75,"tag":171,"props":798,"children":799},{"style":190},[800],{"type":81,"value":801}," sis",{"type":75,"tag":171,"props":803,"children":804},{"style":190},[805],{"type":81,"value":806}," vault-system",{"type":75,"tag":171,"props":808,"children":809},{"style":190},[810],{"type":81,"value":811}," cert-manager",{"type":75,"tag":171,"props":813,"children":814},{"style":184},[815],{"type":81,"value":816},";",{"type":75,"tag":171,"props":818,"children":819},{"style":758},[820],{"type":81,"value":821}," do\n",{"type":75,"tag":171,"props":823,"children":824},{"class":173,"line":275},[825,830,835,840,845,850,855,860,865,870,875,880,885,890],{"type":75,"tag":171,"props":826,"children":827},{"style":178},[828],{"type":81,"value":829},"  kubectl",{"type":75,"tag":171,"props":831,"children":832},{"style":190},[833],{"type":81,"value":834}," create",{"type":75,"tag":171,"props":836,"children":837},{"style":190},[838],{"type":81,"value":839}," namespace",{"type":75,"tag":171,"props":841,"children":842},{"style":184},[843],{"type":81,"value":844}," \"",{"type":75,"tag":171,"props":846,"children":847},{"style":196},[848],{"type":81,"value":849},"$ns",{"type":75,"tag":171,"props":851,"children":852},{"style":184},[853],{"type":81,"value":854},"\"",{"type":75,"tag":171,"props":856,"children":857},{"style":190},[858],{"type":81,"value":859}," --dry-run=client",{"type":75,"tag":171,"props":861,"children":862},{"style":190},[863],{"type":81,"value":864}," -o",{"type":75,"tag":171,"props":866,"children":867},{"style":190},[868],{"type":81,"value":869}," yaml",{"type":75,"tag":171,"props":871,"children":872},{"style":184},[873],{"type":81,"value":874}," |",{"type":75,"tag":171,"props":876,"children":877},{"style":178},[878],{"type":81,"value":879}," kubectl",{"type":75,"tag":171,"props":881,"children":882},{"style":190},[883],{"type":81,"value":884}," apply",{"type":75,"tag":171,"props":886,"children":887},{"style":190},[888],{"type":81,"value":889}," -f",{"type":75,"tag":171,"props":891,"children":892},{"style":190},[893],{"type":81,"value":894}," -\n",{"type":75,"tag":171,"props":896,"children":897},{"class":173,"line":543},[898],{"type":75,"tag":171,"props":899,"children":900},{"style":758},[901],{"type":81,"value":902},"done\n",{"type":75,"tag":171,"props":904,"children":905},{"class":173,"line":566},[906],{"type":75,"tag":171,"props":907,"children":909},{"emptyLinePlaceholder":908},true,[910],{"type":81,"value":911},"\n",{"type":75,"tag":171,"props":913,"children":915},{"class":173,"line":914},5,[916],{"type":75,"tag":171,"props":917,"children":918},{"style":520},[919],{"type":81,"value":920},"# Only if pulling from a private registry (e.g., NGC nvcr.io)\n",{"type":75,"tag":171,"props":922,"children":924},{"class":173,"line":923},6,[925,931,936,941,945,950],{"type":75,"tag":171,"props":926,"children":928},{"style":927},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[929],{"type":81,"value":930},"export",{"type":75,"tag":171,"props":932,"children":933},{"style":196},[934],{"type":81,"value":935}," NGC_API_KEY",{"type":75,"tag":171,"props":937,"children":938},{"style":184},[939],{"type":81,"value":940},"=",{"type":75,"tag":171,"props":942,"children":943},{"style":184},[944],{"type":81,"value":854},{"type":75,"tag":171,"props":946,"children":947},{"style":190},[948],{"type":81,"value":949},"\u003Cyour-key>",{"type":75,"tag":171,"props":951,"children":952},{"style":184},[953],{"type":81,"value":954},"\"\n",{"type":75,"tag":171,"props":956,"children":958},{"class":173,"line":957},7,[959,963,967,971,975,979,983,987,991,995,999,1003,1007],{"type":75,"tag":171,"props":960,"children":961},{"style":758},[962],{"type":81,"value":761},{"type":75,"tag":171,"props":964,"children":965},{"style":196},[966],{"type":81,"value":766},{"type":75,"tag":171,"props":968,"children":969},{"style":758},[970],{"type":81,"value":771},{"type":75,"tag":171,"props":972,"children":973},{"style":190},[974],{"type":81,"value":776},{"type":75,"tag":171,"props":976,"children":977},{"style":190},[978],{"type":81,"value":781},{"type":75,"tag":171,"props":980,"children":981},{"style":190},[982],{"type":81,"value":786},{"type":75,"tag":171,"props":984,"children":985},{"style":190},[986],{"type":81,"value":791},{"type":75,"tag":171,"props":988,"children":989},{"style":190},[990],{"type":81,"value":796},{"type":75,"tag":171,"props":992,"children":993},{"style":190},[994],{"type":81,"value":801},{"type":75,"tag":171,"props":996,"children":997},{"style":190},[998],{"type":81,"value":806},{"type":75,"tag":171,"props":1000,"children":1001},{"style":190},[1002],{"type":81,"value":811},{"type":75,"tag":171,"props":1004,"children":1005},{"style":184},[1006],{"type":81,"value":816},{"type":75,"tag":171,"props":1008,"children":1009},{"style":758},[1010],{"type":81,"value":821},{"type":75,"tag":171,"props":1012,"children":1014},{"class":173,"line":1013},8,[1015,1019,1023,1028,1033,1038],{"type":75,"tag":171,"props":1016,"children":1017},{"style":178},[1018],{"type":81,"value":829},{"type":75,"tag":171,"props":1020,"children":1021},{"style":190},[1022],{"type":81,"value":834},{"type":75,"tag":171,"props":1024,"children":1025},{"style":190},[1026],{"type":81,"value":1027}," secret",{"type":75,"tag":171,"props":1029,"children":1030},{"style":190},[1031],{"type":81,"value":1032}," docker-registry",{"type":75,"tag":171,"props":1034,"children":1035},{"style":190},[1036],{"type":81,"value":1037}," nvcr-creds",{"type":75,"tag":171,"props":1039,"children":1040},{"style":196},[1041],{"type":81,"value":1042}," \\\n",{"type":75,"tag":171,"props":1044,"children":1046},{"class":173,"line":1045},9,[1047,1052],{"type":75,"tag":171,"props":1048,"children":1049},{"style":190},[1050],{"type":81,"value":1051},"    --docker-server=nvcr.io",{"type":75,"tag":171,"props":1053,"children":1054},{"style":196},[1055],{"type":81,"value":1042},{"type":75,"tag":171,"props":1057,"children":1059},{"class":173,"line":1058},10,[1060,1065,1070,1075,1079],{"type":75,"tag":171,"props":1061,"children":1062},{"style":190},[1063],{"type":81,"value":1064},"    --docker-username=",{"type":75,"tag":171,"props":1066,"children":1067},{"style":184},[1068],{"type":81,"value":1069},"'",{"type":75,"tag":171,"props":1071,"children":1072},{"style":190},[1073],{"type":81,"value":1074},"$oauthtoken",{"type":75,"tag":171,"props":1076,"children":1077},{"style":184},[1078],{"type":81,"value":1069},{"type":75,"tag":171,"props":1080,"children":1081},{"style":196},[1082],{"type":81,"value":1042},{"type":75,"tag":171,"props":1084,"children":1086},{"class":173,"line":1085},11,[1087,1092,1096,1101,1105],{"type":75,"tag":171,"props":1088,"children":1089},{"style":190},[1090],{"type":81,"value":1091},"    --docker-password=",{"type":75,"tag":171,"props":1093,"children":1094},{"style":184},[1095],{"type":81,"value":854},{"type":75,"tag":171,"props":1097,"children":1098},{"style":196},[1099],{"type":81,"value":1100},"$NGC_API_KEY",{"type":75,"tag":171,"props":1102,"children":1103},{"style":184},[1104],{"type":81,"value":854},{"type":75,"tag":171,"props":1106,"children":1107},{"style":196},[1108],{"type":81,"value":1042},{"type":75,"tag":171,"props":1110,"children":1112},{"class":173,"line":1111},12,[1113,1118,1122,1126,1130],{"type":75,"tag":171,"props":1114,"children":1115},{"style":190},[1116],{"type":81,"value":1117},"    --namespace=",{"type":75,"tag":171,"props":1119,"children":1120},{"style":184},[1121],{"type":81,"value":854},{"type":75,"tag":171,"props":1123,"children":1124},{"style":196},[1125],{"type":81,"value":849},{"type":75,"tag":171,"props":1127,"children":1128},{"style":184},[1129],{"type":81,"value":854},{"type":75,"tag":171,"props":1131,"children":1132},{"style":196},[1133],{"type":81,"value":1042},{"type":75,"tag":171,"props":1135,"children":1137},{"class":173,"line":1136},13,[1138,1143,1147,1151,1155,1159,1163,1167],{"type":75,"tag":171,"props":1139,"children":1140},{"style":190},[1141],{"type":81,"value":1142},"    --dry-run=client",{"type":75,"tag":171,"props":1144,"children":1145},{"style":190},[1146],{"type":81,"value":864},{"type":75,"tag":171,"props":1148,"children":1149},{"style":190},[1150],{"type":81,"value":869},{"type":75,"tag":171,"props":1152,"children":1153},{"style":184},[1154],{"type":81,"value":874},{"type":75,"tag":171,"props":1156,"children":1157},{"style":178},[1158],{"type":81,"value":879},{"type":75,"tag":171,"props":1160,"children":1161},{"style":190},[1162],{"type":81,"value":884},{"type":75,"tag":171,"props":1164,"children":1165},{"style":190},[1166],{"type":81,"value":889},{"type":75,"tag":171,"props":1168,"children":1169},{"style":190},[1170],{"type":81,"value":894},{"type":75,"tag":171,"props":1172,"children":1174},{"class":173,"line":1173},14,[1175],{"type":75,"tag":171,"props":1176,"children":1177},{"style":758},[1178],{"type":81,"value":902},{"type":75,"tag":84,"props":1180,"children":1181},{},[1182,1184,1190],{"type":81,"value":1183},"If using pull secrets, you must also configure each helmfile release to reference them. See ",{"type":75,"tag":127,"props":1185,"children":1187},{"href":1186},"#image-pull-secrets",[1188],{"type":81,"value":1189},"Image Pull Secrets",{"type":81,"value":134},{"type":75,"tag":740,"props":1192,"children":1194},{"id":1193},"_2-authenticate-helm-to-your-chart-registry",[1195],{"type":81,"value":1196},"2. Authenticate helm to your chart registry",{"type":75,"tag":84,"props":1198,"children":1199},{},[1200,1202,1208,1210,1216],{"type":81,"value":1201},"Both ",{"type":75,"tag":102,"props":1203,"children":1205},{"className":1204},[],[1206],{"type":81,"value":1207},"docker login",{"type":81,"value":1209}," AND ",{"type":75,"tag":102,"props":1211,"children":1213},{"className":1212},[],[1214],{"type":81,"value":1215},"helm registry login",{"type":81,"value":1217}," are required for NGC. Helmfile uses helm (not docker) to pull OCI charts, so the helm credential store must be authenticated separately.",{"type":75,"tag":161,"props":1219,"children":1221},{"className":163,"code":1220,"language":65,"meta":165,"style":165},"# NGC -- both commands are required\ndocker login nvcr.io -u '$oauthtoken' -p \"$NGC_API_KEY\"\nhelm registry login nvcr.io -u '$oauthtoken' -p \"$NGC_API_KEY\"\n\n# ECR\naws ecr get-login-password --region \u003Cregion> | \\\n  helm registry login --username AWS --password-stdin \u003Caccount>.dkr.ecr.\u003Cregion>.amazonaws.com\n",[1222],{"type":75,"tag":102,"props":1223,"children":1224},{"__ignoreMap":165},[1225,1233,1286,1337,1344,1352,1400],{"type":75,"tag":171,"props":1226,"children":1227},{"class":173,"line":174},[1228],{"type":75,"tag":171,"props":1229,"children":1230},{"style":520},[1231],{"type":81,"value":1232},"# NGC -- both commands are required\n",{"type":75,"tag":171,"props":1234,"children":1235},{"class":173,"line":275},[1236,1241,1246,1251,1256,1261,1265,1269,1274,1278,1282],{"type":75,"tag":171,"props":1237,"children":1238},{"style":178},[1239],{"type":81,"value":1240},"docker",{"type":75,"tag":171,"props":1242,"children":1243},{"style":190},[1244],{"type":81,"value":1245}," login",{"type":75,"tag":171,"props":1247,"children":1248},{"style":190},[1249],{"type":81,"value":1250}," nvcr.io",{"type":75,"tag":171,"props":1252,"children":1253},{"style":190},[1254],{"type":81,"value":1255}," -u",{"type":75,"tag":171,"props":1257,"children":1258},{"style":184},[1259],{"type":81,"value":1260}," '",{"type":75,"tag":171,"props":1262,"children":1263},{"style":190},[1264],{"type":81,"value":1074},{"type":75,"tag":171,"props":1266,"children":1267},{"style":184},[1268],{"type":81,"value":1069},{"type":75,"tag":171,"props":1270,"children":1271},{"style":190},[1272],{"type":81,"value":1273}," -p",{"type":75,"tag":171,"props":1275,"children":1276},{"style":184},[1277],{"type":81,"value":844},{"type":75,"tag":171,"props":1279,"children":1280},{"style":196},[1281],{"type":81,"value":1100},{"type":75,"tag":171,"props":1283,"children":1284},{"style":184},[1285],{"type":81,"value":954},{"type":75,"tag":171,"props":1287,"children":1288},{"class":173,"line":543},[1289,1293,1297,1301,1305,1309,1313,1317,1321,1325,1329,1333],{"type":75,"tag":171,"props":1290,"children":1291},{"style":178},[1292],{"type":81,"value":68},{"type":75,"tag":171,"props":1294,"children":1295},{"style":190},[1296],{"type":81,"value":366},{"type":75,"tag":171,"props":1298,"children":1299},{"style":190},[1300],{"type":81,"value":1245},{"type":75,"tag":171,"props":1302,"children":1303},{"style":190},[1304],{"type":81,"value":1250},{"type":75,"tag":171,"props":1306,"children":1307},{"style":190},[1308],{"type":81,"value":1255},{"type":75,"tag":171,"props":1310,"children":1311},{"style":184},[1312],{"type":81,"value":1260},{"type":75,"tag":171,"props":1314,"children":1315},{"style":190},[1316],{"type":81,"value":1074},{"type":75,"tag":171,"props":1318,"children":1319},{"style":184},[1320],{"type":81,"value":1069},{"type":75,"tag":171,"props":1322,"children":1323},{"style":190},[1324],{"type":81,"value":1273},{"type":75,"tag":171,"props":1326,"children":1327},{"style":184},[1328],{"type":81,"value":844},{"type":75,"tag":171,"props":1330,"children":1331},{"style":196},[1332],{"type":81,"value":1100},{"type":75,"tag":171,"props":1334,"children":1335},{"style":184},[1336],{"type":81,"value":954},{"type":75,"tag":171,"props":1338,"children":1339},{"class":173,"line":566},[1340],{"type":75,"tag":171,"props":1341,"children":1342},{"emptyLinePlaceholder":908},[1343],{"type":81,"value":911},{"type":75,"tag":171,"props":1345,"children":1346},{"class":173,"line":914},[1347],{"type":75,"tag":171,"props":1348,"children":1349},{"style":520},[1350],{"type":81,"value":1351},"# ECR\n",{"type":75,"tag":171,"props":1353,"children":1354},{"class":173,"line":923},[1355,1360,1365,1370,1375,1379,1384,1388,1392,1396],{"type":75,"tag":171,"props":1356,"children":1357},{"style":178},[1358],{"type":81,"value":1359},"aws",{"type":75,"tag":171,"props":1361,"children":1362},{"style":190},[1363],{"type":81,"value":1364}," ecr",{"type":75,"tag":171,"props":1366,"children":1367},{"style":190},[1368],{"type":81,"value":1369}," get-login-password",{"type":75,"tag":171,"props":1371,"children":1372},{"style":190},[1373],{"type":81,"value":1374}," --region",{"type":75,"tag":171,"props":1376,"children":1377},{"style":184},[1378],{"type":81,"value":187},{"type":75,"tag":171,"props":1380,"children":1381},{"style":190},[1382],{"type":81,"value":1383},"regio",{"type":75,"tag":171,"props":1385,"children":1386},{"style":196},[1387],{"type":81,"value":414},{"type":75,"tag":171,"props":1389,"children":1390},{"style":184},[1391],{"type":81,"value":204},{"type":75,"tag":171,"props":1393,"children":1394},{"style":184},[1395],{"type":81,"value":874},{"type":75,"tag":171,"props":1397,"children":1398},{"style":196},[1399],{"type":81,"value":1042},{"type":75,"tag":171,"props":1401,"children":1402},{"class":173,"line":957},[1403,1408,1412,1416,1421,1426,1431,1435,1440,1445,1449,1454,1458,1462,1466,1470],{"type":75,"tag":171,"props":1404,"children":1405},{"style":178},[1406],{"type":81,"value":1407},"  helm",{"type":75,"tag":171,"props":1409,"children":1410},{"style":190},[1411],{"type":81,"value":366},{"type":75,"tag":171,"props":1413,"children":1414},{"style":190},[1415],{"type":81,"value":1245},{"type":75,"tag":171,"props":1417,"children":1418},{"style":190},[1419],{"type":81,"value":1420}," --username",{"type":75,"tag":171,"props":1422,"children":1423},{"style":190},[1424],{"type":81,"value":1425}," AWS",{"type":75,"tag":171,"props":1427,"children":1428},{"style":190},[1429],{"type":81,"value":1430}," --password-stdin",{"type":75,"tag":171,"props":1432,"children":1433},{"style":184},[1434],{"type":81,"value":187},{"type":75,"tag":171,"props":1436,"children":1437},{"style":190},[1438],{"type":81,"value":1439},"accoun",{"type":75,"tag":171,"props":1441,"children":1442},{"style":196},[1443],{"type":81,"value":1444},"t",{"type":75,"tag":171,"props":1446,"children":1447},{"style":184},[1448],{"type":81,"value":204},{"type":75,"tag":171,"props":1450,"children":1451},{"style":190},[1452],{"type":81,"value":1453},".dkr.ecr.",{"type":75,"tag":171,"props":1455,"children":1456},{"style":184},[1457],{"type":81,"value":404},{"type":75,"tag":171,"props":1459,"children":1460},{"style":190},[1461],{"type":81,"value":1383},{"type":75,"tag":171,"props":1463,"children":1464},{"style":196},[1465],{"type":81,"value":414},{"type":75,"tag":171,"props":1467,"children":1468},{"style":184},[1469],{"type":81,"value":204},{"type":75,"tag":171,"props":1471,"children":1472},{"style":190},[1473],{"type":81,"value":1474},".amazonaws.com\n",{"type":75,"tag":1476,"props":1477,"children":1478},"blockquote",{},[1479],{"type":75,"tag":84,"props":1480,"children":1481},{},[1482,1484,1489,1491,1497,1499,1505,1507,1512,1514,1520,1522,1528,1530,1536],{"type":81,"value":1483},"Note: ",{"type":75,"tag":102,"props":1485,"children":1487},{"className":1486},[],[1488],{"type":81,"value":1215},{"type":81,"value":1490}," keeps only the last login per host (multi-org gotcha).\nThe stack and the caches span several NGC orgs that need different keys, but they all\nlive under the one host ",{"type":75,"tag":102,"props":1492,"children":1494},{"className":1493},[],[1495],{"type":81,"value":1496},"nvcr.io",{"type":81,"value":1498},". A second ",{"type":75,"tag":102,"props":1500,"children":1502},{"className":1501},[],[1503],{"type":81,"value":1504},"helm registry login nvcr.io",{"type":81,"value":1506}," (or\n",{"type":75,"tag":102,"props":1508,"children":1510},{"className":1509},[],[1511],{"type":81,"value":1207},{"type":81,"value":1513},") with a different key silently overwrites the first, so pulls for the\nfirst org then fail with a misleading ",{"type":75,"tag":102,"props":1515,"children":1517},{"className":1516},[],[1518],{"type":81,"value":1519},"403 Access Denied",{"type":81,"value":1521}," (not a 401: the auth\nsucceeded, it's the wrong-org key). If you pull charts\u002Fimages from more than one org,\neither re-run the login with the correct key immediately before each pull, or pass\n",{"type":75,"tag":102,"props":1523,"children":1525},{"className":1524},[],[1526],{"type":81,"value":1527},"--username '$oauthtoken' --password \u003Ckey>",{"type":81,"value":1529}," per ",{"type":75,"tag":102,"props":1531,"children":1533},{"className":1532},[],[1534],{"type":81,"value":1535},"helm pull",{"type":81,"value":1537}," invocation instead of relying\non the stored login. (Pulling everything from a single registry mirror avoids this\nentirely.)",{"type":75,"tag":740,"props":1539,"children":1541},{"id":1540},"_2b-recommended-pull-from-a-registry-the-cluster-authenticates-to-automatically",[1542],{"type":81,"value":1543},"2b. (Recommended) Pull from a registry the cluster authenticates to automatically",{"type":75,"tag":84,"props":1545,"children":1546},{},[1547],{"type":81,"value":1548},"If you mirror the NVCF images into a registry that the cluster can pull from without an\nexplicit image pull secret (for example, a registry where the node or workload identity is\ngranted pull access automatically), point the stack at it and skip the credential steps\nabove.",{"type":75,"tag":1550,"props":1551,"children":1552},"ol",{},[1553,1785],{"type":75,"tag":1554,"props":1555,"children":1556},"li",{},[1557,1559,1565,1567,1701],{"type":81,"value":1558},"Point the stack at the mirror in ",{"type":75,"tag":102,"props":1560,"children":1562},{"className":1561},[],[1563],{"type":81,"value":1564},"environments\u002F\u003Cenv>.yaml",{"type":81,"value":1566},":",{"type":75,"tag":161,"props":1568,"children":1571},{"className":1569,"code":1570,"language":66,"meta":165,"style":165},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","global:\n  image:\n    registry: \u003Cyour-registry>            # e.g. registry.example.com\n    repository: \u003Cmirror-sub-path>\n  helm:\n    sources:                              # only if you also mirrored the helm charts\n      registry: \u003Cyour-registry>\n      repository: \u003Cmirror-sub-path>\n",[1572],{"type":75,"tag":102,"props":1573,"children":1574},{"__ignoreMap":165},[1575,1589,1601,1623,1640,1651,1668,1685],{"type":75,"tag":171,"props":1576,"children":1577},{"class":173,"line":174},[1578,1584],{"type":75,"tag":171,"props":1579,"children":1581},{"style":1580},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[1582],{"type":81,"value":1583},"global",{"type":75,"tag":171,"props":1585,"children":1586},{"style":184},[1587],{"type":81,"value":1588},":\n",{"type":75,"tag":171,"props":1590,"children":1591},{"class":173,"line":275},[1592,1597],{"type":75,"tag":171,"props":1593,"children":1594},{"style":1580},[1595],{"type":81,"value":1596},"  image",{"type":75,"tag":171,"props":1598,"children":1599},{"style":184},[1600],{"type":81,"value":1588},{"type":75,"tag":171,"props":1602,"children":1603},{"class":173,"line":543},[1604,1609,1613,1618],{"type":75,"tag":171,"props":1605,"children":1606},{"style":1580},[1607],{"type":81,"value":1608},"    registry",{"type":75,"tag":171,"props":1610,"children":1611},{"style":184},[1612],{"type":81,"value":1566},{"type":75,"tag":171,"props":1614,"children":1615},{"style":190},[1616],{"type":81,"value":1617}," \u003Cyour-registry>",{"type":75,"tag":171,"props":1619,"children":1620},{"style":520},[1621],{"type":81,"value":1622},"            # e.g. registry.example.com\n",{"type":75,"tag":171,"props":1624,"children":1625},{"class":173,"line":566},[1626,1631,1635],{"type":75,"tag":171,"props":1627,"children":1628},{"style":1580},[1629],{"type":81,"value":1630},"    repository",{"type":75,"tag":171,"props":1632,"children":1633},{"style":184},[1634],{"type":81,"value":1566},{"type":75,"tag":171,"props":1636,"children":1637},{"style":190},[1638],{"type":81,"value":1639}," \u003Cmirror-sub-path>\n",{"type":75,"tag":171,"props":1641,"children":1642},{"class":173,"line":914},[1643,1647],{"type":75,"tag":171,"props":1644,"children":1645},{"style":1580},[1646],{"type":81,"value":1407},{"type":75,"tag":171,"props":1648,"children":1649},{"style":184},[1650],{"type":81,"value":1588},{"type":75,"tag":171,"props":1652,"children":1653},{"class":173,"line":923},[1654,1659,1663],{"type":75,"tag":171,"props":1655,"children":1656},{"style":1580},[1657],{"type":81,"value":1658},"    sources",{"type":75,"tag":171,"props":1660,"children":1661},{"style":184},[1662],{"type":81,"value":1566},{"type":75,"tag":171,"props":1664,"children":1665},{"style":520},[1666],{"type":81,"value":1667},"                              # only if you also mirrored the helm charts\n",{"type":75,"tag":171,"props":1669,"children":1670},{"class":173,"line":957},[1671,1676,1680],{"type":75,"tag":171,"props":1672,"children":1673},{"style":1580},[1674],{"type":81,"value":1675},"      registry",{"type":75,"tag":171,"props":1677,"children":1678},{"style":184},[1679],{"type":81,"value":1566},{"type":75,"tag":171,"props":1681,"children":1682},{"style":190},[1683],{"type":81,"value":1684}," \u003Cyour-registry>\n",{"type":75,"tag":171,"props":1686,"children":1687},{"class":173,"line":1013},[1688,1693,1697],{"type":75,"tag":171,"props":1689,"children":1690},{"style":1580},[1691],{"type":81,"value":1692},"      repository",{"type":75,"tag":171,"props":1694,"children":1695},{"style":184},[1696],{"type":81,"value":1566},{"type":75,"tag":171,"props":1698,"children":1699},{"style":190},[1700],{"type":81,"value":1639},{"type":75,"tag":1476,"props":1702,"children":1703},{},[1704],{"type":75,"tag":84,"props":1705,"children":1706},{},[1707,1709,1715,1717,1723,1725,1731,1733,1739,1741,1746,1748,1754,1756,1761,1763,1769,1771,1777,1779,1784],{"type":81,"value":1708},"The manifest has a mix of OCI and non-OCI artifacts, so convert non-OCI charts before\nmirroring. Container images (",{"type":75,"tag":102,"props":1710,"children":1712},{"className":1711},[],[1713],{"type":81,"value":1714},"nvcr.io\u002F...",{"type":81,"value":1716},") are already OCI and mirror as-is. Some Helm\ncharts, though, are published to an ",{"type":75,"tag":102,"props":1718,"children":1720},{"className":1719},[],[1721],{"type":81,"value":1722},"https:\u002F\u002F",{"type":81,"value":1724}," Helm repository (for example\n",{"type":75,"tag":102,"props":1726,"children":1728},{"className":1727},[],[1729],{"type":81,"value":1730},"helm.ngc.nvidia.com",{"type":81,"value":1732},"). Those are not OCI and cannot be resolved by ",{"type":75,"tag":102,"props":1734,"children":1736},{"className":1735},[],[1737],{"type":81,"value":1738},"global.helm.sources",{"type":81,"value":1740},"\n(OCI) or mirrored directly. Before pointing the stack at your mirror, convert each\n",{"type":75,"tag":102,"props":1742,"children":1744},{"className":1743},[],[1745],{"type":81,"value":1722},{"type":81,"value":1747}," chart to OCI: ",{"type":75,"tag":102,"props":1749,"children":1751},{"className":1750},[],[1752],{"type":81,"value":1753},"helm repo add",{"type":81,"value":1755},"\u002F",{"type":75,"tag":102,"props":1757,"children":1759},{"className":1758},[],[1760],{"type":81,"value":1535},{"type":81,"value":1762}," the ",{"type":75,"tag":102,"props":1764,"children":1766},{"className":1765},[],[1767],{"type":81,"value":1768},".tgz",{"type":81,"value":1770},", then\n",{"type":75,"tag":102,"props":1772,"children":1774},{"className":1773},[],[1775],{"type":81,"value":1776},"helm push \u003Cchart>.tgz oci:\u002F\u002F\u003Cyour-registry>\u002F\u003Cmirror-sub-path>",{"type":81,"value":1778},". Only OCI charts can live\nunder ",{"type":75,"tag":102,"props":1780,"children":1782},{"className":1781},[],[1783],{"type":81,"value":1738},{"type":81,"value":134},{"type":75,"tag":1554,"props":1786,"children":1787},{},[1788,1790,1796,1798,1804,1806,1811,1812,1817,1819],{"type":81,"value":1789},"Skip the credential steps above. When the cluster authenticates to the registry\nautomatically (the node or workload identity has pull access), the pull needs no\nKubernetes secret, so you do not need: the per-namespace ",{"type":75,"tag":102,"props":1791,"children":1793},{"className":1792},[],[1794],{"type":81,"value":1795},"nvcr-creds",{"type":81,"value":1797}," from step 1\n(still create the namespaces, just not the secret), the ",{"type":75,"tag":102,"props":1799,"children":1801},{"className":1800},[],[1802],{"type":81,"value":1803},"global.imagePullSecrets",{"type":81,"value":1805}," entry,\nor the ",{"type":75,"tag":102,"props":1807,"children":1809},{"className":1808},[],[1810],{"type":81,"value":1207},{"type":81,"value":1755},{"type":75,"tag":102,"props":1813,"children":1815},{"className":1814},[],[1816],{"type":81,"value":1504},{"type":81,"value":1818}," from step 2.",{"type":75,"tag":1476,"props":1820,"children":1821},{},[1822],{"type":75,"tag":84,"props":1823,"children":1824},{},[1825,1827,1833],{"type":81,"value":1826},"One residual credential: nvcf-api still validates the user function image at\n",{"type":75,"tag":102,"props":1828,"children":1830},{"className":1829},[],[1831],{"type":81,"value":1832},"function create",{"type":81,"value":1834}," via the account registry credential, so that credential must point at\nwhichever registry holds the user image (set it to your mirror if the image is mirrored\nthere).",{"type":75,"tag":740,"props":1836,"children":1838},{"id":1837},"_3-set-the-route-domain",[1839],{"type":81,"value":1840},"3. Set the route domain",{"type":75,"tag":84,"props":1842,"children":1843},{},[1844,1850,1852,1858],{"type":75,"tag":102,"props":1845,"children":1847},{"className":1846},[],[1848],{"type":81,"value":1849},"global.domain",{"type":81,"value":1851}," sets the hostname suffix for routes such as ",{"type":75,"tag":102,"props":1853,"children":1855},{"className":1854},[],[1856],{"type":81,"value":1857},"api.\u003Cdomain>",{"type":81,"value":1859},". It does not configure the Gateway or its load balancer. Query the exact existing Gateway:",{"type":75,"tag":161,"props":1861,"children":1863},{"className":163,"code":1862,"language":65,"meta":165,"style":165},"GATEWAY_ADDRESS=$(kubectl get gateway \u003Cgateway-name> -n \u003Cgateway-namespace> -o jsonpath='{.status.addresses[0].value}')\ntest -n \"$GATEWAY_ADDRESS\"\n",[1864],{"type":75,"tag":102,"props":1865,"children":1866},{"__ignoreMap":165},[1867,1961],{"type":75,"tag":171,"props":1868,"children":1869},{"class":173,"line":174},[1870,1875,1880,1884,1889,1894,1898,1903,1908,1912,1917,1921,1926,1930,1934,1938,1943,1947,1952,1956],{"type":75,"tag":171,"props":1871,"children":1872},{"style":196},[1873],{"type":81,"value":1874},"GATEWAY_ADDRESS",{"type":75,"tag":171,"props":1876,"children":1877},{"style":184},[1878],{"type":81,"value":1879},"=$(",{"type":75,"tag":171,"props":1881,"children":1882},{"style":178},[1883],{"type":81,"value":69},{"type":75,"tag":171,"props":1885,"children":1886},{"style":190},[1887],{"type":81,"value":1888}," get",{"type":75,"tag":171,"props":1890,"children":1891},{"style":190},[1892],{"type":81,"value":1893}," gateway",{"type":75,"tag":171,"props":1895,"children":1896},{"style":184},[1897],{"type":81,"value":187},{"type":75,"tag":171,"props":1899,"children":1900},{"style":190},[1901],{"type":81,"value":1902},"gateway-nam",{"type":75,"tag":171,"props":1904,"children":1905},{"style":196},[1906],{"type":81,"value":1907},"e",{"type":75,"tag":171,"props":1909,"children":1910},{"style":184},[1911],{"type":81,"value":204},{"type":75,"tag":171,"props":1913,"children":1914},{"style":190},[1915],{"type":81,"value":1916}," -n",{"type":75,"tag":171,"props":1918,"children":1919},{"style":184},[1920],{"type":81,"value":187},{"type":75,"tag":171,"props":1922,"children":1923},{"style":190},[1924],{"type":81,"value":1925},"gateway-namespac",{"type":75,"tag":171,"props":1927,"children":1928},{"style":196},[1929],{"type":81,"value":1907},{"type":75,"tag":171,"props":1931,"children":1932},{"style":184},[1933],{"type":81,"value":204},{"type":75,"tag":171,"props":1935,"children":1936},{"style":190},[1937],{"type":81,"value":864},{"type":75,"tag":171,"props":1939,"children":1940},{"style":190},[1941],{"type":81,"value":1942}," jsonpath=",{"type":75,"tag":171,"props":1944,"children":1945},{"style":184},[1946],{"type":81,"value":1069},{"type":75,"tag":171,"props":1948,"children":1949},{"style":190},[1950],{"type":81,"value":1951},"{.status.addresses[0].value}",{"type":75,"tag":171,"props":1953,"children":1954},{"style":184},[1955],{"type":81,"value":1069},{"type":75,"tag":171,"props":1957,"children":1958},{"style":184},[1959],{"type":81,"value":1960},")\n",{"type":75,"tag":171,"props":1962,"children":1963},{"class":173,"line":275},[1964,1969,1973,1977,1982],{"type":75,"tag":171,"props":1965,"children":1966},{"style":606},[1967],{"type":81,"value":1968},"test",{"type":75,"tag":171,"props":1970,"children":1971},{"style":190},[1972],{"type":81,"value":1916},{"type":75,"tag":171,"props":1974,"children":1975},{"style":184},[1976],{"type":81,"value":844},{"type":75,"tag":171,"props":1978,"children":1979},{"style":196},[1980],{"type":81,"value":1981},"$GATEWAY_ADDRESS",{"type":75,"tag":171,"props":1983,"children":1984},{"style":184},[1985],{"type":81,"value":954},{"type":75,"tag":84,"props":1987,"children":1988},{},[1989],{"type":81,"value":1990},"The address may be a hostname or IP. Cloud provider and Gateway configuration determine the load balancer type and address format. The self-managed stack does not select either.",{"type":75,"tag":84,"props":1992,"children":1993},{},[1994,1996,2002,2004,2009,2011,2017,2019,2024,2026,2031,2033,2038],{"type":81,"value":1995},"For temporary testing without DNS, use a stable reserved suffix such as ",{"type":75,"tag":102,"props":1997,"children":1999},{"className":1998},[],[2000],{"type":81,"value":2001},"global.domain: \"nvcf.test\"",{"type":81,"value":2003},", connect to ",{"type":75,"tag":102,"props":2005,"children":2007},{"className":2006},[],[2008],{"type":81,"value":1874},{"type":81,"value":2010},", and send ",{"type":75,"tag":102,"props":2012,"children":2014},{"className":2013},[],[2015],{"type":81,"value":2016},"Host: api.nvcf.test",{"type":81,"value":2018},". For production, use your base DNS domain and point its route records to ",{"type":75,"tag":102,"props":2020,"children":2022},{"className":2021},[],[2023],{"type":81,"value":1874},{"type":81,"value":2025},". If the endpoint later changes, keep ",{"type":75,"tag":102,"props":2027,"children":2029},{"className":2028},[],[2030],{"type":81,"value":1849},{"type":81,"value":2032}," stable and update only DNS or the client destination. See Example 4 in ",{"type":75,"tag":127,"props":2034,"children":2036},{"href":2035},"examples.md",[2037],{"type":81,"value":2035},{"type":81,"value":134},{"type":75,"tag":740,"props":2040,"children":2042},{"id":2041},"_4-preview-and-deploy",[2043],{"type":81,"value":2044},"4. Preview and deploy",{"type":75,"tag":161,"props":2046,"children":2048},{"className":163,"code":2047,"language":65,"meta":165,"style":165},"HELMFILE_ENV=\u003Cenv-name> helmfile template   # Preview rendered manifests\nHELMFILE_ENV=\u003Cenv-name> helmfile sync       # Deploy\n",[2049],{"type":75,"tag":102,"props":2050,"children":2051},{"__ignoreMap":165},[2052,2089],{"type":75,"tag":171,"props":2053,"children":2054},{"class":173,"line":174},[2055,2060,2065,2070,2074,2079,2084],{"type":75,"tag":171,"props":2056,"children":2057},{"style":196},[2058],{"type":81,"value":2059},"HELMFILE_ENV",{"type":75,"tag":171,"props":2061,"children":2062},{"style":184},[2063],{"type":81,"value":2064},"=\u003C",{"type":75,"tag":171,"props":2066,"children":2067},{"style":190},[2068],{"type":81,"value":2069},"env-name",{"type":75,"tag":171,"props":2071,"children":2072},{"style":184},[2073],{"type":81,"value":204},{"type":75,"tag":171,"props":2075,"children":2076},{"style":178},[2077],{"type":81,"value":2078}," helmfile",{"type":75,"tag":171,"props":2080,"children":2081},{"style":190},[2082],{"type":81,"value":2083}," template",{"type":75,"tag":171,"props":2085,"children":2086},{"style":520},[2087],{"type":81,"value":2088},"   # Preview rendered manifests\n",{"type":75,"tag":171,"props":2090,"children":2091},{"class":173,"line":275},[2092,2096,2100,2104,2108,2112,2117],{"type":75,"tag":171,"props":2093,"children":2094},{"style":196},[2095],{"type":81,"value":2059},{"type":75,"tag":171,"props":2097,"children":2098},{"style":184},[2099],{"type":81,"value":2064},{"type":75,"tag":171,"props":2101,"children":2102},{"style":190},[2103],{"type":81,"value":2069},{"type":75,"tag":171,"props":2105,"children":2106},{"style":184},[2107],{"type":81,"value":204},{"type":75,"tag":171,"props":2109,"children":2110},{"style":178},[2111],{"type":81,"value":2078},{"type":75,"tag":171,"props":2113,"children":2114},{"style":190},[2115],{"type":81,"value":2116}," sync",{"type":75,"tag":171,"props":2118,"children":2119},{"style":520},[2120],{"type":81,"value":2121},"       # Deploy\n",{"type":75,"tag":1476,"props":2123,"children":2124},{},[2125],{"type":75,"tag":84,"props":2126,"children":2127},{},[2128,2130,2136,2138,2144,2146,2152,2154,2159,2161,2167,2169,2174,2176,2181],{"type":81,"value":2129},"Set BOTH Cassandra passwords in the secrets file. The secrets file documents\n",{"type":75,"tag":102,"props":2131,"children":2133},{"className":2132},[],[2134],{"type":81,"value":2135},"DEFAULT_CASSANDRA_PASSWORD",{"type":81,"value":2137},", but the cassandra chart has a second credential,\n",{"type":75,"tag":102,"props":2139,"children":2141},{"className":2140},[],[2142],{"type":81,"value":2143},"cassandra.serviceRolePassword",{"type":81,"value":2145},", that defaults to ",{"type":75,"tag":102,"props":2147,"children":2149},{"className":2148},[],[2150],{"type":81,"value":2151},"ch@ng3m3",{"type":81,"value":2153},". If you set only\n",{"type":75,"tag":102,"props":2155,"children":2157},{"className":2156},[],[2158],{"type":81,"value":2135},{"type":81,"value":2160},", the service role keeps that default and every DB-backed\nservice crash-loops with ",{"type":75,"tag":102,"props":2162,"children":2164},{"className":2163},[],[2165],{"type":81,"value":2166},"Bad credentials",{"type":81,"value":2168}," once it tries to authenticate. Set\n",{"type":75,"tag":102,"props":2170,"children":2172},{"className":2171},[],[2173],{"type":81,"value":2143},{"type":81,"value":2175}," to the same value as ",{"type":75,"tag":102,"props":2177,"children":2179},{"className":2178},[],[2180],{"type":81,"value":2135},{"type":81,"value":134},{"type":75,"tag":740,"props":2183,"children":2185},{"id":2184},"_5-deployment-phases",[2186],{"type":81,"value":2187},"5. Deployment phases",{"type":75,"tag":84,"props":2189,"children":2190},{},[2191],{"type":81,"value":2192},"Helmfile deploys in order with dependencies:",{"type":75,"tag":2194,"props":2195,"children":2196},"table",{},[2197,2226],{"type":75,"tag":2198,"props":2199,"children":2200},"thead",{},[2201],{"type":75,"tag":2202,"props":2203,"children":2204},"tr",{},[2205,2211,2216,2221],{"type":75,"tag":2206,"props":2207,"children":2208},"th",{},[2209],{"type":81,"value":2210},"Phase",{"type":75,"tag":2206,"props":2212,"children":2213},{},[2214],{"type":81,"value":2215},"Selector",{"type":75,"tag":2206,"props":2217,"children":2218},{},[2219],{"type":81,"value":2220},"Services",{"type":75,"tag":2206,"props":2222,"children":2223},{},[2224],{"type":81,"value":2225},"Wait time",{"type":75,"tag":2227,"props":2228,"children":2229},"tbody",{},[2230,2258,2284,2311],{"type":75,"tag":2202,"props":2231,"children":2232},{},[2233,2239,2248,2253],{"type":75,"tag":2234,"props":2235,"children":2236},"td",{},[2237],{"type":81,"value":2238},"1",{"type":75,"tag":2234,"props":2240,"children":2241},{},[2242],{"type":75,"tag":102,"props":2243,"children":2245},{"className":2244},[],[2246],{"type":81,"value":2247},"release-group=dependencies",{"type":75,"tag":2234,"props":2249,"children":2250},{},[2251],{"type":81,"value":2252},"NATS, OpenBao, Cassandra",{"type":75,"tag":2234,"props":2254,"children":2255},{},[2256],{"type":81,"value":2257},"5-10 min",{"type":75,"tag":2202,"props":2259,"children":2260},{},[2261,2266,2275,2280],{"type":75,"tag":2234,"props":2262,"children":2263},{},[2264],{"type":81,"value":2265},"2",{"type":75,"tag":2234,"props":2267,"children":2268},{},[2269],{"type":75,"tag":102,"props":2270,"children":2272},{"className":2271},[],[2273],{"type":81,"value":2274},"release-group=services",{"type":75,"tag":2234,"props":2276,"children":2277},{},[2278],{"type":81,"value":2279},"API, SIS, ESS, invocation, grpc-proxy, notary, api-keys, optional LLM gateway\u002Frouter, optional Vanity Gateway, NVCF UI when the stack package includes the addon",{"type":75,"tag":2234,"props":2281,"children":2282},{},[2283],{"type":81,"value":2257},{"type":75,"tag":2202,"props":2285,"children":2286},{},[2287,2292,2301,2306],{"type":75,"tag":2234,"props":2288,"children":2289},{},[2290],{"type":81,"value":2291},"3",{"type":75,"tag":2234,"props":2293,"children":2294},{},[2295],{"type":75,"tag":102,"props":2296,"children":2298},{"className":2297},[],[2299],{"type":81,"value":2300},"release-group=ingress",{"type":75,"tag":2234,"props":2302,"children":2303},{},[2304],{"type":81,"value":2305},"Gateway routes",{"type":75,"tag":2234,"props":2307,"children":2308},{},[2309],{"type":81,"value":2310},"1-2 min",{"type":75,"tag":2202,"props":2312,"children":2313},{},[2314,2319,2328,2333],{"type":75,"tag":2234,"props":2315,"children":2316},{},[2317],{"type":81,"value":2318},"4",{"type":75,"tag":2234,"props":2320,"children":2321},{},[2322],{"type":75,"tag":102,"props":2323,"children":2325},{"className":2324},[],[2326],{"type":81,"value":2327},"release-group=observability",{"type":75,"tag":2234,"props":2329,"children":2330},{},[2331],{"type":81,"value":2332},"Observability stack (if enabled)",{"type":75,"tag":2234,"props":2334,"children":2335},{},[2336],{"type":81,"value":2310},{"type":75,"tag":84,"props":2338,"children":2339},{},[2340],{"type":81,"value":2341},"Monitor in a separate terminal:",{"type":75,"tag":161,"props":2343,"children":2345},{"className":163,"code":2344,"language":65,"meta":165,"style":165},"kubectl get pods -A -w\n",[2346],{"type":75,"tag":102,"props":2347,"children":2348},{"__ignoreMap":165},[2349],{"type":75,"tag":171,"props":2350,"children":2351},{"class":173,"line":174},[2352,2356,2360,2365,2370],{"type":75,"tag":171,"props":2353,"children":2354},{"style":178},[2355],{"type":81,"value":69},{"type":75,"tag":171,"props":2357,"children":2358},{"style":190},[2359],{"type":81,"value":1888},{"type":75,"tag":171,"props":2361,"children":2362},{"style":190},[2363],{"type":81,"value":2364}," pods",{"type":75,"tag":171,"props":2366,"children":2367},{"style":190},[2368],{"type":81,"value":2369}," -A",{"type":75,"tag":171,"props":2371,"children":2372},{"style":190},[2373],{"type":81,"value":2374}," -w\n",{"type":75,"tag":740,"props":2376,"children":2378},{"id":2377},"_6-enable-vanity-gateway-optional",[2379],{"type":81,"value":2380},"6. Enable Vanity Gateway (optional)",{"type":75,"tag":84,"props":2382,"children":2383},{},[2384,2386,2392,2394,2400],{"type":81,"value":2385},"Vanity Gateway is disabled by default and is only needed for customer-facing\nhostnames or path mappings in front of the standard NVCF API and invocation\nroutes. It is available only in stack packages that include the Vanity Gateway\naddon. If the extracted stack package does not contain a ",{"type":75,"tag":102,"props":2387,"children":2389},{"className":2388},[],[2390],{"type":81,"value":2391},"vanity-gateway",{"type":81,"value":2393},"\nrelease and ",{"type":75,"tag":102,"props":2395,"children":2397},{"className":2396},[],[2398],{"type":81,"value":2399},"vanityGateway",{"type":81,"value":2401}," route values, skip these commands and use a newer\nstack package that includes them.",{"type":75,"tag":84,"props":2403,"children":2404},{},[2405,2407,2413],{"type":81,"value":2406},"If the stack package includes the addon, enable it in\n",{"type":75,"tag":102,"props":2408,"children":2410},{"className":2409},[],[2411],{"type":81,"value":2412},"environments\u002F\u003Cenv-name>.yaml",{"type":81,"value":1566},{"type":75,"tag":161,"props":2415,"children":2417},{"className":1569,"code":2416,"language":66,"meta":165,"style":165},"addons:\n  vanityGateway:\n    enabled: true\n    mappingConfig: {}\n",[2418],{"type":75,"tag":102,"props":2419,"children":2420},{"__ignoreMap":165},[2421,2433,2445,2463],{"type":75,"tag":171,"props":2422,"children":2423},{"class":173,"line":174},[2424,2429],{"type":75,"tag":171,"props":2425,"children":2426},{"style":1580},[2427],{"type":81,"value":2428},"addons",{"type":75,"tag":171,"props":2430,"children":2431},{"style":184},[2432],{"type":81,"value":1588},{"type":75,"tag":171,"props":2434,"children":2435},{"class":173,"line":275},[2436,2441],{"type":75,"tag":171,"props":2437,"children":2438},{"style":1580},[2439],{"type":81,"value":2440},"  vanityGateway",{"type":75,"tag":171,"props":2442,"children":2443},{"style":184},[2444],{"type":81,"value":1588},{"type":75,"tag":171,"props":2446,"children":2447},{"class":173,"line":543},[2448,2453,2457],{"type":75,"tag":171,"props":2449,"children":2450},{"style":1580},[2451],{"type":81,"value":2452},"    enabled",{"type":75,"tag":171,"props":2454,"children":2455},{"style":184},[2456],{"type":81,"value":1566},{"type":75,"tag":171,"props":2458,"children":2460},{"style":2459},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[2461],{"type":81,"value":2462}," true\n",{"type":75,"tag":171,"props":2464,"children":2465},{"class":173,"line":566},[2466,2471,2475],{"type":75,"tag":171,"props":2467,"children":2468},{"style":1580},[2469],{"type":81,"value":2470},"    mappingConfig",{"type":75,"tag":171,"props":2472,"children":2473},{"style":184},[2474],{"type":81,"value":1566},{"type":75,"tag":171,"props":2476,"children":2477},{"style":184},[2478],{"type":81,"value":2479}," {}\n",{"type":75,"tag":84,"props":2481,"children":2482},{},[2483,2485,2491,2493,2499,2501,2507,2509,2514],{"type":81,"value":2484},"The default route host is ",{"type":75,"tag":102,"props":2486,"children":2488},{"className":2487},[],[2489],{"type":81,"value":2490},"vanity.\u003Cdomain>",{"type":81,"value":2492}," and the backend is\n",{"type":75,"tag":102,"props":2494,"children":2496},{"className":2495},[],[2497],{"type":81,"value":2498},"vanity-gateway.nvcf:8080",{"type":81,"value":2500},". Put deployment-specific host and path mappings under\n",{"type":75,"tag":102,"props":2502,"children":2504},{"className":2503},[],[2505],{"type":81,"value":2506},"addons.vanityGateway.mappingConfig",{"type":81,"value":2508},". If you need custom vanity hosts instead of\n",{"type":75,"tag":102,"props":2510,"children":2512},{"className":2511},[],[2513],{"type":81,"value":2490},{"type":81,"value":2515},", use the route hostname overrides supported by the stack\npackage and create matching DNS records.",{"type":75,"tag":84,"props":2517,"children":2518},{},[2519,2521,2526],{"type":81,"value":2520},"After confirming the package includes the ",{"type":75,"tag":102,"props":2522,"children":2524},{"className":2523},[],[2525],{"type":81,"value":2391},{"type":81,"value":2527}," release, preview and\napply the service and route:",{"type":75,"tag":161,"props":2529,"children":2531},{"className":163,"code":2530,"language":65,"meta":165,"style":165},"HELMFILE_ENV=\u003Cenv-name> helmfile --selector name=vanity-gateway template\nHELMFILE_ENV=\u003Cenv-name> helmfile --selector name=vanity-gateway sync\nHELMFILE_ENV=\u003Cenv-name> helmfile --selector release-group=ingress sync\n",[2532],{"type":75,"tag":102,"props":2533,"children":2534},{"__ignoreMap":165},[2535,2573,2609],{"type":75,"tag":171,"props":2536,"children":2537},{"class":173,"line":174},[2538,2542,2546,2550,2554,2558,2563,2568],{"type":75,"tag":171,"props":2539,"children":2540},{"style":196},[2541],{"type":81,"value":2059},{"type":75,"tag":171,"props":2543,"children":2544},{"style":184},[2545],{"type":81,"value":2064},{"type":75,"tag":171,"props":2547,"children":2548},{"style":190},[2549],{"type":81,"value":2069},{"type":75,"tag":171,"props":2551,"children":2552},{"style":184},[2553],{"type":81,"value":204},{"type":75,"tag":171,"props":2555,"children":2556},{"style":178},[2557],{"type":81,"value":2078},{"type":75,"tag":171,"props":2559,"children":2560},{"style":190},[2561],{"type":81,"value":2562}," --selector",{"type":75,"tag":171,"props":2564,"children":2565},{"style":190},[2566],{"type":81,"value":2567}," name=vanity-gateway",{"type":75,"tag":171,"props":2569,"children":2570},{"style":190},[2571],{"type":81,"value":2572}," template\n",{"type":75,"tag":171,"props":2574,"children":2575},{"class":173,"line":275},[2576,2580,2584,2588,2592,2596,2600,2604],{"type":75,"tag":171,"props":2577,"children":2578},{"style":196},[2579],{"type":81,"value":2059},{"type":75,"tag":171,"props":2581,"children":2582},{"style":184},[2583],{"type":81,"value":2064},{"type":75,"tag":171,"props":2585,"children":2586},{"style":190},[2587],{"type":81,"value":2069},{"type":75,"tag":171,"props":2589,"children":2590},{"style":184},[2591],{"type":81,"value":204},{"type":75,"tag":171,"props":2593,"children":2594},{"style":178},[2595],{"type":81,"value":2078},{"type":75,"tag":171,"props":2597,"children":2598},{"style":190},[2599],{"type":81,"value":2562},{"type":75,"tag":171,"props":2601,"children":2602},{"style":190},[2603],{"type":81,"value":2567},{"type":75,"tag":171,"props":2605,"children":2606},{"style":190},[2607],{"type":81,"value":2608}," sync\n",{"type":75,"tag":171,"props":2610,"children":2611},{"class":173,"line":543},[2612,2616,2620,2624,2628,2632,2636,2641],{"type":75,"tag":171,"props":2613,"children":2614},{"style":196},[2615],{"type":81,"value":2059},{"type":75,"tag":171,"props":2617,"children":2618},{"style":184},[2619],{"type":81,"value":2064},{"type":75,"tag":171,"props":2621,"children":2622},{"style":190},[2623],{"type":81,"value":2069},{"type":75,"tag":171,"props":2625,"children":2626},{"style":184},[2627],{"type":81,"value":204},{"type":75,"tag":171,"props":2629,"children":2630},{"style":178},[2631],{"type":81,"value":2078},{"type":75,"tag":171,"props":2633,"children":2634},{"style":190},[2635],{"type":81,"value":2562},{"type":75,"tag":171,"props":2637,"children":2638},{"style":190},[2639],{"type":81,"value":2640}," release-group=ingress",{"type":75,"tag":171,"props":2642,"children":2643},{"style":190},[2644],{"type":81,"value":2608},{"type":75,"tag":84,"props":2646,"children":2647},{},[2648],{"type":81,"value":2649},"Verify the route only when the addon is present and enabled:",{"type":75,"tag":161,"props":2651,"children":2653},{"className":163,"code":2652,"language":65,"meta":165,"style":165},"kubectl get deploy,svc -n nvcf -l app.kubernetes.io\u002Fname=vanity-gateway\nkubectl get httproute -A | grep -i vanity\ncurl -H \"Host: vanity.\u003Cdomain>\" \"http:\u002F\u002F\u003Cgateway-address>\u002Fhealth\"\n",[2654],{"type":75,"tag":102,"props":2655,"children":2656},{"__ignoreMap":165},[2657,2691,2730],{"type":75,"tag":171,"props":2658,"children":2659},{"class":173,"line":174},[2660,2664,2668,2673,2677,2681,2686],{"type":75,"tag":171,"props":2661,"children":2662},{"style":178},[2663],{"type":81,"value":69},{"type":75,"tag":171,"props":2665,"children":2666},{"style":190},[2667],{"type":81,"value":1888},{"type":75,"tag":171,"props":2669,"children":2670},{"style":190},[2671],{"type":81,"value":2672}," deploy,svc",{"type":75,"tag":171,"props":2674,"children":2675},{"style":190},[2676],{"type":81,"value":1916},{"type":75,"tag":171,"props":2678,"children":2679},{"style":190},[2680],{"type":81,"value":786},{"type":75,"tag":171,"props":2682,"children":2683},{"style":190},[2684],{"type":81,"value":2685}," -l",{"type":75,"tag":171,"props":2687,"children":2688},{"style":190},[2689],{"type":81,"value":2690}," app.kubernetes.io\u002Fname=vanity-gateway\n",{"type":75,"tag":171,"props":2692,"children":2693},{"class":173,"line":275},[2694,2698,2702,2707,2711,2715,2720,2725],{"type":75,"tag":171,"props":2695,"children":2696},{"style":178},[2697],{"type":81,"value":69},{"type":75,"tag":171,"props":2699,"children":2700},{"style":190},[2701],{"type":81,"value":1888},{"type":75,"tag":171,"props":2703,"children":2704},{"style":190},[2705],{"type":81,"value":2706}," httproute",{"type":75,"tag":171,"props":2708,"children":2709},{"style":190},[2710],{"type":81,"value":2369},{"type":75,"tag":171,"props":2712,"children":2713},{"style":184},[2714],{"type":81,"value":874},{"type":75,"tag":171,"props":2716,"children":2717},{"style":178},[2718],{"type":81,"value":2719}," grep",{"type":75,"tag":171,"props":2721,"children":2722},{"style":190},[2723],{"type":81,"value":2724}," -i",{"type":75,"tag":171,"props":2726,"children":2727},{"style":190},[2728],{"type":81,"value":2729}," vanity\n",{"type":75,"tag":171,"props":2731,"children":2732},{"class":173,"line":543},[2733,2738,2743,2747,2752,2756,2760,2765],{"type":75,"tag":171,"props":2734,"children":2735},{"style":178},[2736],{"type":81,"value":2737},"curl",{"type":75,"tag":171,"props":2739,"children":2740},{"style":190},[2741],{"type":81,"value":2742}," -H",{"type":75,"tag":171,"props":2744,"children":2745},{"style":184},[2746],{"type":81,"value":844},{"type":75,"tag":171,"props":2748,"children":2749},{"style":190},[2750],{"type":81,"value":2751},"Host: vanity.\u003Cdomain>",{"type":75,"tag":171,"props":2753,"children":2754},{"style":184},[2755],{"type":81,"value":854},{"type":75,"tag":171,"props":2757,"children":2758},{"style":184},[2759],{"type":81,"value":844},{"type":75,"tag":171,"props":2761,"children":2762},{"style":190},[2763],{"type":81,"value":2764},"http:\u002F\u002F\u003Cgateway-address>\u002Fhealth",{"type":75,"tag":171,"props":2766,"children":2767},{"style":184},[2768],{"type":81,"value":954},{"type":75,"tag":84,"props":2770,"children":2771},{},[2772],{"type":81,"value":2773},"Do not enable Vanity Gateway for standard API, API Keys, invocation, LLM\ninvocation, or gRPC traffic. Those routes are provided by the base gateway\nroutes.",{"type":75,"tag":740,"props":2775,"children":2777},{"id":2776},"_7-enable-nvcf-ui-optional",[2778],{"type":81,"value":2779},"7. Enable NVCF UI (optional)",{"type":75,"tag":84,"props":2781,"children":2782},{},[2783,2785,2791,2793,2799],{"type":81,"value":2784},"NVCF UI is an optional, customer-facing admin-panel UI. It is disabled by\ndefault and available only in stack packages that include the NVCF UI addon. If\nyour extracted stack package does not contain a ",{"type":75,"tag":102,"props":2786,"children":2788},{"className":2787},[],[2789],{"type":81,"value":2790},"nvcf-ui",{"type":81,"value":2792}," release and ",{"type":75,"tag":102,"props":2794,"children":2796},{"className":2795},[],[2797],{"type":81,"value":2798},"nvcfUi",{"type":81,"value":2800},"\nroute values, skip this section. The NVCF UI admin panel is currently\nunauthenticated; do not expose it to the public internet.",{"type":75,"tag":84,"props":2802,"children":2803},{},[2804,2806,2812],{"type":81,"value":2805},"For enablement, namespace and pull secret setup, apply, and verification steps,\nsee ",{"type":75,"tag":127,"props":2807,"children":2809},{"href":2808},"references\u002Fnvcf-ui-addon.md",[2810],{"type":81,"value":2811},"NVCF UI Addon",{"type":81,"value":134},{"type":75,"tag":740,"props":2814,"children":2816},{"id":2815},"_8-install-the-separate-compute-plane-stack",[2817],{"type":81,"value":2818},"8. Install the separate compute-plane stack",{"type":75,"tag":84,"props":2820,"children":2821},{},[2822],{"type":81,"value":2823},"The control-plane stack does not install compute-plane components. After it is healthy, choose exactly one compute-plane handoff:",{"type":75,"tag":2825,"props":2826,"children":2827},"ul",{},[2828,2833],{"type":75,"tag":1554,"props":2829,"children":2830},{},[2831],{"type":81,"value":2832},"Helmfile values flow: author separate environment files, then register and\ninstall with the compute-plane Makefile.",{"type":75,"tag":1554,"props":2834,"children":2835},{},[2836],{"type":81,"value":2837},"CLI profile flow: use the profile generated by the CLI control-plane install,\nthen run the complete CLI register and install commands.",{"type":75,"tag":84,"props":2839,"children":2840},{},[2841,2843,2847],{"type":81,"value":2842},"Do not mix these handoffs. Follow ",{"type":75,"tag":127,"props":2844,"children":2845},{"href":483},[2846],{"type":81,"value":486},{"type":81,"value":2848},"\nfor commands, EKS routing, pull secrets, verification, and teardown order.",{"type":75,"tag":90,"props":2850,"children":2852},{"id":2851},"clean-teardown",[2853],{"type":81,"value":2854},"Clean Teardown",{"type":75,"tag":84,"props":2856,"children":2857},{},[2858,2860,2865],{"type":81,"value":2859},"Destroy every compute plane before the control plane, using the teardown for\nthe same handoff; see ",{"type":75,"tag":127,"props":2861,"children":2863},{"href":2862},"references\u002Fcompute-plane-installation.md#teardown",[2864],{"type":81,"value":486},{"type":81,"value":134},{"type":75,"tag":84,"props":2867,"children":2868},{},[2869,2871,2877,2879,2885,2886,2892,2893,2899,2900,2906,2907,2913,2914,2920,2921,2927,2928,2934,2935,2941,2943,2948,2949,2954,2955,2961,2963,2969,2971,2977,2979,2985,2987,2992,2994,3000],{"type":81,"value":2870},"Scope: Only destroy releases managed by this control-plane helmfile stack. The NVCF releases are: ",{"type":75,"tag":102,"props":2872,"children":2874},{"className":2873},[],[2875],{"type":81,"value":2876},"nats",{"type":81,"value":2878},", ",{"type":75,"tag":102,"props":2880,"children":2882},{"className":2881},[],[2883],{"type":81,"value":2884},"openbao-server",{"type":81,"value":2878},{"type":75,"tag":102,"props":2887,"children":2889},{"className":2888},[],[2890],{"type":81,"value":2891},"cassandra",{"type":81,"value":2878},{"type":75,"tag":102,"props":2894,"children":2896},{"className":2895},[],[2897],{"type":81,"value":2898},"api-keys",{"type":81,"value":2878},{"type":75,"tag":102,"props":2901,"children":2903},{"className":2902},[],[2904],{"type":81,"value":2905},"sis",{"type":81,"value":2878},{"type":75,"tag":102,"props":2908,"children":2910},{"className":2909},[],[2911],{"type":81,"value":2912},"api",{"type":81,"value":2878},{"type":75,"tag":102,"props":2915,"children":2917},{"className":2916},[],[2918],{"type":81,"value":2919},"invocation-service",{"type":81,"value":2878},{"type":75,"tag":102,"props":2922,"children":2924},{"className":2923},[],[2925],{"type":81,"value":2926},"grpc-proxy",{"type":81,"value":2878},{"type":75,"tag":102,"props":2929,"children":2931},{"className":2930},[],[2932],{"type":81,"value":2933},"ess-api",{"type":81,"value":2878},{"type":75,"tag":102,"props":2936,"children":2938},{"className":2937},[],[2939],{"type":81,"value":2940},"notary-service",{"type":81,"value":2942},", optional ",{"type":75,"tag":102,"props":2944,"children":2946},{"className":2945},[],[2947],{"type":81,"value":2391},{"type":81,"value":2878},{"type":75,"tag":102,"props":2950,"children":2952},{"className":2951},[],[2953],{"type":81,"value":2790},{"type":81,"value":2878},{"type":75,"tag":102,"props":2956,"children":2958},{"className":2957},[],[2959],{"type":81,"value":2960},"admin-issuer-proxy",{"type":81,"value":2962},", and ",{"type":75,"tag":102,"props":2964,"children":2966},{"className":2965},[],[2967],{"type":81,"value":2968},"ingress",{"type":81,"value":2970},"; ",{"type":75,"tag":102,"props":2972,"children":2974},{"className":2973},[],[2975],{"type":81,"value":2976},"cert-manager",{"type":81,"value":2978}," is included only when ",{"type":75,"tag":102,"props":2980,"children":2982},{"className":2981},[],[2983],{"type":81,"value":2984},"certManager.enabled: true",{"type":81,"value":2986},". The control-plane namespace inventory also includes ",{"type":75,"tag":102,"props":2988,"children":2990},{"className":2989},[],[2991],{"type":81,"value":2976},{"type":81,"value":2993},", but default namespace cleanup preserves it. Namespace preservation does not preserve the Helm release. External cert-manager must use ",{"type":75,"tag":102,"props":2995,"children":2997},{"className":2996},[],[2998],{"type":81,"value":2999},"certManager.enabled: false",{"type":81,"value":3001},"; otherwise Helmfile treats the release as stack-managed. Do not delete other helm releases or namespaces on the cluster.",{"type":75,"tag":84,"props":3003,"children":3004},{},[3005,3007,3012,3014,3019],{"type":81,"value":3006},"Before control-plane destroy, identify whether cert-manager is external. For an external installation, require effective ",{"type":75,"tag":102,"props":3008,"children":3010},{"className":3009},[],[3011],{"type":81,"value":2999},{"type":81,"value":3013}," and abort if it is true. For an intended stack-managed release with effective ",{"type":75,"tag":102,"props":3015,"children":3017},{"className":3016},[],[3018],{"type":81,"value":2984},{"type":81,"value":3020},", verify matching stack Helm release metadata and abort on missing or mismatched evidence. Preserve the namespace by default; only plan its removal after verifying namespace provenance and getting explicit confirmation.",{"type":75,"tag":740,"props":3022,"children":3024},{"id":3023},"standard-teardown",[3025],{"type":81,"value":3026},"Standard teardown",{"type":75,"tag":84,"props":3028,"children":3029},{},[3030,3032,3037],{"type":81,"value":3031},"Run from inside the ",{"type":75,"tag":102,"props":3033,"children":3035},{"className":3034},[],[3036],{"type":81,"value":591},{"type":81,"value":3038}," directory:",{"type":75,"tag":161,"props":3040,"children":3042},{"className":163,"code":3041,"language":65,"meta":165,"style":165},"HELMFILE_ENV=\u003Cenv-name> helmfile destroy\n",[3043],{"type":75,"tag":102,"props":3044,"children":3045},{"__ignoreMap":165},[3046],{"type":75,"tag":171,"props":3047,"children":3048},{"class":173,"line":174},[3049,3053,3057,3061,3065,3069],{"type":75,"tag":171,"props":3050,"children":3051},{"style":196},[3052],{"type":81,"value":2059},{"type":75,"tag":171,"props":3054,"children":3055},{"style":184},[3056],{"type":81,"value":2064},{"type":75,"tag":171,"props":3058,"children":3059},{"style":190},[3060],{"type":81,"value":2069},{"type":75,"tag":171,"props":3062,"children":3063},{"style":184},[3064],{"type":81,"value":204},{"type":75,"tag":171,"props":3066,"children":3067},{"style":178},[3068],{"type":81,"value":2078},{"type":75,"tag":171,"props":3070,"children":3071},{"style":190},[3072],{"type":81,"value":3073}," destroy\n",{"type":75,"tag":740,"props":3075,"children":3077},{"id":3076},"delete-namespaces",[3078],{"type":81,"value":3079},"Delete namespaces",{"type":75,"tag":161,"props":3081,"children":3083},{"className":163,"code":3082,"language":65,"meta":165,"style":165},"for ns in cassandra-system nats-system nvcf api-keys ess sis nvcf-ui vault-system; do\n  kubectl delete namespace \"$ns\" --ignore-not-found\ndone\n",[3084],{"type":75,"tag":102,"props":3085,"children":3086},{"__ignoreMap":165},[3087,3143,3176],{"type":75,"tag":171,"props":3088,"children":3089},{"class":173,"line":174},[3090,3094,3098,3102,3106,3110,3114,3118,3122,3126,3131,3135,3139],{"type":75,"tag":171,"props":3091,"children":3092},{"style":758},[3093],{"type":81,"value":761},{"type":75,"tag":171,"props":3095,"children":3096},{"style":196},[3097],{"type":81,"value":766},{"type":75,"tag":171,"props":3099,"children":3100},{"style":758},[3101],{"type":81,"value":771},{"type":75,"tag":171,"props":3103,"children":3104},{"style":190},[3105],{"type":81,"value":776},{"type":75,"tag":171,"props":3107,"children":3108},{"style":190},[3109],{"type":81,"value":781},{"type":75,"tag":171,"props":3111,"children":3112},{"style":190},[3113],{"type":81,"value":786},{"type":75,"tag":171,"props":3115,"children":3116},{"style":190},[3117],{"type":81,"value":791},{"type":75,"tag":171,"props":3119,"children":3120},{"style":190},[3121],{"type":81,"value":796},{"type":75,"tag":171,"props":3123,"children":3124},{"style":190},[3125],{"type":81,"value":801},{"type":75,"tag":171,"props":3127,"children":3128},{"style":190},[3129],{"type":81,"value":3130}," nvcf-ui",{"type":75,"tag":171,"props":3132,"children":3133},{"style":190},[3134],{"type":81,"value":806},{"type":75,"tag":171,"props":3136,"children":3137},{"style":184},[3138],{"type":81,"value":816},{"type":75,"tag":171,"props":3140,"children":3141},{"style":758},[3142],{"type":81,"value":821},{"type":75,"tag":171,"props":3144,"children":3145},{"class":173,"line":275},[3146,3150,3155,3159,3163,3167,3171],{"type":75,"tag":171,"props":3147,"children":3148},{"style":178},[3149],{"type":81,"value":829},{"type":75,"tag":171,"props":3151,"children":3152},{"style":190},[3153],{"type":81,"value":3154}," delete",{"type":75,"tag":171,"props":3156,"children":3157},{"style":190},[3158],{"type":81,"value":839},{"type":75,"tag":171,"props":3160,"children":3161},{"style":184},[3162],{"type":81,"value":844},{"type":75,"tag":171,"props":3164,"children":3165},{"style":196},[3166],{"type":81,"value":849},{"type":75,"tag":171,"props":3168,"children":3169},{"style":184},[3170],{"type":81,"value":854},{"type":75,"tag":171,"props":3172,"children":3173},{"style":190},[3174],{"type":81,"value":3175}," --ignore-not-found\n",{"type":75,"tag":171,"props":3177,"children":3178},{"class":173,"line":543},[3179],{"type":75,"tag":171,"props":3180,"children":3181},{"style":758},[3182],{"type":81,"value":902},{"type":75,"tag":740,"props":3184,"children":3186},{"id":3185},"verify-clean",[3187],{"type":81,"value":3188},"Verify clean",{"type":75,"tag":161,"props":3190,"children":3192},{"className":163,"code":3191,"language":65,"meta":165,"style":165},"kubectl get ns | grep -E '(cassandra|nats|vault|nvcf|api-keys|ess|sis)'\n# Should return empty\n",[3193],{"type":75,"tag":102,"props":3194,"children":3195},{"__ignoreMap":165},[3196,3239],{"type":75,"tag":171,"props":3197,"children":3198},{"class":173,"line":174},[3199,3203,3207,3212,3216,3220,3225,3229,3234],{"type":75,"tag":171,"props":3200,"children":3201},{"style":178},[3202],{"type":81,"value":69},{"type":75,"tag":171,"props":3204,"children":3205},{"style":190},[3206],{"type":81,"value":1888},{"type":75,"tag":171,"props":3208,"children":3209},{"style":190},[3210],{"type":81,"value":3211}," ns",{"type":75,"tag":171,"props":3213,"children":3214},{"style":184},[3215],{"type":81,"value":874},{"type":75,"tag":171,"props":3217,"children":3218},{"style":178},[3219],{"type":81,"value":2719},{"type":75,"tag":171,"props":3221,"children":3222},{"style":190},[3223],{"type":81,"value":3224}," -E",{"type":75,"tag":171,"props":3226,"children":3227},{"style":184},[3228],{"type":81,"value":1260},{"type":75,"tag":171,"props":3230,"children":3231},{"style":190},[3232],{"type":81,"value":3233},"(cassandra|nats|vault|nvcf|api-keys|ess|sis)",{"type":75,"tag":171,"props":3235,"children":3236},{"style":184},[3237],{"type":81,"value":3238},"'\n",{"type":75,"tag":171,"props":3240,"children":3241},{"class":173,"line":275},[3242],{"type":75,"tag":171,"props":3243,"children":3244},{"style":520},[3245],{"type":81,"value":3246},"# Should return empty\n",{"type":75,"tag":90,"props":3248,"children":3250},{"id":3249},"overriding-helm-chart-values",[3251],{"type":81,"value":731},{"type":75,"tag":740,"props":3253,"children":3255},{"id":3254},"environment-file-limited",[3256],{"type":81,"value":3257},"Environment file (limited)",{"type":75,"tag":84,"props":3259,"children":3260},{},[3261,3263,3268,3270,3276,3277,3283,3284,3290],{"type":81,"value":3262},"Works only for keys that ",{"type":75,"tag":102,"props":3264,"children":3266},{"className":3265},[],[3267],{"type":81,"value":703},{"type":81,"value":3269}," propagates (e.g., ",{"type":75,"tag":102,"props":3271,"children":3273},{"className":3272},[],[3274],{"type":81,"value":3275},"cassandra.replicaCount",{"type":81,"value":2878},{"type":75,"tag":102,"props":3278,"children":3280},{"className":3279},[],[3281],{"type":81,"value":3282},"global.storageClass",{"type":81,"value":2878},{"type":75,"tag":102,"props":3285,"children":3287},{"className":3286},[],[3288],{"type":81,"value":3289},"global.image",{"type":81,"value":3291},").",{"type":75,"tag":740,"props":3293,"children":3295},{"id":3294},"release-values-block-any-chart-value",[3296],{"type":81,"value":3297},"Release values block (any chart value)",{"type":75,"tag":84,"props":3299,"children":3300},{},[3301,3303,3309,3311,3316,3318,3324,3326,3332],{"type":81,"value":3302},"Edit the release in ",{"type":75,"tag":102,"props":3304,"children":3306},{"className":3305},[],[3307],{"type":81,"value":3308},"helmfile.d\u002F*.yaml.gotmpl",{"type":81,"value":3310}," and add a ",{"type":75,"tag":102,"props":3312,"children":3314},{"className":3313},[],[3315],{"type":81,"value":723},{"type":81,"value":3317}," block. In the examples below, ",{"type":75,"tag":102,"props":3319,"children":3321},{"className":3320},[],[3322],{"type":81,"value":3323},"\u003Cprivate-values>",{"type":81,"value":3325}," refers to the ",{"type":75,"tag":102,"props":3327,"children":3329},{"className":3328},[],[3330],{"type":81,"value":3331},"secrets\u002F",{"type":81,"value":3333}," directory at the helmfile stack root.",{"type":75,"tag":161,"props":3335,"children":3337},{"className":1569,"code":3336,"language":66,"meta":165,"style":165},"- name: cassandra\n  version: 0.8.0\n  condition: cassandra.enabled\n  namespace: cassandra-system\n  \u003C\u003C: *dependency\n  values:\n    - ..\u002Fglobal.yaml.gotmpl\n    - ..\u002F\u003Cprivate-values>\u002F{{ requiredEnv \"HELMFILE_ENV\" }}-secrets.yaml\n    - cassandra:\n        resources:\n          limits:\n            cpu: \"8\"\n            memory: 8192Mi\n          requests:\n            cpu: \"2\"\n            memory: 4096Mi\n",[3338],{"type":75,"tag":102,"props":3339,"children":3340},{"__ignoreMap":165},[3341,3363,3381,3398,3415,3433,3445,3458,3470,3486,3498,3510,3535,3552,3564,3588],{"type":75,"tag":171,"props":3342,"children":3343},{"class":173,"line":174},[3344,3349,3354,3358],{"type":75,"tag":171,"props":3345,"children":3346},{"style":184},[3347],{"type":81,"value":3348},"-",{"type":75,"tag":171,"props":3350,"children":3351},{"style":1580},[3352],{"type":81,"value":3353}," name",{"type":75,"tag":171,"props":3355,"children":3356},{"style":184},[3357],{"type":81,"value":1566},{"type":75,"tag":171,"props":3359,"children":3360},{"style":190},[3361],{"type":81,"value":3362}," cassandra\n",{"type":75,"tag":171,"props":3364,"children":3365},{"class":173,"line":275},[3366,3371,3375],{"type":75,"tag":171,"props":3367,"children":3368},{"style":1580},[3369],{"type":81,"value":3370},"  version",{"type":75,"tag":171,"props":3372,"children":3373},{"style":184},[3374],{"type":81,"value":1566},{"type":75,"tag":171,"props":3376,"children":3378},{"style":3377},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[3379],{"type":81,"value":3380}," 0.8.0\n",{"type":75,"tag":171,"props":3382,"children":3383},{"class":173,"line":543},[3384,3389,3393],{"type":75,"tag":171,"props":3385,"children":3386},{"style":1580},[3387],{"type":81,"value":3388},"  condition",{"type":75,"tag":171,"props":3390,"children":3391},{"style":184},[3392],{"type":81,"value":1566},{"type":75,"tag":171,"props":3394,"children":3395},{"style":190},[3396],{"type":81,"value":3397}," cassandra.enabled\n",{"type":75,"tag":171,"props":3399,"children":3400},{"class":173,"line":566},[3401,3406,3410],{"type":75,"tag":171,"props":3402,"children":3403},{"style":1580},[3404],{"type":81,"value":3405},"  namespace",{"type":75,"tag":171,"props":3407,"children":3408},{"style":184},[3409],{"type":81,"value":1566},{"type":75,"tag":171,"props":3411,"children":3412},{"style":190},[3413],{"type":81,"value":3414}," cassandra-system\n",{"type":75,"tag":171,"props":3416,"children":3417},{"class":173,"line":914},[3418,3423,3428],{"type":75,"tag":171,"props":3419,"children":3420},{"style":184},[3421],{"type":81,"value":3422},"  \u003C\u003C:",{"type":75,"tag":171,"props":3424,"children":3425},{"style":758},[3426],{"type":81,"value":3427}," *",{"type":75,"tag":171,"props":3429,"children":3430},{"style":196},[3431],{"type":81,"value":3432},"dependency\n",{"type":75,"tag":171,"props":3434,"children":3435},{"class":173,"line":923},[3436,3441],{"type":75,"tag":171,"props":3437,"children":3438},{"style":1580},[3439],{"type":81,"value":3440},"  values",{"type":75,"tag":171,"props":3442,"children":3443},{"style":184},[3444],{"type":81,"value":1588},{"type":75,"tag":171,"props":3446,"children":3447},{"class":173,"line":957},[3448,3453],{"type":75,"tag":171,"props":3449,"children":3450},{"style":184},[3451],{"type":81,"value":3452},"    -",{"type":75,"tag":171,"props":3454,"children":3455},{"style":190},[3456],{"type":81,"value":3457}," ..\u002Fglobal.yaml.gotmpl\n",{"type":75,"tag":171,"props":3459,"children":3460},{"class":173,"line":1013},[3461,3465],{"type":75,"tag":171,"props":3462,"children":3463},{"style":184},[3464],{"type":81,"value":3452},{"type":75,"tag":171,"props":3466,"children":3467},{"style":190},[3468],{"type":81,"value":3469}," ..\u002F\u003Cprivate-values>\u002F{{ requiredEnv \"HELMFILE_ENV\" }}-secrets.yaml\n",{"type":75,"tag":171,"props":3471,"children":3472},{"class":173,"line":1045},[3473,3477,3482],{"type":75,"tag":171,"props":3474,"children":3475},{"style":184},[3476],{"type":81,"value":3452},{"type":75,"tag":171,"props":3478,"children":3479},{"style":1580},[3480],{"type":81,"value":3481}," cassandra",{"type":75,"tag":171,"props":3483,"children":3484},{"style":184},[3485],{"type":81,"value":1588},{"type":75,"tag":171,"props":3487,"children":3488},{"class":173,"line":1058},[3489,3494],{"type":75,"tag":171,"props":3490,"children":3491},{"style":1580},[3492],{"type":81,"value":3493},"        resources",{"type":75,"tag":171,"props":3495,"children":3496},{"style":184},[3497],{"type":81,"value":1588},{"type":75,"tag":171,"props":3499,"children":3500},{"class":173,"line":1085},[3501,3506],{"type":75,"tag":171,"props":3502,"children":3503},{"style":1580},[3504],{"type":81,"value":3505},"          limits",{"type":75,"tag":171,"props":3507,"children":3508},{"style":184},[3509],{"type":81,"value":1588},{"type":75,"tag":171,"props":3511,"children":3512},{"class":173,"line":1111},[3513,3518,3522,3526,3531],{"type":75,"tag":171,"props":3514,"children":3515},{"style":1580},[3516],{"type":81,"value":3517},"            cpu",{"type":75,"tag":171,"props":3519,"children":3520},{"style":184},[3521],{"type":81,"value":1566},{"type":75,"tag":171,"props":3523,"children":3524},{"style":184},[3525],{"type":81,"value":844},{"type":75,"tag":171,"props":3527,"children":3528},{"style":190},[3529],{"type":81,"value":3530},"8",{"type":75,"tag":171,"props":3532,"children":3533},{"style":184},[3534],{"type":81,"value":954},{"type":75,"tag":171,"props":3536,"children":3537},{"class":173,"line":1136},[3538,3543,3547],{"type":75,"tag":171,"props":3539,"children":3540},{"style":1580},[3541],{"type":81,"value":3542},"            memory",{"type":75,"tag":171,"props":3544,"children":3545},{"style":184},[3546],{"type":81,"value":1566},{"type":75,"tag":171,"props":3548,"children":3549},{"style":190},[3550],{"type":81,"value":3551}," 8192Mi\n",{"type":75,"tag":171,"props":3553,"children":3554},{"class":173,"line":1173},[3555,3560],{"type":75,"tag":171,"props":3556,"children":3557},{"style":1580},[3558],{"type":81,"value":3559},"          requests",{"type":75,"tag":171,"props":3561,"children":3562},{"style":184},[3563],{"type":81,"value":1588},{"type":75,"tag":171,"props":3565,"children":3567},{"class":173,"line":3566},15,[3568,3572,3576,3580,3584],{"type":75,"tag":171,"props":3569,"children":3570},{"style":1580},[3571],{"type":81,"value":3517},{"type":75,"tag":171,"props":3573,"children":3574},{"style":184},[3575],{"type":81,"value":1566},{"type":75,"tag":171,"props":3577,"children":3578},{"style":184},[3579],{"type":81,"value":844},{"type":75,"tag":171,"props":3581,"children":3582},{"style":190},[3583],{"type":81,"value":2265},{"type":75,"tag":171,"props":3585,"children":3586},{"style":184},[3587],{"type":81,"value":954},{"type":75,"tag":171,"props":3589,"children":3591},{"class":173,"line":3590},16,[3592,3596,3600],{"type":75,"tag":171,"props":3593,"children":3594},{"style":1580},[3595],{"type":81,"value":3542},{"type":75,"tag":171,"props":3597,"children":3598},{"style":184},[3599],{"type":81,"value":1566},{"type":75,"tag":171,"props":3601,"children":3602},{"style":190},[3603],{"type":81,"value":3604}," 4096Mi\n",{"type":75,"tag":84,"props":3606,"children":3607},{},[3608,3610,3616,3618,3624,3626,3631,3633,3638],{"type":81,"value":3609},"YAML merge gotcha: When a release uses ",{"type":75,"tag":102,"props":3611,"children":3613},{"className":3612},[],[3614],{"type":81,"value":3615},"\u003C\u003C: *dependency",{"type":81,"value":3617}," or ",{"type":75,"tag":102,"props":3619,"children":3621},{"className":3620},[],[3622],{"type":81,"value":3623},"inherit",{"type":81,"value":3625},", specifying ",{"type":75,"tag":102,"props":3627,"children":3629},{"className":3628},[],[3630],{"type":81,"value":723},{"type":81,"value":3632}," replaces the template's values list. You must re-include ",{"type":75,"tag":102,"props":3634,"children":3636},{"className":3635},[],[3637],{"type":81,"value":703},{"type":81,"value":3639}," and the secrets file.",{"type":75,"tag":740,"props":3641,"children":3643},{"id":3642},"preview-and-apply-a-single-release",[3644],{"type":81,"value":3645},"Preview and apply a single release",{"type":75,"tag":161,"props":3647,"children":3649},{"className":163,"code":3648,"language":65,"meta":165,"style":165},"HELMFILE_ENV=\u003Cenv> helmfile --selector name=cassandra template  # Preview\nHELMFILE_ENV=\u003Cenv> helmfile --selector name=cassandra sync      # Apply\n",[3650],{"type":75,"tag":102,"props":3651,"children":3652},{"__ignoreMap":165},[3653,3695],{"type":75,"tag":171,"props":3654,"children":3655},{"class":173,"line":174},[3656,3660,3664,3669,3673,3677,3681,3686,3690],{"type":75,"tag":171,"props":3657,"children":3658},{"style":196},[3659],{"type":81,"value":2059},{"type":75,"tag":171,"props":3661,"children":3662},{"style":184},[3663],{"type":81,"value":2064},{"type":75,"tag":171,"props":3665,"children":3666},{"style":190},[3667],{"type":81,"value":3668},"env",{"type":75,"tag":171,"props":3670,"children":3671},{"style":184},[3672],{"type":81,"value":204},{"type":75,"tag":171,"props":3674,"children":3675},{"style":178},[3676],{"type":81,"value":2078},{"type":75,"tag":171,"props":3678,"children":3679},{"style":190},[3680],{"type":81,"value":2562},{"type":75,"tag":171,"props":3682,"children":3683},{"style":190},[3684],{"type":81,"value":3685}," name=cassandra",{"type":75,"tag":171,"props":3687,"children":3688},{"style":190},[3689],{"type":81,"value":2083},{"type":75,"tag":171,"props":3691,"children":3692},{"style":520},[3693],{"type":81,"value":3694},"  # Preview\n",{"type":75,"tag":171,"props":3696,"children":3697},{"class":173,"line":275},[3698,3702,3706,3710,3714,3718,3722,3726,3730],{"type":75,"tag":171,"props":3699,"children":3700},{"style":196},[3701],{"type":81,"value":2059},{"type":75,"tag":171,"props":3703,"children":3704},{"style":184},[3705],{"type":81,"value":2064},{"type":75,"tag":171,"props":3707,"children":3708},{"style":190},[3709],{"type":81,"value":3668},{"type":75,"tag":171,"props":3711,"children":3712},{"style":184},[3713],{"type":81,"value":204},{"type":75,"tag":171,"props":3715,"children":3716},{"style":178},[3717],{"type":81,"value":2078},{"type":75,"tag":171,"props":3719,"children":3720},{"style":190},[3721],{"type":81,"value":2562},{"type":75,"tag":171,"props":3723,"children":3724},{"style":190},[3725],{"type":81,"value":3685},{"type":75,"tag":171,"props":3727,"children":3728},{"style":190},[3729],{"type":81,"value":2116},{"type":75,"tag":171,"props":3731,"children":3732},{"style":520},[3733],{"type":81,"value":3734},"      # Apply\n",{"type":75,"tag":90,"props":3736,"children":3738},{"id":3737},"image-pull-secrets",[3739],{"type":81,"value":1189},{"type":75,"tag":84,"props":3741,"children":3742},{},[3743],{"type":81,"value":3744},"There are three distinct credential types. Do not conflate them, and mind the\nformat rule below (two of them fail even with the correct key if the format is wrong):",{"type":75,"tag":2194,"props":3746,"children":3747},{},[3748,3772],{"type":75,"tag":2198,"props":3749,"children":3750},{},[3751],{"type":75,"tag":2202,"props":3752,"children":3753},{},[3754,3757,3762,3767],{"type":75,"tag":2206,"props":3755,"children":3756},{},[],{"type":75,"tag":2206,"props":3758,"children":3759},{},[3760],{"type":81,"value":3761},"Control Plane Pull Secrets",{"type":75,"tag":2206,"props":3763,"children":3764},{},[3765],{"type":81,"value":3766},"API Bootstrap Registry Creds",{"type":75,"tag":2206,"props":3768,"children":3769},{},[3770],{"type":81,"value":3771},"Sidecar Image-Pull Secret",{"type":75,"tag":2227,"props":3773,"children":3774},{},[3775,3803,3870],{"type":75,"tag":2202,"props":3776,"children":3777},{},[3778,3783,3788,3798],{"type":75,"tag":2234,"props":3779,"children":3780},{},[3781],{"type":81,"value":3782},"Purpose",{"type":75,"tag":2234,"props":3784,"children":3785},{},[3786],{"type":81,"value":3787},"K8s pulls NVCF service images",{"type":75,"tag":2234,"props":3789,"children":3790},{},[3791,3793],{"type":81,"value":3792},"nvcf-api validates the user function image at ",{"type":75,"tag":102,"props":3794,"children":3796},{"className":3795},[],[3797],{"type":81,"value":1832},{"type":75,"tag":2234,"props":3799,"children":3800},{},[3801],{"type":81,"value":3802},"nvcf-api pulls the platform sidecars injected into every worker pod",{"type":75,"tag":2202,"props":3804,"children":3805},{},[3806,3811,3824,3835],{"type":75,"tag":2234,"props":3807,"children":3808},{},[3809],{"type":81,"value":3810},"Where",{"type":75,"tag":2234,"props":3812,"children":3813},{},[3814,3816,3822],{"type":81,"value":3815},"K8s ",{"type":75,"tag":102,"props":3817,"children":3819},{"className":3818},[],[3820],{"type":81,"value":3821},"docker-registry",{"type":81,"value":3823}," Secrets + Kyverno ClusterPolicy",{"type":75,"tag":2234,"props":3825,"children":3826},{},[3827,3833],{"type":75,"tag":102,"props":3828,"children":3830},{"className":3829},[],[3831],{"type":81,"value":3832},"\u003Cprivate-values>\u002F\u003Cenv>-secrets.yaml",{"type":81,"value":3834}," (account-bootstrap)",{"type":75,"tag":2234,"props":3836,"children":3837},{},[3838,3844,3846,3852,3854,3860,3862,3868],{"type":75,"tag":102,"props":3839,"children":3841},{"className":3840},[],[3842],{"type":81,"value":3843},"NVCF_API_SIDECARS_IMAGE_PULL_SECRET",{"type":81,"value":3845}," in ",{"type":75,"tag":102,"props":3847,"children":3849},{"className":3848},[],[3850],{"type":81,"value":3851},"\u003Cenv>-secrets.yaml",{"type":81,"value":3853}," -> OpenBao ",{"type":75,"tag":102,"props":3855,"children":3857},{"className":3856},[],[3858],{"type":81,"value":3859},"services\u002Fnvcf-api\u002Fkv\u002Fsidecars\u002Fimage-pull-secret",{"type":81,"value":3861}," (field ",{"type":75,"tag":102,"props":3863,"children":3865},{"className":3864},[],[3866],{"type":81,"value":3867},"secret",{"type":81,"value":3869},")",{"type":75,"tag":2202,"props":3871,"children":3872},{},[3873,3878,3891,3900],{"type":75,"tag":2234,"props":3874,"children":3875},{},[3876],{"type":81,"value":3877},"Format",{"type":75,"tag":2234,"props":3879,"children":3880},{},[3881,3883,3889],{"type":81,"value":3882},"standard ",{"type":75,"tag":102,"props":3884,"children":3886},{"className":3885},[],[3887],{"type":81,"value":3888},".dockerconfigjson",{"type":81,"value":3890}," (kubectl builds it for you)",{"type":75,"tag":2234,"props":3892,"children":3893},{},[3894],{"type":75,"tag":102,"props":3895,"children":3897},{"className":3896},[],[3898],{"type":81,"value":3899},"base64(\"$oauthtoken:\u003Ckey>\")",{"type":75,"tag":2234,"props":3901,"children":3902},{},[3903],{"type":75,"tag":102,"props":3904,"children":3906},{"className":3905},[],[3907],{"type":81,"value":3899},{"type":75,"tag":1476,"props":3909,"children":3910},{},[3911,3961],{"type":75,"tag":84,"props":3912,"children":3913},{},[3914,3916,3922,3924,3930,3932,3937,3939,3944,3946,3952,3954,3960],{"type":81,"value":3915},"Note: the account-bootstrap cred and the sidecar secret are NOT a\ndockerconfigjson. The secrets template ships the placeholder\n",{"type":75,"tag":102,"props":3917,"children":3919},{"className":3918},[],[3920],{"type":81,"value":3921},"REPLACE_WITH_BASE64_DOCKER_CREDENTIAL",{"type":81,"value":3923},", which reads as \"base64 of a dockerconfigjson.\"\nIt isn't. Both values are consumed as a raw docker ",{"type":75,"tag":102,"props":3925,"children":3927},{"className":3926},[],[3928],{"type":81,"value":3929},"auth",{"type":81,"value":3931}," string: nvcf-api takes the\nvalue verbatim and drops it into the ",{"type":75,"tag":102,"props":3933,"children":3935},{"className":3934},[],[3936],{"type":81,"value":3929},{"type":81,"value":3938}," field of the dockerconfig it builds. So the\nvalue must be ",{"type":75,"tag":102,"props":3940,"children":3942},{"className":3941},[],[3943],{"type":81,"value":3899},{"type":81,"value":3945},", e.g. ",{"type":75,"tag":102,"props":3947,"children":3949},{"className":3948},[],[3950],{"type":81,"value":3951},"printf '$oauthtoken:%s' \"$KEY\" | openssl base64 -A",{"type":81,"value":3953},",\nnot ",{"type":75,"tag":102,"props":3955,"children":3957},{"className":3956},[],[3958],{"type":81,"value":3959},"base64(\u003Cdockerconfigjson>)",{"type":81,"value":134},{"type":75,"tag":2825,"props":3962,"children":3963},{},[3964,3983],{"type":75,"tag":1554,"props":3965,"children":3966},{},[3967,3969,3974,3976,3982],{"type":81,"value":3968},"Wrong account-bootstrap value: ",{"type":75,"tag":102,"props":3970,"children":3972},{"className":3971},[],[3973],{"type":81,"value":1832},{"type":81,"value":3975}," returns\n",{"type":75,"tag":102,"props":3977,"children":3979},{"className":3978},[],[3980],{"type":81,"value":3981},"400 \"must be base64 encoded username:password format\"",{"type":81,"value":134},{"type":75,"tag":1554,"props":3984,"children":3985},{},[3986,3988,3994,3996],{"type":81,"value":3987},"Wrong sidecar value: every worker's sidecar pull gets a malformed credential and\n",{"type":75,"tag":102,"props":3989,"children":3991},{"className":3990},[],[3992],{"type":81,"value":3993},"403",{"type":81,"value":3995},"s, regardless of which key is inside. The symptom looks like a bad or wrong-org\nkey, but the key is fine and the encoding is the bug. To repair an already-installed\nstack without a full re-sync:\n",{"type":75,"tag":161,"props":3997,"children":3999},{"className":163,"code":3998,"language":65,"meta":165,"style":165},"AUTH=$(printf '$oauthtoken:%s' \"$KEY\" | openssl base64 -A)\nbao kv put services\u002Fnvcf-api\u002Fkv\u002Fsidecars\u002Fimage-pull-secret secret=\"$AUTH\"\n# if NVIDIA Cloud Tasks is enabled, repeat for services\u002Fnvct-api\u002Fkv\u002Fsidecars\u002Fimage-pull-secret\nkubectl rollout restart -n nvcf deploy\u002Fnvcf-api   # picks up the new sidecar cred\n",[4000],{"type":75,"tag":102,"props":4001,"children":4002},{"__ignoreMap":165},[4003,4068,4109,4117],{"type":75,"tag":171,"props":4004,"children":4005},{"class":173,"line":174},[4006,4011,4015,4020,4024,4029,4033,4037,4042,4046,4050,4055,4060,4064],{"type":75,"tag":171,"props":4007,"children":4008},{"style":196},[4009],{"type":81,"value":4010},"AUTH",{"type":75,"tag":171,"props":4012,"children":4013},{"style":184},[4014],{"type":81,"value":1879},{"type":75,"tag":171,"props":4016,"children":4017},{"style":606},[4018],{"type":81,"value":4019},"printf",{"type":75,"tag":171,"props":4021,"children":4022},{"style":184},[4023],{"type":81,"value":1260},{"type":75,"tag":171,"props":4025,"children":4026},{"style":190},[4027],{"type":81,"value":4028},"$oauthtoken:%s",{"type":75,"tag":171,"props":4030,"children":4031},{"style":184},[4032],{"type":81,"value":1069},{"type":75,"tag":171,"props":4034,"children":4035},{"style":184},[4036],{"type":81,"value":844},{"type":75,"tag":171,"props":4038,"children":4039},{"style":196},[4040],{"type":81,"value":4041},"$KEY",{"type":75,"tag":171,"props":4043,"children":4044},{"style":184},[4045],{"type":81,"value":854},{"type":75,"tag":171,"props":4047,"children":4048},{"style":184},[4049],{"type":81,"value":874},{"type":75,"tag":171,"props":4051,"children":4052},{"style":178},[4053],{"type":81,"value":4054}," openssl",{"type":75,"tag":171,"props":4056,"children":4057},{"style":190},[4058],{"type":81,"value":4059}," base64",{"type":75,"tag":171,"props":4061,"children":4062},{"style":190},[4063],{"type":81,"value":2369},{"type":75,"tag":171,"props":4065,"children":4066},{"style":184},[4067],{"type":81,"value":1960},{"type":75,"tag":171,"props":4069,"children":4070},{"class":173,"line":275},[4071,4076,4081,4086,4091,4096,4100,4105],{"type":75,"tag":171,"props":4072,"children":4073},{"style":178},[4074],{"type":81,"value":4075},"bao",{"type":75,"tag":171,"props":4077,"children":4078},{"style":190},[4079],{"type":81,"value":4080}," kv",{"type":75,"tag":171,"props":4082,"children":4083},{"style":190},[4084],{"type":81,"value":4085}," put",{"type":75,"tag":171,"props":4087,"children":4088},{"style":190},[4089],{"type":81,"value":4090}," services\u002Fnvcf-api\u002Fkv\u002Fsidecars\u002Fimage-pull-secret",{"type":75,"tag":171,"props":4092,"children":4093},{"style":190},[4094],{"type":81,"value":4095}," secret=",{"type":75,"tag":171,"props":4097,"children":4098},{"style":184},[4099],{"type":81,"value":854},{"type":75,"tag":171,"props":4101,"children":4102},{"style":196},[4103],{"type":81,"value":4104},"$AUTH",{"type":75,"tag":171,"props":4106,"children":4107},{"style":184},[4108],{"type":81,"value":954},{"type":75,"tag":171,"props":4110,"children":4111},{"class":173,"line":543},[4112],{"type":75,"tag":171,"props":4113,"children":4114},{"style":520},[4115],{"type":81,"value":4116},"# if NVIDIA Cloud Tasks is enabled, repeat for services\u002Fnvct-api\u002Fkv\u002Fsidecars\u002Fimage-pull-secret\n",{"type":75,"tag":171,"props":4118,"children":4119},{"class":173,"line":566},[4120,4124,4129,4134,4138,4142,4147],{"type":75,"tag":171,"props":4121,"children":4122},{"style":178},[4123],{"type":81,"value":69},{"type":75,"tag":171,"props":4125,"children":4126},{"style":190},[4127],{"type":81,"value":4128}," rollout",{"type":75,"tag":171,"props":4130,"children":4131},{"style":190},[4132],{"type":81,"value":4133}," restart",{"type":75,"tag":171,"props":4135,"children":4136},{"style":190},[4137],{"type":81,"value":1916},{"type":75,"tag":171,"props":4139,"children":4140},{"style":190},[4141],{"type":81,"value":786},{"type":75,"tag":171,"props":4143,"children":4144},{"style":190},[4145],{"type":81,"value":4146}," deploy\u002Fnvcf-api",{"type":75,"tag":171,"props":4148,"children":4149},{"style":520},[4150],{"type":81,"value":4151},"   # picks up the new sidecar cred\n",{"type":75,"tag":1476,"props":4153,"children":4154},{},[4155],{"type":75,"tag":84,"props":4156,"children":4157},{},[4158],{"type":81,"value":4159},"Note (multi-org): when pulling from NGC (not a mirror), the user function image and\nthe platform sidecars often live in different nvcr.io orgs, each needing a different\nkey, so the account-bootstrap cred (function image) and the sidecar secret (sidecars)\nare frequently two different keys. Mirroring everything into a single registry the\ncluster authenticates to automatically collapses this to one credential path.",{"type":75,"tag":740,"props":4161,"children":4163},{"id":4162},"configuring-with-kyverno-recommended",[4164],{"type":81,"value":4165},"Configuring with Kyverno (recommended)",{"type":75,"tag":84,"props":4167,"children":4168},{},[4169,4171,4177],{"type":81,"value":4170},"Use a Kyverno mutating admission policy to automatically inject ",{"type":75,"tag":102,"props":4172,"children":4174},{"className":4173},[],[4175],{"type":81,"value":4176},"imagePullSecrets",{"type":81,"value":4178}," into all pods in NVCF namespaces. This works uniformly for all charts -- no per-chart configuration or helmfile modifications needed.",{"type":75,"tag":161,"props":4180,"children":4182},{"className":163,"code":4181,"language":65,"meta":165,"style":165},"# 1. Install Kyverno\nhelm repo add kyverno https:\u002F\u002Fkyverno.github.io\u002Fkyverno\u002F\nhelm repo update\nhelm install kyverno kyverno\u002Fkyverno -n kyverno --create-namespace\n\n# 2. Create pull secret in each namespace\nexport NGC_API_KEY=\"\u003Cyour-key>\"\nfor ns in cassandra-system nats-system nvcf api-keys ess sis vault-system cert-manager; do\n  kubectl create secret docker-registry nvcr-creds \\\n    --docker-server=nvcr.io \\\n    --docker-username='$oauthtoken' \\\n    --docker-password=\"$NGC_API_KEY\" \\\n    --namespace=\"$ns\" \\\n    --dry-run=client -o yaml | kubectl apply -f -\ndone\n\n# 3. Apply the Kyverno ClusterPolicy\nkubectl apply -f kyverno-imagepullsecret-policy.yaml\n",[4183],{"type":75,"tag":102,"props":4184,"children":4185},{"__ignoreMap":165},[4186,4194,4221,4237,4271,4278,4286,4313,4368,4395,4406,4429,4452,4475,4510,4517,4524,4533],{"type":75,"tag":171,"props":4187,"children":4188},{"class":173,"line":174},[4189],{"type":75,"tag":171,"props":4190,"children":4191},{"style":520},[4192],{"type":81,"value":4193},"# 1. Install Kyverno\n",{"type":75,"tag":171,"props":4195,"children":4196},{"class":173,"line":275},[4197,4201,4206,4211,4216],{"type":75,"tag":171,"props":4198,"children":4199},{"style":178},[4200],{"type":81,"value":68},{"type":75,"tag":171,"props":4202,"children":4203},{"style":190},[4204],{"type":81,"value":4205}," repo",{"type":75,"tag":171,"props":4207,"children":4208},{"style":190},[4209],{"type":81,"value":4210}," add",{"type":75,"tag":171,"props":4212,"children":4213},{"style":190},[4214],{"type":81,"value":4215}," kyverno",{"type":75,"tag":171,"props":4217,"children":4218},{"style":190},[4219],{"type":81,"value":4220}," https:\u002F\u002Fkyverno.github.io\u002Fkyverno\u002F\n",{"type":75,"tag":171,"props":4222,"children":4223},{"class":173,"line":543},[4224,4228,4232],{"type":75,"tag":171,"props":4225,"children":4226},{"style":178},[4227],{"type":81,"value":68},{"type":75,"tag":171,"props":4229,"children":4230},{"style":190},[4231],{"type":81,"value":4205},{"type":75,"tag":171,"props":4233,"children":4234},{"style":190},[4235],{"type":81,"value":4236}," update\n",{"type":75,"tag":171,"props":4238,"children":4239},{"class":173,"line":566},[4240,4244,4249,4253,4258,4262,4266],{"type":75,"tag":171,"props":4241,"children":4242},{"style":178},[4243],{"type":81,"value":68},{"type":75,"tag":171,"props":4245,"children":4246},{"style":190},[4247],{"type":81,"value":4248}," install",{"type":75,"tag":171,"props":4250,"children":4251},{"style":190},[4252],{"type":81,"value":4215},{"type":75,"tag":171,"props":4254,"children":4255},{"style":190},[4256],{"type":81,"value":4257}," kyverno\u002Fkyverno",{"type":75,"tag":171,"props":4259,"children":4260},{"style":190},[4261],{"type":81,"value":1916},{"type":75,"tag":171,"props":4263,"children":4264},{"style":190},[4265],{"type":81,"value":4215},{"type":75,"tag":171,"props":4267,"children":4268},{"style":190},[4269],{"type":81,"value":4270}," --create-namespace\n",{"type":75,"tag":171,"props":4272,"children":4273},{"class":173,"line":914},[4274],{"type":75,"tag":171,"props":4275,"children":4276},{"emptyLinePlaceholder":908},[4277],{"type":81,"value":911},{"type":75,"tag":171,"props":4279,"children":4280},{"class":173,"line":923},[4281],{"type":75,"tag":171,"props":4282,"children":4283},{"style":520},[4284],{"type":81,"value":4285},"# 2. Create pull secret in each namespace\n",{"type":75,"tag":171,"props":4287,"children":4288},{"class":173,"line":957},[4289,4293,4297,4301,4305,4309],{"type":75,"tag":171,"props":4290,"children":4291},{"style":927},[4292],{"type":81,"value":930},{"type":75,"tag":171,"props":4294,"children":4295},{"style":196},[4296],{"type":81,"value":935},{"type":75,"tag":171,"props":4298,"children":4299},{"style":184},[4300],{"type":81,"value":940},{"type":75,"tag":171,"props":4302,"children":4303},{"style":184},[4304],{"type":81,"value":854},{"type":75,"tag":171,"props":4306,"children":4307},{"style":190},[4308],{"type":81,"value":949},{"type":75,"tag":171,"props":4310,"children":4311},{"style":184},[4312],{"type":81,"value":954},{"type":75,"tag":171,"props":4314,"children":4315},{"class":173,"line":1013},[4316,4320,4324,4328,4332,4336,4340,4344,4348,4352,4356,4360,4364],{"type":75,"tag":171,"props":4317,"children":4318},{"style":758},[4319],{"type":81,"value":761},{"type":75,"tag":171,"props":4321,"children":4322},{"style":196},[4323],{"type":81,"value":766},{"type":75,"tag":171,"props":4325,"children":4326},{"style":758},[4327],{"type":81,"value":771},{"type":75,"tag":171,"props":4329,"children":4330},{"style":190},[4331],{"type":81,"value":776},{"type":75,"tag":171,"props":4333,"children":4334},{"style":190},[4335],{"type":81,"value":781},{"type":75,"tag":171,"props":4337,"children":4338},{"style":190},[4339],{"type":81,"value":786},{"type":75,"tag":171,"props":4341,"children":4342},{"style":190},[4343],{"type":81,"value":791},{"type":75,"tag":171,"props":4345,"children":4346},{"style":190},[4347],{"type":81,"value":796},{"type":75,"tag":171,"props":4349,"children":4350},{"style":190},[4351],{"type":81,"value":801},{"type":75,"tag":171,"props":4353,"children":4354},{"style":190},[4355],{"type":81,"value":806},{"type":75,"tag":171,"props":4357,"children":4358},{"style":190},[4359],{"type":81,"value":811},{"type":75,"tag":171,"props":4361,"children":4362},{"style":184},[4363],{"type":81,"value":816},{"type":75,"tag":171,"props":4365,"children":4366},{"style":758},[4367],{"type":81,"value":821},{"type":75,"tag":171,"props":4369,"children":4370},{"class":173,"line":1045},[4371,4375,4379,4383,4387,4391],{"type":75,"tag":171,"props":4372,"children":4373},{"style":178},[4374],{"type":81,"value":829},{"type":75,"tag":171,"props":4376,"children":4377},{"style":190},[4378],{"type":81,"value":834},{"type":75,"tag":171,"props":4380,"children":4381},{"style":190},[4382],{"type":81,"value":1027},{"type":75,"tag":171,"props":4384,"children":4385},{"style":190},[4386],{"type":81,"value":1032},{"type":75,"tag":171,"props":4388,"children":4389},{"style":190},[4390],{"type":81,"value":1037},{"type":75,"tag":171,"props":4392,"children":4393},{"style":196},[4394],{"type":81,"value":1042},{"type":75,"tag":171,"props":4396,"children":4397},{"class":173,"line":1058},[4398,4402],{"type":75,"tag":171,"props":4399,"children":4400},{"style":190},[4401],{"type":81,"value":1051},{"type":75,"tag":171,"props":4403,"children":4404},{"style":196},[4405],{"type":81,"value":1042},{"type":75,"tag":171,"props":4407,"children":4408},{"class":173,"line":1085},[4409,4413,4417,4421,4425],{"type":75,"tag":171,"props":4410,"children":4411},{"style":190},[4412],{"type":81,"value":1064},{"type":75,"tag":171,"props":4414,"children":4415},{"style":184},[4416],{"type":81,"value":1069},{"type":75,"tag":171,"props":4418,"children":4419},{"style":190},[4420],{"type":81,"value":1074},{"type":75,"tag":171,"props":4422,"children":4423},{"style":184},[4424],{"type":81,"value":1069},{"type":75,"tag":171,"props":4426,"children":4427},{"style":196},[4428],{"type":81,"value":1042},{"type":75,"tag":171,"props":4430,"children":4431},{"class":173,"line":1111},[4432,4436,4440,4444,4448],{"type":75,"tag":171,"props":4433,"children":4434},{"style":190},[4435],{"type":81,"value":1091},{"type":75,"tag":171,"props":4437,"children":4438},{"style":184},[4439],{"type":81,"value":854},{"type":75,"tag":171,"props":4441,"children":4442},{"style":196},[4443],{"type":81,"value":1100},{"type":75,"tag":171,"props":4445,"children":4446},{"style":184},[4447],{"type":81,"value":854},{"type":75,"tag":171,"props":4449,"children":4450},{"style":196},[4451],{"type":81,"value":1042},{"type":75,"tag":171,"props":4453,"children":4454},{"class":173,"line":1136},[4455,4459,4463,4467,4471],{"type":75,"tag":171,"props":4456,"children":4457},{"style":190},[4458],{"type":81,"value":1117},{"type":75,"tag":171,"props":4460,"children":4461},{"style":184},[4462],{"type":81,"value":854},{"type":75,"tag":171,"props":4464,"children":4465},{"style":196},[4466],{"type":81,"value":849},{"type":75,"tag":171,"props":4468,"children":4469},{"style":184},[4470],{"type":81,"value":854},{"type":75,"tag":171,"props":4472,"children":4473},{"style":196},[4474],{"type":81,"value":1042},{"type":75,"tag":171,"props":4476,"children":4477},{"class":173,"line":1173},[4478,4482,4486,4490,4494,4498,4502,4506],{"type":75,"tag":171,"props":4479,"children":4480},{"style":190},[4481],{"type":81,"value":1142},{"type":75,"tag":171,"props":4483,"children":4484},{"style":190},[4485],{"type":81,"value":864},{"type":75,"tag":171,"props":4487,"children":4488},{"style":190},[4489],{"type":81,"value":869},{"type":75,"tag":171,"props":4491,"children":4492},{"style":184},[4493],{"type":81,"value":874},{"type":75,"tag":171,"props":4495,"children":4496},{"style":178},[4497],{"type":81,"value":879},{"type":75,"tag":171,"props":4499,"children":4500},{"style":190},[4501],{"type":81,"value":884},{"type":75,"tag":171,"props":4503,"children":4504},{"style":190},[4505],{"type":81,"value":889},{"type":75,"tag":171,"props":4507,"children":4508},{"style":190},[4509],{"type":81,"value":894},{"type":75,"tag":171,"props":4511,"children":4512},{"class":173,"line":3566},[4513],{"type":75,"tag":171,"props":4514,"children":4515},{"style":758},[4516],{"type":81,"value":902},{"type":75,"tag":171,"props":4518,"children":4519},{"class":173,"line":3590},[4520],{"type":75,"tag":171,"props":4521,"children":4522},{"emptyLinePlaceholder":908},[4523],{"type":81,"value":911},{"type":75,"tag":171,"props":4525,"children":4527},{"class":173,"line":4526},17,[4528],{"type":75,"tag":171,"props":4529,"children":4530},{"style":520},[4531],{"type":81,"value":4532},"# 3. Apply the Kyverno ClusterPolicy\n",{"type":75,"tag":171,"props":4534,"children":4536},{"class":173,"line":4535},18,[4537,4541,4545,4549],{"type":75,"tag":171,"props":4538,"children":4539},{"style":178},[4540],{"type":81,"value":69},{"type":75,"tag":171,"props":4542,"children":4543},{"style":190},[4544],{"type":81,"value":884},{"type":75,"tag":171,"props":4546,"children":4547},{"style":190},[4548],{"type":81,"value":889},{"type":75,"tag":171,"props":4550,"children":4551},{"style":190},[4552],{"type":81,"value":4553}," kyverno-imagepullsecret-policy.yaml\n",{"type":75,"tag":84,"props":4555,"children":4556},{},[4557,4559,4565],{"type":81,"value":4558},"The policy mutates every pod at admission time, adding ",{"type":75,"tag":102,"props":4560,"children":4562},{"className":4561},[],[4563],{"type":81,"value":4564},"imagePullSecrets: [{name: nvcr-creds}]",{"type":81,"value":4566},". Verify with:",{"type":75,"tag":161,"props":4568,"children":4570},{"className":163,"code":4569,"language":65,"meta":165,"style":165},"kubectl get pod -n \u003Cnamespace> \u003Cpod-name> -o jsonpath='{.spec.imagePullSecrets}'\n# Should show: [{\"name\":\"nvcr-creds\"}]\n",[4571],{"type":75,"tag":102,"props":4572,"children":4573},{"__ignoreMap":165},[4574,4649],{"type":75,"tag":171,"props":4575,"children":4576},{"class":173,"line":174},[4577,4581,4585,4590,4594,4598,4603,4607,4611,4615,4620,4624,4628,4632,4636,4640,4645],{"type":75,"tag":171,"props":4578,"children":4579},{"style":178},[4580],{"type":81,"value":69},{"type":75,"tag":171,"props":4582,"children":4583},{"style":190},[4584],{"type":81,"value":1888},{"type":75,"tag":171,"props":4586,"children":4587},{"style":190},[4588],{"type":81,"value":4589}," pod",{"type":75,"tag":171,"props":4591,"children":4592},{"style":190},[4593],{"type":81,"value":1916},{"type":75,"tag":171,"props":4595,"children":4596},{"style":184},[4597],{"type":81,"value":187},{"type":75,"tag":171,"props":4599,"children":4600},{"style":190},[4601],{"type":81,"value":4602},"namespac",{"type":75,"tag":171,"props":4604,"children":4605},{"style":196},[4606],{"type":81,"value":1907},{"type":75,"tag":171,"props":4608,"children":4609},{"style":184},[4610],{"type":81,"value":204},{"type":75,"tag":171,"props":4612,"children":4613},{"style":184},[4614],{"type":81,"value":187},{"type":75,"tag":171,"props":4616,"children":4617},{"style":190},[4618],{"type":81,"value":4619},"pod-nam",{"type":75,"tag":171,"props":4621,"children":4622},{"style":196},[4623],{"type":81,"value":1907},{"type":75,"tag":171,"props":4625,"children":4626},{"style":184},[4627],{"type":81,"value":204},{"type":75,"tag":171,"props":4629,"children":4630},{"style":190},[4631],{"type":81,"value":864},{"type":75,"tag":171,"props":4633,"children":4634},{"style":190},[4635],{"type":81,"value":1942},{"type":75,"tag":171,"props":4637,"children":4638},{"style":184},[4639],{"type":81,"value":1069},{"type":75,"tag":171,"props":4641,"children":4642},{"style":190},[4643],{"type":81,"value":4644},"{.spec.imagePullSecrets}",{"type":75,"tag":171,"props":4646,"children":4647},{"style":184},[4648],{"type":81,"value":3238},{"type":75,"tag":171,"props":4650,"children":4651},{"class":173,"line":275},[4652],{"type":75,"tag":171,"props":4653,"children":4654},{"style":520},[4655],{"type":81,"value":4656},"# Should show: [{\"name\":\"nvcr-creds\"}]\n",{"type":75,"tag":84,"props":4658,"children":4659},{},[4660],{"type":81,"value":4661},"Not needed if using a CSP built-in credential helper (e.g., ECR with IAM node roles).",{"type":75,"tag":84,"props":4663,"children":4664},{},[4665,4667,4672],{"type":81,"value":4666},"For the Kyverno policy YAML and pull secret creation script, see ",{"type":75,"tag":127,"props":4668,"children":4670},{"href":4669},"references\u002Fpull-secrets.md",[4671],{"type":81,"value":4669},{"type":81,"value":134},{"type":75,"tag":90,"props":4674,"children":4676},{"id":4675},"debugging",[4677],{"type":81,"value":4678},"Debugging",{"type":75,"tag":740,"props":4680,"children":4682},{"id":4681},"quick-status-check",[4683],{"type":81,"value":4684},"Quick status check",{"type":75,"tag":161,"props":4686,"children":4688},{"className":163,"code":4687,"language":65,"meta":165,"style":165},"kubectl get pods -A -o wide                        # All pods\nhelm list -A                                        # All helm releases\nkubectl get events -n \u003Cns> --sort-by='.lastTimestamp'  # Recent events\n",[4689],{"type":75,"tag":102,"props":4690,"children":4691},{"__ignoreMap":165},[4692,4725,4745],{"type":75,"tag":171,"props":4693,"children":4694},{"class":173,"line":174},[4695,4699,4703,4707,4711,4715,4720],{"type":75,"tag":171,"props":4696,"children":4697},{"style":178},[4698],{"type":81,"value":69},{"type":75,"tag":171,"props":4700,"children":4701},{"style":190},[4702],{"type":81,"value":1888},{"type":75,"tag":171,"props":4704,"children":4705},{"style":190},[4706],{"type":81,"value":2364},{"type":75,"tag":171,"props":4708,"children":4709},{"style":190},[4710],{"type":81,"value":2369},{"type":75,"tag":171,"props":4712,"children":4713},{"style":190},[4714],{"type":81,"value":864},{"type":75,"tag":171,"props":4716,"children":4717},{"style":190},[4718],{"type":81,"value":4719}," wide",{"type":75,"tag":171,"props":4721,"children":4722},{"style":520},[4723],{"type":81,"value":4724},"                        # All pods\n",{"type":75,"tag":171,"props":4726,"children":4727},{"class":173,"line":275},[4728,4732,4736,4740],{"type":75,"tag":171,"props":4729,"children":4730},{"style":178},[4731],{"type":81,"value":68},{"type":75,"tag":171,"props":4733,"children":4734},{"style":190},[4735],{"type":81,"value":558},{"type":75,"tag":171,"props":4737,"children":4738},{"style":190},[4739],{"type":81,"value":2369},{"type":75,"tag":171,"props":4741,"children":4742},{"style":520},[4743],{"type":81,"value":4744},"                                        # All helm releases\n",{"type":75,"tag":171,"props":4746,"children":4747},{"class":173,"line":543},[4748,4752,4756,4761,4765,4769,4773,4778,4782,4787,4791,4796,4800],{"type":75,"tag":171,"props":4749,"children":4750},{"style":178},[4751],{"type":81,"value":69},{"type":75,"tag":171,"props":4753,"children":4754},{"style":190},[4755],{"type":81,"value":1888},{"type":75,"tag":171,"props":4757,"children":4758},{"style":190},[4759],{"type":81,"value":4760}," events",{"type":75,"tag":171,"props":4762,"children":4763},{"style":190},[4764],{"type":81,"value":1916},{"type":75,"tag":171,"props":4766,"children":4767},{"style":184},[4768],{"type":81,"value":187},{"type":75,"tag":171,"props":4770,"children":4771},{"style":190},[4772],{"type":81,"value":414},{"type":75,"tag":171,"props":4774,"children":4775},{"style":196},[4776],{"type":81,"value":4777},"s",{"type":75,"tag":171,"props":4779,"children":4780},{"style":184},[4781],{"type":81,"value":204},{"type":75,"tag":171,"props":4783,"children":4784},{"style":190},[4785],{"type":81,"value":4786}," --sort-by=",{"type":75,"tag":171,"props":4788,"children":4789},{"style":184},[4790],{"type":81,"value":1069},{"type":75,"tag":171,"props":4792,"children":4793},{"style":190},[4794],{"type":81,"value":4795},".lastTimestamp",{"type":75,"tag":171,"props":4797,"children":4798},{"style":184},[4799],{"type":81,"value":1069},{"type":75,"tag":171,"props":4801,"children":4802},{"style":520},[4803],{"type":81,"value":4804},"  # Recent events\n",{"type":75,"tag":740,"props":4806,"children":4808},{"id":4807},"common-failure-patterns",[4809],{"type":81,"value":4810},"Common failure patterns",{"type":75,"tag":2194,"props":4812,"children":4813},{},[4814,4835],{"type":75,"tag":2198,"props":4815,"children":4816},{},[4817],{"type":75,"tag":2202,"props":4818,"children":4819},{},[4820,4825,4830],{"type":75,"tag":2206,"props":4821,"children":4822},{},[4823],{"type":81,"value":4824},"Symptom",{"type":75,"tag":2206,"props":4826,"children":4827},{},[4828],{"type":81,"value":4829},"Cause",{"type":75,"tag":2206,"props":4831,"children":4832},{},[4833],{"type":81,"value":4834},"Fix",{"type":75,"tag":2227,"props":4836,"children":4837},{},[4838,4868,4892,4974,5006,5062,5091,5131,5155,5192,5253],{"type":75,"tag":2202,"props":4839,"children":4840},{},[4841,4858,4863],{"type":75,"tag":2234,"props":4842,"children":4843},{},[4844,4850,4852],{"type":75,"tag":102,"props":4845,"children":4847},{"className":4846},[],[4848],{"type":81,"value":4849},"ImagePullBackOff",{"type":81,"value":4851}," + ",{"type":75,"tag":102,"props":4853,"children":4855},{"className":4854},[],[4856],{"type":81,"value":4857},"401 Unauthorized",{"type":75,"tag":2234,"props":4859,"children":4860},{},[4861],{"type":81,"value":4862},"Missing or wrong pull secret",{"type":75,"tag":2234,"props":4864,"children":4865},{},[4866],{"type":81,"value":4867},"Check secret exists, check SA has imagePullSecrets",{"type":75,"tag":2202,"props":4869,"children":4870},{},[4871,4882,4887],{"type":75,"tag":2234,"props":4872,"children":4873},{},[4874,4880],{"type":75,"tag":102,"props":4875,"children":4877},{"className":4876},[],[4878],{"type":81,"value":4879},"Init:0\u002F1",{"type":81,"value":4881}," stuck on service pods",{"type":75,"tag":2234,"props":4883,"children":4884},{},[4885],{"type":81,"value":4886},"Vault-agent waiting for OpenBao",{"type":75,"tag":2234,"props":4888,"children":4889},{},[4890],{"type":81,"value":4891},"Check OpenBao pods + migration job status",{"type":75,"tag":2202,"props":4893,"children":4894},{},[4895,4926,4939],{"type":75,"tag":2234,"props":4896,"children":4897},{},[4898,4903,4904,4910,4912,4918,4920],{"type":75,"tag":102,"props":4899,"children":4901},{"className":4900},[],[4902],{"type":81,"value":4879},{"type":81,"value":4851},{"type":75,"tag":102,"props":4905,"children":4907},{"className":4906},[],[4908],{"type":81,"value":4909},"vault-agent-init",{"type":81,"value":4911}," shows ",{"type":75,"tag":102,"props":4913,"children":4915},{"className":4914},[],[4916],{"type":81,"value":4917},"auth\u002Fjwt\u002Flogin",{"type":81,"value":4919}," -> ",{"type":75,"tag":102,"props":4921,"children":4923},{"className":4922},[],[4924],{"type":81,"value":4925},"400 ... no known key successfully validated the token signature",{"type":75,"tag":2234,"props":4927,"children":4928},{},[4929,4931,4937],{"type":81,"value":4930},"OpenBao JWT auth configured with a static pubkey instead of the cluster's live JWKS -- the base default ",{"type":75,"tag":102,"props":4932,"children":4934},{"className":4933},[],[4935],{"type":81,"value":4936},"openbao.migrations.issuerDiscovery.enabled: false",{"type":81,"value":4938},". Common on AKS \u002F any OIDC-issuer cluster",{"type":75,"tag":2234,"props":4940,"children":4941},{},[4942,4944,4950,4952,4958,4960,4965,4967,4972],{"type":81,"value":4943},"Set ",{"type":75,"tag":102,"props":4945,"children":4947},{"className":4946},[],[4948],{"type":81,"value":4949},"openbao.migrations.issuerDiscovery.enabled: true",{"type":81,"value":4951},", delete job ",{"type":75,"tag":102,"props":4953,"children":4955},{"className":4954},[],[4956],{"type":81,"value":4957},"openbao-server-migrations",{"type":81,"value":4959},", re-sync ",{"type":75,"tag":102,"props":4961,"children":4963},{"className":4962},[],[4964],{"type":81,"value":2884},{"type":81,"value":4966},". See ",{"type":75,"tag":127,"props":4968,"children":4970},{"href":4969},"references\u002Fdebugging.md",[4971],{"type":81,"value":4969},{"type":81,"value":4973}," \"Init:0\u002F1 Stuck\" fix #4.",{"type":75,"tag":2202,"props":4975,"children":4976},{},[4977,4988,4993],{"type":75,"tag":2234,"props":4978,"children":4979},{},[4980,4986],{"type":75,"tag":102,"props":4981,"children":4983},{"className":4982},[],[4984],{"type":81,"value":4985},"OOMKilled",{"type":81,"value":4987}," on Cassandra",{"type":75,"tag":2234,"props":4989,"children":4990},{},[4991],{"type":81,"value":4992},"Default resources too small",{"type":75,"tag":2234,"props":4994,"children":4995},{},[4996,4998,5004],{"type":81,"value":4997},"Override ",{"type":75,"tag":102,"props":4999,"children":5001},{"className":5000},[],[5002],{"type":81,"value":5003},"cassandra.resources",{"type":81,"value":5005}," via values block",{"type":75,"tag":2202,"props":5007,"children":5008},{},[5009,5021,5044],{"type":75,"tag":2234,"props":5010,"children":5011},{},[5012,5014,5019],{"type":81,"value":5013},"DB-backed services crash-loop ",{"type":75,"tag":102,"props":5015,"children":5017},{"className":5016},[],[5018],{"type":81,"value":2166},{"type":81,"value":5020}," connecting to Cassandra",{"type":75,"tag":2234,"props":5022,"children":5023},{},[5024,5026,5031,5032,5037,5039],{"type":81,"value":5025},"Secrets file set only ",{"type":75,"tag":102,"props":5027,"children":5029},{"className":5028},[],[5030],{"type":81,"value":2135},{"type":81,"value":2970},{"type":75,"tag":102,"props":5033,"children":5035},{"className":5034},[],[5036],{"type":81,"value":2143},{"type":81,"value":5038}," still defaults to ",{"type":75,"tag":102,"props":5040,"children":5042},{"className":5041},[],[5043],{"type":81,"value":2151},{"type":75,"tag":2234,"props":5045,"children":5046},{},[5047,5048,5053,5055,5060],{"type":81,"value":4943},{"type":75,"tag":102,"props":5049,"children":5051},{"className":5050},[],[5052],{"type":81,"value":2143},{"type":81,"value":5054}," in the secrets file equal to ",{"type":75,"tag":102,"props":5056,"children":5058},{"className":5057},[],[5059],{"type":81,"value":2135},{"type":81,"value":5061},", then re-sync cassandra + the DB-backed services",{"type":75,"tag":2202,"props":5063,"children":5064},{},[5065,5075,5080],{"type":75,"tag":2234,"props":5066,"children":5067},{},[5068,5074],{"type":75,"tag":102,"props":5069,"children":5071},{"className":5070},[],[5072],{"type":81,"value":5073},"Pending",{"type":81,"value":2364},{"type":75,"tag":2234,"props":5076,"children":5077},{},[5078],{"type":81,"value":5079},"Node selector mismatch or no storage class",{"type":75,"tag":2234,"props":5081,"children":5082},{},[5083,5089],{"type":75,"tag":102,"props":5084,"children":5086},{"className":5085},[],[5087],{"type":81,"value":5088},"kubectl describe pod",{"type":81,"value":5090},", check labels and storage",{"type":75,"tag":2202,"props":5092,"children":5093},{},[5094,5107,5112],{"type":75,"tag":2234,"props":5095,"children":5096},{},[5097,5099,5105],{"type":81,"value":5098},"Helm release in ",{"type":75,"tag":102,"props":5100,"children":5102},{"className":5101},[],[5103],{"type":81,"value":5104},"failed",{"type":81,"value":5106}," state",{"type":75,"tag":2234,"props":5108,"children":5109},{},[5110],{"type":81,"value":5111},"First install failed partway",{"type":75,"tag":2234,"props":5113,"children":5114},{},[5115,5121,5123,5129],{"type":75,"tag":102,"props":5116,"children":5118},{"className":5117},[],[5119],{"type":81,"value":5120},"helmfile destroy",{"type":81,"value":5122}," the release, then ",{"type":75,"tag":102,"props":5124,"children":5126},{"className":5125},[],[5127],{"type":81,"value":5128},"sync",{"type":81,"value":5130}," again",{"type":75,"tag":2202,"props":5132,"children":5133},{},[5134,5139,5144],{"type":75,"tag":2234,"props":5135,"children":5136},{},[5137],{"type":81,"value":5138},"Account bootstrap timeout",{"type":75,"tag":2234,"props":5140,"children":5141},{},[5142],{"type":81,"value":5143},"Wrong base64 credentials in secrets file",{"type":75,"tag":2234,"props":5145,"children":5146},{},[5147,5149],{"type":81,"value":5148},"Check ",{"type":75,"tag":102,"props":5150,"children":5152},{"className":5151},[],[5153],{"type":81,"value":5154},"kubectl logs job\u002Fnvcf-api-account-bootstrap -n nvcf",{"type":75,"tag":2202,"props":5156,"children":5157},{},[5158,5169,5181],{"type":75,"tag":2234,"props":5159,"children":5160},{},[5161,5163],{"type":81,"value":5162},"function \u002F account calls return ",{"type":75,"tag":102,"props":5164,"children":5166},{"className":5165},[],[5167],{"type":81,"value":5168},"404 Unknown client_id",{"type":75,"tag":2234,"props":5170,"children":5171},{},[5172,5174,5179],{"type":81,"value":5173},"The account-bootstrap post-install hook never ran: the ",{"type":75,"tag":102,"props":5175,"children":5177},{"className":5176},[],[5178],{"type":81,"value":2912},{"type":81,"value":5180}," release install failed, or was repaired with live patches instead of a clean re-sync, so the hook (and the account) was skipped",{"type":75,"tag":2234,"props":5182,"children":5183},{},[5184,5186],{"type":81,"value":5185},"Verify and re-run the hook; see ",{"type":75,"tag":127,"props":5187,"children":5189},{"href":5188},"#verify-the-account-bootstrap-ran",[5190],{"type":81,"value":5191},"Verify the account bootstrap ran",{"type":75,"tag":2202,"props":5193,"children":5194},{},[5195,5208,5233],{"type":75,"tag":2234,"props":5196,"children":5197},{},[5198,5200,5206],{"type":81,"value":5199},"Services fail to read vault secrets; ",{"type":75,"tag":102,"props":5201,"children":5203},{"className":5202},[],[5204],{"type":81,"value":5205},"secrets.json",{"type":81,"value":5207}," not found",{"type":75,"tag":2234,"props":5209,"children":5210},{},[5211,5213,5219,5220,5226,5228],{"type":81,"value":5212},"Vault path hardcoded to ",{"type":75,"tag":102,"props":5214,"children":5216},{"className":5215},[],[5217],{"type":81,"value":5218},"\u002Fhome\u002Fapp\u002Fvault\u002F",{"type":81,"value":3845},{"type":75,"tag":102,"props":5221,"children":5223},{"className":5222},[],[5224],{"type":81,"value":5225},"_helpers.tpl",{"type":81,"value":5227},"; the runtime resolves the mounted path relative to the working directory and drops the leading ",{"type":75,"tag":102,"props":5229,"children":5231},{"className":5230},[],[5232],{"type":81,"value":1755},{"type":75,"tag":2234,"props":5234,"children":5235},{},[5236,5237,5243,5245,5251],{"type":81,"value":4997},{"type":75,"tag":102,"props":5238,"children":5240},{"className":5239},[],[5241],{"type":81,"value":5242},"podAnnotations",{"type":81,"value":5244}," and set ",{"type":75,"tag":102,"props":5246,"children":5248},{"className":5247},[],[5249],{"type":81,"value":5250},"JAVA_TOOL_OPTIONS: \"-Duser.dir=\u002F\"",{"type":81,"value":5252}," in release values",{"type":75,"tag":2202,"props":5254,"children":5255},{},[5256,5261,5289],{"type":75,"tag":2234,"props":5257,"children":5258},{},[5259],{"type":81,"value":5260},"NATS connection fails at startup; placement tag mismatch",{"type":75,"tag":2234,"props":5262,"children":5263},{},[5264,5266,5272,5274,5280,5282,5288],{"type":81,"value":5265},"NATS server tags hardcoded to ",{"type":75,"tag":102,"props":5267,"children":5269},{"className":5268},[],[5270],{"type":81,"value":5271},"dc:ncp",{"type":81,"value":5273},"; app derives tag from ",{"type":75,"tag":102,"props":5275,"children":5277},{"className":5276},[],[5278],{"type":81,"value":5279},"AWS_REGION",{"type":81,"value":5281}," (e.g., ",{"type":75,"tag":102,"props":5283,"children":5285},{"className":5284},[],[5286],{"type":81,"value":5287},"us-gov-west-1",{"type":81,"value":3869},{"type":75,"tag":2234,"props":5290,"children":5291},{},[5292,5293,5299,5300,5306],{"type":81,"value":4943},{"type":75,"tag":102,"props":5294,"children":5296},{"className":5295},[],[5297],{"type":81,"value":5298},"AWS_REGION=ncp",{"type":81,"value":109},{"type":75,"tag":102,"props":5301,"children":5303},{"className":5302},[],[5304],{"type":81,"value":5305},"NVCF_AWS_REGION=ncp",{"type":81,"value":5307}," in env config",{"type":75,"tag":740,"props":5309,"children":5311},{"id":5310},"verify-the-account-bootstrap-ran",[5312],{"type":81,"value":5191},{"type":75,"tag":84,"props":5314,"children":5315},{},[5316,5318,5337,5339,5345,5346,5351,5353,5358,5360,5365],{"type":81,"value":5317},"The default account is created by a ",{"type":75,"tag":5319,"props":5320,"children":5321},"strong",{},[5322,5328,5330,5335],{"type":75,"tag":102,"props":5323,"children":5325},{"className":5324},[],[5326],{"type":81,"value":5327},"post-install",{"type":81,"value":5329}," hook on the ",{"type":75,"tag":102,"props":5331,"children":5333},{"className":5332},[],[5334],{"type":81,"value":2912},{"type":81,"value":5336}," release",{"type":81,"value":5338},"\n(",{"type":75,"tag":102,"props":5340,"children":5342},{"className":5341},[],[5343],{"type":81,"value":5344},"job\u002Fnvcf-api-account-bootstrap",{"type":81,"value":3845},{"type":75,"tag":102,"props":5347,"children":5349},{"className":5348},[],[5350],{"type":81,"value":45},{"type":81,"value":5352},"). Because it is a post-install hook, it only\nfires on a successful ",{"type":75,"tag":102,"props":5354,"children":5356},{"className":5355},[],[5357],{"type":81,"value":2912},{"type":81,"value":5359}," install. If that install failed and you repaired it with live\npatches (rather than a clean re-sync), the hook never runs, no account is created, and\nevery subsequent function\u002Faccount call returns a cryptic ",{"type":75,"tag":102,"props":5361,"children":5363},{"className":5362},[],[5364],{"type":81,"value":5168},{"type":81,"value":5366}," with\nnothing obviously wrong in the running pods.",{"type":75,"tag":84,"props":5368,"children":5369},{},[5370],{"type":81,"value":5371},"Verify it ran after the first sync:",{"type":75,"tag":161,"props":5373,"children":5375},{"className":163,"code":5374,"language":65,"meta":165,"style":165},"kubectl get job -n nvcf nvcf-api-account-bootstrap \\\n  -o jsonpath='{.status.succeeded}'   # expect 1\n",[5376],{"type":75,"tag":102,"props":5377,"children":5378},{"__ignoreMap":165},[5379,5412],{"type":75,"tag":171,"props":5380,"children":5381},{"class":173,"line":174},[5382,5386,5390,5395,5399,5403,5408],{"type":75,"tag":171,"props":5383,"children":5384},{"style":178},[5385],{"type":81,"value":69},{"type":75,"tag":171,"props":5387,"children":5388},{"style":190},[5389],{"type":81,"value":1888},{"type":75,"tag":171,"props":5391,"children":5392},{"style":190},[5393],{"type":81,"value":5394}," job",{"type":75,"tag":171,"props":5396,"children":5397},{"style":190},[5398],{"type":81,"value":1916},{"type":75,"tag":171,"props":5400,"children":5401},{"style":190},[5402],{"type":81,"value":786},{"type":75,"tag":171,"props":5404,"children":5405},{"style":190},[5406],{"type":81,"value":5407}," nvcf-api-account-bootstrap",{"type":75,"tag":171,"props":5409,"children":5410},{"style":196},[5411],{"type":81,"value":1042},{"type":75,"tag":171,"props":5413,"children":5414},{"class":173,"line":275},[5415,5420,5424,5428,5433,5437],{"type":75,"tag":171,"props":5416,"children":5417},{"style":190},[5418],{"type":81,"value":5419},"  -o",{"type":75,"tag":171,"props":5421,"children":5422},{"style":190},[5423],{"type":81,"value":1942},{"type":75,"tag":171,"props":5425,"children":5426},{"style":184},[5427],{"type":81,"value":1069},{"type":75,"tag":171,"props":5429,"children":5430},{"style":190},[5431],{"type":81,"value":5432},"{.status.succeeded}",{"type":75,"tag":171,"props":5434,"children":5435},{"style":184},[5436],{"type":81,"value":1069},{"type":75,"tag":171,"props":5438,"children":5439},{"style":520},[5440],{"type":81,"value":5441},"   # expect 1\n",{"type":75,"tag":84,"props":5443,"children":5444},{},[5445,5447,5453,5455,5460],{"type":81,"value":5446},"If the job is absent or ",{"type":75,"tag":102,"props":5448,"children":5450},{"className":5449},[],[5451],{"type":81,"value":5452},"0",{"type":81,"value":5454},", re-run it by re-syncing the ",{"type":75,"tag":102,"props":5456,"children":5458},{"className":5457},[],[5459],{"type":81,"value":2912},{"type":81,"value":5461}," release so the hook fires\nagain (delete the prior job first if it lingers in a completed\u002Ffailed state and blocks the\nhook):",{"type":75,"tag":161,"props":5463,"children":5465},{"className":163,"code":5464,"language":65,"meta":165,"style":165},"kubectl delete job -n nvcf nvcf-api-account-bootstrap --ignore-not-found\nHELMFILE_ENV=\u003Cenv> helmfile --selector name=api sync\nkubectl logs -n nvcf job\u002Fnvcf-api-account-bootstrap   # confirm the account was created\n",[5466],{"type":75,"tag":102,"props":5467,"children":5468},{"__ignoreMap":165},[5469,5500,5536],{"type":75,"tag":171,"props":5470,"children":5471},{"class":173,"line":174},[5472,5476,5480,5484,5488,5492,5496],{"type":75,"tag":171,"props":5473,"children":5474},{"style":178},[5475],{"type":81,"value":69},{"type":75,"tag":171,"props":5477,"children":5478},{"style":190},[5479],{"type":81,"value":3154},{"type":75,"tag":171,"props":5481,"children":5482},{"style":190},[5483],{"type":81,"value":5394},{"type":75,"tag":171,"props":5485,"children":5486},{"style":190},[5487],{"type":81,"value":1916},{"type":75,"tag":171,"props":5489,"children":5490},{"style":190},[5491],{"type":81,"value":786},{"type":75,"tag":171,"props":5493,"children":5494},{"style":190},[5495],{"type":81,"value":5407},{"type":75,"tag":171,"props":5497,"children":5498},{"style":190},[5499],{"type":81,"value":3175},{"type":75,"tag":171,"props":5501,"children":5502},{"class":173,"line":275},[5503,5507,5511,5515,5519,5523,5527,5532],{"type":75,"tag":171,"props":5504,"children":5505},{"style":196},[5506],{"type":81,"value":2059},{"type":75,"tag":171,"props":5508,"children":5509},{"style":184},[5510],{"type":81,"value":2064},{"type":75,"tag":171,"props":5512,"children":5513},{"style":190},[5514],{"type":81,"value":3668},{"type":75,"tag":171,"props":5516,"children":5517},{"style":184},[5518],{"type":81,"value":204},{"type":75,"tag":171,"props":5520,"children":5521},{"style":178},[5522],{"type":81,"value":2078},{"type":75,"tag":171,"props":5524,"children":5525},{"style":190},[5526],{"type":81,"value":2562},{"type":75,"tag":171,"props":5528,"children":5529},{"style":190},[5530],{"type":81,"value":5531}," name=api",{"type":75,"tag":171,"props":5533,"children":5534},{"style":190},[5535],{"type":81,"value":2608},{"type":75,"tag":171,"props":5537,"children":5538},{"class":173,"line":543},[5539,5543,5548,5552,5556,5561],{"type":75,"tag":171,"props":5540,"children":5541},{"style":178},[5542],{"type":81,"value":69},{"type":75,"tag":171,"props":5544,"children":5545},{"style":190},[5546],{"type":81,"value":5547}," logs",{"type":75,"tag":171,"props":5549,"children":5550},{"style":190},[5551],{"type":81,"value":1916},{"type":75,"tag":171,"props":5553,"children":5554},{"style":190},[5555],{"type":81,"value":786},{"type":75,"tag":171,"props":5557,"children":5558},{"style":190},[5559],{"type":81,"value":5560}," job\u002Fnvcf-api-account-bootstrap",{"type":75,"tag":171,"props":5562,"children":5563},{"style":520},[5564],{"type":81,"value":5565},"   # confirm the account was created\n",{"type":75,"tag":90,"props":5567,"children":5569},{"id":5568},"additional-resources",[5570],{"type":81,"value":5571},"Additional Resources",{"type":75,"tag":2825,"props":5573,"children":5574},{},[5575,5597],{"type":75,"tag":1554,"props":5576,"children":5577},{},[5578,5580,5584,5586,5591,5592],{"type":81,"value":5579},"Installation: ",{"type":75,"tag":127,"props":5581,"children":5582},{"href":2035},[5583],{"type":81,"value":2035},{"type":81,"value":5585},", the ",{"type":75,"tag":127,"props":5587,"children":5589},{"href":665,"rel":5588},[667],[5590],{"type":81,"value":670},{"type":81,"value":2962},{"type":75,"tag":127,"props":5593,"children":5594},{"href":483},[5595],{"type":81,"value":5596},"split compute-plane workflows",{"type":75,"tag":1554,"props":5598,"children":5599},{},[5600,5602,5608,5609,5614,5615],{"type":81,"value":5601},"Operations: ",{"type":75,"tag":127,"props":5603,"children":5605},{"href":5604},"references\u002Fhelmfile-structure.md",[5606],{"type":81,"value":5607},"Helmfile structure",{"type":81,"value":2878},{"type":75,"tag":127,"props":5610,"children":5611},{"href":4669},[5612],{"type":81,"value":5613},"pull secrets",{"type":81,"value":2962},{"type":75,"tag":127,"props":5616,"children":5617},{"href":4969},[5618],{"type":81,"value":4675},{"type":75,"tag":84,"props":5620,"children":5621},{},[5622,5624,5630],{"type":81,"value":5623},"After deployment, use the ",{"type":75,"tag":102,"props":5625,"children":5627},{"className":5626},[],[5628],{"type":81,"value":5629},"nvcf-self-managed-cli",{"type":81,"value":5631}," skill to create functions, create tasks, manage API keys, and invoke endpoints.",{"type":75,"tag":5633,"props":5634,"children":5635},"style",{},[5636],{"type":81,"value":5637},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":5639,"total":543},[5640,5652,5659],{"slug":5629,"name":5629,"fn":5641,"description":5642,"org":5643,"tags":5644,"stars":23,"repoUrl":24,"updatedAt":5651},"deploy and manage NVIDIA Cloud Functions","Install, operate, and tear down self-hosted NVIDIA Cloud Functions (NVCF)\ndeployments with nvcf-cli. Use for control-plane or compute-plane install,\nstatus checks, cluster registration, function deploy\u002Finvoke, task\ncreate\u002Flist\u002Fcancel\u002Fdelete, API keys, admin tokens, JWKS rotation,\nfailed-install diagnosis, and uninstall or down workflows. Trigger keywords:\nnvcf, nvcf-cli, self-hosted nvcf, self-managed nvcf, NVCFBackend, NVCA, NCP,\nICMS, helmfile, control plane, compute plane, LLM function,\nOpenAI-compatible invocation, Responses API, embeddings, batch task, task\nmonitor, cluster rotate, cluster delete, helm task, helm-based task,\ntask secrets, update task secrets, retrieve task results, bulk fetch tasks,\ntask results, NVCT, NVCT task, NVCT batch job.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5645,5646,5649,5650],{"name":13,"slug":14,"type":15},{"name":5647,"slug":5648,"type":15},"Cloud","cloud",{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},"2026-07-30T05:29:41.025233",{"slug":4,"name":4,"fn":5,"description":6,"org":5653,"tags":5654,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5655,5656,5657,5658],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"slug":5660,"name":5660,"fn":5661,"description":5662,"org":5663,"tags":5664,"stars":23,"repoUrl":24,"updatedAt":5669},"nvcf-self-managed-prerequisite","install NVCF self-managed prerequisites","Install the prerequisites the NVCA operator \u002F compute plane needs before nvcf-nvca-install can succeed: the operator tool nvcf-cli (required by the compute-plane stack's register-cluster step), KAI Scheduler (for the KAIScheduler feature gate), and the SMB CSI driver (for the sharedStorage Samba sidecar PVCs). The two cluster components are cloud-neutral helm installs at the NVCF-validated version pins; same install on AKS, EKS, GKE, k3d, or bare metal. Use when the user mentions NVCA prereqs, nvcf-cli, \"nvcf-cli not found\", ensure-nvcf-cli, register-cluster, KAI Scheduler, SMB CSI, csi-driver-smb, queue quotas, default-parent-queue, NVCA shared-storage PVCs stuck Pending, or asks how to prepare a cluster before installing the NVCA operator.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5665,5666,5667,5668],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:33:51.764422",{"items":5671,"total":5824},[5672,5690,5706,5717,5729,5741,5754,5768,5781,5790,5804,5813],{"slug":5673,"name":5673,"fn":5674,"description":5675,"org":5676,"tags":5677,"stars":5687,"repoUrl":5688,"updatedAt":5689},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5678,5681,5684],{"name":5679,"slug":5680,"type":15},"Documentation","documentation",{"name":5682,"slug":5683,"type":15},"MCP","mcp",{"name":5685,"slug":5686,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":5691,"name":5691,"fn":5692,"description":5693,"org":5694,"tags":5695,"stars":5703,"repoUrl":5704,"updatedAt":5705},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5696,5699,5700],{"name":5697,"slug":5698,"type":15},"Containers","containers",{"name":17,"slug":18,"type":15},{"name":5701,"slug":5702,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":5707,"name":5707,"fn":5708,"description":5709,"org":5710,"tags":5711,"stars":5703,"repoUrl":5704,"updatedAt":5716},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5712,5715],{"name":5713,"slug":5714,"type":15},"CI\u002FCD","ci-cd",{"name":17,"slug":18,"type":15},"2026-07-14T05:25:59.97109",{"slug":5718,"name":5718,"fn":5719,"description":5720,"org":5721,"tags":5722,"stars":5703,"repoUrl":5704,"updatedAt":5728},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5723,5724,5725],{"name":5713,"slug":5714,"type":15},{"name":17,"slug":18,"type":15},{"name":5726,"slug":5727,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":5730,"name":5730,"fn":5731,"description":5732,"org":5733,"tags":5734,"stars":5703,"repoUrl":5704,"updatedAt":5740},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5735,5736,5737],{"name":4678,"slug":4675,"type":15},{"name":5726,"slug":5727,"type":15},{"name":5738,"slug":5739,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":5742,"name":5742,"fn":5743,"description":5744,"org":5745,"tags":5746,"stars":5703,"repoUrl":5704,"updatedAt":5753},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5747,5750],{"name":5748,"slug":5749,"type":15},"Best Practices","best-practices",{"name":5751,"slug":5752,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":5755,"name":5755,"fn":5756,"description":5757,"org":5758,"tags":5759,"stars":5703,"repoUrl":5704,"updatedAt":5767},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5760,5763,5766],{"name":5761,"slug":5762,"type":15},"Machine Learning","machine-learning",{"name":5764,"slug":5765,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":5769,"name":5769,"fn":5770,"description":5771,"org":5772,"tags":5773,"stars":5703,"repoUrl":5704,"updatedAt":5780},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5774,5777],{"name":5775,"slug":5776,"type":15},"QA","qa",{"name":5778,"slug":5779,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":5782,"name":5782,"fn":5783,"description":5784,"org":5785,"tags":5786,"stars":5703,"repoUrl":5704,"updatedAt":5789},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5787,5788],{"name":17,"slug":18,"type":15},{"name":21,"slug":22,"type":15},"2026-07-14T05:25:49.362534",{"slug":5791,"name":5791,"fn":5792,"description":5793,"org":5794,"tags":5795,"stars":5703,"repoUrl":5704,"updatedAt":5803},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5796,5799,5800],{"name":5797,"slug":5798,"type":15},"Code Review","code-review",{"name":5726,"slug":5727,"type":15},{"name":5801,"slug":5802,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":5805,"name":5805,"fn":5806,"description":5807,"org":5808,"tags":5809,"stars":5703,"repoUrl":5704,"updatedAt":5812},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5810,5811],{"name":5775,"slug":5776,"type":15},{"name":5778,"slug":5779,"type":15},"2026-07-14T05:25:54.928983",{"slug":5814,"name":5814,"fn":5815,"description":5816,"org":5817,"tags":5818,"stars":5703,"repoUrl":5704,"updatedAt":5823},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[5819,5822],{"name":5820,"slug":5821,"type":15},"Automation","automation",{"name":5713,"slug":5714,"type":15},"2026-07-30T05:29:03.275638",496]