[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-migrate-mongodb-to-percona":3,"mdc--llyvl8-key":44,"related-repo-nvidia-migrate-mongodb-to-percona":1023,"related-org-nvidia-migrate-mongodb-to-percona":1032},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":39,"sourceUrl":42,"mdContent":43},"migrate-mongodb-to-percona","migrate MongoDB to Percona Operator","Execute the NVSentinel MongoDB backend migration from Bitnami to the Percona Operator after readiness is confirmed: data dump (default preserve path), removal of the installation, cleanup of surviving objects, values preparation, and the Percona deployment. Destructive: on the opt-out clean path all health event data is wiped. Use only after check-mongodb-migration-readiness reports READY and the operator confirmed the decisions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"MongoDB","mongodb","tag",{"name":17,"slug":18,"type":15},"Database","database",{"name":20,"slug":21,"type":15},"Migration","migration",{"name":23,"slug":24,"type":15},"Deployment","deployment",378,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNVSentinel","2026-08-31T09:19:19.418291",null,117,[31,32,33,34,35,36,37,38],"accelerated-computing","automation","breakfix","gpu","gpu-monitoring","kubernetes","observability","resilience",{"repoUrl":26,"stars":25,"forks":29,"topics":40,"description":41},[31,32,33,34,35,36,37,38],"NVSentinel detects and remediates GPU faults on Kubernetes nodes","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNVSentinel\u002Ftree\u002FHEAD\u002Fskills\u002Fmongodb-migration\u002Fbitnami-to-percona\u002Fmigrate-mongodb-to-percona","---\nname: migrate-mongodb-to-percona\ndescription: >-\n  Execute the NVSentinel MongoDB backend migration from Bitnami to the\n  Percona Operator after readiness is confirmed: data dump (default\n  preserve path), removal of the installation, cleanup of surviving\n  objects, values preparation, and the Percona deployment. Destructive:\n  on the opt-out clean path all health event data is wiped. Use only\n  after check-mongodb-migration-readiness reports READY and the operator\n  confirmed the decisions.\nmaturity: experimental\nlifecycle: evergreen\napi-version: nvsentinel.skills\u002Fv1\nallowed-tools: Bash(kubectl *), Bash(helm *), Bash(argocd *), Bash(flux *), Bash(git *), Bash(scripts\u002Fmongodb-migration\u002F*), Read, Grep\n---\n\n# Migrate MongoDB to Percona\n\n## When to use\n\nUse this skill **after** `check-mongodb-migration-readiness` reports READY.\n\nHard gate: do NOT run any step below until the operator has confirmed, in\nthis conversation, (a) the data-handling decision (the default preserve\npath, or the clean path with data loss explicitly accepted) and (b) what\nhappens to each quarantined node. If either is missing, go back to the\nreadiness skill.\n\n## Inputs\n\n- `NVSENTINEL_NAMESPACE`, `NVSENTINEL_RELEASE` (default `nvsentinel`)\n- `DATA_PATH` (`restore` is the default, `clean` is the opt-out): the\n  confirmed data-handling decision\n- `VALUES_FILES`: the operator's values file(s) for the install\n- `CHART_REF`: the chart to deploy in step 5 (path or repo reference plus\n  version), captured during readiness so the migration does not silently\n  upgrade or downgrade NVSentinel as a side effect\n- `ARCHIVE`: dump archive path (restore path only)\n- GitOps status from readiness (if managed: reconciliation suspended before\n  step 1, git updated before resuming; runbook steps 2, 4a and 6a)\n\n## Setup\n\n```bash\nexport NVSENTINEL_NAMESPACE=\"nvsentinel\"\nexport NVSENTINEL_RELEASE=\"nvsentinel\"\n```\n\n## Trigger order (required)\n\n0. **GitOps only: confirm reconciliation is stopped** (runbook step 2)\n   BEFORE anything below mutates the cluster; with automated sync or\n   self-heal active, the controller reverts the step 1 scale-downs and\n   re-creates whatever step 2 removes. On ArgoCD, automated sync is\n   disabled or a deny sync window covers the migration; on Flux, the\n   HelmRelease is suspended. If it is not stopped, stop it now and verify\n   before continuing.\n\n1. **Dump (default preserve path; skipped only on the clean path):**\n\n   ```bash\n   scripts\u002Fmongodb-migration\u002Fmigrate-data.sh dump \u003CARCHIVE> --stop-writers\n   ```\n\n   `--stop-writers` scales fault-quarantine, node-drainer, and\n   fault-remediation to zero and waits for their pods to be GONE, not just\n   scaling down (a terminating pod can still write references). This stops\n   fault handling until step 8, so it is part of the migration the operator\n   confirmed, not an extra decision. Either way the script fails closed: it\n   refuses while any writer pod exists, including terminating ones, or\n   while their state cannot be determined. Never work around a refusal on\n   the preserve path: an archive taken with active writers can contain\n   dangling references and must not be used to preserve fault state. The\n   script auto-detects the source backend and always excludes\n   `ResumeTokens` (change-stream tokens are only valid on the cluster that\n   created them). Gate: the script must report a non-empty archive.\n\n2. **Remove the current installation.** How depends on who manages it:\n   - GitOps-managed (the common case): reconciliation must already be\n     stopped (runbook step 2) BEFORE the step 1 scale-downs, otherwise the\n     controller reverts them. There may be no Helm release at all (ArgoCD\n     renders charts without creating release records), so `helm uninstall`\n     has nothing to act on; remove the rendered resources through the\n     controller instead, for example by deleting the NVSentinel application\n     with cascading deletion (runbook step 4a). Confirm the workloads are\n     gone before continuing. The cleanup script works the same either way.\n   - Helm-managed (a release exists, `helm status` succeeds):\n\n     ```bash\n     helm uninstall \"$NVSENTINEL_RELEASE\" -n \"$NVSENTINEL_NAMESPACE\"\n     ```\n\n3. **Cleanup:**\n\n   ```bash\n   scripts\u002Fmongodb-migration\u002Fcleanup.sh --yes\n   ```\n\n   Add `--clear-fault-state` ONLY on the clean path. On the restore path,\n   fault state (node annotations, remediation resources) must be kept:\n   restored documents keep their IDs, so those references become valid\n   again. Gate: exit 0. Exit 3 means the script refused (a Helm release\n   still exists: finish step 2 first, or the confirmation was declined).\n   Exit 1 means an error or leftovers remain; leftover TLS secrets cause\n   opaque connection-closed crash loops on the next install, and\n   remediation resources stuck on finalizers are reported with the manual\n   patch command. Do not continue until cleanup verifies clean.\n\n4. **Values.** Confirm the operator's values contain:\n   - `mongodb-store.useBitnami: false` and\n     `mongodb-store.usePerconaOperator: true` (both, always together)\n   - volume size at or above the provider minimum (OCI block volumes: 50Gi):\n     `mongodb-store.psmdb-db.replsets.rs0.volumeSpec.pvc.resources.requests.storage`\n   - scheduling for `mongodb-store.job`, `mongodb-store.psmdb-operator`,\n     and `mongodb-store.psmdb-db.replsets.rs0` where the cluster uses node\n     selectors or taints\n   - on single-node test clusters only:\n     `mongodb-store.psmdb-db.replsets.rs0.affinity.antiAffinityTopologyKey: \"none\"`\n     (the psmdb default is required anti-affinity across hostnames)\n\n5. **Deploy.** How depends on who manages it:\n   - GitOps-managed (the common case): commit the updated values to git\n     first, then let the controller deploy (recreate the application, or\n     resume the reconciliation stopped in runbook step 2). Git must carry\n     the new values BEFORE the controller reconciles, otherwise the first\n     sync redeploys the old backend.\n   - Helm-managed:\n\n     ```bash\n     helm upgrade --install \"$NVSENTINEL_RELEASE\" \u003CCHART_REF> -n \"$NVSENTINEL_NAMESPACE\" \\\n       -f \u003CVALUES_FILES...> --timeout 20m --wait --wait-for-jobs\n     ```\n\n     `--wait-for-jobs` matters: `--wait` alone does not wait for the\n     database initialization job.\n\n## Failure branches (validated)\n\n| Symptom | Action |\n| ------- | ------ |\n| Operator log `requested storage (...) is less than actual storage (...)`, psmdb `error`, no primary | Volume below the provider minimum. Fix values, delete the `mongod-data-*` PVCs, re-run step 5. |\n| MongoDB init Job (`-l app.kubernetes.io\u002Fname=create-mongodb-database`) `Failed` with `DeadlineExceeded` | The job never retries on its own. `kubectl delete job -l app.kubernetes.io\u002Fname=create-mongodb-database -n \"$NVSENTINEL_NAMESPACE\"`, then re-run the same `helm upgrade`; Helm recreates it and it completes against the healthy replica set. |\n| Consumers crash-loop with TLS\u002Fconnection-closed errors right after install | Leftover TLS secrets from the old backend. Re-run step 3 fully, then step 5. |\n| `helm upgrade` fails with `field is immutable` on the create-mongodb-database Job | An in-place backend switch was attempted on a live release. Follow the runbook troubleshooting row (rollback, manual deletion of everything the failed revision created), then restart from the readiness skill. |\n\n## Next skill to run\n\n- `verify-mongodb-percona-migration` (always; it also performs the restore\n  on the restore path)\n\n## References\n\n| Topic | Reference |\n|-------|-----------|\n| Runbook (source of truth) | `docs\u002Frunbooks\u002Fmongodb-bitnami-to-percona-migration.md` |\n| Scripts | `scripts\u002Fmongodb-migration\u002F` |\n| Readiness | [check-mongodb-migration-readiness](..\u002Fcheck-mongodb-migration-readiness\u002FSKILL.md) |\n",{"data":45,"body":50},{"name":4,"description":6,"maturity":46,"lifecycle":47,"api-version":48,"allowed-tools":49},"experimental","evergreen","nvsentinel.skills\u002Fv1","Bash(kubectl *), Bash(helm *), Bash(argocd *), Bash(flux *), Bash(git *), Bash(scripts\u002Fmongodb-migration\u002F*), Read, Grep",{"type":51,"children":52},"root",[53,61,68,91,96,102,199,205,284,290,757,763,918,924,938,944,1017],{"type":54,"tag":55,"props":56,"children":57},"element","h1",{"id":4},[58],{"type":59,"value":60},"text","Migrate MongoDB to Percona",{"type":54,"tag":62,"props":63,"children":65},"h2",{"id":64},"when-to-use",[66],{"type":59,"value":67},"When to use",{"type":54,"tag":69,"props":70,"children":71},"p",{},[72,74,80,82,89],{"type":59,"value":73},"Use this skill ",{"type":54,"tag":75,"props":76,"children":77},"strong",{},[78],{"type":59,"value":79},"after",{"type":59,"value":81}," ",{"type":54,"tag":83,"props":84,"children":86},"code",{"className":85},[],[87],{"type":59,"value":88},"check-mongodb-migration-readiness",{"type":59,"value":90}," reports READY.",{"type":54,"tag":69,"props":92,"children":93},{},[94],{"type":59,"value":95},"Hard gate: do NOT run any step below until the operator has confirmed, in\nthis conversation, (a) the data-handling decision (the default preserve\npath, or the clean path with data loss explicitly accepted) and (b) what\nhappens to each quarantined node. If either is missing, go back to the\nreadiness skill.",{"type":54,"tag":62,"props":97,"children":99},{"id":98},"inputs",[100],{"type":59,"value":101},"Inputs",{"type":54,"tag":103,"props":104,"children":105},"ul",{},[106,134,161,172,183,194],{"type":54,"tag":107,"props":108,"children":109},"li",{},[110,116,118,124,126,132],{"type":54,"tag":83,"props":111,"children":113},{"className":112},[],[114],{"type":59,"value":115},"NVSENTINEL_NAMESPACE",{"type":59,"value":117},", ",{"type":54,"tag":83,"props":119,"children":121},{"className":120},[],[122],{"type":59,"value":123},"NVSENTINEL_RELEASE",{"type":59,"value":125}," (default ",{"type":54,"tag":83,"props":127,"children":129},{"className":128},[],[130],{"type":59,"value":131},"nvsentinel",{"type":59,"value":133},")",{"type":54,"tag":107,"props":135,"children":136},{},[137,143,145,151,153,159],{"type":54,"tag":83,"props":138,"children":140},{"className":139},[],[141],{"type":59,"value":142},"DATA_PATH",{"type":59,"value":144}," (",{"type":54,"tag":83,"props":146,"children":148},{"className":147},[],[149],{"type":59,"value":150},"restore",{"type":59,"value":152}," is the default, ",{"type":54,"tag":83,"props":154,"children":156},{"className":155},[],[157],{"type":59,"value":158},"clean",{"type":59,"value":160}," is the opt-out): the\nconfirmed data-handling decision",{"type":54,"tag":107,"props":162,"children":163},{},[164,170],{"type":54,"tag":83,"props":165,"children":167},{"className":166},[],[168],{"type":59,"value":169},"VALUES_FILES",{"type":59,"value":171},": the operator's values file(s) for the install",{"type":54,"tag":107,"props":173,"children":174},{},[175,181],{"type":54,"tag":83,"props":176,"children":178},{"className":177},[],[179],{"type":59,"value":180},"CHART_REF",{"type":59,"value":182},": the chart to deploy in step 5 (path or repo reference plus\nversion), captured during readiness so the migration does not silently\nupgrade or downgrade NVSentinel as a side effect",{"type":54,"tag":107,"props":184,"children":185},{},[186,192],{"type":54,"tag":83,"props":187,"children":189},{"className":188},[],[190],{"type":59,"value":191},"ARCHIVE",{"type":59,"value":193},": dump archive path (restore path only)",{"type":54,"tag":107,"props":195,"children":196},{},[197],{"type":59,"value":198},"GitOps status from readiness (if managed: reconciliation suspended before\nstep 1, git updated before resuming; runbook steps 2, 4a and 6a)",{"type":54,"tag":62,"props":200,"children":202},{"id":201},"setup",[203],{"type":59,"value":204},"Setup",{"type":54,"tag":206,"props":207,"children":212},"pre",{"className":208,"code":209,"language":210,"meta":211,"style":211},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","export NVSENTINEL_NAMESPACE=\"nvsentinel\"\nexport NVSENTINEL_RELEASE=\"nvsentinel\"\n","bash","",[213],{"type":54,"tag":83,"props":214,"children":215},{"__ignoreMap":211},[216,255],{"type":54,"tag":217,"props":218,"children":221},"span",{"class":219,"line":220},"line",1,[222,228,234,240,245,250],{"type":54,"tag":217,"props":223,"children":225},{"style":224},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[226],{"type":59,"value":227},"export",{"type":54,"tag":217,"props":229,"children":231},{"style":230},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[232],{"type":59,"value":233}," NVSENTINEL_NAMESPACE",{"type":54,"tag":217,"props":235,"children":237},{"style":236},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[238],{"type":59,"value":239},"=",{"type":54,"tag":217,"props":241,"children":242},{"style":236},[243],{"type":59,"value":244},"\"",{"type":54,"tag":217,"props":246,"children":248},{"style":247},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[249],{"type":59,"value":131},{"type":54,"tag":217,"props":251,"children":252},{"style":236},[253],{"type":59,"value":254},"\"\n",{"type":54,"tag":217,"props":256,"children":258},{"class":219,"line":257},2,[259,263,268,272,276,280],{"type":54,"tag":217,"props":260,"children":261},{"style":224},[262],{"type":59,"value":227},{"type":54,"tag":217,"props":264,"children":265},{"style":230},[266],{"type":59,"value":267}," NVSENTINEL_RELEASE",{"type":54,"tag":217,"props":269,"children":270},{"style":236},[271],{"type":59,"value":239},{"type":54,"tag":217,"props":273,"children":274},{"style":236},[275],{"type":59,"value":244},{"type":54,"tag":217,"props":277,"children":278},{"style":247},[279],{"type":59,"value":131},{"type":54,"tag":217,"props":281,"children":282},{"style":236},[283],{"type":59,"value":254},{"type":54,"tag":62,"props":285,"children":287},{"id":286},"trigger-order-required",[288],{"type":59,"value":289},"Trigger order (required)",{"type":54,"tag":291,"props":292,"children":294},"ol",{"start":293},0,[295,305,379,470,511,595],{"type":54,"tag":107,"props":296,"children":297},{},[298,303],{"type":54,"tag":75,"props":299,"children":300},{},[301],{"type":59,"value":302},"GitOps only: confirm reconciliation is stopped",{"type":59,"value":304}," (runbook step 2)\nBEFORE anything below mutates the cluster; with automated sync or\nself-heal active, the controller reverts the step 1 scale-downs and\nre-creates whatever step 2 removes. On ArgoCD, automated sync is\ndisabled or a deny sync window covers the migration; on Flux, the\nHelmRelease is suspended. If it is not stopped, stop it now and verify\nbefore continuing.",{"type":54,"tag":107,"props":306,"children":307},{},[308,313,359,363,369,371,377],{"type":54,"tag":75,"props":309,"children":310},{},[311],{"type":59,"value":312},"Dump (default preserve path; skipped only on the clean path):",{"type":54,"tag":206,"props":314,"children":316},{"className":208,"code":315,"language":210,"meta":211,"style":211},"scripts\u002Fmongodb-migration\u002Fmigrate-data.sh dump \u003CARCHIVE> --stop-writers\n",[317],{"type":54,"tag":83,"props":318,"children":319},{"__ignoreMap":211},[320],{"type":54,"tag":217,"props":321,"children":322},{"class":219,"line":220},[323,329,334,339,344,349,354],{"type":54,"tag":217,"props":324,"children":326},{"style":325},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[327],{"type":59,"value":328},"scripts\u002Fmongodb-migration\u002Fmigrate-data.sh",{"type":54,"tag":217,"props":330,"children":331},{"style":247},[332],{"type":59,"value":333}," dump",{"type":54,"tag":217,"props":335,"children":336},{"style":236},[337],{"type":59,"value":338}," \u003C",{"type":54,"tag":217,"props":340,"children":341},{"style":247},[342],{"type":59,"value":343},"ARCHIV",{"type":54,"tag":217,"props":345,"children":346},{"style":230},[347],{"type":59,"value":348},"E",{"type":54,"tag":217,"props":350,"children":351},{"style":236},[352],{"type":59,"value":353},">",{"type":54,"tag":217,"props":355,"children":356},{"style":247},[357],{"type":59,"value":358}," --stop-writers\n",{"type":54,"tag":360,"props":361,"children":362},"br",{},[],{"type":54,"tag":83,"props":364,"children":366},{"className":365},[],[367],{"type":59,"value":368},"--stop-writers",{"type":59,"value":370}," scales fault-quarantine, node-drainer, and\nfault-remediation to zero and waits for their pods to be GONE, not just\nscaling down (a terminating pod can still write references). This stops\nfault handling until step 8, so it is part of the migration the operator\nconfirmed, not an extra decision. Either way the script fails closed: it\nrefuses while any writer pod exists, including terminating ones, or\nwhile their state cannot be determined. Never work around a refusal on\nthe preserve path: an archive taken with active writers can contain\ndangling references and must not be used to preserve fault state. The\nscript auto-detects the source backend and always excludes\n",{"type":54,"tag":83,"props":372,"children":374},{"className":373},[],[375],{"type":59,"value":376},"ResumeTokens",{"type":59,"value":378}," (change-stream tokens are only valid on the cluster that\ncreated them). Gate: the script must report a non-empty archive.",{"type":54,"tag":107,"props":380,"children":381},{},[382,387,389],{"type":54,"tag":75,"props":383,"children":384},{},[385],{"type":59,"value":386},"Remove the current installation.",{"type":59,"value":388}," How depends on who manages it:",{"type":54,"tag":103,"props":390,"children":391},{},[392,405],{"type":54,"tag":107,"props":393,"children":394},{},[395,397,403],{"type":59,"value":396},"GitOps-managed (the common case): reconciliation must already be\nstopped (runbook step 2) BEFORE the step 1 scale-downs, otherwise the\ncontroller reverts them. There may be no Helm release at all (ArgoCD\nrenders charts without creating release records), so ",{"type":54,"tag":83,"props":398,"children":400},{"className":399},[],[401],{"type":59,"value":402},"helm uninstall",{"type":59,"value":404},"\nhas nothing to act on; remove the rendered resources through the\ncontroller instead, for example by deleting the NVSentinel application\nwith cascading deletion (runbook step 4a). Confirm the workloads are\ngone before continuing. The cleanup script works the same either way.",{"type":54,"tag":107,"props":406,"children":407},{},[408,410,416,418],{"type":59,"value":409},"Helm-managed (a release exists, ",{"type":54,"tag":83,"props":411,"children":413},{"className":412},[],[414],{"type":59,"value":415},"helm status",{"type":59,"value":417}," succeeds):",{"type":54,"tag":206,"props":419,"children":421},{"className":208,"code":420,"language":210,"meta":211,"style":211},"helm uninstall \"$NVSENTINEL_RELEASE\" -n \"$NVSENTINEL_NAMESPACE\"\n",[422],{"type":54,"tag":83,"props":423,"children":424},{"__ignoreMap":211},[425],{"type":54,"tag":217,"props":426,"children":427},{"class":219,"line":220},[428,433,438,443,448,452,457,461,466],{"type":54,"tag":217,"props":429,"children":430},{"style":325},[431],{"type":59,"value":432},"helm",{"type":54,"tag":217,"props":434,"children":435},{"style":247},[436],{"type":59,"value":437}," uninstall",{"type":54,"tag":217,"props":439,"children":440},{"style":236},[441],{"type":59,"value":442}," \"",{"type":54,"tag":217,"props":444,"children":445},{"style":230},[446],{"type":59,"value":447},"$NVSENTINEL_RELEASE",{"type":54,"tag":217,"props":449,"children":450},{"style":236},[451],{"type":59,"value":244},{"type":54,"tag":217,"props":453,"children":454},{"style":247},[455],{"type":59,"value":456}," -n",{"type":54,"tag":217,"props":458,"children":459},{"style":236},[460],{"type":59,"value":442},{"type":54,"tag":217,"props":462,"children":463},{"style":230},[464],{"type":59,"value":465},"$NVSENTINEL_NAMESPACE",{"type":54,"tag":217,"props":467,"children":468},{"style":236},[469],{"type":59,"value":254},{"type":54,"tag":107,"props":471,"children":472},{},[473,478,498,501,503,509],{"type":54,"tag":75,"props":474,"children":475},{},[476],{"type":59,"value":477},"Cleanup:",{"type":54,"tag":206,"props":479,"children":481},{"className":208,"code":480,"language":210,"meta":211,"style":211},"scripts\u002Fmongodb-migration\u002Fcleanup.sh --yes\n",[482],{"type":54,"tag":83,"props":483,"children":484},{"__ignoreMap":211},[485],{"type":54,"tag":217,"props":486,"children":487},{"class":219,"line":220},[488,493],{"type":54,"tag":217,"props":489,"children":490},{"style":325},[491],{"type":59,"value":492},"scripts\u002Fmongodb-migration\u002Fcleanup.sh",{"type":54,"tag":217,"props":494,"children":495},{"style":247},[496],{"type":59,"value":497}," --yes\n",{"type":54,"tag":360,"props":499,"children":500},{},[],{"type":59,"value":502},"Add ",{"type":54,"tag":83,"props":504,"children":506},{"className":505},[],[507],{"type":59,"value":508},"--clear-fault-state",{"type":59,"value":510}," ONLY on the clean path. On the restore path,\nfault state (node annotations, remediation resources) must be kept:\nrestored documents keep their IDs, so those references become valid\nagain. Gate: exit 0. Exit 3 means the script refused (a Helm release\nstill exists: finish step 2 first, or the confirmation was declined).\nExit 1 means an error or leftovers remain; leftover TLS secrets cause\nopaque connection-closed crash loops on the next install, and\nremediation resources stuck on finalizers are reported with the manual\npatch command. Do not continue until cleanup verifies clean.",{"type":54,"tag":107,"props":512,"children":513},{},[514,519,521],{"type":54,"tag":75,"props":515,"children":516},{},[517],{"type":59,"value":518},"Values.",{"type":59,"value":520}," Confirm the operator's values contain:",{"type":54,"tag":103,"props":522,"children":523},{},[524,543,554,582],{"type":54,"tag":107,"props":525,"children":526},{},[527,533,535,541],{"type":54,"tag":83,"props":528,"children":530},{"className":529},[],[531],{"type":59,"value":532},"mongodb-store.useBitnami: false",{"type":59,"value":534}," and\n",{"type":54,"tag":83,"props":536,"children":538},{"className":537},[],[539],{"type":59,"value":540},"mongodb-store.usePerconaOperator: true",{"type":59,"value":542}," (both, always together)",{"type":54,"tag":107,"props":544,"children":545},{},[546,548],{"type":59,"value":547},"volume size at or above the provider minimum (OCI block volumes: 50Gi):\n",{"type":54,"tag":83,"props":549,"children":551},{"className":550},[],[552],{"type":59,"value":553},"mongodb-store.psmdb-db.replsets.rs0.volumeSpec.pvc.resources.requests.storage",{"type":54,"tag":107,"props":555,"children":556},{},[557,559,565,566,572,574,580],{"type":59,"value":558},"scheduling for ",{"type":54,"tag":83,"props":560,"children":562},{"className":561},[],[563],{"type":59,"value":564},"mongodb-store.job",{"type":59,"value":117},{"type":54,"tag":83,"props":567,"children":569},{"className":568},[],[570],{"type":59,"value":571},"mongodb-store.psmdb-operator",{"type":59,"value":573},",\nand ",{"type":54,"tag":83,"props":575,"children":577},{"className":576},[],[578],{"type":59,"value":579},"mongodb-store.psmdb-db.replsets.rs0",{"type":59,"value":581}," where the cluster uses node\nselectors or taints",{"type":54,"tag":107,"props":583,"children":584},{},[585,587,593],{"type":59,"value":586},"on single-node test clusters only:\n",{"type":54,"tag":83,"props":588,"children":590},{"className":589},[],[591],{"type":59,"value":592},"mongodb-store.psmdb-db.replsets.rs0.affinity.antiAffinityTopologyKey: \"none\"",{"type":59,"value":594},"\n(the psmdb default is required anti-affinity across hostnames)",{"type":54,"tag":107,"props":596,"children":597},{},[598,603,604],{"type":54,"tag":75,"props":599,"children":600},{},[601],{"type":59,"value":602},"Deploy.",{"type":59,"value":388},{"type":54,"tag":103,"props":605,"children":606},{},[607,612],{"type":54,"tag":107,"props":608,"children":609},{},[610],{"type":59,"value":611},"GitOps-managed (the common case): commit the updated values to git\nfirst, then let the controller deploy (recreate the application, or\nresume the reconciliation stopped in runbook step 2). Git must carry\nthe new values BEFORE the controller reconciles, otherwise the first\nsync redeploys the old backend.",{"type":54,"tag":107,"props":613,"children":614},{},[615,617,738,741,747,749,755],{"type":59,"value":616},"Helm-managed:",{"type":54,"tag":206,"props":618,"children":620},{"className":208,"code":619,"language":210,"meta":211,"style":211},"helm upgrade --install \"$NVSENTINEL_RELEASE\" \u003CCHART_REF> -n \"$NVSENTINEL_NAMESPACE\" \\\n  -f \u003CVALUES_FILES...> --timeout 20m --wait --wait-for-jobs\n",[621],{"type":54,"tag":83,"props":622,"children":623},{"__ignoreMap":211},[624,692],{"type":54,"tag":217,"props":625,"children":626},{"class":219,"line":220},[627,631,636,641,645,649,653,657,662,667,671,675,679,683,687],{"type":54,"tag":217,"props":628,"children":629},{"style":325},[630],{"type":59,"value":432},{"type":54,"tag":217,"props":632,"children":633},{"style":247},[634],{"type":59,"value":635}," upgrade",{"type":54,"tag":217,"props":637,"children":638},{"style":247},[639],{"type":59,"value":640}," --install",{"type":54,"tag":217,"props":642,"children":643},{"style":236},[644],{"type":59,"value":442},{"type":54,"tag":217,"props":646,"children":647},{"style":230},[648],{"type":59,"value":447},{"type":54,"tag":217,"props":650,"children":651},{"style":236},[652],{"type":59,"value":244},{"type":54,"tag":217,"props":654,"children":655},{"style":236},[656],{"type":59,"value":338},{"type":54,"tag":217,"props":658,"children":659},{"style":247},[660],{"type":59,"value":661},"CHART_RE",{"type":54,"tag":217,"props":663,"children":664},{"style":230},[665],{"type":59,"value":666},"F",{"type":54,"tag":217,"props":668,"children":669},{"style":236},[670],{"type":59,"value":353},{"type":54,"tag":217,"props":672,"children":673},{"style":247},[674],{"type":59,"value":456},{"type":54,"tag":217,"props":676,"children":677},{"style":236},[678],{"type":59,"value":442},{"type":54,"tag":217,"props":680,"children":681},{"style":230},[682],{"type":59,"value":465},{"type":54,"tag":217,"props":684,"children":685},{"style":236},[686],{"type":59,"value":244},{"type":54,"tag":217,"props":688,"children":689},{"style":230},[690],{"type":59,"value":691}," \\\n",{"type":54,"tag":217,"props":693,"children":694},{"class":219,"line":257},[695,700,704,709,714,718,723,728,733],{"type":54,"tag":217,"props":696,"children":697},{"style":247},[698],{"type":59,"value":699},"  -f",{"type":54,"tag":217,"props":701,"children":702},{"style":236},[703],{"type":59,"value":338},{"type":54,"tag":217,"props":705,"children":706},{"style":247},[707],{"type":59,"value":708},"VALUES_FILES..",{"type":54,"tag":217,"props":710,"children":711},{"style":230},[712],{"type":59,"value":713},".",{"type":54,"tag":217,"props":715,"children":716},{"style":236},[717],{"type":59,"value":353},{"type":54,"tag":217,"props":719,"children":720},{"style":247},[721],{"type":59,"value":722}," --timeout",{"type":54,"tag":217,"props":724,"children":725},{"style":247},[726],{"type":59,"value":727}," 20m",{"type":54,"tag":217,"props":729,"children":730},{"style":247},[731],{"type":59,"value":732}," --wait",{"type":54,"tag":217,"props":734,"children":735},{"style":247},[736],{"type":59,"value":737}," --wait-for-jobs\n",{"type":54,"tag":360,"props":739,"children":740},{},[],{"type":54,"tag":83,"props":742,"children":744},{"className":743},[],[745],{"type":59,"value":746},"--wait-for-jobs",{"type":59,"value":748}," matters: ",{"type":54,"tag":83,"props":750,"children":752},{"className":751},[],[753],{"type":59,"value":754},"--wait",{"type":59,"value":756}," alone does not wait for the\ndatabase initialization job.",{"type":54,"tag":62,"props":758,"children":760},{"id":759},"failure-branches-validated",[761],{"type":59,"value":762},"Failure branches (validated)",{"type":54,"tag":764,"props":765,"children":766},"table",{},[767,786],{"type":54,"tag":768,"props":769,"children":770},"thead",{},[771],{"type":54,"tag":772,"props":773,"children":774},"tr",{},[775,781],{"type":54,"tag":776,"props":777,"children":778},"th",{},[779],{"type":59,"value":780},"Symptom",{"type":54,"tag":776,"props":782,"children":783},{},[784],{"type":59,"value":785},"Action",{"type":54,"tag":787,"props":788,"children":789},"tbody",{},[790,828,879,892],{"type":54,"tag":772,"props":791,"children":792},{},[793,815],{"type":54,"tag":794,"props":795,"children":796},"td",{},[797,799,805,807,813],{"type":59,"value":798},"Operator log ",{"type":54,"tag":83,"props":800,"children":802},{"className":801},[],[803],{"type":59,"value":804},"requested storage (...) is less than actual storage (...)",{"type":59,"value":806},", psmdb ",{"type":54,"tag":83,"props":808,"children":810},{"className":809},[],[811],{"type":59,"value":812},"error",{"type":59,"value":814},", no primary",{"type":54,"tag":794,"props":816,"children":817},{},[818,820,826],{"type":59,"value":819},"Volume below the provider minimum. Fix values, delete the ",{"type":54,"tag":83,"props":821,"children":823},{"className":822},[],[824],{"type":59,"value":825},"mongod-data-*",{"type":59,"value":827}," PVCs, re-run step 5.",{"type":54,"tag":772,"props":829,"children":830},{},[831,858],{"type":54,"tag":794,"props":832,"children":833},{},[834,836,842,844,850,852],{"type":59,"value":835},"MongoDB init Job (",{"type":54,"tag":83,"props":837,"children":839},{"className":838},[],[840],{"type":59,"value":841},"-l app.kubernetes.io\u002Fname=create-mongodb-database",{"type":59,"value":843},") ",{"type":54,"tag":83,"props":845,"children":847},{"className":846},[],[848],{"type":59,"value":849},"Failed",{"type":59,"value":851}," with ",{"type":54,"tag":83,"props":853,"children":855},{"className":854},[],[856],{"type":59,"value":857},"DeadlineExceeded",{"type":54,"tag":794,"props":859,"children":860},{},[861,863,869,871,877],{"type":59,"value":862},"The job never retries on its own. ",{"type":54,"tag":83,"props":864,"children":866},{"className":865},[],[867],{"type":59,"value":868},"kubectl delete job -l app.kubernetes.io\u002Fname=create-mongodb-database -n \"$NVSENTINEL_NAMESPACE\"",{"type":59,"value":870},", then re-run the same ",{"type":54,"tag":83,"props":872,"children":874},{"className":873},[],[875],{"type":59,"value":876},"helm upgrade",{"type":59,"value":878},"; Helm recreates it and it completes against the healthy replica set.",{"type":54,"tag":772,"props":880,"children":881},{},[882,887],{"type":54,"tag":794,"props":883,"children":884},{},[885],{"type":59,"value":886},"Consumers crash-loop with TLS\u002Fconnection-closed errors right after install",{"type":54,"tag":794,"props":888,"children":889},{},[890],{"type":59,"value":891},"Leftover TLS secrets from the old backend. Re-run step 3 fully, then step 5.",{"type":54,"tag":772,"props":893,"children":894},{},[895,913],{"type":54,"tag":794,"props":896,"children":897},{},[898,903,905,911],{"type":54,"tag":83,"props":899,"children":901},{"className":900},[],[902],{"type":59,"value":876},{"type":59,"value":904}," fails with ",{"type":54,"tag":83,"props":906,"children":908},{"className":907},[],[909],{"type":59,"value":910},"field is immutable",{"type":59,"value":912}," on the create-mongodb-database Job",{"type":54,"tag":794,"props":914,"children":915},{},[916],{"type":59,"value":917},"An in-place backend switch was attempted on a live release. Follow the runbook troubleshooting row (rollback, manual deletion of everything the failed revision created), then restart from the readiness skill.",{"type":54,"tag":62,"props":919,"children":921},{"id":920},"next-skill-to-run",[922],{"type":59,"value":923},"Next skill to run",{"type":54,"tag":103,"props":925,"children":926},{},[927],{"type":54,"tag":107,"props":928,"children":929},{},[930,936],{"type":54,"tag":83,"props":931,"children":933},{"className":932},[],[934],{"type":59,"value":935},"verify-mongodb-percona-migration",{"type":59,"value":937}," (always; it also performs the restore\non the restore path)",{"type":54,"tag":62,"props":939,"children":941},{"id":940},"references",[942],{"type":59,"value":943},"References",{"type":54,"tag":764,"props":945,"children":946},{},[947,963],{"type":54,"tag":768,"props":948,"children":949},{},[950],{"type":54,"tag":772,"props":951,"children":952},{},[953,958],{"type":54,"tag":776,"props":954,"children":955},{},[956],{"type":59,"value":957},"Topic",{"type":54,"tag":776,"props":959,"children":960},{},[961],{"type":59,"value":962},"Reference",{"type":54,"tag":787,"props":964,"children":965},{},[966,983,1000],{"type":54,"tag":772,"props":967,"children":968},{},[969,974],{"type":54,"tag":794,"props":970,"children":971},{},[972],{"type":59,"value":973},"Runbook (source of truth)",{"type":54,"tag":794,"props":975,"children":976},{},[977],{"type":54,"tag":83,"props":978,"children":980},{"className":979},[],[981],{"type":59,"value":982},"docs\u002Frunbooks\u002Fmongodb-bitnami-to-percona-migration.md",{"type":54,"tag":772,"props":984,"children":985},{},[986,991],{"type":54,"tag":794,"props":987,"children":988},{},[989],{"type":59,"value":990},"Scripts",{"type":54,"tag":794,"props":992,"children":993},{},[994],{"type":54,"tag":83,"props":995,"children":997},{"className":996},[],[998],{"type":59,"value":999},"scripts\u002Fmongodb-migration\u002F",{"type":54,"tag":772,"props":1001,"children":1002},{},[1003,1008],{"type":54,"tag":794,"props":1004,"children":1005},{},[1006],{"type":59,"value":1007},"Readiness",{"type":54,"tag":794,"props":1009,"children":1010},{},[1011],{"type":54,"tag":1012,"props":1013,"children":1015},"a",{"href":1014},"..\u002Fcheck-mongodb-migration-readiness\u002FSKILL.md",[1016],{"type":59,"value":88},{"type":54,"tag":1018,"props":1019,"children":1020},"style",{},[1021],{"type":59,"value":1022},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1024,"total":220},[1025],{"slug":4,"name":4,"fn":5,"description":6,"org":1026,"tags":1027,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1028,1029,1030,1031],{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"items":1033,"total":1187},[1034,1052,1068,1079,1091,1105,1118,1130,1143,1154,1168,1177],{"slug":1035,"name":1035,"fn":1036,"description":1037,"org":1038,"tags":1039,"stars":1049,"repoUrl":1050,"updatedAt":1051},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1040,1043,1046],{"name":1041,"slug":1042,"type":15},"Documentation","documentation",{"name":1044,"slug":1045,"type":15},"MCP","mcp",{"name":1047,"slug":1048,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-08-25T03:29:57.273192",{"slug":1053,"name":1053,"fn":1054,"description":1055,"org":1056,"tags":1057,"stars":1065,"repoUrl":1066,"updatedAt":1067},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1058,1061,1062],{"name":1059,"slug":1060,"type":15},"Containers","containers",{"name":23,"slug":24,"type":15},{"name":1063,"slug":1064,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1069,"name":1069,"fn":1070,"description":1071,"org":1072,"tags":1073,"stars":1065,"repoUrl":1066,"updatedAt":1078},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1074,1077],{"name":1075,"slug":1076,"type":15},"CI\u002FCD","ci-cd",{"name":23,"slug":24,"type":15},"2026-07-14T05:25:59.97109",{"slug":1080,"name":1080,"fn":1081,"description":1082,"org":1083,"tags":1084,"stars":1065,"repoUrl":1066,"updatedAt":1090},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1085,1086,1087],{"name":1075,"slug":1076,"type":15},{"name":23,"slug":24,"type":15},{"name":1088,"slug":1089,"type":15},"GitHub","github","2026-08-28T14:38:16.959248",{"slug":1092,"name":1092,"fn":1093,"description":1094,"org":1095,"tags":1096,"stars":1065,"repoUrl":1066,"updatedAt":1104},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1097,1100,1101],{"name":1098,"slug":1099,"type":15},"Debugging","debugging",{"name":1088,"slug":1089,"type":15},{"name":1102,"slug":1103,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1106,"name":1106,"fn":1107,"description":1108,"org":1109,"tags":1110,"stars":1065,"repoUrl":1066,"updatedAt":1117},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1111,1114],{"name":1112,"slug":1113,"type":15},"Best Practices","best-practices",{"name":1115,"slug":1116,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1119,"name":1119,"fn":1120,"description":1121,"org":1122,"tags":1123,"stars":1065,"repoUrl":1066,"updatedAt":1129},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1124,1127,1128],{"name":1125,"slug":1126,"type":15},"Machine Learning","machine-learning",{"name":20,"slug":21,"type":15},{"name":9,"slug":8,"type":15},"2026-08-21T03:36:57.470256",{"slug":1131,"name":1131,"fn":1132,"description":1133,"org":1134,"tags":1135,"stars":1065,"repoUrl":1066,"updatedAt":1142},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1136,1139],{"name":1137,"slug":1138,"type":15},"QA","qa",{"name":1140,"slug":1141,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":1144,"name":1144,"fn":1145,"description":1146,"org":1147,"tags":1148,"stars":1065,"repoUrl":1066,"updatedAt":1153},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1149,1150],{"name":23,"slug":24,"type":15},{"name":1151,"slug":1152,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":1155,"name":1155,"fn":1156,"description":1157,"org":1158,"tags":1159,"stars":1065,"repoUrl":1066,"updatedAt":1167},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1160,1163,1164],{"name":1161,"slug":1162,"type":15},"Code Review","code-review",{"name":1088,"slug":1089,"type":15},{"name":1165,"slug":1166,"type":15},"Pull Requests","pull-requests","2026-08-25T03:29:16.211287",{"slug":1169,"name":1169,"fn":1170,"description":1171,"org":1172,"tags":1173,"stars":1065,"repoUrl":1066,"updatedAt":1176},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1174,1175],{"name":1137,"slug":1138,"type":15},{"name":1140,"slug":1141,"type":15},"2026-07-14T05:25:54.928983",{"slug":1178,"name":1178,"fn":1179,"description":1180,"org":1181,"tags":1182,"stars":1065,"repoUrl":1066,"updatedAt":1186},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1183,1185],{"name":1184,"slug":32,"type":15},"Automation",{"name":1075,"slug":1076,"type":15},"2026-07-30T05:29:03.275638",563]