[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-google-cloud-gke-ai-troubleshooting-handle-disruption-gpu-tpu":3,"mdc--c10p2-key":34,"related-org-google-cloud-gke-ai-troubleshooting-handle-disruption-gpu-tpu":728,"related-repo-google-cloud-gke-ai-troubleshooting-handle-disruption-gpu-tpu":917},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":24,"repoUrl":25,"updatedAt":26,"license":27,"forks":28,"topics":29,"repo":30,"sourceUrl":32,"mdContent":33},"gke-ai-troubleshooting-handle-disruption-gpu-tpu","diagnose GPU and TPU workload disruptions","Diagnose and predict node disruption during Compute Engine host maintenance for GPU and TPU workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"google-cloud","Google Cloud","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgoogle-cloud.png","GoogleCloudPlatform",[13,17,20,21],{"name":14,"slug":15,"type":16},"Performance","performance","tag",{"name":18,"slug":19,"type":16},"Kubernetes","kubernetes",{"name":9,"slug":8,"type":16},{"name":22,"slug":23,"type":16},"Debugging","debugging",161,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fgke-mcp","2026-07-28T05:34:18.149515",null,78,[],{"repoUrl":25,"stars":24,"forks":28,"topics":31,"description":27},[],"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fgke-mcp\u002Ftree\u002FHEAD\u002Fskills\u002Fgke-ai-troubleshooting-handle-disruption-gpu-tpu","---\nname: gke-ai-troubleshooting-handle-disruption-gpu-tpu\ndescription: Diagnose and predict node disruption during Compute Engine host maintenance for GPU and TPU workloads.\n---\n\n# Handle Disruption on GPUs and TPUs Troubleshooting\n\n## 🔍 Diagnostic Workflow\n\n### Step 0: Context Acquisition & Execution Trigger\n\n- **Mandatory Context**: `project_id`, `location`, `cluster_name`, `timestamp`.\n- **Optional Context**: `node_name`, `workload_name`, `workload_namespace`, `nodepool_name`.\n- **CRITICAL EXECUTION DIRECTIVE (STRICTLY ENFORCED)**:\n  - **IF ALL 4 MANDATORY PARAMETERS ARE PRESENT**: DO NOT ask the user for confirmation, DO NOT echo parameters back asking for verification, and DO NOT output a text-only plan waiting for user input. YOU MUST IMMEDIATELY INVOKE DIAGNOSTIC TOOLS (`get_k8s_resource`, `query_prometheus`, `query_logs`) IN YOUR VERY FIRST TURN.\n  - **IF ANY MANDATORY PARAMETER IS MISSING**: Stop immediately and ask the user to provide ONLY the missing mandatory parameters.\n\n### Step 1: [Low Risk] Check for Upcoming Scheduled Maintenance\n\n- **Action**: Propose running `kubectl` to check if nodes have the scheduled maintenance label indicating an upcoming disruption.\n- **Example Command**:\n  ```bash\n  kubectl get nodes -l cloud.google.com\u002Fscheduled-maintenance-time -L cloud.google.com\u002Fscheduled-maintenance-time\n  ```\n- **Interpretation**: The `SCHEDULED-MAINTENANCE-TIME` column shows the Unix epoch time when the VM is scheduled for maintenance. If this label exists, a disruption is guaranteed to occur.\n\n### Step 2: [Low Risk] Investigation via Cloud Monitoring (PromQL)\n\n- **Action**: Call `query_prometheus` tool to query Cloud Monitoring PromQL metrics for the cluster. If `query_prometheus` is unavailable or fails due to missing permissions, provide the PromQL queries below to the user for manual verification in Google Cloud Console.\n- **For Past Disruptions**: When investigating past disruptions (e.g., historical timestamps or past\n  interruptions), immediately pass the `timestamp` parameter to `query_prometheus` to check\n  `kubernetes_io:node_interruption_count{monitored_resource=\"k8s_node\", interruption_reason=\"HW\u002FSW Maintenance\"}`\n  around that specific time. If metric value > 0, conclude that Compute Engine host maintenance WAS\n  confirmed as the cause of the past disruption.\n- **Example PromQL Queries**:\n  ```promql\n  # Fetch host maintenance events for nodes\n  sum by (interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_interruption_count{monitored_resource=\"k8s_node\", interruption_reason=\"HW\u002FSW Maintenance\"}[${__interval}]))\n  ```\n  ```promql\n  # See the interruption count aggregated by node pool\n  sum by (node_pool_name,interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_pool_interruption_count{monitored_resource=\"k8s_node_pool\", interruption_reason=\"HW\u002FSW Maintenance\", node_pool_name=\"\u003Cnodepool_name>\" }[${__interval}]))\n  ```\n- **Interpretation**: If `kubernetes_io:node_interruption_count` shows values > 0 for `interruption_reason=\"HW\u002FSW Maintenance\"`, it indicates the underlying Compute Engine VM was interrupted due to scheduled host maintenance.\n\n### Step 3: [Low Risk] Investigation via Cloud Logging & Node Status\n\n- **Action**: Call `query_logs` or `describe_k8s_resource` to filter GKE logs and check node status\u002Ftaints for active ongoing node maintenance.\n- **Active Ongoing Maintenance Detection**:\n  - Look for `cloud.google.com\u002Factive-node-maintenance` set to `ONGOING` or the `cloud.google.com\u002Fimpending-node-termination:NoSchedule` taint on the node.\n  - **Explicit Conclusion Required**: If `active-node-maintenance` is `ONGOING` or the\n    `impending-node-termination:NoSchedule` taint is present, explicitly inform the user:\n    _\"GKE is actively stopping workloads due to ongoing Compute Engine host maintenance.\"_\n  - **Mandatory Taint Warning**: Explicitly advise the user **NOT to tolerate** the\n    `cloud.google.com\u002Fimpending-node-termination:NoSchedule` taint, as the node will be\n    terminated by GKE regardless.\n\n### Step 4: Conclusion and Resolution\n\n- **Action**: Provide a summary of findings to the user and suggest appropriate mitigation\n  strategies ONLY IF host maintenance events were confirmed or scheduled.\n- **Negative Finding Rule**: If NO evidence of scheduled or past host maintenance is found,\n  explicitly conclude that the disruption was NOT caused by Compute Engine host maintenance and\n  report this negative finding. DO NOT recommend configuring graceful termination or opportunistic\n  maintenance when no maintenance events are detected.\n- **Reporting Rule**: Signal Only. Report high-signal information indicating that the disruption was\n  caused by Compute Engine host maintenance, specifically affecting the underlying GPU\u002FTPU nodes.\n  DO NOT dump raw logs.\n- **Resolutions to Suggest (Only when maintenance is confirmed or scheduled)**:\n  1. **Configure Graceful Termination**: Recommend configuring graceful termination by setting\n     `spec.terminationGracePeriodSeconds` (up to 60 minutes) to allow ML workloads (e.g., Orbax\n     checkpointing) to save state upon receiving `SIGTERM` before the node terminates.\n  2. **Opportunistic Maintenance**: Recommend configuring Opportunistic Maintenance to trigger host\n     updates automatically when GPU\u002FTPU nodes are idle.\n  3. **Capacity Buffer \u002F Resiliency**: Recommend configuring a `PodDisruptionBudget` (PDB)\n     specifying `minAvailable` replicas to maintain availability during disruptions.\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,48,55,62,201,215,315,327,468,480,609,615,722],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"handle-disruption-on-gpus-and-tpus-troubleshooting",[45],{"type":46,"value":47},"text","Handle Disruption on GPUs and TPUs Troubleshooting",{"type":40,"tag":49,"props":50,"children":52},"h2",{"id":51},"diagnostic-workflow",[53],{"type":46,"value":54},"🔍 Diagnostic Workflow",{"type":40,"tag":56,"props":57,"children":59},"h3",{"id":58},"step-0-context-acquisition-execution-trigger",[60],{"type":46,"value":61},"Step 0: Context Acquisition & Execution Trigger",{"type":40,"tag":63,"props":64,"children":65},"ul",{},[66,109,146],{"type":40,"tag":67,"props":68,"children":69},"li",{},[70,76,78,85,87,93,94,100,101,107],{"type":40,"tag":71,"props":72,"children":73},"strong",{},[74],{"type":46,"value":75},"Mandatory Context",{"type":46,"value":77},": ",{"type":40,"tag":79,"props":80,"children":82},"code",{"className":81},[],[83],{"type":46,"value":84},"project_id",{"type":46,"value":86},", ",{"type":40,"tag":79,"props":88,"children":90},{"className":89},[],[91],{"type":46,"value":92},"location",{"type":46,"value":86},{"type":40,"tag":79,"props":95,"children":97},{"className":96},[],[98],{"type":46,"value":99},"cluster_name",{"type":46,"value":86},{"type":40,"tag":79,"props":102,"children":104},{"className":103},[],[105],{"type":46,"value":106},"timestamp",{"type":46,"value":108},".",{"type":40,"tag":67,"props":110,"children":111},{},[112,117,118,124,125,131,132,138,139,145],{"type":40,"tag":71,"props":113,"children":114},{},[115],{"type":46,"value":116},"Optional Context",{"type":46,"value":77},{"type":40,"tag":79,"props":119,"children":121},{"className":120},[],[122],{"type":46,"value":123},"node_name",{"type":46,"value":86},{"type":40,"tag":79,"props":126,"children":128},{"className":127},[],[129],{"type":46,"value":130},"workload_name",{"type":46,"value":86},{"type":40,"tag":79,"props":133,"children":135},{"className":134},[],[136],{"type":46,"value":137},"workload_namespace",{"type":46,"value":86},{"type":40,"tag":79,"props":140,"children":142},{"className":141},[],[143],{"type":46,"value":144},"nodepool_name",{"type":46,"value":108},{"type":40,"tag":67,"props":147,"children":148},{},[149,154,156],{"type":40,"tag":71,"props":150,"children":151},{},[152],{"type":46,"value":153},"CRITICAL EXECUTION DIRECTIVE (STRICTLY ENFORCED)",{"type":46,"value":155},":\n",{"type":40,"tag":63,"props":157,"children":158},{},[159,191],{"type":40,"tag":67,"props":160,"children":161},{},[162,167,169,175,176,182,183,189],{"type":40,"tag":71,"props":163,"children":164},{},[165],{"type":46,"value":166},"IF ALL 4 MANDATORY PARAMETERS ARE PRESENT",{"type":46,"value":168},": DO NOT ask the user for confirmation, DO NOT echo parameters back asking for verification, and DO NOT output a text-only plan waiting for user input. YOU MUST IMMEDIATELY INVOKE DIAGNOSTIC TOOLS (",{"type":40,"tag":79,"props":170,"children":172},{"className":171},[],[173],{"type":46,"value":174},"get_k8s_resource",{"type":46,"value":86},{"type":40,"tag":79,"props":177,"children":179},{"className":178},[],[180],{"type":46,"value":181},"query_prometheus",{"type":46,"value":86},{"type":40,"tag":79,"props":184,"children":186},{"className":185},[],[187],{"type":46,"value":188},"query_logs",{"type":46,"value":190},") IN YOUR VERY FIRST TURN.",{"type":40,"tag":67,"props":192,"children":193},{},[194,199],{"type":40,"tag":71,"props":195,"children":196},{},[197],{"type":46,"value":198},"IF ANY MANDATORY PARAMETER IS MISSING",{"type":46,"value":200},": Stop immediately and ask the user to provide ONLY the missing mandatory parameters.",{"type":40,"tag":56,"props":202,"children":204},{"id":203},"step-1-low-risk-check-for-upcoming-scheduled-maintenance",[205,207,213],{"type":46,"value":206},"Step 1: ",{"type":40,"tag":208,"props":209,"children":210},"span",{},[211],{"type":46,"value":212},"Low Risk",{"type":46,"value":214}," Check for Upcoming Scheduled Maintenance",{"type":40,"tag":63,"props":216,"children":217},{},[218,236,297],{"type":40,"tag":67,"props":219,"children":220},{},[221,226,228,234],{"type":40,"tag":71,"props":222,"children":223},{},[224],{"type":46,"value":225},"Action",{"type":46,"value":227},": Propose running ",{"type":40,"tag":79,"props":229,"children":231},{"className":230},[],[232],{"type":46,"value":233},"kubectl",{"type":46,"value":235}," to check if nodes have the scheduled maintenance label indicating an upcoming disruption.",{"type":40,"tag":67,"props":237,"children":238},{},[239,244,245],{"type":40,"tag":71,"props":240,"children":241},{},[242],{"type":46,"value":243},"Example Command",{"type":46,"value":155},{"type":40,"tag":246,"props":247,"children":252},"pre",{"className":248,"code":249,"language":250,"meta":251,"style":251},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","kubectl get nodes -l cloud.google.com\u002Fscheduled-maintenance-time -L cloud.google.com\u002Fscheduled-maintenance-time\n","bash","",[253],{"type":40,"tag":79,"props":254,"children":255},{"__ignoreMap":251},[256],{"type":40,"tag":208,"props":257,"children":260},{"class":258,"line":259},"line",1,[261,266,272,277,282,287,292],{"type":40,"tag":208,"props":262,"children":264},{"style":263},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[265],{"type":46,"value":233},{"type":40,"tag":208,"props":267,"children":269},{"style":268},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[270],{"type":46,"value":271}," get",{"type":40,"tag":208,"props":273,"children":274},{"style":268},[275],{"type":46,"value":276}," nodes",{"type":40,"tag":208,"props":278,"children":279},{"style":268},[280],{"type":46,"value":281}," -l",{"type":40,"tag":208,"props":283,"children":284},{"style":268},[285],{"type":46,"value":286}," cloud.google.com\u002Fscheduled-maintenance-time",{"type":40,"tag":208,"props":288,"children":289},{"style":268},[290],{"type":46,"value":291}," -L",{"type":40,"tag":208,"props":293,"children":294},{"style":268},[295],{"type":46,"value":296}," cloud.google.com\u002Fscheduled-maintenance-time\n",{"type":40,"tag":67,"props":298,"children":299},{},[300,305,307,313],{"type":40,"tag":71,"props":301,"children":302},{},[303],{"type":46,"value":304},"Interpretation",{"type":46,"value":306},": The ",{"type":40,"tag":79,"props":308,"children":310},{"className":309},[],[311],{"type":46,"value":312},"SCHEDULED-MAINTENANCE-TIME",{"type":46,"value":314}," column shows the Unix epoch time when the VM is scheduled for maintenance. If this label exists, a disruption is guaranteed to occur.",{"type":40,"tag":56,"props":316,"children":318},{"id":317},"step-2-low-risk-investigation-via-cloud-monitoring-promql",[319,321,325],{"type":46,"value":320},"Step 2: ",{"type":40,"tag":208,"props":322,"children":323},{},[324],{"type":46,"value":212},{"type":46,"value":326}," Investigation via Cloud Monitoring (PromQL)",{"type":40,"tag":63,"props":328,"children":329},{},[330,353,385,443],{"type":40,"tag":67,"props":331,"children":332},{},[333,337,339,344,346,351],{"type":40,"tag":71,"props":334,"children":335},{},[336],{"type":46,"value":225},{"type":46,"value":338},": Call ",{"type":40,"tag":79,"props":340,"children":342},{"className":341},[],[343],{"type":46,"value":181},{"type":46,"value":345}," tool to query Cloud Monitoring PromQL metrics for the cluster. If ",{"type":40,"tag":79,"props":347,"children":349},{"className":348},[],[350],{"type":46,"value":181},{"type":46,"value":352}," is unavailable or fails due to missing permissions, provide the PromQL queries below to the user for manual verification in Google Cloud Console.",{"type":40,"tag":67,"props":354,"children":355},{},[356,361,363,368,370,375,377,383],{"type":40,"tag":71,"props":357,"children":358},{},[359],{"type":46,"value":360},"For Past Disruptions",{"type":46,"value":362},": When investigating past disruptions (e.g., historical timestamps or past\ninterruptions), immediately pass the ",{"type":40,"tag":79,"props":364,"children":366},{"className":365},[],[367],{"type":46,"value":106},{"type":46,"value":369}," parameter to ",{"type":40,"tag":79,"props":371,"children":373},{"className":372},[],[374],{"type":46,"value":181},{"type":46,"value":376}," to check\n",{"type":40,"tag":79,"props":378,"children":380},{"className":379},[],[381],{"type":46,"value":382},"kubernetes_io:node_interruption_count{monitored_resource=\"k8s_node\", interruption_reason=\"HW\u002FSW Maintenance\"}",{"type":46,"value":384},"\naround that specific time. If metric value > 0, conclude that Compute Engine host maintenance WAS\nconfirmed as the cause of the past disruption.",{"type":40,"tag":67,"props":386,"children":387},{},[388,393,394,420],{"type":40,"tag":71,"props":389,"children":390},{},[391],{"type":46,"value":392},"Example PromQL Queries",{"type":46,"value":155},{"type":40,"tag":246,"props":395,"children":399},{"className":396,"code":397,"language":398,"meta":251,"style":251},"language-promql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Fetch host maintenance events for nodes\nsum by (interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_interruption_count{monitored_resource=\"k8s_node\", interruption_reason=\"HW\u002FSW Maintenance\"}[${__interval}]))\n","promql",[400],{"type":40,"tag":79,"props":401,"children":402},{"__ignoreMap":251},[403,411],{"type":40,"tag":208,"props":404,"children":405},{"class":258,"line":259},[406],{"type":40,"tag":208,"props":407,"children":408},{},[409],{"type":46,"value":410},"# Fetch host maintenance events for nodes\n",{"type":40,"tag":208,"props":412,"children":414},{"class":258,"line":413},2,[415],{"type":40,"tag":208,"props":416,"children":417},{},[418],{"type":46,"value":419},"sum by (interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_interruption_count{monitored_resource=\"k8s_node\", interruption_reason=\"HW\u002FSW Maintenance\"}[${__interval}]))\n",{"type":40,"tag":246,"props":421,"children":423},{"className":396,"code":422,"language":398,"meta":251,"style":251},"# See the interruption count aggregated by node pool\nsum by (node_pool_name,interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_pool_interruption_count{monitored_resource=\"k8s_node_pool\", interruption_reason=\"HW\u002FSW Maintenance\", node_pool_name=\"\u003Cnodepool_name>\" }[${__interval}]))\n",[424],{"type":40,"tag":79,"props":425,"children":426},{"__ignoreMap":251},[427,435],{"type":40,"tag":208,"props":428,"children":429},{"class":258,"line":259},[430],{"type":40,"tag":208,"props":431,"children":432},{},[433],{"type":46,"value":434},"# See the interruption count aggregated by node pool\n",{"type":40,"tag":208,"props":436,"children":437},{"class":258,"line":413},[438],{"type":40,"tag":208,"props":439,"children":440},{},[441],{"type":46,"value":442},"sum by (node_pool_name,interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_pool_interruption_count{monitored_resource=\"k8s_node_pool\", interruption_reason=\"HW\u002FSW Maintenance\", node_pool_name=\"\u003Cnodepool_name>\" }[${__interval}]))\n",{"type":40,"tag":67,"props":444,"children":445},{},[446,450,452,458,460,466],{"type":40,"tag":71,"props":447,"children":448},{},[449],{"type":46,"value":304},{"type":46,"value":451},": If ",{"type":40,"tag":79,"props":453,"children":455},{"className":454},[],[456],{"type":46,"value":457},"kubernetes_io:node_interruption_count",{"type":46,"value":459}," shows values > 0 for ",{"type":40,"tag":79,"props":461,"children":463},{"className":462},[],[464],{"type":46,"value":465},"interruption_reason=\"HW\u002FSW Maintenance\"",{"type":46,"value":467},", it indicates the underlying Compute Engine VM was interrupted due to scheduled host maintenance.",{"type":40,"tag":56,"props":469,"children":471},{"id":470},"step-3-low-risk-investigation-via-cloud-logging-node-status",[472,474,478],{"type":46,"value":473},"Step 3: ",{"type":40,"tag":208,"props":475,"children":476},{},[477],{"type":46,"value":212},{"type":46,"value":479}," Investigation via Cloud Logging & Node Status",{"type":40,"tag":63,"props":481,"children":482},{},[483,506],{"type":40,"tag":67,"props":484,"children":485},{},[486,490,491,496,498,504],{"type":40,"tag":71,"props":487,"children":488},{},[489],{"type":46,"value":225},{"type":46,"value":338},{"type":40,"tag":79,"props":492,"children":494},{"className":493},[],[495],{"type":46,"value":188},{"type":46,"value":497}," or ",{"type":40,"tag":79,"props":499,"children":501},{"className":500},[],[502],{"type":46,"value":503},"describe_k8s_resource",{"type":46,"value":505}," to filter GKE logs and check node status\u002Ftaints for active ongoing node maintenance.",{"type":40,"tag":67,"props":507,"children":508},{},[509,514,515],{"type":40,"tag":71,"props":510,"children":511},{},[512],{"type":46,"value":513},"Active Ongoing Maintenance Detection",{"type":46,"value":155},{"type":40,"tag":63,"props":516,"children":517},{},[518,547,585],{"type":40,"tag":67,"props":519,"children":520},{},[521,523,529,531,537,539,545],{"type":46,"value":522},"Look for ",{"type":40,"tag":79,"props":524,"children":526},{"className":525},[],[527],{"type":46,"value":528},"cloud.google.com\u002Factive-node-maintenance",{"type":46,"value":530}," set to ",{"type":40,"tag":79,"props":532,"children":534},{"className":533},[],[535],{"type":46,"value":536},"ONGOING",{"type":46,"value":538}," or the ",{"type":40,"tag":79,"props":540,"children":542},{"className":541},[],[543],{"type":46,"value":544},"cloud.google.com\u002Fimpending-node-termination:NoSchedule",{"type":46,"value":546}," taint on the node.",{"type":40,"tag":67,"props":548,"children":549},{},[550,555,556,562,564,569,571,577,579],{"type":40,"tag":71,"props":551,"children":552},{},[553],{"type":46,"value":554},"Explicit Conclusion Required",{"type":46,"value":451},{"type":40,"tag":79,"props":557,"children":559},{"className":558},[],[560],{"type":46,"value":561},"active-node-maintenance",{"type":46,"value":563}," is ",{"type":40,"tag":79,"props":565,"children":567},{"className":566},[],[568],{"type":46,"value":536},{"type":46,"value":570}," or the\n",{"type":40,"tag":79,"props":572,"children":574},{"className":573},[],[575],{"type":46,"value":576},"impending-node-termination:NoSchedule",{"type":46,"value":578}," taint is present, explicitly inform the user:\n",{"type":40,"tag":580,"props":581,"children":582},"em",{},[583],{"type":46,"value":584},"\"GKE is actively stopping workloads due to ongoing Compute Engine host maintenance.\"",{"type":40,"tag":67,"props":586,"children":587},{},[588,593,595,600,602,607],{"type":40,"tag":71,"props":589,"children":590},{},[591],{"type":46,"value":592},"Mandatory Taint Warning",{"type":46,"value":594},": Explicitly advise the user ",{"type":40,"tag":71,"props":596,"children":597},{},[598],{"type":46,"value":599},"NOT to tolerate",{"type":46,"value":601}," the\n",{"type":40,"tag":79,"props":603,"children":605},{"className":604},[],[606],{"type":46,"value":544},{"type":46,"value":608}," taint, as the node will be\nterminated by GKE regardless.",{"type":40,"tag":56,"props":610,"children":612},{"id":611},"step-4-conclusion-and-resolution",[613],{"type":46,"value":614},"Step 4: Conclusion and Resolution",{"type":40,"tag":63,"props":616,"children":617},{},[618,627,637,647],{"type":40,"tag":67,"props":619,"children":620},{},[621,625],{"type":40,"tag":71,"props":622,"children":623},{},[624],{"type":46,"value":225},{"type":46,"value":626},": Provide a summary of findings to the user and suggest appropriate mitigation\nstrategies ONLY IF host maintenance events were confirmed or scheduled.",{"type":40,"tag":67,"props":628,"children":629},{},[630,635],{"type":40,"tag":71,"props":631,"children":632},{},[633],{"type":46,"value":634},"Negative Finding Rule",{"type":46,"value":636},": If NO evidence of scheduled or past host maintenance is found,\nexplicitly conclude that the disruption was NOT caused by Compute Engine host maintenance and\nreport this negative finding. DO NOT recommend configuring graceful termination or opportunistic\nmaintenance when no maintenance events are detected.",{"type":40,"tag":67,"props":638,"children":639},{},[640,645],{"type":40,"tag":71,"props":641,"children":642},{},[643],{"type":46,"value":644},"Reporting Rule",{"type":46,"value":646},": Signal Only. Report high-signal information indicating that the disruption was\ncaused by Compute Engine host maintenance, specifically affecting the underlying GPU\u002FTPU nodes.\nDO NOT dump raw logs.",{"type":40,"tag":67,"props":648,"children":649},{},[650,655,656],{"type":40,"tag":71,"props":651,"children":652},{},[653],{"type":46,"value":654},"Resolutions to Suggest (Only when maintenance is confirmed or scheduled)",{"type":46,"value":155},{"type":40,"tag":657,"props":658,"children":659},"ol",{},[660,686,696],{"type":40,"tag":67,"props":661,"children":662},{},[663,668,670,676,678,684],{"type":40,"tag":71,"props":664,"children":665},{},[666],{"type":46,"value":667},"Configure Graceful Termination",{"type":46,"value":669},": Recommend configuring graceful termination by setting\n",{"type":40,"tag":79,"props":671,"children":673},{"className":672},[],[674],{"type":46,"value":675},"spec.terminationGracePeriodSeconds",{"type":46,"value":677}," (up to 60 minutes) to allow ML workloads (e.g., Orbax\ncheckpointing) to save state upon receiving ",{"type":40,"tag":79,"props":679,"children":681},{"className":680},[],[682],{"type":46,"value":683},"SIGTERM",{"type":46,"value":685}," before the node terminates.",{"type":40,"tag":67,"props":687,"children":688},{},[689,694],{"type":40,"tag":71,"props":690,"children":691},{},[692],{"type":46,"value":693},"Opportunistic Maintenance",{"type":46,"value":695},": Recommend configuring Opportunistic Maintenance to trigger host\nupdates automatically when GPU\u002FTPU nodes are idle.",{"type":40,"tag":67,"props":697,"children":698},{},[699,704,706,712,714,720],{"type":40,"tag":71,"props":700,"children":701},{},[702],{"type":46,"value":703},"Capacity Buffer \u002F Resiliency",{"type":46,"value":705},": Recommend configuring a ",{"type":40,"tag":79,"props":707,"children":709},{"className":708},[],[710],{"type":46,"value":711},"PodDisruptionBudget",{"type":46,"value":713}," (PDB)\nspecifying ",{"type":40,"tag":79,"props":715,"children":717},{"className":716},[],[718],{"type":46,"value":719},"minAvailable",{"type":46,"value":721}," replicas to maintain availability during disruptions.",{"type":40,"tag":723,"props":724,"children":725},"style",{},[726],{"type":46,"value":727},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":729,"total":916},[730,748,764,786,800,811,825,842,859,872,888,898],{"slug":731,"name":731,"fn":732,"description":733,"org":734,"tags":735,"stars":745,"repoUrl":746,"updatedAt":747},"kb-search","search and extract local knowledge base documents","Allows listing, searching and extracting information from local knowledge base documents for information about tables\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[736,739,742],{"name":737,"slug":738,"type":16},"Documentation","documentation",{"name":740,"slug":741,"type":16},"Knowledge Base","knowledge-base",{"name":743,"slug":744,"type":16},"Search","search",6749,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fknowledge-catalog","2026-07-12T07:38:52.157375",{"slug":749,"name":750,"fn":751,"description":752,"org":753,"tags":754,"stars":745,"repoUrl":746,"updatedAt":763},"knowledgecatalogdiscoveryagent","knowledge_catalog_discovery_agent","search and rank Knowledge Catalog data entries","Analyzes user queries, extracts relevant predicates, and utilizes Knowledge Catalog Search to find and rank the most relevant data entries. Engages with the user throughout the process.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[755,758,759,762],{"name":756,"slug":757,"type":16},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":16},{"name":760,"slug":761,"type":16},"Knowledge Management","knowledge-management",{"name":743,"slug":744,"type":16},"2026-07-12T07:38:22.196851",{"slug":765,"name":765,"fn":766,"description":767,"org":768,"tags":769,"stars":783,"repoUrl":784,"updatedAt":785},"contributing","contribute to Cloud Foundation Fabric","End-to-end workflow for contributing to Cloud Foundation Fabric: triaging GitHub issues, proactive feature development, validating with tests and Policy Troubleshooter, and submitting sanitized Pull Requests. Use when addressing a Fabric GitHub issue, developing a module or FAST stage change, or preparing a branch for a pull request.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[770,773,776,779,780],{"name":771,"slug":772,"type":16},"Automation","automation",{"name":774,"slug":775,"type":16},"Engineering","engineering",{"name":777,"slug":778,"type":16},"GitHub","github",{"name":9,"slug":8,"type":16},{"name":781,"slug":782,"type":16},"Pull Requests","pull-requests",2062,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fcloud-foundation-fabric","2026-07-31T06:23:36.935005",{"slug":787,"name":787,"fn":788,"description":789,"org":790,"tags":791,"stars":783,"repoUrl":784,"updatedAt":799},"fabric-builder","generate Terraform code for Google Cloud","Generates idiomatic Cloud Foundation Fabric (CFF) Terraform code using CFF modules. Use when users ask to create GCP resources, use Fabric modules, or generate Terraform code for Google Cloud.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[792,793,796],{"name":9,"slug":8,"type":16},{"name":794,"slug":795,"type":16},"Infrastructure as Code","infrastructure-as-code",{"name":797,"slug":798,"type":16},"Terraform","terraform","2026-07-12T07:38:23.514555",{"slug":801,"name":801,"fn":802,"description":803,"org":804,"tags":805,"stars":783,"repoUrl":784,"updatedAt":810},"fast-0-org-setup-prereqs","prepare prerequisites for FAST 0-org-setup","Guides the user step-by-step through the prerequisites for the FAST 0-org-setup stage, supporting both Standard GCP and Google Cloud Dedicated (GCD) environments. Use when a user asks to prepare or run prerequisites for 0-org-setup or bootstrap the FAST landing zone.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[806,807],{"name":9,"slug":8,"type":16},{"name":808,"slug":809,"type":16},"Operations","operations","2026-07-12T07:38:28.127148",{"slug":812,"name":812,"fn":813,"description":814,"org":815,"tags":816,"stars":822,"repoUrl":823,"updatedAt":824},"agent-aware-cli","design agent-aware command-line interfaces","Guide for designing and implementing command-line interfaces (CLIs) that are equally usable by human developers and automated coding agents. Use when the user wants to build a CLI, apply CLI best practices, or use Go with Cobra and Viper.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[817,820,821],{"name":818,"slug":819,"type":16},"CLI","cli",{"name":774,"slug":775,"type":16},{"name":9,"slug":8,"type":16},1150,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fvertex-ai-creative-studio","2026-07-12T07:39:08.41406",{"slug":826,"name":826,"fn":827,"description":828,"org":829,"tags":830,"stars":822,"repoUrl":823,"updatedAt":841},"build-mcp-genmedia","build and configure GenAI MCP servers","Builds the mcp-genmedia Go MCP servers (nanobanana, veo, lyria, gemini-multimodal, chirp3-hd, avtool) from source and wires them into settings.json. Use this skill whenever the MCP tools are missing or broken — typically at the start of a new session, after a container restart, or when \u002Ftmp has been wiped. The prebuilt binaries in \u002Fworkspace\u002F.local\u002Fbin\u002F have no exec bit and live on a noexec mount; this skill compiles fresh executables into \u002Ftmp\u002Fbin\u002F where execution is allowed.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[831,834,835,838],{"name":832,"slug":833,"type":16},"API Development","api-development",{"name":9,"slug":8,"type":16},{"name":836,"slug":837,"type":16},"LLM","llm",{"name":839,"slug":840,"type":16},"MCP","mcp","2026-07-12T07:39:10.911302",{"slug":843,"name":843,"fn":844,"description":845,"org":846,"tags":847,"stars":822,"repoUrl":823,"updatedAt":858},"genmedia-audio-engineer","synthesize and mix audio content","Expert in audio synthesis, music generation, and mixing. Use when creating podcasts, background scores, or multi-track audio layering using mcp-chirp3-go, mcp-lyria-go, mcp-gemini-go, mcp-nanobanana-go, and mcp-avtool-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[848,851,854,855],{"name":849,"slug":850,"type":16},"Audio","audio",{"name":852,"slug":853,"type":16},"Creative","creative",{"name":9,"slug":8,"type":16},{"name":856,"slug":857,"type":16},"Vertex AI","vertex-ai","2026-07-12T07:39:16.623879",{"slug":860,"name":860,"fn":861,"description":862,"org":863,"tags":864,"stars":822,"repoUrl":823,"updatedAt":871},"genmedia-image-artist","generate and edit AI images","Expert in AI image generation and editing. Use when the user needs high-quality textures, character-consistent visuals, or image-to-image editing using mcp-nanobanana-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[865,866,867,870],{"name":852,"slug":853,"type":16},{"name":9,"slug":8,"type":16},{"name":868,"slug":869,"type":16},"Image Generation","image-generation",{"name":856,"slug":857,"type":16},"2026-07-12T07:39:15.372822",{"slug":873,"name":873,"fn":874,"description":875,"org":876,"tags":877,"stars":822,"repoUrl":823,"updatedAt":887},"genmedia-producer","produce multi-step media content","Expert media production assistant. Use when requested to help with storyboarding, podcast creation, audio assembly, or complex multi-step media workflows using the GenMedia MCP servers (Veo, Lyria, Gemini TTS, NanoBanana).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[878,879,880,881,884],{"name":849,"slug":850,"type":16},{"name":852,"slug":853,"type":16},{"name":9,"slug":8,"type":16},{"name":882,"slug":883,"type":16},"Media","media",{"name":885,"slug":886,"type":16},"Video","video","2026-07-12T07:39:09.672849",{"slug":889,"name":889,"fn":890,"description":891,"org":892,"tags":893,"stars":822,"repoUrl":823,"updatedAt":897},"genmedia-video-editor","edit and compose video content","Expert in video composition, editing, and format conversion. Use when the user wants to generate high-quality video, overlay images on video, concatenate clips, create GIFs, or sync audio to video using mcp-avtool-go and mcp-veo-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[894,895,896],{"name":852,"slug":853,"type":16},{"name":9,"slug":8,"type":16},{"name":885,"slug":886,"type":16},"2026-07-12T07:39:13.749081",{"slug":899,"name":899,"fn":900,"description":901,"org":902,"tags":903,"stars":822,"repoUrl":823,"updatedAt":915},"genmedia-voice-director","generate expressive text-to-speech with Gemini","Expert in casting, directing, and generating expressive text-to-speech using Gemini TTS. Use this when the user needs virtual voice actor personas, expressive speech generation, or multiple variations of a voiceover (like \"take 3 on the bounce\").",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[904,905,906,909,912],{"name":849,"slug":850,"type":16},{"name":852,"slug":853,"type":16},{"name":907,"slug":908,"type":16},"Gemini","gemini",{"name":910,"slug":911,"type":16},"Speech","speech",{"name":913,"slug":914,"type":16},"Text-to-Speech","text-to-speech","2026-07-12T07:39:17.86673",80,{"items":918,"total":1002},[919,931,938,951,964,975,991],{"slug":920,"name":920,"fn":921,"description":922,"org":923,"tags":924,"stars":24,"repoUrl":25,"updatedAt":930},"custom-golden-image-discovery","discover golden base images for GKE nodes","Expert at discovering golden base images for GKE custom nodes using technical specs or context clues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[925,928,929],{"name":926,"slug":927,"type":16},"Deployment","deployment",{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},"2026-07-12T07:39:30.888879",{"slug":4,"name":4,"fn":5,"description":6,"org":932,"tags":933,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[934,935,936,937],{"name":22,"slug":23,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":939,"name":939,"fn":940,"description":941,"org":942,"tags":943,"stars":24,"repoUrl":25,"updatedAt":950},"gke-ai-troubleshooting-jobset-interruption","diagnose GKE JobSet interruptions","Systematically diagnose GKE JobSet interruptions, restarts, and preemptions for AI\u002FML training workloads. Identifies preemption events, maintenance interruptions, bad host VMs, unhealthy pods, and coordinator worker failures.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[944,945,946,947],{"name":22,"slug":23,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":948,"slug":949,"type":16},"Observability","observability","2026-07-12T07:40:04.511878",{"slug":952,"name":952,"fn":953,"description":954,"org":955,"tags":956,"stars":24,"repoUrl":25,"updatedAt":963},"gke-ai-troubleshooting-skill-creation-guide","create GKE troubleshooting skill bundles","Expert instructions for building high-quality GKE troubleshooting skills. Codifies Step 0 context rules, zero-hallucination signatures, and explicit LQL\u002FPromQL query requirements.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[957,958,959,960],{"name":737,"slug":738,"type":16},{"name":774,"slug":775,"type":16},{"name":9,"slug":8,"type":16},{"name":961,"slug":962,"type":16},"Technical Writing","technical-writing","2026-07-12T07:39:50.73484",{"slug":965,"name":965,"fn":966,"description":967,"org":968,"tags":969,"stars":24,"repoUrl":25,"updatedAt":974},"gke-ai-troubleshooting-tpu-connection-failure-vbar-oom","diagnose GKE TPU connection failures","Diagnose and prevent `vbar_control_agent` segfaults and OOMs caused by race conditions during TPU device resets and frequent metrics collection (e.g. every 3s). Use when TPU slice initialization fails or `vbar_control_agent` crashes on TPU v6e nodes.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[970,971,972,973],{"name":22,"slug":23,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-07-12T07:39:49.482979",{"slug":976,"name":976,"fn":977,"description":978,"org":979,"tags":980,"stars":24,"repoUrl":25,"updatedAt":990},"gke-app-onboarding","containerize and deploy apps to GKE","Workflows for containerizing and deploying applications to GKE for the first time.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[981,984,985,986,987],{"name":982,"slug":983,"type":16},"Containers","containers",{"name":926,"slug":927,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":988,"slug":989,"type":16},"Onboarding","onboarding","2026-07-12T07:39:41.935837",{"slug":992,"name":992,"fn":993,"description":994,"org":995,"tags":996,"stars":24,"repoUrl":25,"updatedAt":1001},"gke-backup-dr","configure GKE backup and disaster recovery","Workflows for configuring Backup for GKE and disaster recovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[997,998,999,1000],{"name":926,"slug":927,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":808,"slug":809,"type":16},"2026-07-12T07:39:34.806995",25]