[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-google-cloud-gke-ai-troubleshooting-jobset-interruption":3,"mdc--pp2e1k-key":34,"related-repo-google-cloud-gke-ai-troubleshooting-jobset-interruption":1879,"related-org-google-cloud-gke-ai-troubleshooting-jobset-interruption":1971},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":24,"repoUrl":25,"updatedAt":26,"license":27,"forks":28,"topics":29,"repo":30,"sourceUrl":32,"mdContent":33},"gke-ai-troubleshooting-jobset-interruption","diagnose GKE JobSet interruptions","Systematically diagnose GKE JobSet interruptions, restarts, and preemptions for AI\u002FML training workloads. Identifies preemption events, maintenance interruptions, bad host VMs, unhealthy pods, and coordinator worker failures.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"google-cloud","Google Cloud","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgoogle-cloud.png","GoogleCloudPlatform",[13,17,20,21],{"name":14,"slug":15,"type":16},"Observability","observability","tag",{"name":18,"slug":19,"type":16},"Kubernetes","kubernetes",{"name":9,"slug":8,"type":16},{"name":22,"slug":23,"type":16},"Debugging","debugging",161,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fgke-mcp","2026-07-12T07:40:04.511878",null,78,[],{"repoUrl":25,"stars":24,"forks":28,"topics":31,"description":27},[],"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fgke-mcp\u002Ftree\u002FHEAD\u002Fskills\u002Fgke-ai-troubleshooting-jobset-interruption","---\nname: gke-ai-troubleshooting-jobset-interruption\ndescription: >\n  Systematically diagnose GKE JobSet interruptions, restarts, and preemptions for AI\u002FML training workloads.\n  Identifies preemption events, maintenance interruptions, bad host VMs, unhealthy pods, and coordinator worker failures.\n---\n\n# GKE JobSet Interruption Troubleshooting\n\nUse this skill to systematically diagnose and resolve JobSet interruptions, restarts, and preemptions on GKE clusters hosting large-scale AI\u002FML workloads.\n\n## ⚠️ Prerequisites\n\n- [ ] JobSet metrics package must be enabled in `kube-state-metrics` for your cluster (see [KSM JobSet Metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fkube-state-metrics#ksm-jobset-metrics)).\n- [ ] Cloud Logging and Cloud Monitoring must be enabled for the Google Cloud Project.\n\n## 🔍 Diagnostic Workflow\n\n### Step 0: Context Acquisition & Time Window Definition\n\nTo begin troubleshooting, acquire the following context from the user:\n\n- **Project ID** (e.g., `customer-ai-project-123`)\n- **Cluster Name** (e.g., `tpu-cluster-prod`)\n- **Workload Name (JobSet Name)** (e.g., `llama3-70b-training`)\n- **Workload Namespace** (e.g., `default`)\n- **Issue Time** (e.g., `2026-05-20T08:15:00Z`)\n\n#### Time Handling Rules\n\n1.  **Reject Relative Time**: If the user says \"X minutes ago\" or \"just now\", stop and request the exact timestamp or specific time window.\n2.  **Window Calculation**: If the user provides a single timestamp `T`, calculate the query window as **`[T - 30m]` to `[T + 30m]`**.\n    - Let `Start_Time` = `T - 30m`\n    - Let `End_Time` = `T + 30m`\n\n---\n\n### Step 1: Identify JobSet Restarts and Attempts [Low Risk]\n\nVerify if the JobSet is experiencing restart loops and determine the frequency of restarts.\n\n#### Option A: Visual Chart (MQL) - restarts\n\n- **Tool to use**: `monitoring_time_series_chart`\n- **MQL Query**:\n  ```mql\n  fetch prometheus_target\n  | metric 'prometheus.googleapis.com\u002Fkube_jobset_restarts\u002Fgauge'\n  | filter resource.cluster_name == '\u003Ccluster_name>' && metric.jobset_name == '\u003Cworkload_name>'\n  | align next_older(1m)\n  | every 1m\n  | group_by [metric.jobset_name], [val: max(value)]\n  ```\n\n#### Option B: PromQL Query - restarts\n\n- **Method**: Execute via shell command (requires `BypassSandbox: true`).\n- **Command Template**:\n  ```bash\n  curl -sSf -H \"Authorization: Bearer \\$(gcloud auth print-access-token)\" \\\n    \"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=kube_jobset_restarts%7Bjobset_name%3D%22\u003Cworkload_name>%22%2Ccluster%3D%22\u003Ccluster_name>%22%7D\"\n  ```\n- **Logic**: A non-zero or increasing value for restarts indicates that the JobSet is being actively restarted by the controller due to worker failure or interruption.\n- **Automation**: Proceed to Step 2 automatically after reporting findings.\n\n---\n\n### Step 2: Inspect Nodepool Interruptions [Low Risk]\n\nDetermine if the JobSet restarts were triggered by physical nodepool-level events (such as spot preemptions, maintenance, or host terminations).\n\n#### A. Metrics Query (Nodepool Interruption Counts)\n\n##### Option A: Visual Chart (MQL) - interruptions\n\n- **Tool to use**: `monitoring_time_series_chart`\n- **MQL Query**:\n  ```mql\n  fetch k8s_node_pool\n  | metric 'kubernetes.io\u002Fnode_pool\u002Finterruption_count'\n  | filter cluster_name == '\u003Ccluster_name>'\n  | align next_older(10m)\n  | every 10m\n  | group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]\n  ```\n\n##### Option B: PromQL Query - interruptions\n\n- **Method**: Execute via shell command (requires `BypassSandbox: true`).\n- **Command Template**:\n  ```bash\n  curl -sSf -H \"Authorization: Bearer \\$(gcloud auth print-access-token)\" \\\n    \"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=sum%20by%20%28interruption_type%2C%20interruption_reason%2C%20node_pool_name%2C%20cluster_name%29%20%28avg_over_time%28%7B__name__%3D%22kubernetes.io%2Fnode_pool%2Finterruption_count%22%2C%20monitored_resource%3D%22k8s_node_pool%22%2C%20cluster_name%3D%22\u003Ccluster_name>%22%7D%5B10m%5D%29%29\"\n  ```\n\n#### B. Log Query (Nodepool Life Events)\n\n- **Tool to use**: `query_logs`\n- **LQL Log Filter Template**:\n  ```sql\n  resource.type=\"gke_nodepool\"\n  AND resource.labels.cluster_name=\"\u003Ccluster_name>\"\n  AND timestamp >= \"\u003CStart_Time>\"\n  AND timestamp \u003C= \"\u003CEnd_Time>\"\n  ```\n- **Logic**:\n  - **PreemptionEvent**: Spot VMs were preempted, or node was scale-down.\n  - **MaintenanceEvent**: Node pool updated or Google scheduled maintenance.\n  - **TerminationEvent**: Serious host failures. Check `interruption_reason` or logs for host issues.\n- **Automation**: Proceed to Step 3 automatically.\n\n---\n\n### Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]\n\nCorrelate node readiness failures with physical host VMs to see if a single faulty host repeatedly fails coordinator pods.\n\n#### A. Metrics Query (Node Ready Status Check)\n\n##### Option A: Visual Chart (MQL) - node status\n\n- **Tool to use**: `monitoring_time_series_chart`\n- **MQL Query**:\n  ```mql\n  fetch k8s_node\n  | metric 'kubernetes.io\u002Fnode\u002Fstatus_condition'\n  | filter cluster_name == '\u003Ccluster_name>' && metric.condition == 'Ready' && metric.status == 'False'\n  | align next_older(1m)\n  | every 1m\n  | group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]\n  ```\n\n##### Option B: PromQL Query - node status\n\n- **Method**: Execute via shell command (requires `BypassSandbox: true`).\n- **Command Template**:\n  ```bash\n  curl -sSf -H \"Authorization: Bearer \\$(gcloud auth print-access-token)\" \\\n    \"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=sum%20by%20%28status%2C%20condition%2C%20node_pool_name%29%20%28%7B__name__%3D%22kubernetes.io%2Fnode%2Fstatus_condition%22%2C%20monitored_resource%3D%22k8s_node%22%2C%20cluster_name%3D%22\u003Ccluster_name>%22%2C%20condition%3D%22Ready%22%2C%20status%3D%22False%22%7D%29\"\n  ```\n\n#### B. Metrics Query (Node-to-Host Metadata Topology Correlation)\n\n- **Tool to use**: `monitoring_time_series_chart`\n- **MQL Query**:\n  ```mql\n  fetch k8s_node\n  | metric 'kubernetes.io\u002Fnode\u002Fcpu\u002Ftotal_cores'\n  | filter cluster_name == '\u003Ccluster_name>'\n  | align next_older(1m)\n  | every 1m\n  | group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]\n  ```\n\n#### C. Log Query (Node Fault Logs)\n\n- **Tool to use**: `query_logs`\n- **LQL Log Filter Template**:\n  ```sql\n  resource.type=\"k8s_node\"\n  AND resource.labels.cluster_name=\"\u003Ccluster_name>\"\n  AND (textPayload:\"host error\" OR textPayload:\"kernel panic\" OR textPayload:\"hardware failure\" OR textPayload:\"NodeNotReady\")\n  AND timestamp >= \"\u003CStart_Time>\"\n  AND timestamp \u003C= \"\u003CEnd_Time>\"\n  ```\n- **Logic**: Identify if specific nodes are unhealthy (`Ready=False` or `Unknown`) and correlate them to their GCE physical host ID via `metadata.user.gce_topology_host`. Check if the same host is repeatedly failing.\n- **Automation**: Proceed to Step 4 automatically.\n\n---\n\n### Step 4: Inspect Pod and Worker \u002F Container Failures [Low Risk]\n\nAnalyze pod status phases and retrieve coordinator worker logs to identify application-level crashes or network deadlocks.\n\n#### A. Metrics Query (Pod Lifecycle Phases)\n\n##### Option A: Visual Chart (MQL) - pod phase\n\n- **Tool to use**: `monitoring_time_series_chart`\n- **MQL Query**:\n  ```mql\n  fetch k8s_pod\n  | metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Fphase'\n  | filter cluster_name == '\u003Ccluster_name>' && pod_name ==~ '\u003Cworkload_name>.*'\n  | align next_older(10m)\n  | every 10m\n  | group_by [metric.phase], [val: count()]\n  ```\n\n##### Option B: PromQL Query - pod phase\n\n- **Method**: Execute via shell command (requires `BypassSandbox: true`).\n- **Command Template**:\n  ```bash\n  curl -sSf -H \"Authorization: Bearer \\$(gcloud auth print-access-token)\" \\\n    \"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=sum%20by%20%28phase%29%20%28avg_over_time%28%7B__name__%3D%22kube_pod_status_phase%22%2C%20cluster%3D%22\u003Ccluster_name>%22%2C%20pod%3D~%22\u003Cworkload_name>.*%22%7D%5B10m%5D%29%29\"\n  ```\n\n#### B. Metrics Query (Unschedulable Pod Count)\n\n- **Tool to use**: `monitoring_time_series_chart`\n- **MQL Query**:\n  ```mql\n  fetch k8s_pod\n  | metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Funschedulable'\n  | filter cluster_name == '\u003Ccluster_name>' && pod_name ==~ '\u003Cworkload_name>.*'\n  | align next_older(10m)\n  | every 10m\n  | group_by [pod_name], [val: max(value)]\n  ```\n\n#### C. Log Query (Worker Container Logs)\n\n- **Tool to use**: `query_logs`\n- **LQL Log Filter Template**:\n  ```sql\n  resource.type=\"k8s_container\"\n  AND resource.labels.cluster_name=\"\u003Ccluster_name>\"\n  AND labels.\"k8s-pod\u002Fjobset_sigs_k8s_io\u002Fjobset-name\"=\"\u003Cworkload_name>\"\n  AND timestamp >= \"\u003CStart_Time>\"\n  AND timestamp \u003C= \"\u003CEnd_Time>\"\n  ```\n- **Logic**:\n  1. Check the pod timeline to spot pending or unschedulable pods.\n  2. Use the worker container logs to analyze worker 0 in slice 0 (coordinator) for NCCL timeouts, collective communication issues, or MegaScale hangs.\n- **Automation**: Proceed to Resolution.\n\n---\n\n## 🛠️ Resolution Workflow\n\n### Resolution 1: Preemption & Autoscaling Optimizations [Low Risk]\n\nIf Step 2 showed high preemption counts on Spot VMs:\n\n- **Action**: Suggest switching critical long-running training workloads to **GKE Reserved\u002FOn-Demand VMs** or utilizing **Compact Placement Policies** to minimize defragmentation interruptions.\n- **Justification**: Eliminates spot-market preemptions and reduces training restarts.\n\n### Resolution 2: Quarantine Faulty Host VMs [High Risk]\n\nIf Step 3 identified a specific host ID (`gce-topology-host`) that consistently fails or triggers restarts across multiple attempts:\n\n- **Action**: Draft a recommendation to cordon\u002Fdrain the GKE node, delete the underlying GCE VM instance to trigger instance recreation, and open a support ticket with Google Cloud Support specifying the physical host ID.\n- **Justification**: GKE auto-repair will recreate the VM instance on healthy physical hardware, preventing infinite restart loops.\n- **Automation**: Stop and request explicit user confirmation before cordoning\u002Fdraining any node.\n\n---\n\n## 📋 Copypaste Checklist\n\n- [ ] Acquire context and compute `[T - 30m, T + 30m]` window.\n- [ ] Query JobSet restart attempts.\n- [ ] Check Nodepool interruptions (spot preemptions vs. hardware terminations).\n- [ ] Query node-to-host mapping and check node logs for physical host errors.\n- [ ] Inspect pod timeline status and coordinator worker container logs.\n- [ ] Recommend appropriate scheduling strategy (On-demand vs Spot) or host VM quarantining.\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,48,54,61,110,116,123,128,214,221,311,315,327,332,338,433,439,567,570,580,585,591,598,677,683,770,776,901,904,914,919,925,931,1008,1014,1101,1107,1182,1188,1298,1301,1311,1316,1322,1328,1405,1411,1498,1504,1579,1585,1683,1686,1692,1702,1707,1744,1755,1768,1798,1801,1807,1873],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"gke-jobset-interruption-troubleshooting",[45],{"type":46,"value":47},"text","GKE JobSet Interruption Troubleshooting",{"type":40,"tag":49,"props":50,"children":51},"p",{},[52],{"type":46,"value":53},"Use this skill to systematically diagnose and resolve JobSet interruptions, restarts, and preemptions on GKE clusters hosting large-scale AI\u002FML workloads.",{"type":40,"tag":55,"props":56,"children":58},"h2",{"id":57},"️-prerequisites",[59],{"type":46,"value":60},"⚠️ Prerequisites",{"type":40,"tag":62,"props":63,"children":66},"ul",{"className":64},[65],"contains-task-list",[67,101],{"type":40,"tag":68,"props":69,"children":72},"li",{"className":70},[71],"task-list-item",[73,79,81,88,90,99],{"type":40,"tag":74,"props":75,"children":78},"input",{"disabled":76,"type":77},true,"checkbox",[],{"type":46,"value":80}," JobSet metrics package must be enabled in ",{"type":40,"tag":82,"props":83,"children":85},"code",{"className":84},[],[86],{"type":46,"value":87},"kube-state-metrics",{"type":46,"value":89}," for your cluster (see ",{"type":40,"tag":91,"props":92,"children":96},"a",{"href":93,"rel":94},"https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fkube-state-metrics#ksm-jobset-metrics",[95],"nofollow",[97],{"type":46,"value":98},"KSM JobSet Metrics",{"type":46,"value":100},").",{"type":40,"tag":68,"props":102,"children":104},{"className":103},[71],[105,108],{"type":40,"tag":74,"props":106,"children":107},{"disabled":76,"type":77},[],{"type":46,"value":109}," Cloud Logging and Cloud Monitoring must be enabled for the Google Cloud Project.",{"type":40,"tag":55,"props":111,"children":113},{"id":112},"diagnostic-workflow",[114],{"type":46,"value":115},"🔍 Diagnostic Workflow",{"type":40,"tag":117,"props":118,"children":120},"h3",{"id":119},"step-0-context-acquisition-time-window-definition",[121],{"type":46,"value":122},"Step 0: Context Acquisition & Time Window Definition",{"type":40,"tag":49,"props":124,"children":125},{},[126],{"type":46,"value":127},"To begin troubleshooting, acquire the following context from the user:",{"type":40,"tag":62,"props":129,"children":130},{},[131,150,166,182,198],{"type":40,"tag":68,"props":132,"children":133},{},[134,140,142,148],{"type":40,"tag":135,"props":136,"children":137},"strong",{},[138],{"type":46,"value":139},"Project ID",{"type":46,"value":141}," (e.g., ",{"type":40,"tag":82,"props":143,"children":145},{"className":144},[],[146],{"type":46,"value":147},"customer-ai-project-123",{"type":46,"value":149},")",{"type":40,"tag":68,"props":151,"children":152},{},[153,158,159,165],{"type":40,"tag":135,"props":154,"children":155},{},[156],{"type":46,"value":157},"Cluster Name",{"type":46,"value":141},{"type":40,"tag":82,"props":160,"children":162},{"className":161},[],[163],{"type":46,"value":164},"tpu-cluster-prod",{"type":46,"value":149},{"type":40,"tag":68,"props":167,"children":168},{},[169,174,175,181],{"type":40,"tag":135,"props":170,"children":171},{},[172],{"type":46,"value":173},"Workload Name (JobSet Name)",{"type":46,"value":141},{"type":40,"tag":82,"props":176,"children":178},{"className":177},[],[179],{"type":46,"value":180},"llama3-70b-training",{"type":46,"value":149},{"type":40,"tag":68,"props":183,"children":184},{},[185,190,191,197],{"type":40,"tag":135,"props":186,"children":187},{},[188],{"type":46,"value":189},"Workload Namespace",{"type":46,"value":141},{"type":40,"tag":82,"props":192,"children":194},{"className":193},[],[195],{"type":46,"value":196},"default",{"type":46,"value":149},{"type":40,"tag":68,"props":199,"children":200},{},[201,206,207,213],{"type":40,"tag":135,"props":202,"children":203},{},[204],{"type":46,"value":205},"Issue Time",{"type":46,"value":141},{"type":40,"tag":82,"props":208,"children":210},{"className":209},[],[211],{"type":46,"value":212},"2026-05-20T08:15:00Z",{"type":46,"value":149},{"type":40,"tag":215,"props":216,"children":218},"h4",{"id":217},"time-handling-rules",[219],{"type":46,"value":220},"Time Handling Rules",{"type":40,"tag":222,"props":223,"children":224},"ol",{},[225,235],{"type":40,"tag":68,"props":226,"children":227},{},[228,233],{"type":40,"tag":135,"props":229,"children":230},{},[231],{"type":46,"value":232},"Reject Relative Time",{"type":46,"value":234},": If the user says \"X minutes ago\" or \"just now\", stop and request the exact timestamp or specific time window.",{"type":40,"tag":68,"props":236,"children":237},{},[238,243,245,251,253,270,272],{"type":40,"tag":135,"props":239,"children":240},{},[241],{"type":46,"value":242},"Window Calculation",{"type":46,"value":244},": If the user provides a single timestamp ",{"type":40,"tag":82,"props":246,"children":248},{"className":247},[],[249],{"type":46,"value":250},"T",{"type":46,"value":252},", calculate the query window as ",{"type":40,"tag":135,"props":254,"children":255},{},[256,262,264],{"type":40,"tag":82,"props":257,"children":259},{"className":258},[],[260],{"type":46,"value":261},"[T - 30m]",{"type":46,"value":263}," to ",{"type":40,"tag":82,"props":265,"children":267},{"className":266},[],[268],{"type":46,"value":269},"[T + 30m]",{"type":46,"value":271},".\n",{"type":40,"tag":62,"props":273,"children":274},{},[275,294],{"type":40,"tag":68,"props":276,"children":277},{},[278,280,286,288],{"type":46,"value":279},"Let ",{"type":40,"tag":82,"props":281,"children":283},{"className":282},[],[284],{"type":46,"value":285},"Start_Time",{"type":46,"value":287}," = ",{"type":40,"tag":82,"props":289,"children":291},{"className":290},[],[292],{"type":46,"value":293},"T - 30m",{"type":40,"tag":68,"props":295,"children":296},{},[297,298,304,305],{"type":46,"value":279},{"type":40,"tag":82,"props":299,"children":301},{"className":300},[],[302],{"type":46,"value":303},"End_Time",{"type":46,"value":287},{"type":40,"tag":82,"props":306,"children":308},{"className":307},[],[309],{"type":46,"value":310},"T + 30m",{"type":40,"tag":312,"props":313,"children":314},"hr",{},[],{"type":40,"tag":117,"props":316,"children":318},{"id":317},"step-1-identify-jobset-restarts-and-attempts-low-risk",[319,321],{"type":46,"value":320},"Step 1: Identify JobSet Restarts and Attempts ",{"type":40,"tag":322,"props":323,"children":324},"span",{},[325],{"type":46,"value":326},"Low Risk",{"type":40,"tag":49,"props":328,"children":329},{},[330],{"type":46,"value":331},"Verify if the JobSet is experiencing restart loops and determine the frequency of restarts.",{"type":40,"tag":215,"props":333,"children":335},{"id":334},"option-a-visual-chart-mql-restarts",[336],{"type":46,"value":337},"Option A: Visual Chart (MQL) - restarts",{"type":40,"tag":62,"props":339,"children":340},{},[341,357],{"type":40,"tag":68,"props":342,"children":343},{},[344,349,351],{"type":40,"tag":135,"props":345,"children":346},{},[347],{"type":46,"value":348},"Tool to use",{"type":46,"value":350},": ",{"type":40,"tag":82,"props":352,"children":354},{"className":353},[],[355],{"type":46,"value":356},"monitoring_time_series_chart",{"type":40,"tag":68,"props":358,"children":359},{},[360,365,367],{"type":40,"tag":135,"props":361,"children":362},{},[363],{"type":46,"value":364},"MQL Query",{"type":46,"value":366},":\n",{"type":40,"tag":368,"props":369,"children":374},"pre",{"className":370,"code":371,"language":372,"meta":373,"style":373},"language-mql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","fetch prometheus_target\n| metric 'prometheus.googleapis.com\u002Fkube_jobset_restarts\u002Fgauge'\n| filter resource.cluster_name == '\u003Ccluster_name>' && metric.jobset_name == '\u003Cworkload_name>'\n| align next_older(1m)\n| every 1m\n| group_by [metric.jobset_name], [val: max(value)]\n","mql","",[375],{"type":40,"tag":82,"props":376,"children":377},{"__ignoreMap":373},[378,388,397,406,415,424],{"type":40,"tag":322,"props":379,"children":382},{"class":380,"line":381},"line",1,[383],{"type":40,"tag":322,"props":384,"children":385},{},[386],{"type":46,"value":387},"fetch prometheus_target\n",{"type":40,"tag":322,"props":389,"children":391},{"class":380,"line":390},2,[392],{"type":40,"tag":322,"props":393,"children":394},{},[395],{"type":46,"value":396},"| metric 'prometheus.googleapis.com\u002Fkube_jobset_restarts\u002Fgauge'\n",{"type":40,"tag":322,"props":398,"children":400},{"class":380,"line":399},3,[401],{"type":40,"tag":322,"props":402,"children":403},{},[404],{"type":46,"value":405},"| filter resource.cluster_name == '\u003Ccluster_name>' && metric.jobset_name == '\u003Cworkload_name>'\n",{"type":40,"tag":322,"props":407,"children":409},{"class":380,"line":408},4,[410],{"type":40,"tag":322,"props":411,"children":412},{},[413],{"type":46,"value":414},"| align next_older(1m)\n",{"type":40,"tag":322,"props":416,"children":418},{"class":380,"line":417},5,[419],{"type":40,"tag":322,"props":420,"children":421},{},[422],{"type":46,"value":423},"| every 1m\n",{"type":40,"tag":322,"props":425,"children":427},{"class":380,"line":426},6,[428],{"type":40,"tag":322,"props":429,"children":430},{},[431],{"type":46,"value":432},"| group_by [metric.jobset_name], [val: max(value)]\n",{"type":40,"tag":215,"props":434,"children":436},{"id":435},"option-b-promql-query-restarts",[437],{"type":46,"value":438},"Option B: PromQL Query - restarts",{"type":40,"tag":62,"props":440,"children":441},{},[442,459,547,557],{"type":40,"tag":68,"props":443,"children":444},{},[445,450,452,458],{"type":40,"tag":135,"props":446,"children":447},{},[448],{"type":46,"value":449},"Method",{"type":46,"value":451},": Execute via shell command (requires ",{"type":40,"tag":82,"props":453,"children":455},{"className":454},[],[456],{"type":46,"value":457},"BypassSandbox: true",{"type":46,"value":100},{"type":40,"tag":68,"props":460,"children":461},{},[462,467,468],{"type":40,"tag":135,"props":463,"children":464},{},[465],{"type":46,"value":466},"Command Template",{"type":46,"value":366},{"type":40,"tag":368,"props":469,"children":473},{"className":470,"code":471,"language":472,"meta":373,"style":373},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","curl -sSf -H \"Authorization: Bearer \\$(gcloud auth print-access-token)\" \\\n  \"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=kube_jobset_restarts%7Bjobset_name%3D%22\u003Cworkload_name>%22%2Ccluster%3D%22\u003Ccluster_name>%22%7D\"\n","bash",[474],{"type":40,"tag":82,"props":475,"children":476},{"__ignoreMap":373},[477,529],{"type":40,"tag":322,"props":478,"children":479},{"class":380,"line":381},[480,486,492,497,503,508,514,519,524],{"type":40,"tag":322,"props":481,"children":483},{"style":482},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[484],{"type":46,"value":485},"curl",{"type":40,"tag":322,"props":487,"children":489},{"style":488},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[490],{"type":46,"value":491}," -sSf",{"type":40,"tag":322,"props":493,"children":494},{"style":488},[495],{"type":46,"value":496}," -H",{"type":40,"tag":322,"props":498,"children":500},{"style":499},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[501],{"type":46,"value":502}," \"",{"type":40,"tag":322,"props":504,"children":505},{"style":488},[506],{"type":46,"value":507},"Authorization: Bearer ",{"type":40,"tag":322,"props":509,"children":511},{"style":510},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[512],{"type":46,"value":513},"\\$",{"type":40,"tag":322,"props":515,"children":516},{"style":488},[517],{"type":46,"value":518},"(gcloud auth print-access-token)",{"type":40,"tag":322,"props":520,"children":521},{"style":499},[522],{"type":46,"value":523},"\"",{"type":40,"tag":322,"props":525,"children":526},{"style":510},[527],{"type":46,"value":528}," \\\n",{"type":40,"tag":322,"props":530,"children":531},{"class":380,"line":390},[532,537,542],{"type":40,"tag":322,"props":533,"children":534},{"style":499},[535],{"type":46,"value":536},"  \"",{"type":40,"tag":322,"props":538,"children":539},{"style":488},[540],{"type":46,"value":541},"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=kube_jobset_restarts%7Bjobset_name%3D%22\u003Cworkload_name>%22%2Ccluster%3D%22\u003Ccluster_name>%22%7D",{"type":40,"tag":322,"props":543,"children":544},{"style":499},[545],{"type":46,"value":546},"\"\n",{"type":40,"tag":68,"props":548,"children":549},{},[550,555],{"type":40,"tag":135,"props":551,"children":552},{},[553],{"type":46,"value":554},"Logic",{"type":46,"value":556},": A non-zero or increasing value for restarts indicates that the JobSet is being actively restarted by the controller due to worker failure or interruption.",{"type":40,"tag":68,"props":558,"children":559},{},[560,565],{"type":40,"tag":135,"props":561,"children":562},{},[563],{"type":46,"value":564},"Automation",{"type":46,"value":566},": Proceed to Step 2 automatically after reporting findings.",{"type":40,"tag":312,"props":568,"children":569},{},[],{"type":40,"tag":117,"props":571,"children":573},{"id":572},"step-2-inspect-nodepool-interruptions-low-risk",[574,576],{"type":46,"value":575},"Step 2: Inspect Nodepool Interruptions ",{"type":40,"tag":322,"props":577,"children":578},{},[579],{"type":46,"value":326},{"type":40,"tag":49,"props":581,"children":582},{},[583],{"type":46,"value":584},"Determine if the JobSet restarts were triggered by physical nodepool-level events (such as spot preemptions, maintenance, or host terminations).",{"type":40,"tag":215,"props":586,"children":588},{"id":587},"a-metrics-query-nodepool-interruption-counts",[589],{"type":46,"value":590},"A. Metrics Query (Nodepool Interruption Counts)",{"type":40,"tag":592,"props":593,"children":595},"h5",{"id":594},"option-a-visual-chart-mql-interruptions",[596],{"type":46,"value":597},"Option A: Visual Chart (MQL) - interruptions",{"type":40,"tag":62,"props":599,"children":600},{},[601,614],{"type":40,"tag":68,"props":602,"children":603},{},[604,608,609],{"type":40,"tag":135,"props":605,"children":606},{},[607],{"type":46,"value":348},{"type":46,"value":350},{"type":40,"tag":82,"props":610,"children":612},{"className":611},[],[613],{"type":46,"value":356},{"type":40,"tag":68,"props":615,"children":616},{},[617,621,622],{"type":40,"tag":135,"props":618,"children":619},{},[620],{"type":46,"value":364},{"type":46,"value":366},{"type":40,"tag":368,"props":623,"children":625},{"className":370,"code":624,"language":372,"meta":373,"style":373},"fetch k8s_node_pool\n| metric 'kubernetes.io\u002Fnode_pool\u002Finterruption_count'\n| filter cluster_name == '\u003Ccluster_name>'\n| align next_older(10m)\n| every 10m\n| group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]\n",[626],{"type":40,"tag":82,"props":627,"children":628},{"__ignoreMap":373},[629,637,645,653,661,669],{"type":40,"tag":322,"props":630,"children":631},{"class":380,"line":381},[632],{"type":40,"tag":322,"props":633,"children":634},{},[635],{"type":46,"value":636},"fetch k8s_node_pool\n",{"type":40,"tag":322,"props":638,"children":639},{"class":380,"line":390},[640],{"type":40,"tag":322,"props":641,"children":642},{},[643],{"type":46,"value":644},"| metric 'kubernetes.io\u002Fnode_pool\u002Finterruption_count'\n",{"type":40,"tag":322,"props":646,"children":647},{"class":380,"line":399},[648],{"type":40,"tag":322,"props":649,"children":650},{},[651],{"type":46,"value":652},"| filter cluster_name == '\u003Ccluster_name>'\n",{"type":40,"tag":322,"props":654,"children":655},{"class":380,"line":408},[656],{"type":40,"tag":322,"props":657,"children":658},{},[659],{"type":46,"value":660},"| align next_older(10m)\n",{"type":40,"tag":322,"props":662,"children":663},{"class":380,"line":417},[664],{"type":40,"tag":322,"props":665,"children":666},{},[667],{"type":46,"value":668},"| every 10m\n",{"type":40,"tag":322,"props":670,"children":671},{"class":380,"line":426},[672],{"type":40,"tag":322,"props":673,"children":674},{},[675],{"type":46,"value":676},"| group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]\n",{"type":40,"tag":592,"props":678,"children":680},{"id":679},"option-b-promql-query-interruptions",[681],{"type":46,"value":682},"Option B: PromQL Query - interruptions",{"type":40,"tag":62,"props":684,"children":685},{},[686,700],{"type":40,"tag":68,"props":687,"children":688},{},[689,693,694,699],{"type":40,"tag":135,"props":690,"children":691},{},[692],{"type":46,"value":449},{"type":46,"value":451},{"type":40,"tag":82,"props":695,"children":697},{"className":696},[],[698],{"type":46,"value":457},{"type":46,"value":100},{"type":40,"tag":68,"props":701,"children":702},{},[703,707,708],{"type":40,"tag":135,"props":704,"children":705},{},[706],{"type":46,"value":466},{"type":46,"value":366},{"type":40,"tag":368,"props":709,"children":711},{"className":470,"code":710,"language":472,"meta":373,"style":373},"curl -sSf -H \"Authorization: Bearer \\$(gcloud auth print-access-token)\" \\\n  \"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=sum%20by%20%28interruption_type%2C%20interruption_reason%2C%20node_pool_name%2C%20cluster_name%29%20%28avg_over_time%28%7B__name__%3D%22kubernetes.io%2Fnode_pool%2Finterruption_count%22%2C%20monitored_resource%3D%22k8s_node_pool%22%2C%20cluster_name%3D%22\u003Ccluster_name>%22%7D%5B10m%5D%29%29\"\n",[712],{"type":40,"tag":82,"props":713,"children":714},{"__ignoreMap":373},[715,754],{"type":40,"tag":322,"props":716,"children":717},{"class":380,"line":381},[718,722,726,730,734,738,742,746,750],{"type":40,"tag":322,"props":719,"children":720},{"style":482},[721],{"type":46,"value":485},{"type":40,"tag":322,"props":723,"children":724},{"style":488},[725],{"type":46,"value":491},{"type":40,"tag":322,"props":727,"children":728},{"style":488},[729],{"type":46,"value":496},{"type":40,"tag":322,"props":731,"children":732},{"style":499},[733],{"type":46,"value":502},{"type":40,"tag":322,"props":735,"children":736},{"style":488},[737],{"type":46,"value":507},{"type":40,"tag":322,"props":739,"children":740},{"style":510},[741],{"type":46,"value":513},{"type":40,"tag":322,"props":743,"children":744},{"style":488},[745],{"type":46,"value":518},{"type":40,"tag":322,"props":747,"children":748},{"style":499},[749],{"type":46,"value":523},{"type":40,"tag":322,"props":751,"children":752},{"style":510},[753],{"type":46,"value":528},{"type":40,"tag":322,"props":755,"children":756},{"class":380,"line":390},[757,761,766],{"type":40,"tag":322,"props":758,"children":759},{"style":499},[760],{"type":46,"value":536},{"type":40,"tag":322,"props":762,"children":763},{"style":488},[764],{"type":46,"value":765},"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=sum%20by%20%28interruption_type%2C%20interruption_reason%2C%20node_pool_name%2C%20cluster_name%29%20%28avg_over_time%28%7B__name__%3D%22kubernetes.io%2Fnode_pool%2Finterruption_count%22%2C%20monitored_resource%3D%22k8s_node_pool%22%2C%20cluster_name%3D%22\u003Ccluster_name>%22%7D%5B10m%5D%29%29",{"type":40,"tag":322,"props":767,"children":768},{"style":499},[769],{"type":46,"value":546},{"type":40,"tag":215,"props":771,"children":773},{"id":772},"b-log-query-nodepool-life-events",[774],{"type":46,"value":775},"B. Log Query (Nodepool Life Events)",{"type":40,"tag":62,"props":777,"children":778},{},[779,793,843,892],{"type":40,"tag":68,"props":780,"children":781},{},[782,786,787],{"type":40,"tag":135,"props":783,"children":784},{},[785],{"type":46,"value":348},{"type":46,"value":350},{"type":40,"tag":82,"props":788,"children":790},{"className":789},[],[791],{"type":46,"value":792},"query_logs",{"type":40,"tag":68,"props":794,"children":795},{},[796,801,802],{"type":40,"tag":135,"props":797,"children":798},{},[799],{"type":46,"value":800},"LQL Log Filter Template",{"type":46,"value":366},{"type":40,"tag":368,"props":803,"children":807},{"className":804,"code":805,"language":806,"meta":373,"style":373},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","resource.type=\"gke_nodepool\"\nAND resource.labels.cluster_name=\"\u003Ccluster_name>\"\nAND timestamp >= \"\u003CStart_Time>\"\nAND timestamp \u003C= \"\u003CEnd_Time>\"\n","sql",[808],{"type":40,"tag":82,"props":809,"children":810},{"__ignoreMap":373},[811,819,827,835],{"type":40,"tag":322,"props":812,"children":813},{"class":380,"line":381},[814],{"type":40,"tag":322,"props":815,"children":816},{},[817],{"type":46,"value":818},"resource.type=\"gke_nodepool\"\n",{"type":40,"tag":322,"props":820,"children":821},{"class":380,"line":390},[822],{"type":40,"tag":322,"props":823,"children":824},{},[825],{"type":46,"value":826},"AND resource.labels.cluster_name=\"\u003Ccluster_name>\"\n",{"type":40,"tag":322,"props":828,"children":829},{"class":380,"line":399},[830],{"type":40,"tag":322,"props":831,"children":832},{},[833],{"type":46,"value":834},"AND timestamp >= \"\u003CStart_Time>\"\n",{"type":40,"tag":322,"props":836,"children":837},{"class":380,"line":408},[838],{"type":40,"tag":322,"props":839,"children":840},{},[841],{"type":46,"value":842},"AND timestamp \u003C= \"\u003CEnd_Time>\"\n",{"type":40,"tag":68,"props":844,"children":845},{},[846,850,851],{"type":40,"tag":135,"props":847,"children":848},{},[849],{"type":46,"value":554},{"type":46,"value":366},{"type":40,"tag":62,"props":852,"children":853},{},[854,864,874],{"type":40,"tag":68,"props":855,"children":856},{},[857,862],{"type":40,"tag":135,"props":858,"children":859},{},[860],{"type":46,"value":861},"PreemptionEvent",{"type":46,"value":863},": Spot VMs were preempted, or node was scale-down.",{"type":40,"tag":68,"props":865,"children":866},{},[867,872],{"type":40,"tag":135,"props":868,"children":869},{},[870],{"type":46,"value":871},"MaintenanceEvent",{"type":46,"value":873},": Node pool updated or Google scheduled maintenance.",{"type":40,"tag":68,"props":875,"children":876},{},[877,882,884,890],{"type":40,"tag":135,"props":878,"children":879},{},[880],{"type":46,"value":881},"TerminationEvent",{"type":46,"value":883},": Serious host failures. Check ",{"type":40,"tag":82,"props":885,"children":887},{"className":886},[],[888],{"type":46,"value":889},"interruption_reason",{"type":46,"value":891}," or logs for host issues.",{"type":40,"tag":68,"props":893,"children":894},{},[895,899],{"type":40,"tag":135,"props":896,"children":897},{},[898],{"type":46,"value":564},{"type":46,"value":900},": Proceed to Step 3 automatically.",{"type":40,"tag":312,"props":902,"children":903},{},[],{"type":40,"tag":117,"props":905,"children":907},{"id":906},"step-3-inspect-nodes-and-underlying-host-vms-low-risk",[908,910],{"type":46,"value":909},"Step 3: Inspect Nodes and Underlying Host VMs ",{"type":40,"tag":322,"props":911,"children":912},{},[913],{"type":46,"value":326},{"type":40,"tag":49,"props":915,"children":916},{},[917],{"type":46,"value":918},"Correlate node readiness failures with physical host VMs to see if a single faulty host repeatedly fails coordinator pods.",{"type":40,"tag":215,"props":920,"children":922},{"id":921},"a-metrics-query-node-ready-status-check",[923],{"type":46,"value":924},"A. Metrics Query (Node Ready Status Check)",{"type":40,"tag":592,"props":926,"children":928},{"id":927},"option-a-visual-chart-mql-node-status",[929],{"type":46,"value":930},"Option A: Visual Chart (MQL) - node status",{"type":40,"tag":62,"props":932,"children":933},{},[934,947],{"type":40,"tag":68,"props":935,"children":936},{},[937,941,942],{"type":40,"tag":135,"props":938,"children":939},{},[940],{"type":46,"value":348},{"type":46,"value":350},{"type":40,"tag":82,"props":943,"children":945},{"className":944},[],[946],{"type":46,"value":356},{"type":40,"tag":68,"props":948,"children":949},{},[950,954,955],{"type":40,"tag":135,"props":951,"children":952},{},[953],{"type":46,"value":364},{"type":46,"value":366},{"type":40,"tag":368,"props":956,"children":958},{"className":370,"code":957,"language":372,"meta":373,"style":373},"fetch k8s_node\n| metric 'kubernetes.io\u002Fnode\u002Fstatus_condition'\n| filter cluster_name == '\u003Ccluster_name>' && metric.condition == 'Ready' && metric.status == 'False'\n| align next_older(1m)\n| every 1m\n| group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]\n",[959],{"type":40,"tag":82,"props":960,"children":961},{"__ignoreMap":373},[962,970,978,986,993,1000],{"type":40,"tag":322,"props":963,"children":964},{"class":380,"line":381},[965],{"type":40,"tag":322,"props":966,"children":967},{},[968],{"type":46,"value":969},"fetch k8s_node\n",{"type":40,"tag":322,"props":971,"children":972},{"class":380,"line":390},[973],{"type":40,"tag":322,"props":974,"children":975},{},[976],{"type":46,"value":977},"| metric 'kubernetes.io\u002Fnode\u002Fstatus_condition'\n",{"type":40,"tag":322,"props":979,"children":980},{"class":380,"line":399},[981],{"type":40,"tag":322,"props":982,"children":983},{},[984],{"type":46,"value":985},"| filter cluster_name == '\u003Ccluster_name>' && metric.condition == 'Ready' && metric.status == 'False'\n",{"type":40,"tag":322,"props":987,"children":988},{"class":380,"line":408},[989],{"type":40,"tag":322,"props":990,"children":991},{},[992],{"type":46,"value":414},{"type":40,"tag":322,"props":994,"children":995},{"class":380,"line":417},[996],{"type":40,"tag":322,"props":997,"children":998},{},[999],{"type":46,"value":423},{"type":40,"tag":322,"props":1001,"children":1002},{"class":380,"line":426},[1003],{"type":40,"tag":322,"props":1004,"children":1005},{},[1006],{"type":46,"value":1007},"| group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]\n",{"type":40,"tag":592,"props":1009,"children":1011},{"id":1010},"option-b-promql-query-node-status",[1012],{"type":46,"value":1013},"Option B: PromQL Query - node status",{"type":40,"tag":62,"props":1015,"children":1016},{},[1017,1031],{"type":40,"tag":68,"props":1018,"children":1019},{},[1020,1024,1025,1030],{"type":40,"tag":135,"props":1021,"children":1022},{},[1023],{"type":46,"value":449},{"type":46,"value":451},{"type":40,"tag":82,"props":1026,"children":1028},{"className":1027},[],[1029],{"type":46,"value":457},{"type":46,"value":100},{"type":40,"tag":68,"props":1032,"children":1033},{},[1034,1038,1039],{"type":40,"tag":135,"props":1035,"children":1036},{},[1037],{"type":46,"value":466},{"type":46,"value":366},{"type":40,"tag":368,"props":1040,"children":1042},{"className":470,"code":1041,"language":472,"meta":373,"style":373},"curl -sSf -H \"Authorization: Bearer \\$(gcloud auth print-access-token)\" \\\n  \"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=sum%20by%20%28status%2C%20condition%2C%20node_pool_name%29%20%28%7B__name__%3D%22kubernetes.io%2Fnode%2Fstatus_condition%22%2C%20monitored_resource%3D%22k8s_node%22%2C%20cluster_name%3D%22\u003Ccluster_name>%22%2C%20condition%3D%22Ready%22%2C%20status%3D%22False%22%7D%29\"\n",[1043],{"type":40,"tag":82,"props":1044,"children":1045},{"__ignoreMap":373},[1046,1085],{"type":40,"tag":322,"props":1047,"children":1048},{"class":380,"line":381},[1049,1053,1057,1061,1065,1069,1073,1077,1081],{"type":40,"tag":322,"props":1050,"children":1051},{"style":482},[1052],{"type":46,"value":485},{"type":40,"tag":322,"props":1054,"children":1055},{"style":488},[1056],{"type":46,"value":491},{"type":40,"tag":322,"props":1058,"children":1059},{"style":488},[1060],{"type":46,"value":496},{"type":40,"tag":322,"props":1062,"children":1063},{"style":499},[1064],{"type":46,"value":502},{"type":40,"tag":322,"props":1066,"children":1067},{"style":488},[1068],{"type":46,"value":507},{"type":40,"tag":322,"props":1070,"children":1071},{"style":510},[1072],{"type":46,"value":513},{"type":40,"tag":322,"props":1074,"children":1075},{"style":488},[1076],{"type":46,"value":518},{"type":40,"tag":322,"props":1078,"children":1079},{"style":499},[1080],{"type":46,"value":523},{"type":40,"tag":322,"props":1082,"children":1083},{"style":510},[1084],{"type":46,"value":528},{"type":40,"tag":322,"props":1086,"children":1087},{"class":380,"line":390},[1088,1092,1097],{"type":40,"tag":322,"props":1089,"children":1090},{"style":499},[1091],{"type":46,"value":536},{"type":40,"tag":322,"props":1093,"children":1094},{"style":488},[1095],{"type":46,"value":1096},"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=sum%20by%20%28status%2C%20condition%2C%20node_pool_name%29%20%28%7B__name__%3D%22kubernetes.io%2Fnode%2Fstatus_condition%22%2C%20monitored_resource%3D%22k8s_node%22%2C%20cluster_name%3D%22\u003Ccluster_name>%22%2C%20condition%3D%22Ready%22%2C%20status%3D%22False%22%7D%29",{"type":40,"tag":322,"props":1098,"children":1099},{"style":499},[1100],{"type":46,"value":546},{"type":40,"tag":215,"props":1102,"children":1104},{"id":1103},"b-metrics-query-node-to-host-metadata-topology-correlation",[1105],{"type":46,"value":1106},"B. Metrics Query (Node-to-Host Metadata Topology Correlation)",{"type":40,"tag":62,"props":1108,"children":1109},{},[1110,1123],{"type":40,"tag":68,"props":1111,"children":1112},{},[1113,1117,1118],{"type":40,"tag":135,"props":1114,"children":1115},{},[1116],{"type":46,"value":348},{"type":46,"value":350},{"type":40,"tag":82,"props":1119,"children":1121},{"className":1120},[],[1122],{"type":46,"value":356},{"type":40,"tag":68,"props":1124,"children":1125},{},[1126,1130,1131],{"type":40,"tag":135,"props":1127,"children":1128},{},[1129],{"type":46,"value":364},{"type":46,"value":366},{"type":40,"tag":368,"props":1132,"children":1134},{"className":370,"code":1133,"language":372,"meta":373,"style":373},"fetch k8s_node\n| metric 'kubernetes.io\u002Fnode\u002Fcpu\u002Ftotal_cores'\n| filter cluster_name == '\u003Ccluster_name>'\n| align next_older(1m)\n| every 1m\n| group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]\n",[1135],{"type":40,"tag":82,"props":1136,"children":1137},{"__ignoreMap":373},[1138,1145,1153,1160,1167,1174],{"type":40,"tag":322,"props":1139,"children":1140},{"class":380,"line":381},[1141],{"type":40,"tag":322,"props":1142,"children":1143},{},[1144],{"type":46,"value":969},{"type":40,"tag":322,"props":1146,"children":1147},{"class":380,"line":390},[1148],{"type":40,"tag":322,"props":1149,"children":1150},{},[1151],{"type":46,"value":1152},"| metric 'kubernetes.io\u002Fnode\u002Fcpu\u002Ftotal_cores'\n",{"type":40,"tag":322,"props":1154,"children":1155},{"class":380,"line":399},[1156],{"type":40,"tag":322,"props":1157,"children":1158},{},[1159],{"type":46,"value":652},{"type":40,"tag":322,"props":1161,"children":1162},{"class":380,"line":408},[1163],{"type":40,"tag":322,"props":1164,"children":1165},{},[1166],{"type":46,"value":414},{"type":40,"tag":322,"props":1168,"children":1169},{"class":380,"line":417},[1170],{"type":40,"tag":322,"props":1171,"children":1172},{},[1173],{"type":46,"value":423},{"type":40,"tag":322,"props":1175,"children":1176},{"class":380,"line":426},[1177],{"type":40,"tag":322,"props":1178,"children":1179},{},[1180],{"type":46,"value":1181},"| group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]\n",{"type":40,"tag":215,"props":1183,"children":1185},{"id":1184},"c-log-query-node-fault-logs",[1186],{"type":46,"value":1187},"C. Log Query (Node Fault Logs)",{"type":40,"tag":62,"props":1189,"children":1190},{},[1191,1204,1256,1289],{"type":40,"tag":68,"props":1192,"children":1193},{},[1194,1198,1199],{"type":40,"tag":135,"props":1195,"children":1196},{},[1197],{"type":46,"value":348},{"type":46,"value":350},{"type":40,"tag":82,"props":1200,"children":1202},{"className":1201},[],[1203],{"type":46,"value":792},{"type":40,"tag":68,"props":1205,"children":1206},{},[1207,1211,1212],{"type":40,"tag":135,"props":1208,"children":1209},{},[1210],{"type":46,"value":800},{"type":46,"value":366},{"type":40,"tag":368,"props":1213,"children":1215},{"className":804,"code":1214,"language":806,"meta":373,"style":373},"resource.type=\"k8s_node\"\nAND resource.labels.cluster_name=\"\u003Ccluster_name>\"\nAND (textPayload:\"host error\" OR textPayload:\"kernel panic\" OR textPayload:\"hardware failure\" OR textPayload:\"NodeNotReady\")\nAND timestamp >= \"\u003CStart_Time>\"\nAND timestamp \u003C= \"\u003CEnd_Time>\"\n",[1216],{"type":40,"tag":82,"props":1217,"children":1218},{"__ignoreMap":373},[1219,1227,1234,1242,1249],{"type":40,"tag":322,"props":1220,"children":1221},{"class":380,"line":381},[1222],{"type":40,"tag":322,"props":1223,"children":1224},{},[1225],{"type":46,"value":1226},"resource.type=\"k8s_node\"\n",{"type":40,"tag":322,"props":1228,"children":1229},{"class":380,"line":390},[1230],{"type":40,"tag":322,"props":1231,"children":1232},{},[1233],{"type":46,"value":826},{"type":40,"tag":322,"props":1235,"children":1236},{"class":380,"line":399},[1237],{"type":40,"tag":322,"props":1238,"children":1239},{},[1240],{"type":46,"value":1241},"AND (textPayload:\"host error\" OR textPayload:\"kernel panic\" OR textPayload:\"hardware failure\" OR textPayload:\"NodeNotReady\")\n",{"type":40,"tag":322,"props":1243,"children":1244},{"class":380,"line":408},[1245],{"type":40,"tag":322,"props":1246,"children":1247},{},[1248],{"type":46,"value":834},{"type":40,"tag":322,"props":1250,"children":1251},{"class":380,"line":417},[1252],{"type":40,"tag":322,"props":1253,"children":1254},{},[1255],{"type":46,"value":842},{"type":40,"tag":68,"props":1257,"children":1258},{},[1259,1263,1265,1271,1273,1279,1281,1287],{"type":40,"tag":135,"props":1260,"children":1261},{},[1262],{"type":46,"value":554},{"type":46,"value":1264},": Identify if specific nodes are unhealthy (",{"type":40,"tag":82,"props":1266,"children":1268},{"className":1267},[],[1269],{"type":46,"value":1270},"Ready=False",{"type":46,"value":1272}," or ",{"type":40,"tag":82,"props":1274,"children":1276},{"className":1275},[],[1277],{"type":46,"value":1278},"Unknown",{"type":46,"value":1280},") and correlate them to their GCE physical host ID via ",{"type":40,"tag":82,"props":1282,"children":1284},{"className":1283},[],[1285],{"type":46,"value":1286},"metadata.user.gce_topology_host",{"type":46,"value":1288},". Check if the same host is repeatedly failing.",{"type":40,"tag":68,"props":1290,"children":1291},{},[1292,1296],{"type":40,"tag":135,"props":1293,"children":1294},{},[1295],{"type":46,"value":564},{"type":46,"value":1297},": Proceed to Step 4 automatically.",{"type":40,"tag":312,"props":1299,"children":1300},{},[],{"type":40,"tag":117,"props":1302,"children":1304},{"id":1303},"step-4-inspect-pod-and-worker-container-failures-low-risk",[1305,1307],{"type":46,"value":1306},"Step 4: Inspect Pod and Worker \u002F Container Failures ",{"type":40,"tag":322,"props":1308,"children":1309},{},[1310],{"type":46,"value":326},{"type":40,"tag":49,"props":1312,"children":1313},{},[1314],{"type":46,"value":1315},"Analyze pod status phases and retrieve coordinator worker logs to identify application-level crashes or network deadlocks.",{"type":40,"tag":215,"props":1317,"children":1319},{"id":1318},"a-metrics-query-pod-lifecycle-phases",[1320],{"type":46,"value":1321},"A. Metrics Query (Pod Lifecycle Phases)",{"type":40,"tag":592,"props":1323,"children":1325},{"id":1324},"option-a-visual-chart-mql-pod-phase",[1326],{"type":46,"value":1327},"Option A: Visual Chart (MQL) - pod phase",{"type":40,"tag":62,"props":1329,"children":1330},{},[1331,1344],{"type":40,"tag":68,"props":1332,"children":1333},{},[1334,1338,1339],{"type":40,"tag":135,"props":1335,"children":1336},{},[1337],{"type":46,"value":348},{"type":46,"value":350},{"type":40,"tag":82,"props":1340,"children":1342},{"className":1341},[],[1343],{"type":46,"value":356},{"type":40,"tag":68,"props":1345,"children":1346},{},[1347,1351,1352],{"type":40,"tag":135,"props":1348,"children":1349},{},[1350],{"type":46,"value":364},{"type":46,"value":366},{"type":40,"tag":368,"props":1353,"children":1355},{"className":370,"code":1354,"language":372,"meta":373,"style":373},"fetch k8s_pod\n| metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Fphase'\n| filter cluster_name == '\u003Ccluster_name>' && pod_name ==~ '\u003Cworkload_name>.*'\n| align next_older(10m)\n| every 10m\n| group_by [metric.phase], [val: count()]\n",[1356],{"type":40,"tag":82,"props":1357,"children":1358},{"__ignoreMap":373},[1359,1367,1375,1383,1390,1397],{"type":40,"tag":322,"props":1360,"children":1361},{"class":380,"line":381},[1362],{"type":40,"tag":322,"props":1363,"children":1364},{},[1365],{"type":46,"value":1366},"fetch k8s_pod\n",{"type":40,"tag":322,"props":1368,"children":1369},{"class":380,"line":390},[1370],{"type":40,"tag":322,"props":1371,"children":1372},{},[1373],{"type":46,"value":1374},"| metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Fphase'\n",{"type":40,"tag":322,"props":1376,"children":1377},{"class":380,"line":399},[1378],{"type":40,"tag":322,"props":1379,"children":1380},{},[1381],{"type":46,"value":1382},"| filter cluster_name == '\u003Ccluster_name>' && pod_name ==~ '\u003Cworkload_name>.*'\n",{"type":40,"tag":322,"props":1384,"children":1385},{"class":380,"line":408},[1386],{"type":40,"tag":322,"props":1387,"children":1388},{},[1389],{"type":46,"value":660},{"type":40,"tag":322,"props":1391,"children":1392},{"class":380,"line":417},[1393],{"type":40,"tag":322,"props":1394,"children":1395},{},[1396],{"type":46,"value":668},{"type":40,"tag":322,"props":1398,"children":1399},{"class":380,"line":426},[1400],{"type":40,"tag":322,"props":1401,"children":1402},{},[1403],{"type":46,"value":1404},"| group_by [metric.phase], [val: count()]\n",{"type":40,"tag":592,"props":1406,"children":1408},{"id":1407},"option-b-promql-query-pod-phase",[1409],{"type":46,"value":1410},"Option B: PromQL Query - pod phase",{"type":40,"tag":62,"props":1412,"children":1413},{},[1414,1428],{"type":40,"tag":68,"props":1415,"children":1416},{},[1417,1421,1422,1427],{"type":40,"tag":135,"props":1418,"children":1419},{},[1420],{"type":46,"value":449},{"type":46,"value":451},{"type":40,"tag":82,"props":1423,"children":1425},{"className":1424},[],[1426],{"type":46,"value":457},{"type":46,"value":100},{"type":40,"tag":68,"props":1429,"children":1430},{},[1431,1435,1436],{"type":40,"tag":135,"props":1432,"children":1433},{},[1434],{"type":46,"value":466},{"type":46,"value":366},{"type":40,"tag":368,"props":1437,"children":1439},{"className":470,"code":1438,"language":472,"meta":373,"style":373},"curl -sSf -H \"Authorization: Bearer \\$(gcloud auth print-access-token)\" \\\n  \"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=sum%20by%20%28phase%29%20%28avg_over_time%28%7B__name__%3D%22kube_pod_status_phase%22%2C%20cluster%3D%22\u003Ccluster_name>%22%2C%20pod%3D~%22\u003Cworkload_name>.*%22%7D%5B10m%5D%29%29\"\n",[1440],{"type":40,"tag":82,"props":1441,"children":1442},{"__ignoreMap":373},[1443,1482],{"type":40,"tag":322,"props":1444,"children":1445},{"class":380,"line":381},[1446,1450,1454,1458,1462,1466,1470,1474,1478],{"type":40,"tag":322,"props":1447,"children":1448},{"style":482},[1449],{"type":46,"value":485},{"type":40,"tag":322,"props":1451,"children":1452},{"style":488},[1453],{"type":46,"value":491},{"type":40,"tag":322,"props":1455,"children":1456},{"style":488},[1457],{"type":46,"value":496},{"type":40,"tag":322,"props":1459,"children":1460},{"style":499},[1461],{"type":46,"value":502},{"type":40,"tag":322,"props":1463,"children":1464},{"style":488},[1465],{"type":46,"value":507},{"type":40,"tag":322,"props":1467,"children":1468},{"style":510},[1469],{"type":46,"value":513},{"type":40,"tag":322,"props":1471,"children":1472},{"style":488},[1473],{"type":46,"value":518},{"type":40,"tag":322,"props":1475,"children":1476},{"style":499},[1477],{"type":46,"value":523},{"type":40,"tag":322,"props":1479,"children":1480},{"style":510},[1481],{"type":46,"value":528},{"type":40,"tag":322,"props":1483,"children":1484},{"class":380,"line":390},[1485,1489,1494],{"type":40,"tag":322,"props":1486,"children":1487},{"style":499},[1488],{"type":46,"value":536},{"type":40,"tag":322,"props":1490,"children":1491},{"style":488},[1492],{"type":46,"value":1493},"https:\u002F\u002Fmonitoring.googleapis.com\u002Fv1\u002Fprojects\u002F\u003Cproject_id>\u002Flocation\u002Fglobal\u002Fprometheus\u002Fapi\u002Fv1\u002Fquery?query=sum%20by%20%28phase%29%20%28avg_over_time%28%7B__name__%3D%22kube_pod_status_phase%22%2C%20cluster%3D%22\u003Ccluster_name>%22%2C%20pod%3D~%22\u003Cworkload_name>.*%22%7D%5B10m%5D%29%29",{"type":40,"tag":322,"props":1495,"children":1496},{"style":499},[1497],{"type":46,"value":546},{"type":40,"tag":215,"props":1499,"children":1501},{"id":1500},"b-metrics-query-unschedulable-pod-count",[1502],{"type":46,"value":1503},"B. Metrics Query (Unschedulable Pod Count)",{"type":40,"tag":62,"props":1505,"children":1506},{},[1507,1520],{"type":40,"tag":68,"props":1508,"children":1509},{},[1510,1514,1515],{"type":40,"tag":135,"props":1511,"children":1512},{},[1513],{"type":46,"value":348},{"type":46,"value":350},{"type":40,"tag":82,"props":1516,"children":1518},{"className":1517},[],[1519],{"type":46,"value":356},{"type":40,"tag":68,"props":1521,"children":1522},{},[1523,1527,1528],{"type":40,"tag":135,"props":1524,"children":1525},{},[1526],{"type":46,"value":364},{"type":46,"value":366},{"type":40,"tag":368,"props":1529,"children":1531},{"className":370,"code":1530,"language":372,"meta":373,"style":373},"fetch k8s_pod\n| metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Funschedulable'\n| filter cluster_name == '\u003Ccluster_name>' && pod_name ==~ '\u003Cworkload_name>.*'\n| align next_older(10m)\n| every 10m\n| group_by [pod_name], [val: max(value)]\n",[1532],{"type":40,"tag":82,"props":1533,"children":1534},{"__ignoreMap":373},[1535,1542,1550,1557,1564,1571],{"type":40,"tag":322,"props":1536,"children":1537},{"class":380,"line":381},[1538],{"type":40,"tag":322,"props":1539,"children":1540},{},[1541],{"type":46,"value":1366},{"type":40,"tag":322,"props":1543,"children":1544},{"class":380,"line":390},[1545],{"type":40,"tag":322,"props":1546,"children":1547},{},[1548],{"type":46,"value":1549},"| metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Funschedulable'\n",{"type":40,"tag":322,"props":1551,"children":1552},{"class":380,"line":399},[1553],{"type":40,"tag":322,"props":1554,"children":1555},{},[1556],{"type":46,"value":1382},{"type":40,"tag":322,"props":1558,"children":1559},{"class":380,"line":408},[1560],{"type":40,"tag":322,"props":1561,"children":1562},{},[1563],{"type":46,"value":660},{"type":40,"tag":322,"props":1565,"children":1566},{"class":380,"line":417},[1567],{"type":40,"tag":322,"props":1568,"children":1569},{},[1570],{"type":46,"value":668},{"type":40,"tag":322,"props":1572,"children":1573},{"class":380,"line":426},[1574],{"type":40,"tag":322,"props":1575,"children":1576},{},[1577],{"type":46,"value":1578},"| group_by [pod_name], [val: max(value)]\n",{"type":40,"tag":215,"props":1580,"children":1582},{"id":1581},"c-log-query-worker-container-logs",[1583],{"type":46,"value":1584},"C. Log Query (Worker Container Logs)",{"type":40,"tag":62,"props":1586,"children":1587},{},[1588,1601,1653,1674],{"type":40,"tag":68,"props":1589,"children":1590},{},[1591,1595,1596],{"type":40,"tag":135,"props":1592,"children":1593},{},[1594],{"type":46,"value":348},{"type":46,"value":350},{"type":40,"tag":82,"props":1597,"children":1599},{"className":1598},[],[1600],{"type":46,"value":792},{"type":40,"tag":68,"props":1602,"children":1603},{},[1604,1608,1609],{"type":40,"tag":135,"props":1605,"children":1606},{},[1607],{"type":46,"value":800},{"type":46,"value":366},{"type":40,"tag":368,"props":1610,"children":1612},{"className":804,"code":1611,"language":806,"meta":373,"style":373},"resource.type=\"k8s_container\"\nAND resource.labels.cluster_name=\"\u003Ccluster_name>\"\nAND labels.\"k8s-pod\u002Fjobset_sigs_k8s_io\u002Fjobset-name\"=\"\u003Cworkload_name>\"\nAND timestamp >= \"\u003CStart_Time>\"\nAND timestamp \u003C= \"\u003CEnd_Time>\"\n",[1613],{"type":40,"tag":82,"props":1614,"children":1615},{"__ignoreMap":373},[1616,1624,1631,1639,1646],{"type":40,"tag":322,"props":1617,"children":1618},{"class":380,"line":381},[1619],{"type":40,"tag":322,"props":1620,"children":1621},{},[1622],{"type":46,"value":1623},"resource.type=\"k8s_container\"\n",{"type":40,"tag":322,"props":1625,"children":1626},{"class":380,"line":390},[1627],{"type":40,"tag":322,"props":1628,"children":1629},{},[1630],{"type":46,"value":826},{"type":40,"tag":322,"props":1632,"children":1633},{"class":380,"line":399},[1634],{"type":40,"tag":322,"props":1635,"children":1636},{},[1637],{"type":46,"value":1638},"AND labels.\"k8s-pod\u002Fjobset_sigs_k8s_io\u002Fjobset-name\"=\"\u003Cworkload_name>\"\n",{"type":40,"tag":322,"props":1640,"children":1641},{"class":380,"line":408},[1642],{"type":40,"tag":322,"props":1643,"children":1644},{},[1645],{"type":46,"value":834},{"type":40,"tag":322,"props":1647,"children":1648},{"class":380,"line":417},[1649],{"type":40,"tag":322,"props":1650,"children":1651},{},[1652],{"type":46,"value":842},{"type":40,"tag":68,"props":1654,"children":1655},{},[1656,1660,1661],{"type":40,"tag":135,"props":1657,"children":1658},{},[1659],{"type":46,"value":554},{"type":46,"value":366},{"type":40,"tag":222,"props":1662,"children":1663},{},[1664,1669],{"type":40,"tag":68,"props":1665,"children":1666},{},[1667],{"type":46,"value":1668},"Check the pod timeline to spot pending or unschedulable pods.",{"type":40,"tag":68,"props":1670,"children":1671},{},[1672],{"type":46,"value":1673},"Use the worker container logs to analyze worker 0 in slice 0 (coordinator) for NCCL timeouts, collective communication issues, or MegaScale hangs.",{"type":40,"tag":68,"props":1675,"children":1676},{},[1677,1681],{"type":40,"tag":135,"props":1678,"children":1679},{},[1680],{"type":46,"value":564},{"type":46,"value":1682},": Proceed to Resolution.",{"type":40,"tag":312,"props":1684,"children":1685},{},[],{"type":40,"tag":55,"props":1687,"children":1689},{"id":1688},"️-resolution-workflow",[1690],{"type":46,"value":1691},"🛠️ Resolution Workflow",{"type":40,"tag":117,"props":1693,"children":1695},{"id":1694},"resolution-1-preemption-autoscaling-optimizations-low-risk",[1696,1698],{"type":46,"value":1697},"Resolution 1: Preemption & Autoscaling Optimizations ",{"type":40,"tag":322,"props":1699,"children":1700},{},[1701],{"type":46,"value":326},{"type":40,"tag":49,"props":1703,"children":1704},{},[1705],{"type":46,"value":1706},"If Step 2 showed high preemption counts on Spot VMs:",{"type":40,"tag":62,"props":1708,"children":1709},{},[1710,1734],{"type":40,"tag":68,"props":1711,"children":1712},{},[1713,1718,1720,1725,1727,1732],{"type":40,"tag":135,"props":1714,"children":1715},{},[1716],{"type":46,"value":1717},"Action",{"type":46,"value":1719},": Suggest switching critical long-running training workloads to ",{"type":40,"tag":135,"props":1721,"children":1722},{},[1723],{"type":46,"value":1724},"GKE Reserved\u002FOn-Demand VMs",{"type":46,"value":1726}," or utilizing ",{"type":40,"tag":135,"props":1728,"children":1729},{},[1730],{"type":46,"value":1731},"Compact Placement Policies",{"type":46,"value":1733}," to minimize defragmentation interruptions.",{"type":40,"tag":68,"props":1735,"children":1736},{},[1737,1742],{"type":40,"tag":135,"props":1738,"children":1739},{},[1740],{"type":46,"value":1741},"Justification",{"type":46,"value":1743},": Eliminates spot-market preemptions and reduces training restarts.",{"type":40,"tag":117,"props":1745,"children":1747},{"id":1746},"resolution-2-quarantine-faulty-host-vms-high-risk",[1748,1750],{"type":46,"value":1749},"Resolution 2: Quarantine Faulty Host VMs ",{"type":40,"tag":322,"props":1751,"children":1752},{},[1753],{"type":46,"value":1754},"High Risk",{"type":40,"tag":49,"props":1756,"children":1757},{},[1758,1760,1766],{"type":46,"value":1759},"If Step 3 identified a specific host ID (",{"type":40,"tag":82,"props":1761,"children":1763},{"className":1762},[],[1764],{"type":46,"value":1765},"gce-topology-host",{"type":46,"value":1767},") that consistently fails or triggers restarts across multiple attempts:",{"type":40,"tag":62,"props":1769,"children":1770},{},[1771,1780,1789],{"type":40,"tag":68,"props":1772,"children":1773},{},[1774,1778],{"type":40,"tag":135,"props":1775,"children":1776},{},[1777],{"type":46,"value":1717},{"type":46,"value":1779},": Draft a recommendation to cordon\u002Fdrain the GKE node, delete the underlying GCE VM instance to trigger instance recreation, and open a support ticket with Google Cloud Support specifying the physical host ID.",{"type":40,"tag":68,"props":1781,"children":1782},{},[1783,1787],{"type":40,"tag":135,"props":1784,"children":1785},{},[1786],{"type":46,"value":1741},{"type":46,"value":1788},": GKE auto-repair will recreate the VM instance on healthy physical hardware, preventing infinite restart loops.",{"type":40,"tag":68,"props":1790,"children":1791},{},[1792,1796],{"type":40,"tag":135,"props":1793,"children":1794},{},[1795],{"type":46,"value":564},{"type":46,"value":1797},": Stop and request explicit user confirmation before cordoning\u002Fdraining any node.",{"type":40,"tag":312,"props":1799,"children":1800},{},[],{"type":40,"tag":55,"props":1802,"children":1804},{"id":1803},"copypaste-checklist",[1805],{"type":46,"value":1806},"📋 Copypaste Checklist",{"type":40,"tag":62,"props":1808,"children":1810},{"className":1809},[65],[1811,1828,1837,1846,1855,1864],{"type":40,"tag":68,"props":1812,"children":1814},{"className":1813},[71],[1815,1818,1820,1826],{"type":40,"tag":74,"props":1816,"children":1817},{"disabled":76,"type":77},[],{"type":46,"value":1819}," Acquire context and compute ",{"type":40,"tag":82,"props":1821,"children":1823},{"className":1822},[],[1824],{"type":46,"value":1825},"[T - 30m, T + 30m]",{"type":46,"value":1827}," window.",{"type":40,"tag":68,"props":1829,"children":1831},{"className":1830},[71],[1832,1835],{"type":40,"tag":74,"props":1833,"children":1834},{"disabled":76,"type":77},[],{"type":46,"value":1836}," Query JobSet restart attempts.",{"type":40,"tag":68,"props":1838,"children":1840},{"className":1839},[71],[1841,1844],{"type":40,"tag":74,"props":1842,"children":1843},{"disabled":76,"type":77},[],{"type":46,"value":1845}," Check Nodepool interruptions (spot preemptions vs. hardware terminations).",{"type":40,"tag":68,"props":1847,"children":1849},{"className":1848},[71],[1850,1853],{"type":40,"tag":74,"props":1851,"children":1852},{"disabled":76,"type":77},[],{"type":46,"value":1854}," Query node-to-host mapping and check node logs for physical host errors.",{"type":40,"tag":68,"props":1856,"children":1858},{"className":1857},[71],[1859,1862],{"type":40,"tag":74,"props":1860,"children":1861},{"disabled":76,"type":77},[],{"type":46,"value":1863}," Inspect pod timeline status and coordinator worker container logs.",{"type":40,"tag":68,"props":1865,"children":1867},{"className":1866},[71],[1868,1871],{"type":40,"tag":74,"props":1869,"children":1870},{"disabled":76,"type":77},[],{"type":46,"value":1872}," Recommend appropriate scheduling strategy (On-demand vs Spot) or host VM quarantining.",{"type":40,"tag":1874,"props":1875,"children":1876},"style",{},[1877],{"type":46,"value":1878},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1880,"total":1970},[1881,1893,1906,1913,1930,1941,1957],{"slug":1882,"name":1882,"fn":1883,"description":1884,"org":1885,"tags":1886,"stars":24,"repoUrl":25,"updatedAt":1892},"custom-golden-image-discovery","discover golden base images for GKE nodes","Expert at discovering golden base images for GKE custom nodes using technical specs or context clues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1887,1890,1891],{"name":1888,"slug":1889,"type":16},"Deployment","deployment",{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},"2026-07-12T07:39:30.888879",{"slug":1894,"name":1894,"fn":1895,"description":1896,"org":1897,"tags":1898,"stars":24,"repoUrl":25,"updatedAt":1905},"gke-ai-troubleshooting-handle-disruption-gpu-tpu","diagnose GPU and TPU workload disruptions","Diagnose and predict node disruption during Compute Engine host maintenance for GPU and TPU workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1899,1900,1901,1902],{"name":22,"slug":23,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":1903,"slug":1904,"type":16},"Performance","performance","2026-07-28T05:34:18.149515",{"slug":4,"name":4,"fn":5,"description":6,"org":1907,"tags":1908,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1909,1910,1911,1912],{"name":22,"slug":23,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":1914,"name":1914,"fn":1915,"description":1916,"org":1917,"tags":1918,"stars":24,"repoUrl":25,"updatedAt":1929},"gke-ai-troubleshooting-skill-creation-guide","create GKE troubleshooting skill bundles","Expert instructions for building high-quality GKE troubleshooting skills. Codifies Step 0 context rules, zero-hallucination signatures, and explicit LQL\u002FPromQL query requirements.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1919,1922,1925,1926],{"name":1920,"slug":1921,"type":16},"Documentation","documentation",{"name":1923,"slug":1924,"type":16},"Engineering","engineering",{"name":9,"slug":8,"type":16},{"name":1927,"slug":1928,"type":16},"Technical Writing","technical-writing","2026-07-12T07:39:50.73484",{"slug":1931,"name":1931,"fn":1932,"description":1933,"org":1934,"tags":1935,"stars":24,"repoUrl":25,"updatedAt":1940},"gke-ai-troubleshooting-tpu-connection-failure-vbar-oom","diagnose GKE TPU connection failures","Diagnose and prevent `vbar_control_agent` segfaults and OOMs caused by race conditions during TPU device resets and frequent metrics collection (e.g. every 3s). Use when TPU slice initialization fails or `vbar_control_agent` crashes on TPU v6e nodes.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1936,1937,1938,1939],{"name":22,"slug":23,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":1903,"slug":1904,"type":16},"2026-07-12T07:39:49.482979",{"slug":1942,"name":1942,"fn":1943,"description":1944,"org":1945,"tags":1946,"stars":24,"repoUrl":25,"updatedAt":1956},"gke-app-onboarding","containerize and deploy apps to GKE","Workflows for containerizing and deploying applications to GKE for the first time.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1947,1950,1951,1952,1953],{"name":1948,"slug":1949,"type":16},"Containers","containers",{"name":1888,"slug":1889,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":1954,"slug":1955,"type":16},"Onboarding","onboarding","2026-07-12T07:39:41.935837",{"slug":1958,"name":1958,"fn":1959,"description":1960,"org":1961,"tags":1962,"stars":24,"repoUrl":25,"updatedAt":1969},"gke-backup-dr","configure GKE backup and disaster recovery","Workflows for configuring Backup for GKE and disaster recovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1963,1964,1965,1966],{"name":1888,"slug":1889,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":1967,"slug":1968,"type":16},"Operations","operations","2026-07-12T07:39:34.806995",25,{"items":1972,"total":2152},[1973,1989,2005,2024,2038,2047,2061,2078,2095,2108,2124,2134],{"slug":1974,"name":1974,"fn":1975,"description":1976,"org":1977,"tags":1978,"stars":1986,"repoUrl":1987,"updatedAt":1988},"kb-search","search and extract local knowledge base documents","Allows listing, searching and extracting information from local knowledge base documents for information about tables\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1979,1980,1983],{"name":1920,"slug":1921,"type":16},{"name":1981,"slug":1982,"type":16},"Knowledge Base","knowledge-base",{"name":1984,"slug":1985,"type":16},"Search","search",6749,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fknowledge-catalog","2026-07-12T07:38:52.157375",{"slug":1990,"name":1991,"fn":1992,"description":1993,"org":1994,"tags":1995,"stars":1986,"repoUrl":1987,"updatedAt":2004},"knowledgecatalogdiscoveryagent","knowledge_catalog_discovery_agent","search and rank Knowledge Catalog data entries","Analyzes user queries, extracts relevant predicates, and utilizes Knowledge Catalog Search to find and rank the most relevant data entries. Engages with the user throughout the process.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1996,1999,2000,2003],{"name":1997,"slug":1998,"type":16},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":16},{"name":2001,"slug":2002,"type":16},"Knowledge Management","knowledge-management",{"name":1984,"slug":1985,"type":16},"2026-07-12T07:38:22.196851",{"slug":2006,"name":2006,"fn":2007,"description":2008,"org":2009,"tags":2010,"stars":2021,"repoUrl":2022,"updatedAt":2023},"contributing","contribute to Cloud Foundation Fabric","End-to-end workflow for contributing to Cloud Foundation Fabric: triaging GitHub issues, proactive feature development, validating with tests and Policy Troubleshooter, and submitting sanitized Pull Requests. Use when addressing a Fabric GitHub issue, developing a module or FAST stage change, or preparing a branch for a pull request.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2011,2013,2014,2017,2018],{"name":564,"slug":2012,"type":16},"automation",{"name":1923,"slug":1924,"type":16},{"name":2015,"slug":2016,"type":16},"GitHub","github",{"name":9,"slug":8,"type":16},{"name":2019,"slug":2020,"type":16},"Pull Requests","pull-requests",2062,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fcloud-foundation-fabric","2026-07-31T06:23:36.935005",{"slug":2025,"name":2025,"fn":2026,"description":2027,"org":2028,"tags":2029,"stars":2021,"repoUrl":2022,"updatedAt":2037},"fabric-builder","generate Terraform code for Google Cloud","Generates idiomatic Cloud Foundation Fabric (CFF) Terraform code using CFF modules. Use when users ask to create GCP resources, use Fabric modules, or generate Terraform code for Google Cloud.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2030,2031,2034],{"name":9,"slug":8,"type":16},{"name":2032,"slug":2033,"type":16},"Infrastructure as Code","infrastructure-as-code",{"name":2035,"slug":2036,"type":16},"Terraform","terraform","2026-07-12T07:38:23.514555",{"slug":2039,"name":2039,"fn":2040,"description":2041,"org":2042,"tags":2043,"stars":2021,"repoUrl":2022,"updatedAt":2046},"fast-0-org-setup-prereqs","prepare prerequisites for FAST 0-org-setup","Guides the user step-by-step through the prerequisites for the FAST 0-org-setup stage, supporting both Standard GCP and Google Cloud Dedicated (GCD) environments. Use when a user asks to prepare or run prerequisites for 0-org-setup or bootstrap the FAST landing zone.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2044,2045],{"name":9,"slug":8,"type":16},{"name":1967,"slug":1968,"type":16},"2026-07-12T07:38:28.127148",{"slug":2048,"name":2048,"fn":2049,"description":2050,"org":2051,"tags":2052,"stars":2058,"repoUrl":2059,"updatedAt":2060},"agent-aware-cli","design agent-aware command-line interfaces","Guide for designing and implementing command-line interfaces (CLIs) that are equally usable by human developers and automated coding agents. Use when the user wants to build a CLI, apply CLI best practices, or use Go with Cobra and Viper.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2053,2056,2057],{"name":2054,"slug":2055,"type":16},"CLI","cli",{"name":1923,"slug":1924,"type":16},{"name":9,"slug":8,"type":16},1150,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fvertex-ai-creative-studio","2026-07-12T07:39:08.41406",{"slug":2062,"name":2062,"fn":2063,"description":2064,"org":2065,"tags":2066,"stars":2058,"repoUrl":2059,"updatedAt":2077},"build-mcp-genmedia","build and configure GenAI MCP servers","Builds the mcp-genmedia Go MCP servers (nanobanana, veo, lyria, gemini-multimodal, chirp3-hd, avtool) from source and wires them into settings.json. Use this skill whenever the MCP tools are missing or broken — typically at the start of a new session, after a container restart, or when \u002Ftmp has been wiped. The prebuilt binaries in \u002Fworkspace\u002F.local\u002Fbin\u002F have no exec bit and live on a noexec mount; this skill compiles fresh executables into \u002Ftmp\u002Fbin\u002F where execution is allowed.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2067,2070,2071,2074],{"name":2068,"slug":2069,"type":16},"API Development","api-development",{"name":9,"slug":8,"type":16},{"name":2072,"slug":2073,"type":16},"LLM","llm",{"name":2075,"slug":2076,"type":16},"MCP","mcp","2026-07-12T07:39:10.911302",{"slug":2079,"name":2079,"fn":2080,"description":2081,"org":2082,"tags":2083,"stars":2058,"repoUrl":2059,"updatedAt":2094},"genmedia-audio-engineer","synthesize and mix audio content","Expert in audio synthesis, music generation, and mixing. Use when creating podcasts, background scores, or multi-track audio layering using mcp-chirp3-go, mcp-lyria-go, mcp-gemini-go, mcp-nanobanana-go, and mcp-avtool-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2084,2087,2090,2091],{"name":2085,"slug":2086,"type":16},"Audio","audio",{"name":2088,"slug":2089,"type":16},"Creative","creative",{"name":9,"slug":8,"type":16},{"name":2092,"slug":2093,"type":16},"Vertex AI","vertex-ai","2026-07-12T07:39:16.623879",{"slug":2096,"name":2096,"fn":2097,"description":2098,"org":2099,"tags":2100,"stars":2058,"repoUrl":2059,"updatedAt":2107},"genmedia-image-artist","generate and edit AI images","Expert in AI image generation and editing. Use when the user needs high-quality textures, character-consistent visuals, or image-to-image editing using mcp-nanobanana-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2101,2102,2103,2106],{"name":2088,"slug":2089,"type":16},{"name":9,"slug":8,"type":16},{"name":2104,"slug":2105,"type":16},"Image Generation","image-generation",{"name":2092,"slug":2093,"type":16},"2026-07-12T07:39:15.372822",{"slug":2109,"name":2109,"fn":2110,"description":2111,"org":2112,"tags":2113,"stars":2058,"repoUrl":2059,"updatedAt":2123},"genmedia-producer","produce multi-step media content","Expert media production assistant. Use when requested to help with storyboarding, podcast creation, audio assembly, or complex multi-step media workflows using the GenMedia MCP servers (Veo, Lyria, Gemini TTS, NanoBanana).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2114,2115,2116,2117,2120],{"name":2085,"slug":2086,"type":16},{"name":2088,"slug":2089,"type":16},{"name":9,"slug":8,"type":16},{"name":2118,"slug":2119,"type":16},"Media","media",{"name":2121,"slug":2122,"type":16},"Video","video","2026-07-12T07:39:09.672849",{"slug":2125,"name":2125,"fn":2126,"description":2127,"org":2128,"tags":2129,"stars":2058,"repoUrl":2059,"updatedAt":2133},"genmedia-video-editor","edit and compose video content","Expert in video composition, editing, and format conversion. Use when the user wants to generate high-quality video, overlay images on video, concatenate clips, create GIFs, or sync audio to video using mcp-avtool-go and mcp-veo-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2130,2131,2132],{"name":2088,"slug":2089,"type":16},{"name":9,"slug":8,"type":16},{"name":2121,"slug":2122,"type":16},"2026-07-12T07:39:13.749081",{"slug":2135,"name":2135,"fn":2136,"description":2137,"org":2138,"tags":2139,"stars":2058,"repoUrl":2059,"updatedAt":2151},"genmedia-voice-director","generate expressive text-to-speech with Gemini","Expert in casting, directing, and generating expressive text-to-speech using Gemini TTS. Use this when the user needs virtual voice actor personas, expressive speech generation, or multiple variations of a voiceover (like \"take 3 on the bounce\").",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2140,2141,2142,2145,2148],{"name":2085,"slug":2086,"type":16},{"name":2088,"slug":2089,"type":16},{"name":2143,"slug":2144,"type":16},"Gemini","gemini",{"name":2146,"slug":2147,"type":16},"Speech","speech",{"name":2149,"slug":2150,"type":16},"Text-to-Speech","text-to-speech","2026-07-12T07:39:17.86673",80]