[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-google-cloud-gke-ai-troubleshooting-tpu-connection-failure-vbar-oom":3,"mdc--1a81c8-key":34,"related-org-google-cloud-gke-ai-troubleshooting-tpu-connection-failure-vbar-oom":935,"related-repo-google-cloud-gke-ai-troubleshooting-tpu-connection-failure-vbar-oom":1123},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":24,"repoUrl":25,"updatedAt":26,"license":27,"forks":28,"topics":29,"repo":30,"sourceUrl":32,"mdContent":33},"gke-ai-troubleshooting-tpu-connection-failure-vbar-oom","diagnose GKE TPU connection failures","Diagnose and prevent `vbar_control_agent` segfaults and OOMs caused by race conditions during TPU device resets and frequent metrics collection (e.g. every 3s). Use when TPU slice initialization fails or `vbar_control_agent` crashes on TPU v6e nodes.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"google-cloud","Google Cloud","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgoogle-cloud.png","GoogleCloudPlatform",[13,17,20,21],{"name":14,"slug":15,"type":16},"Performance","performance","tag",{"name":18,"slug":19,"type":16},"Kubernetes","kubernetes",{"name":9,"slug":8,"type":16},{"name":22,"slug":23,"type":16},"Debugging","debugging",161,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fgke-mcp","2026-07-12T07:39:49.482979",null,78,[],{"repoUrl":25,"stars":24,"forks":28,"topics":31,"description":27},[],"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fgke-mcp\u002Ftree\u002FHEAD\u002Fskills\u002Fgke-ai-troubleshooting-tpu-connection-failure-vbar-oom","---\nname: gke-ai-troubleshooting-tpu-connection-failure-vbar-oom\ndescription: >\n  Diagnose and prevent `vbar_control_agent` segfaults and OOMs caused by race\n  conditions during TPU device resets and frequent metrics collection (e.g.\n  every 3s). Use when TPU slice initialization fails or `vbar_control_agent`\n  crashes on TPU v6e nodes.\n---\n\n# TPU Connection Failure and VBAR OOM Troubleshooting\n\nUse this skill to systematically diagnose and prevent `vbar_control_agent`\nsegfaults and Out-Of-Memory (OOM) errors on TPU v6e nodes.\n\n## ⚠️ Prerequisites\n\n- [ ] Cloud Logging must be enabled for the project.\n- [ ] Access to the project and cluster via `gcloud` or equivalent tool.\n\n## 🔍 Diagnostic Workflow\n\n### Step 0: Context Acquisition & Time Window Definition\n\nTo begin troubleshooting, acquire the following context from the user:\n\n- **Project ID** (e.g., `customer-ai-project-123`)\n- **Cluster Name** (e.g., `tpu-cluster-prod`)\n- **Node Name or Instance ID** (e.g., `tpu-node-1`)\n- **Workload Name (JobSet Name)** (e.g., `my-training-job-456`)\n- **Workload Namespace**\n- **Issue Time** (e.g., `2026-04-14T20:00:00Z`)\n\n#### Time Handling Rules\n\n1.  **Reject Relative Time**: If the user says \"X minutes ago\" or \"just now\",\n    stop and ask for the exact timestamp or a specific time window.\n2.  **Window Calculation**: If the user provides a start time or an \"around\"\n    time `T`, calculate the query window as **`[T - 30m]` to `[T + 30m]`**.\n    - Let `Start_Time` = `T - 30m`\n    - Let `End_Time` = `T + 30m`\n\n### Step 1: Check for `vbar_control_agent` OOMs\n\nLook for specific `out of memory` messages from `vbar_control_agent` in serial\nconsole logs.\n\n- **Tool to use**: `query_logs`\n- **Filter Templates**:\n\n**Serial Console Logs (OOMs):**\n\n```sql\nlogName=\"projects\u002F\u003Cproject_id>\u002Flogs\u002Fserialconsole.googleapis.com%2fserial_port_1_output\"\nAND labels.\"compute.googleapis.com\u002Fresource_name\"=\"\u003Cnode_name>\"\nAND SEARCH(text_payload, \"Memory cgroup out of memory: Killed process .* (vbar_control_ag)\")\nAND timestamp >= \"\u003CStart_Time>\"\nAND timestamp \u003C= \"\u003CEnd_Time>\"\n```\n\n- **Logic**: Presence of `Memory cgroup out of memory` messages related to\n  `vbar_control_agent`. Stack traces pointing to\n  `libtpu::tpunetd::VBARControlHelper::MetricsReadFromVBAR` are a strong indicator.\n- **Automation**: Proceed to next step automatically after reporting findings.\n- **Reference**: See `references\u002Ffailure_signatures.md` for example log patterns.\n\n### Step 2: Investigate `tpu-device-plugin` Metrics Fetch Failures [Low Risk]\n\nCheck if `tpu-device-plugin` is reporting metric fetch failures.\n\n- **Tool to use**: `query_logs`\n- **Filter Template**:\n\n```sql\nresource.type=\"k8s_container\"\nAND resource.labels.project_id=\"\u003Cproject_id>\"\nAND resource.labels.cluster_name=\"\u003Ccluster_name>\"\nAND resource.labels.container_name=\"tpu-device-plugin\"\nAND severity=ERROR\nAND textPayload:\"metrics fetch failed for .* deviceID and .* device path with error: checksum didn't match with the metrics data. Corrupt data found\"\nAND timestamp >= \"\u003CStart_Time>\"\nAND timestamp \u003C= \"\u003CEnd_Time>\"\n```\n\n- **Logic**: Errors indicating \"metrics fetch failed\" with \"checksum didn't\n  match\" suggest vBAR memory corruption.\n- **Automation**: Proceed to next step automatically after reporting findings.\n\n### Step 3: Check for Custom Metrics Collection Usage [Low Risk]\n\nInquire with the user about any custom TPU metrics collection mechanisms they\nhave deployed.\n\n- **Action**: Ask the user if they are using custom scripts or agents (e.g.,\n  using `libtpu.sdk.tpumonitoring`) that frequently query `GetHostMetrics` from\n  `vBAR Control Agent`.\n- **Logic**: Confirmation of custom metrics collection helps confirm the race\n  condition hypothesis.\n- **Automation**: Stop and wait for user response before proceeding to resolution.\n\n## 🛠️ Resolution Workflow\n\n### Resolution 1: Temporarily Disable Custom Metrics Collection [High Risk]\n\nIf a custom metrics collection agent is identified, advise the user to\ntemporarily disable it.\n\n- **Action**: Recommend disabling the custom metrics collector.\n- **Justification**: Prevents reads from vBAR during device resets, stopping\n  crashes and OOMs.\n- **Automation**: Stop and request explicit user approval in the bug thread\n  before making this recommendation or taking action.\n\n### Resolution 2: Await `vbar_control_agent` Resiliency Update [Low Risk]\n\nAdvise the user that a permanent fix will be available in a future GKE version.\n\n- **Action**: Recommend upgrading GKE when the fix is available.\n- **Justification**: The updated agent will be resilient to memory corruption\n  and gracefully handle reads from unbound vBARs.\n- **Automation**: Proceed to report this finding.\n\n## 📋 copypaste checklist\n\n- [ ] Acquire context and compute `[T - 30m, T + 30m]` window.\n- [ ] Check for `vbar_control_agent` segfaults and OOMs using `query_logs`.\n- [ ] Investigate `tpu-device-plugin` failures using `query_logs`.\n- [ ] Ask user about custom metrics collection usage.\n- [ ] Advise disabling custom metrics collection (High Risk) if applicable.\n- [ ] Advise awaiting resiliency update.\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,48,63,70,107,113,120,125,219,226,316,329,349,378,386,444,508,527,539,564,636,656,666,671,726,732,743,748,779,796,801,831,837,929],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"tpu-connection-failure-and-vbar-oom-troubleshooting",[45],{"type":46,"value":47},"text","TPU Connection Failure and VBAR OOM Troubleshooting",{"type":40,"tag":49,"props":50,"children":51},"p",{},[52,54,61],{"type":46,"value":53},"Use this skill to systematically diagnose and prevent ",{"type":40,"tag":55,"props":56,"children":58},"code",{"className":57},[],[59],{"type":46,"value":60},"vbar_control_agent",{"type":46,"value":62},"\nsegfaults and Out-Of-Memory (OOM) errors on TPU v6e nodes.",{"type":40,"tag":64,"props":65,"children":67},"h2",{"id":66},"️-prerequisites",[68],{"type":46,"value":69},"⚠️ Prerequisites",{"type":40,"tag":71,"props":72,"children":75},"ul",{"className":73},[74],"contains-task-list",[76,90],{"type":40,"tag":77,"props":78,"children":81},"li",{"className":79},[80],"task-list-item",[82,88],{"type":40,"tag":83,"props":84,"children":87},"input",{"disabled":85,"type":86},true,"checkbox",[],{"type":46,"value":89}," Cloud Logging must be enabled for the project.",{"type":40,"tag":77,"props":91,"children":93},{"className":92},[80],[94,97,99,105],{"type":40,"tag":83,"props":95,"children":96},{"disabled":85,"type":86},[],{"type":46,"value":98}," Access to the project and cluster via ",{"type":40,"tag":55,"props":100,"children":102},{"className":101},[],[103],{"type":46,"value":104},"gcloud",{"type":46,"value":106}," or equivalent tool.",{"type":40,"tag":64,"props":108,"children":110},{"id":109},"diagnostic-workflow",[111],{"type":46,"value":112},"🔍 Diagnostic Workflow",{"type":40,"tag":114,"props":115,"children":117},"h3",{"id":116},"step-0-context-acquisition-time-window-definition",[118],{"type":46,"value":119},"Step 0: Context Acquisition & Time Window Definition",{"type":40,"tag":49,"props":121,"children":122},{},[123],{"type":46,"value":124},"To begin troubleshooting, acquire the following context from the user:",{"type":40,"tag":71,"props":126,"children":127},{},[128,147,163,179,195,203],{"type":40,"tag":77,"props":129,"children":130},{},[131,137,139,145],{"type":40,"tag":132,"props":133,"children":134},"strong",{},[135],{"type":46,"value":136},"Project ID",{"type":46,"value":138}," (e.g., ",{"type":40,"tag":55,"props":140,"children":142},{"className":141},[],[143],{"type":46,"value":144},"customer-ai-project-123",{"type":46,"value":146},")",{"type":40,"tag":77,"props":148,"children":149},{},[150,155,156,162],{"type":40,"tag":132,"props":151,"children":152},{},[153],{"type":46,"value":154},"Cluster Name",{"type":46,"value":138},{"type":40,"tag":55,"props":157,"children":159},{"className":158},[],[160],{"type":46,"value":161},"tpu-cluster-prod",{"type":46,"value":146},{"type":40,"tag":77,"props":164,"children":165},{},[166,171,172,178],{"type":40,"tag":132,"props":167,"children":168},{},[169],{"type":46,"value":170},"Node Name or Instance ID",{"type":46,"value":138},{"type":40,"tag":55,"props":173,"children":175},{"className":174},[],[176],{"type":46,"value":177},"tpu-node-1",{"type":46,"value":146},{"type":40,"tag":77,"props":180,"children":181},{},[182,187,188,194],{"type":40,"tag":132,"props":183,"children":184},{},[185],{"type":46,"value":186},"Workload Name (JobSet Name)",{"type":46,"value":138},{"type":40,"tag":55,"props":189,"children":191},{"className":190},[],[192],{"type":46,"value":193},"my-training-job-456",{"type":46,"value":146},{"type":40,"tag":77,"props":196,"children":197},{},[198],{"type":40,"tag":132,"props":199,"children":200},{},[201],{"type":46,"value":202},"Workload Namespace",{"type":40,"tag":77,"props":204,"children":205},{},[206,211,212,218],{"type":40,"tag":132,"props":207,"children":208},{},[209],{"type":46,"value":210},"Issue Time",{"type":46,"value":138},{"type":40,"tag":55,"props":213,"children":215},{"className":214},[],[216],{"type":46,"value":217},"2026-04-14T20:00:00Z",{"type":46,"value":146},{"type":40,"tag":220,"props":221,"children":223},"h4",{"id":222},"time-handling-rules",[224],{"type":46,"value":225},"Time Handling Rules",{"type":40,"tag":227,"props":228,"children":229},"ol",{},[230,240],{"type":40,"tag":77,"props":231,"children":232},{},[233,238],{"type":40,"tag":132,"props":234,"children":235},{},[236],{"type":46,"value":237},"Reject Relative Time",{"type":46,"value":239},": If the user says \"X minutes ago\" or \"just now\",\nstop and ask for the exact timestamp or a specific time window.",{"type":40,"tag":77,"props":241,"children":242},{},[243,248,250,256,258,275,277],{"type":40,"tag":132,"props":244,"children":245},{},[246],{"type":46,"value":247},"Window Calculation",{"type":46,"value":249},": If the user provides a start time or an \"around\"\ntime ",{"type":40,"tag":55,"props":251,"children":253},{"className":252},[],[254],{"type":46,"value":255},"T",{"type":46,"value":257},", calculate the query window as ",{"type":40,"tag":132,"props":259,"children":260},{},[261,267,269],{"type":40,"tag":55,"props":262,"children":264},{"className":263},[],[265],{"type":46,"value":266},"[T - 30m]",{"type":46,"value":268}," to ",{"type":40,"tag":55,"props":270,"children":272},{"className":271},[],[273],{"type":46,"value":274},"[T + 30m]",{"type":46,"value":276},".\n",{"type":40,"tag":71,"props":278,"children":279},{},[280,299],{"type":40,"tag":77,"props":281,"children":282},{},[283,285,291,293],{"type":46,"value":284},"Let ",{"type":40,"tag":55,"props":286,"children":288},{"className":287},[],[289],{"type":46,"value":290},"Start_Time",{"type":46,"value":292}," = ",{"type":40,"tag":55,"props":294,"children":296},{"className":295},[],[297],{"type":46,"value":298},"T - 30m",{"type":40,"tag":77,"props":300,"children":301},{},[302,303,309,310],{"type":46,"value":284},{"type":40,"tag":55,"props":304,"children":306},{"className":305},[],[307],{"type":46,"value":308},"End_Time",{"type":46,"value":292},{"type":40,"tag":55,"props":311,"children":313},{"className":312},[],[314],{"type":46,"value":315},"T + 30m",{"type":40,"tag":114,"props":317,"children":319},{"id":318},"step-1-check-for-vbar_control_agent-ooms",[320,322,327],{"type":46,"value":321},"Step 1: Check for ",{"type":40,"tag":55,"props":323,"children":325},{"className":324},[],[326],{"type":46,"value":60},{"type":46,"value":328}," OOMs",{"type":40,"tag":49,"props":330,"children":331},{},[332,334,340,342,347],{"type":46,"value":333},"Look for specific ",{"type":40,"tag":55,"props":335,"children":337},{"className":336},[],[338],{"type":46,"value":339},"out of memory",{"type":46,"value":341}," messages from ",{"type":40,"tag":55,"props":343,"children":345},{"className":344},[],[346],{"type":46,"value":60},{"type":46,"value":348}," in serial\nconsole logs.",{"type":40,"tag":71,"props":350,"children":351},{},[352,368],{"type":40,"tag":77,"props":353,"children":354},{},[355,360,362],{"type":40,"tag":132,"props":356,"children":357},{},[358],{"type":46,"value":359},"Tool to use",{"type":46,"value":361},": ",{"type":40,"tag":55,"props":363,"children":365},{"className":364},[],[366],{"type":46,"value":367},"query_logs",{"type":40,"tag":77,"props":369,"children":370},{},[371,376],{"type":40,"tag":132,"props":372,"children":373},{},[374],{"type":46,"value":375},"Filter Templates",{"type":46,"value":377},":",{"type":40,"tag":49,"props":379,"children":380},{},[381],{"type":40,"tag":132,"props":382,"children":383},{},[384],{"type":46,"value":385},"Serial Console Logs (OOMs):",{"type":40,"tag":387,"props":388,"children":393},"pre",{"className":389,"code":390,"language":391,"meta":392,"style":392},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","logName=\"projects\u002F\u003Cproject_id>\u002Flogs\u002Fserialconsole.googleapis.com%2fserial_port_1_output\"\nAND labels.\"compute.googleapis.com\u002Fresource_name\"=\"\u003Cnode_name>\"\nAND SEARCH(text_payload, \"Memory cgroup out of memory: Killed process .* (vbar_control_ag)\")\nAND timestamp >= \"\u003CStart_Time>\"\nAND timestamp \u003C= \"\u003CEnd_Time>\"\n","sql","",[394],{"type":40,"tag":55,"props":395,"children":396},{"__ignoreMap":392},[397,408,417,426,435],{"type":40,"tag":398,"props":399,"children":402},"span",{"class":400,"line":401},"line",1,[403],{"type":40,"tag":398,"props":404,"children":405},{},[406],{"type":46,"value":407},"logName=\"projects\u002F\u003Cproject_id>\u002Flogs\u002Fserialconsole.googleapis.com%2fserial_port_1_output\"\n",{"type":40,"tag":398,"props":409,"children":411},{"class":400,"line":410},2,[412],{"type":40,"tag":398,"props":413,"children":414},{},[415],{"type":46,"value":416},"AND labels.\"compute.googleapis.com\u002Fresource_name\"=\"\u003Cnode_name>\"\n",{"type":40,"tag":398,"props":418,"children":420},{"class":400,"line":419},3,[421],{"type":40,"tag":398,"props":422,"children":423},{},[424],{"type":46,"value":425},"AND SEARCH(text_payload, \"Memory cgroup out of memory: Killed process .* (vbar_control_ag)\")\n",{"type":40,"tag":398,"props":427,"children":429},{"class":400,"line":428},4,[430],{"type":40,"tag":398,"props":431,"children":432},{},[433],{"type":46,"value":434},"AND timestamp >= \"\u003CStart_Time>\"\n",{"type":40,"tag":398,"props":436,"children":438},{"class":400,"line":437},5,[439],{"type":40,"tag":398,"props":440,"children":441},{},[442],{"type":46,"value":443},"AND timestamp \u003C= \"\u003CEnd_Time>\"\n",{"type":40,"tag":71,"props":445,"children":446},{},[447,480,490],{"type":40,"tag":77,"props":448,"children":449},{},[450,455,457,463,465,470,472,478],{"type":40,"tag":132,"props":451,"children":452},{},[453],{"type":46,"value":454},"Logic",{"type":46,"value":456},": Presence of ",{"type":40,"tag":55,"props":458,"children":460},{"className":459},[],[461],{"type":46,"value":462},"Memory cgroup out of memory",{"type":46,"value":464}," messages related to\n",{"type":40,"tag":55,"props":466,"children":468},{"className":467},[],[469],{"type":46,"value":60},{"type":46,"value":471},". Stack traces pointing to\n",{"type":40,"tag":55,"props":473,"children":475},{"className":474},[],[476],{"type":46,"value":477},"libtpu::tpunetd::VBARControlHelper::MetricsReadFromVBAR",{"type":46,"value":479}," are a strong indicator.",{"type":40,"tag":77,"props":481,"children":482},{},[483,488],{"type":40,"tag":132,"props":484,"children":485},{},[486],{"type":46,"value":487},"Automation",{"type":46,"value":489},": Proceed to next step automatically after reporting findings.",{"type":40,"tag":77,"props":491,"children":492},{},[493,498,500,506],{"type":40,"tag":132,"props":494,"children":495},{},[496],{"type":46,"value":497},"Reference",{"type":46,"value":499},": See ",{"type":40,"tag":55,"props":501,"children":503},{"className":502},[],[504],{"type":46,"value":505},"references\u002Ffailure_signatures.md",{"type":46,"value":507}," for example log patterns.",{"type":40,"tag":114,"props":509,"children":511},{"id":510},"step-2-investigate-tpu-device-plugin-metrics-fetch-failures-low-risk",[512,514,520,522],{"type":46,"value":513},"Step 2: Investigate ",{"type":40,"tag":55,"props":515,"children":517},{"className":516},[],[518],{"type":46,"value":519},"tpu-device-plugin",{"type":46,"value":521}," Metrics Fetch Failures ",{"type":40,"tag":398,"props":523,"children":524},{},[525],{"type":46,"value":526},"Low Risk",{"type":40,"tag":49,"props":528,"children":529},{},[530,532,537],{"type":46,"value":531},"Check if ",{"type":40,"tag":55,"props":533,"children":535},{"className":534},[],[536],{"type":46,"value":519},{"type":46,"value":538}," is reporting metric fetch failures.",{"type":40,"tag":71,"props":540,"children":541},{},[542,555],{"type":40,"tag":77,"props":543,"children":544},{},[545,549,550],{"type":40,"tag":132,"props":546,"children":547},{},[548],{"type":46,"value":359},{"type":46,"value":361},{"type":40,"tag":55,"props":551,"children":553},{"className":552},[],[554],{"type":46,"value":367},{"type":40,"tag":77,"props":556,"children":557},{},[558,563],{"type":40,"tag":132,"props":559,"children":560},{},[561],{"type":46,"value":562},"Filter Template",{"type":46,"value":377},{"type":40,"tag":387,"props":565,"children":567},{"className":389,"code":566,"language":391,"meta":392,"style":392},"resource.type=\"k8s_container\"\nAND resource.labels.project_id=\"\u003Cproject_id>\"\nAND resource.labels.cluster_name=\"\u003Ccluster_name>\"\nAND resource.labels.container_name=\"tpu-device-plugin\"\nAND severity=ERROR\nAND textPayload:\"metrics fetch failed for .* deviceID and .* device path with error: checksum didn't match with the metrics data. Corrupt data found\"\nAND timestamp >= \"\u003CStart_Time>\"\nAND timestamp \u003C= \"\u003CEnd_Time>\"\n",[568],{"type":40,"tag":55,"props":569,"children":570},{"__ignoreMap":392},[571,579,587,595,603,611,620,628],{"type":40,"tag":398,"props":572,"children":573},{"class":400,"line":401},[574],{"type":40,"tag":398,"props":575,"children":576},{},[577],{"type":46,"value":578},"resource.type=\"k8s_container\"\n",{"type":40,"tag":398,"props":580,"children":581},{"class":400,"line":410},[582],{"type":40,"tag":398,"props":583,"children":584},{},[585],{"type":46,"value":586},"AND resource.labels.project_id=\"\u003Cproject_id>\"\n",{"type":40,"tag":398,"props":588,"children":589},{"class":400,"line":419},[590],{"type":40,"tag":398,"props":591,"children":592},{},[593],{"type":46,"value":594},"AND resource.labels.cluster_name=\"\u003Ccluster_name>\"\n",{"type":40,"tag":398,"props":596,"children":597},{"class":400,"line":428},[598],{"type":40,"tag":398,"props":599,"children":600},{},[601],{"type":46,"value":602},"AND resource.labels.container_name=\"tpu-device-plugin\"\n",{"type":40,"tag":398,"props":604,"children":605},{"class":400,"line":437},[606],{"type":40,"tag":398,"props":607,"children":608},{},[609],{"type":46,"value":610},"AND severity=ERROR\n",{"type":40,"tag":398,"props":612,"children":614},{"class":400,"line":613},6,[615],{"type":40,"tag":398,"props":616,"children":617},{},[618],{"type":46,"value":619},"AND textPayload:\"metrics fetch failed for .* deviceID and .* device path with error: checksum didn't match with the metrics data. Corrupt data found\"\n",{"type":40,"tag":398,"props":621,"children":623},{"class":400,"line":622},7,[624],{"type":40,"tag":398,"props":625,"children":626},{},[627],{"type":46,"value":434},{"type":40,"tag":398,"props":629,"children":631},{"class":400,"line":630},8,[632],{"type":40,"tag":398,"props":633,"children":634},{},[635],{"type":46,"value":443},{"type":40,"tag":71,"props":637,"children":638},{},[639,648],{"type":40,"tag":77,"props":640,"children":641},{},[642,646],{"type":40,"tag":132,"props":643,"children":644},{},[645],{"type":46,"value":454},{"type":46,"value":647},": Errors indicating \"metrics fetch failed\" with \"checksum didn't\nmatch\" suggest vBAR memory corruption.",{"type":40,"tag":77,"props":649,"children":650},{},[651,655],{"type":40,"tag":132,"props":652,"children":653},{},[654],{"type":46,"value":487},{"type":46,"value":489},{"type":40,"tag":114,"props":657,"children":659},{"id":658},"step-3-check-for-custom-metrics-collection-usage-low-risk",[660,662],{"type":46,"value":661},"Step 3: Check for Custom Metrics Collection Usage ",{"type":40,"tag":398,"props":663,"children":664},{},[665],{"type":46,"value":526},{"type":40,"tag":49,"props":667,"children":668},{},[669],{"type":46,"value":670},"Inquire with the user about any custom TPU metrics collection mechanisms they\nhave deployed.",{"type":40,"tag":71,"props":672,"children":673},{},[674,708,717],{"type":40,"tag":77,"props":675,"children":676},{},[677,682,684,690,692,698,700,706],{"type":40,"tag":132,"props":678,"children":679},{},[680],{"type":46,"value":681},"Action",{"type":46,"value":683},": Ask the user if they are using custom scripts or agents (e.g.,\nusing ",{"type":40,"tag":55,"props":685,"children":687},{"className":686},[],[688],{"type":46,"value":689},"libtpu.sdk.tpumonitoring",{"type":46,"value":691},") that frequently query ",{"type":40,"tag":55,"props":693,"children":695},{"className":694},[],[696],{"type":46,"value":697},"GetHostMetrics",{"type":46,"value":699}," from\n",{"type":40,"tag":55,"props":701,"children":703},{"className":702},[],[704],{"type":46,"value":705},"vBAR Control Agent",{"type":46,"value":707},".",{"type":40,"tag":77,"props":709,"children":710},{},[711,715],{"type":40,"tag":132,"props":712,"children":713},{},[714],{"type":46,"value":454},{"type":46,"value":716},": Confirmation of custom metrics collection helps confirm the race\ncondition hypothesis.",{"type":40,"tag":77,"props":718,"children":719},{},[720,724],{"type":40,"tag":132,"props":721,"children":722},{},[723],{"type":46,"value":487},{"type":46,"value":725},": Stop and wait for user response before proceeding to resolution.",{"type":40,"tag":64,"props":727,"children":729},{"id":728},"️-resolution-workflow",[730],{"type":46,"value":731},"🛠️ Resolution Workflow",{"type":40,"tag":114,"props":733,"children":735},{"id":734},"resolution-1-temporarily-disable-custom-metrics-collection-high-risk",[736,738],{"type":46,"value":737},"Resolution 1: Temporarily Disable Custom Metrics Collection ",{"type":40,"tag":398,"props":739,"children":740},{},[741],{"type":46,"value":742},"High Risk",{"type":40,"tag":49,"props":744,"children":745},{},[746],{"type":46,"value":747},"If a custom metrics collection agent is identified, advise the user to\ntemporarily disable it.",{"type":40,"tag":71,"props":749,"children":750},{},[751,760,770],{"type":40,"tag":77,"props":752,"children":753},{},[754,758],{"type":40,"tag":132,"props":755,"children":756},{},[757],{"type":46,"value":681},{"type":46,"value":759},": Recommend disabling the custom metrics collector.",{"type":40,"tag":77,"props":761,"children":762},{},[763,768],{"type":40,"tag":132,"props":764,"children":765},{},[766],{"type":46,"value":767},"Justification",{"type":46,"value":769},": Prevents reads from vBAR during device resets, stopping\ncrashes and OOMs.",{"type":40,"tag":77,"props":771,"children":772},{},[773,777],{"type":40,"tag":132,"props":774,"children":775},{},[776],{"type":46,"value":487},{"type":46,"value":778},": Stop and request explicit user approval in the bug thread\nbefore making this recommendation or taking action.",{"type":40,"tag":114,"props":780,"children":782},{"id":781},"resolution-2-await-vbar_control_agent-resiliency-update-low-risk",[783,785,790,792],{"type":46,"value":784},"Resolution 2: Await ",{"type":40,"tag":55,"props":786,"children":788},{"className":787},[],[789],{"type":46,"value":60},{"type":46,"value":791}," Resiliency Update ",{"type":40,"tag":398,"props":793,"children":794},{},[795],{"type":46,"value":526},{"type":40,"tag":49,"props":797,"children":798},{},[799],{"type":46,"value":800},"Advise the user that a permanent fix will be available in a future GKE version.",{"type":40,"tag":71,"props":802,"children":803},{},[804,813,822],{"type":40,"tag":77,"props":805,"children":806},{},[807,811],{"type":40,"tag":132,"props":808,"children":809},{},[810],{"type":46,"value":681},{"type":46,"value":812},": Recommend upgrading GKE when the fix is available.",{"type":40,"tag":77,"props":814,"children":815},{},[816,820],{"type":40,"tag":132,"props":817,"children":818},{},[819],{"type":46,"value":767},{"type":46,"value":821},": The updated agent will be resilient to memory corruption\nand gracefully handle reads from unbound vBARs.",{"type":40,"tag":77,"props":823,"children":824},{},[825,829],{"type":40,"tag":132,"props":826,"children":827},{},[828],{"type":46,"value":487},{"type":46,"value":830},": Proceed to report this finding.",{"type":40,"tag":64,"props":832,"children":834},{"id":833},"copypaste-checklist",[835],{"type":46,"value":836},"📋 copypaste checklist",{"type":40,"tag":71,"props":838,"children":840},{"className":839},[74],[841,858,880,902,911,920],{"type":40,"tag":77,"props":842,"children":844},{"className":843},[80],[845,848,850,856],{"type":40,"tag":83,"props":846,"children":847},{"disabled":85,"type":86},[],{"type":46,"value":849}," Acquire context and compute ",{"type":40,"tag":55,"props":851,"children":853},{"className":852},[],[854],{"type":46,"value":855},"[T - 30m, T + 30m]",{"type":46,"value":857}," window.",{"type":40,"tag":77,"props":859,"children":861},{"className":860},[80],[862,865,867,872,874,879],{"type":40,"tag":83,"props":863,"children":864},{"disabled":85,"type":86},[],{"type":46,"value":866}," Check for ",{"type":40,"tag":55,"props":868,"children":870},{"className":869},[],[871],{"type":46,"value":60},{"type":46,"value":873}," segfaults and OOMs using ",{"type":40,"tag":55,"props":875,"children":877},{"className":876},[],[878],{"type":46,"value":367},{"type":46,"value":707},{"type":40,"tag":77,"props":881,"children":883},{"className":882},[80],[884,887,889,894,896,901],{"type":40,"tag":83,"props":885,"children":886},{"disabled":85,"type":86},[],{"type":46,"value":888}," Investigate ",{"type":40,"tag":55,"props":890,"children":892},{"className":891},[],[893],{"type":46,"value":519},{"type":46,"value":895}," failures using ",{"type":40,"tag":55,"props":897,"children":899},{"className":898},[],[900],{"type":46,"value":367},{"type":46,"value":707},{"type":40,"tag":77,"props":903,"children":905},{"className":904},[80],[906,909],{"type":40,"tag":83,"props":907,"children":908},{"disabled":85,"type":86},[],{"type":46,"value":910}," Ask user about custom metrics collection usage.",{"type":40,"tag":77,"props":912,"children":914},{"className":913},[80],[915,918],{"type":40,"tag":83,"props":916,"children":917},{"disabled":85,"type":86},[],{"type":46,"value":919}," Advise disabling custom metrics collection (High Risk) if applicable.",{"type":40,"tag":77,"props":921,"children":923},{"className":922},[80],[924,927],{"type":40,"tag":83,"props":925,"children":926},{"disabled":85,"type":86},[],{"type":46,"value":928}," Advise awaiting resiliency update.",{"type":40,"tag":930,"props":931,"children":932},"style",{},[933],{"type":46,"value":934},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":936,"total":1122},[937,955,971,992,1006,1017,1031,1048,1065,1078,1094,1104],{"slug":938,"name":938,"fn":939,"description":940,"org":941,"tags":942,"stars":952,"repoUrl":953,"updatedAt":954},"kb-search","search and extract local knowledge base documents","Allows listing, searching and extracting information from local knowledge base documents for information about tables\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[943,946,949],{"name":944,"slug":945,"type":16},"Documentation","documentation",{"name":947,"slug":948,"type":16},"Knowledge Base","knowledge-base",{"name":950,"slug":951,"type":16},"Search","search",6749,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fknowledge-catalog","2026-07-12T07:38:52.157375",{"slug":956,"name":957,"fn":958,"description":959,"org":960,"tags":961,"stars":952,"repoUrl":953,"updatedAt":970},"knowledgecatalogdiscoveryagent","knowledge_catalog_discovery_agent","search and rank Knowledge Catalog data entries","Analyzes user queries, extracts relevant predicates, and utilizes Knowledge Catalog Search to find and rank the most relevant data entries. Engages with the user throughout the process.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[962,965,966,969],{"name":963,"slug":964,"type":16},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":16},{"name":967,"slug":968,"type":16},"Knowledge Management","knowledge-management",{"name":950,"slug":951,"type":16},"2026-07-12T07:38:22.196851",{"slug":972,"name":972,"fn":973,"description":974,"org":975,"tags":976,"stars":989,"repoUrl":990,"updatedAt":991},"contributing","contribute to Cloud Foundation Fabric","End-to-end workflow for contributing to Cloud Foundation Fabric: triaging GitHub issues, proactive feature development, validating with tests and Policy Troubleshooter, and submitting sanitized Pull Requests. Use when addressing a Fabric GitHub issue, developing a module or FAST stage change, or preparing a branch for a pull request.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[977,979,982,985,986],{"name":487,"slug":978,"type":16},"automation",{"name":980,"slug":981,"type":16},"Engineering","engineering",{"name":983,"slug":984,"type":16},"GitHub","github",{"name":9,"slug":8,"type":16},{"name":987,"slug":988,"type":16},"Pull Requests","pull-requests",2062,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fcloud-foundation-fabric","2026-07-31T06:23:36.935005",{"slug":993,"name":993,"fn":994,"description":995,"org":996,"tags":997,"stars":989,"repoUrl":990,"updatedAt":1005},"fabric-builder","generate Terraform code for Google Cloud","Generates idiomatic Cloud Foundation Fabric (CFF) Terraform code using CFF modules. Use when users ask to create GCP resources, use Fabric modules, or generate Terraform code for Google Cloud.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[998,999,1002],{"name":9,"slug":8,"type":16},{"name":1000,"slug":1001,"type":16},"Infrastructure as Code","infrastructure-as-code",{"name":1003,"slug":1004,"type":16},"Terraform","terraform","2026-07-12T07:38:23.514555",{"slug":1007,"name":1007,"fn":1008,"description":1009,"org":1010,"tags":1011,"stars":989,"repoUrl":990,"updatedAt":1016},"fast-0-org-setup-prereqs","prepare prerequisites for FAST 0-org-setup","Guides the user step-by-step through the prerequisites for the FAST 0-org-setup stage, supporting both Standard GCP and Google Cloud Dedicated (GCD) environments. Use when a user asks to prepare or run prerequisites for 0-org-setup or bootstrap the FAST landing zone.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1012,1013],{"name":9,"slug":8,"type":16},{"name":1014,"slug":1015,"type":16},"Operations","operations","2026-07-12T07:38:28.127148",{"slug":1018,"name":1018,"fn":1019,"description":1020,"org":1021,"tags":1022,"stars":1028,"repoUrl":1029,"updatedAt":1030},"agent-aware-cli","design agent-aware command-line interfaces","Guide for designing and implementing command-line interfaces (CLIs) that are equally usable by human developers and automated coding agents. Use when the user wants to build a CLI, apply CLI best practices, or use Go with Cobra and Viper.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1023,1026,1027],{"name":1024,"slug":1025,"type":16},"CLI","cli",{"name":980,"slug":981,"type":16},{"name":9,"slug":8,"type":16},1150,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fvertex-ai-creative-studio","2026-07-12T07:39:08.41406",{"slug":1032,"name":1032,"fn":1033,"description":1034,"org":1035,"tags":1036,"stars":1028,"repoUrl":1029,"updatedAt":1047},"build-mcp-genmedia","build and configure GenAI MCP servers","Builds the mcp-genmedia Go MCP servers (nanobanana, veo, lyria, gemini-multimodal, chirp3-hd, avtool) from source and wires them into settings.json. Use this skill whenever the MCP tools are missing or broken — typically at the start of a new session, after a container restart, or when \u002Ftmp has been wiped. The prebuilt binaries in \u002Fworkspace\u002F.local\u002Fbin\u002F have no exec bit and live on a noexec mount; this skill compiles fresh executables into \u002Ftmp\u002Fbin\u002F where execution is allowed.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1037,1040,1041,1044],{"name":1038,"slug":1039,"type":16},"API Development","api-development",{"name":9,"slug":8,"type":16},{"name":1042,"slug":1043,"type":16},"LLM","llm",{"name":1045,"slug":1046,"type":16},"MCP","mcp","2026-07-12T07:39:10.911302",{"slug":1049,"name":1049,"fn":1050,"description":1051,"org":1052,"tags":1053,"stars":1028,"repoUrl":1029,"updatedAt":1064},"genmedia-audio-engineer","synthesize and mix audio content","Expert in audio synthesis, music generation, and mixing. Use when creating podcasts, background scores, or multi-track audio layering using mcp-chirp3-go, mcp-lyria-go, mcp-gemini-go, mcp-nanobanana-go, and mcp-avtool-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1054,1057,1060,1061],{"name":1055,"slug":1056,"type":16},"Audio","audio",{"name":1058,"slug":1059,"type":16},"Creative","creative",{"name":9,"slug":8,"type":16},{"name":1062,"slug":1063,"type":16},"Vertex AI","vertex-ai","2026-07-12T07:39:16.623879",{"slug":1066,"name":1066,"fn":1067,"description":1068,"org":1069,"tags":1070,"stars":1028,"repoUrl":1029,"updatedAt":1077},"genmedia-image-artist","generate and edit AI images","Expert in AI image generation and editing. Use when the user needs high-quality textures, character-consistent visuals, or image-to-image editing using mcp-nanobanana-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1071,1072,1073,1076],{"name":1058,"slug":1059,"type":16},{"name":9,"slug":8,"type":16},{"name":1074,"slug":1075,"type":16},"Image Generation","image-generation",{"name":1062,"slug":1063,"type":16},"2026-07-12T07:39:15.372822",{"slug":1079,"name":1079,"fn":1080,"description":1081,"org":1082,"tags":1083,"stars":1028,"repoUrl":1029,"updatedAt":1093},"genmedia-producer","produce multi-step media content","Expert media production assistant. Use when requested to help with storyboarding, podcast creation, audio assembly, or complex multi-step media workflows using the GenMedia MCP servers (Veo, Lyria, Gemini TTS, NanoBanana).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1084,1085,1086,1087,1090],{"name":1055,"slug":1056,"type":16},{"name":1058,"slug":1059,"type":16},{"name":9,"slug":8,"type":16},{"name":1088,"slug":1089,"type":16},"Media","media",{"name":1091,"slug":1092,"type":16},"Video","video","2026-07-12T07:39:09.672849",{"slug":1095,"name":1095,"fn":1096,"description":1097,"org":1098,"tags":1099,"stars":1028,"repoUrl":1029,"updatedAt":1103},"genmedia-video-editor","edit and compose video content","Expert in video composition, editing, and format conversion. Use when the user wants to generate high-quality video, overlay images on video, concatenate clips, create GIFs, or sync audio to video using mcp-avtool-go and mcp-veo-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1100,1101,1102],{"name":1058,"slug":1059,"type":16},{"name":9,"slug":8,"type":16},{"name":1091,"slug":1092,"type":16},"2026-07-12T07:39:13.749081",{"slug":1105,"name":1105,"fn":1106,"description":1107,"org":1108,"tags":1109,"stars":1028,"repoUrl":1029,"updatedAt":1121},"genmedia-voice-director","generate expressive text-to-speech with Gemini","Expert in casting, directing, and generating expressive text-to-speech using Gemini TTS. Use this when the user needs virtual voice actor personas, expressive speech generation, or multiple variations of a voiceover (like \"take 3 on the bounce\").",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1110,1111,1112,1115,1118],{"name":1055,"slug":1056,"type":16},{"name":1058,"slug":1059,"type":16},{"name":1113,"slug":1114,"type":16},"Gemini","gemini",{"name":1116,"slug":1117,"type":16},"Speech","speech",{"name":1119,"slug":1120,"type":16},"Text-to-Speech","text-to-speech","2026-07-12T07:39:17.86673",80,{"items":1124,"total":1208},[1125,1137,1148,1161,1174,1181,1197],{"slug":1126,"name":1126,"fn":1127,"description":1128,"org":1129,"tags":1130,"stars":24,"repoUrl":25,"updatedAt":1136},"custom-golden-image-discovery","discover golden base images for GKE nodes","Expert at discovering golden base images for GKE custom nodes using technical specs or context clues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1131,1134,1135],{"name":1132,"slug":1133,"type":16},"Deployment","deployment",{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},"2026-07-12T07:39:30.888879",{"slug":1138,"name":1138,"fn":1139,"description":1140,"org":1141,"tags":1142,"stars":24,"repoUrl":25,"updatedAt":1147},"gke-ai-troubleshooting-handle-disruption-gpu-tpu","diagnose GPU and TPU workload disruptions","Diagnose and predict node disruption during Compute Engine host maintenance for GPU and TPU workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1143,1144,1145,1146],{"name":22,"slug":23,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-07-28T05:34:18.149515",{"slug":1149,"name":1149,"fn":1150,"description":1151,"org":1152,"tags":1153,"stars":24,"repoUrl":25,"updatedAt":1160},"gke-ai-troubleshooting-jobset-interruption","diagnose GKE JobSet interruptions","Systematically diagnose GKE JobSet interruptions, restarts, and preemptions for AI\u002FML training workloads. Identifies preemption events, maintenance interruptions, bad host VMs, unhealthy pods, and coordinator worker failures.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1154,1155,1156,1157],{"name":22,"slug":23,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":1158,"slug":1159,"type":16},"Observability","observability","2026-07-12T07:40:04.511878",{"slug":1162,"name":1162,"fn":1163,"description":1164,"org":1165,"tags":1166,"stars":24,"repoUrl":25,"updatedAt":1173},"gke-ai-troubleshooting-skill-creation-guide","create GKE troubleshooting skill bundles","Expert instructions for building high-quality GKE troubleshooting skills. Codifies Step 0 context rules, zero-hallucination signatures, and explicit LQL\u002FPromQL query requirements.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1167,1168,1169,1170],{"name":944,"slug":945,"type":16},{"name":980,"slug":981,"type":16},{"name":9,"slug":8,"type":16},{"name":1171,"slug":1172,"type":16},"Technical Writing","technical-writing","2026-07-12T07:39:50.73484",{"slug":4,"name":4,"fn":5,"description":6,"org":1175,"tags":1176,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1177,1178,1179,1180],{"name":22,"slug":23,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":1182,"name":1182,"fn":1183,"description":1184,"org":1185,"tags":1186,"stars":24,"repoUrl":25,"updatedAt":1196},"gke-app-onboarding","containerize and deploy apps to GKE","Workflows for containerizing and deploying applications to GKE for the first time.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1187,1190,1191,1192,1193],{"name":1188,"slug":1189,"type":16},"Containers","containers",{"name":1132,"slug":1133,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":1194,"slug":1195,"type":16},"Onboarding","onboarding","2026-07-12T07:39:41.935837",{"slug":1198,"name":1198,"fn":1199,"description":1200,"org":1201,"tags":1202,"stars":24,"repoUrl":25,"updatedAt":1207},"gke-backup-dr","configure GKE backup and disaster recovery","Workflows for configuring Backup for GKE and disaster recovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1203,1204,1205,1206],{"name":1132,"slug":1133,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":1014,"slug":1015,"type":16},"2026-07-12T07:39:34.806995",25]