[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-deepgram-infrastructure-ops":3,"mdc--iib5gr-key":29,"related-org-deepgram-infrastructure-ops":1098,"related-repo-deepgram-infrastructure-ops":1259},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":19,"repoUrl":20,"updatedAt":21,"license":22,"forks":23,"topics":24,"repo":25,"sourceUrl":27,"mdContent":28},"infrastructure-ops","manage infrastructure across multiple servers","Use when you need to manage infrastructure across multiple servers interactively via wsh — deploying applications, configuring services, managing packages, performing rolling updates, and handling the prompts and judgment calls that declarative tools cannot. Examples: \"deploy this application across 10 servers with health checks between each\", \"upgrade packages across the fleet and handle diverse prompts\", \"inspect and modify configuration across servers\", \"roll back a failed deployment\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"deepgram","Deepgram","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdeepgram.png",[12,16],{"name":13,"slug":14,"type":15},"CLI","cli","tag",{"name":17,"slug":18,"type":15},"Infrastructure","infrastructure",5,"https:\u002F\u002Fgithub.com\u002Fdeepgram\u002Fwsh","2026-07-12T08:29:57.659886",null,2,[],{"repoUrl":20,"stars":19,"forks":23,"topics":26,"description":22},[],"https:\u002F\u002Fgithub.com\u002Fdeepgram\u002Fwsh\u002Ftree\u002FHEAD\u002Fskills\u002Finfrastructure-ops","---\nname: infrastructure-ops\ndescription: >\n Use when you need to manage infrastructure across multiple servers\n interactively via wsh — deploying applications, configuring services,\n managing packages, performing rolling updates, and handling the prompts\n and judgment calls that declarative tools cannot. Examples: \"deploy this\n application across 10 servers with health checks between each\",\n \"upgrade packages across the fleet and handle diverse prompts\",\n \"inspect and modify configuration across servers\", \"roll back a\n failed deployment\".\n---\n\n> **IMPORTANT: EXECUTION CONTEXT**\n> This skill describes *what to do* — domain patterns and decision-making.\n> It does NOT describe *how* to call the API.\n>\n> 1. **If you have `wsh_*` tools** (check your toolkit for `wsh_send_input`,\n> `wsh_get_screen`, etc.): use them directly. Operation names in this\n> skill generally map to tool names (e.g., \"send input\" → `wsh_send_input`).\n> When in doubt, list your available `wsh_*` tools.\n> 2. **If you do NOT have `wsh_*` tools**: you are in HTTP\u002Fcurl fallback mode.\n> **DO NOT GUESS endpoints or CLI subcommands.**\n> Load the full API reference first: search your workspace for\n> `skills\u002Fcore\u002F` and read `SKILL.md`. It contains every endpoint\n> with working curl examples and a bootstrap sequence.\n> 3. **Quick bootstrap**: `curl -sf --unix-socket ${XDG_RUNTIME_DIR:-\u002Ftmp}\u002Fwsh\u002Fdefault.http.sock http:\u002F\u002Flocalhost\u002Fhealth`\n> — if that fails: `wsh server -L agent-$$ --ephemeral &` and retry.\n\n# wsh:infrastructure-ops -- Fleet Management\n\nAnsible, Puppet, and Chef exist because you can't interactively\noperate 50 machines at once. They solve this with declarative\nconfigs and idempotent operations -- you describe the desired state,\nthe tool converges toward it. This works brilliantly for the 80%\nof infrastructure work that's predictable and repeatable.\n\nBut 20% of real infrastructure work isn't predictable. Interactive\ninstallers that ask questions nobody anticipated. Package upgrades\nthat present merge conflicts in config files. Services that fail\nin ways that require investigation, not just a restart. Approval\nprompts that need human judgment. Diagnostics that require poking\naround, reading logs, trying things.\n\nwsh changes the equation. An AI agent can sit at every terminal\nsimultaneously -- reading screens, handling prompts, making\ndecisions. This enables an imperative, interactive model for the\nwork that declarative tools can't handle. Not a replacement for\nAnsible. A complement for the cases where Ansible isn't enough.\n\n## When to Use This\n\n**Use infrastructure-ops when:**\n- Operations require interactive input -- installers, prompts,\n approval dialogs, merge conflict resolution\n- Each host may behave differently and needs per-host judgment\n- You need to verify results interactively between steps\n (read logs, check endpoints, inspect state)\n- The operation is exploratory -- diagnosing an issue across\n a fleet, where what you do on host N depends on what you\n found on hosts 1 through N-1\n- Rolling deployments need health verification with real\n traffic before proceeding to the next batch\n\n**Don't use infrastructure-ops when:**\n- The operation is fully predictable and idempotent -- use\n Ansible, Terraform, or your existing config management\n- You're provisioning infrastructure from scratch -- use\n Terraform or CloudFormation\n- The operation is a single command with no interaction --\n use `parallel-ssh` or `ansible -m shell`\n- You don't need per-host decision-making -- fan out with\n a simple script instead\n\n## Prerequisites\n\nYou need a federated wsh cluster: a hub server with backends\nregistered for each target machine. Each backend is a wsh server\nrunning on the target host.\n\n list servers\n # Expect: hub (local), plus one backend per target host\n\nIf backends aren't registered yet, add them:\n\n add server at address http:\u002F\u002F10.0.1.10:8080\n add server at address http:\u002F\u002F10.0.1.11:8080\n add server at address http:\u002F\u002F10.0.1.12:8080\n\n # Wait for all to become healthy\n loop:\n list servers\n if all target servers are healthy: break\n wait briefly\n retry\n\nSee the **wsh:cluster-orchestration** skill for details on server\nregistration, health monitoring, and authentication.\n\nYou also need the patterns from **wsh:drive-process** (the\nsend\u002Fwait\u002Fread loop) and **wsh:multi-session** (parallel session\nmanagement, tagging, fan-out). This skill composes on top of both.\n\n## Core Pattern: Same Operation Across N Hosts\n\nThe fundamental infrastructure pattern: run the same operation on\nevery host, but handle each host's response individually. This is\ndifferent from `parallel-ssh` because you react to what each host\ndoes -- you don't just fire and forget.\n\n### Sequential Fan-Out\n\nThe safe default. Operate on one host at a time, verify success\nbefore moving to the next:\n\n hosts = [\"web-1\", \"web-2\", \"web-3\", \"web-4\", \"web-5\"]\n results = {}\n\n for host in hosts:\n create session \"op-{host}\" on server \"{host}\"\n send to \"op-{host}\": sudo systemctl restart myapp\\n\n wait for idle on \"op-{host}\"\n read screen from \"op-{host}\"\n\n if password prompt detected:\n send to \"op-{host}\": {sudo_password}\\n\n wait for idle on \"op-{host}\"\n read screen from \"op-{host}\"\n\n if success:\n results[host] = \"ok\"\n else:\n results[host] = \"failed\"\n # Decide: continue to next host, or stop?\n\n kill session \"op-{host}\"\n\n report results\n\nSequential is slower but safer. If the operation breaks the first\nhost, you haven't touched the other four.\n\n### Parallel Fan-Out\n\nWhen the operation is safe to run everywhere simultaneously --\nread-only commands, status checks, log collection:\n\n hosts = [\"web-1\", \"web-2\", \"web-3\", \"web-4\", \"web-5\"]\n\n # Launch all at once\n for host in hosts:\n create session \"check-{host}\" on server \"{host}\", tagged: fleet-check\n send to \"check-{host}\": systemctl status myapp\\n\n\n # Poll for completion\n completed = {}\n while not all hosts completed:\n wait for idle on sessions tagged \"fleet-check\" (timeout 2000ms)\n # Returns whichever session settled first\n read screen from that session\n record result\n mark as completed\n # Use last_session + last_generation to advance\n\n # Clean up\n for host in hosts:\n kill session \"check-{host}\"\n\n report results\n\n### Handling Per-Host Prompt Diversity\n\nThe real reason this pattern exists. When you run `apt upgrade`\nacross 10 servers, each one may present different prompts:\n\n- Server 1: no prompts, clean upgrade\n- Server 2: \"Configuration file changed, keep or replace? [Y\u002Fn]\"\n- Server 3: \"Service nginx needs to be restarted. Restart now? [y\u002FN]\"\n- Server 4: \"Pending kernel upgrade. Reboot required.\"\n- Server 5: dependency conflict requiring manual resolution\n\nA declarative tool must handle all these cases upfront with\nconfiguration flags. An agent reads each screen and responds\nwith judgment:\n\n read screen from \"upgrade-{host}\"\n\n if \"keep or replace\" in screen:\n # Inspect the diff, decide based on whether we've\n # customized this config\n send: d\\n # show diff first\n wait, read\n if config has local customizations:\n send: N\\n # keep ours\n else:\n send: Y\\n # take the new version\n\n if \"restart now\" in screen:\n # Check if this is a load-balanced service that can restart\n send: y\\n\n\n if \"reboot required\" in screen:\n # Note for later -- don't reboot mid-upgrade-sweep\n results[host].needs_reboot = true\n\nThis per-host judgment is the core value proposition. The agent\nadapts to what each host actually presents.\n\n## Rolling Deployments\n\nDeploy to a fleet incrementally, verifying health between each\nbatch. If something goes wrong, stop before it affects the whole\nfleet.\n\n### The Rolling Deploy Pattern\n\n hosts = [\"web-1\", \"web-2\", \"web-3\", \"web-4\", \"web-5\", \"web-6\"]\n batch_size = 2\n failed = []\n\n for batch in chunk(hosts, batch_size):\n # Deploy this batch in parallel\n for host in batch:\n create session \"deploy-{host}\" on server \"{host}\", tagged: deploy\n send to \"deploy-{host}\": cd \u002Fopt\u002Fmyapp && .\u002Fdeploy.sh v2.1.0\\n\n\n # Wait for all in this batch to finish\n for host in batch:\n wait for idle on \"deploy-{host}\" (timeout 30000ms)\n read screen from \"deploy-{host}\"\n\n if deployment prompt detected:\n handle prompt (see per-host prompt handling)\n\n wait for idle on \"deploy-{host}\"\n read screen from \"deploy-{host}\"\n\n if deployment failed:\n failed.append(host)\n\n # Health check this batch before proceeding\n for host in batch:\n create session \"health-{host}\" on server \"{host}\", tagged: health\n send to \"health-{host}\": curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth\\n\n wait for idle on \"health-{host}\"\n read screen from \"health-{host}\"\n\n if health check failed:\n failed.append(host)\n\n kill session \"health-{host}\"\n\n # Clean up deploy sessions for this batch\n for host in batch:\n kill session \"deploy-{host}\"\n\n # Gate: stop if this batch had failures\n if failed is not empty:\n report: \"Stopping rolling deploy. Failed hosts: {failed}\"\n report: \"Remaining hosts not deployed: {remaining}\"\n break\n\n # Optional: wait for the new version to soak\n wait 30 seconds\n # Re-check health after soak period\n for host in batch:\n create session \"soak-{host}\" on server \"{host}\"\n send to \"soak-{host}\": curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth\\n\n wait for idle, read screen\n if unhealthy:\n failed.append(host)\n kill session \"soak-{host}\"\n\n if failed is not empty:\n report: \"Soak check failed. Stopping.\"\n break\n\n if failed is empty:\n report: \"Rolling deploy complete. All hosts healthy.\"\n\n### Choosing Batch Size\n\n- **Batch of 1**: Safest. Every host is individually verified.\n Slowest. Use for the first deploy of a risky change.\n- **Batch of N\u002F3**: Good balance. You always have 2\u002F3 of the\n fleet on the old version if something goes wrong.\n- **Batch of N-1**: Deploy to all but one (the canary). If the\n canary is still fine on the old version after you've deployed\n everywhere else, you know the issue is with the new version.\n\n### Canary Pattern\n\nA special case of rolling deployment: deploy to one host first,\nsoak for longer, then proceed if healthy:\n\n canary = hosts[0]\n rest = hosts[1:]\n\n # Deploy to canary\n create session \"deploy-canary\" on server \"{canary}\"\n send to \"deploy-canary\": .\u002Fdeploy.sh v2.1.0\\n\n wait for idle, read screen, handle prompts\n kill session \"deploy-canary\"\n\n # Extended health check on canary\n for i in range(5):\n wait 60 seconds\n create session \"canary-check\" on server \"{canary}\"\n send to \"canary-check\": curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth && echo \"HEALTHY\" || echo \"UNHEALTHY\"\\n\n wait for idle, read screen\n if \"UNHEALTHY\" in screen:\n report: \"Canary failed health check. Aborting.\"\n initiate rollback on canary\n stop\n kill session \"canary-check\"\n\n # Canary is healthy after 5 minutes. Deploy the rest.\n proceed with rolling deploy on rest\n\n## Configuration Management\n\nInspect current configuration, apply changes, and verify the\nresult -- interactively, with the ability to inspect and judge\nat each step.\n\n### Inspect-Modify-Verify\n\nThe fundamental configuration pattern:\n\n for host in hosts:\n create session \"config-{host}\" on server \"{host}\"\n\n # 1. Inspect current state\n send to \"config-{host}\": cat \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n wait for idle, read screen\n record current_config[host]\n\n # 2. Decide whether to modify\n if config needs updating:\n # Back up first\n send: cp \u002Fetc\u002Fmyapp\u002Fconfig.yaml \u002Fetc\u002Fmyapp\u002Fconfig.yaml.bak\\n\n wait for idle\n\n # Apply the change (using sed, patch, or writing a new file)\n send: sed -i 's\u002Fmax_connections: 100\u002Fmax_connections: 200\u002F' \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n wait for idle\n\n # 3. Verify the change\n send: cat \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n wait for idle, read screen\n if change is correct:\n results[host] = \"updated\"\n else:\n # Restore backup\n send: cp \u002Fetc\u002Fmyapp\u002Fconfig.yaml.bak \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n wait for idle\n results[host] = \"reverted -- change didn't apply correctly\"\n else:\n results[host] = \"no change needed\"\n\n kill session \"config-{host}\"\n\n### Configuration Drift Detection\n\nCompare configurations across the fleet to find hosts that\nhave drifted from the expected state:\n\n expected_config = \"...\" # known-good configuration\n drifted = []\n\n for host in hosts:\n create session \"audit-{host}\" on server \"{host}\", tagged: audit\n send to \"audit-{host}\": cat \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n\n for host in hosts:\n wait for idle on \"audit-{host}\"\n read screen from \"audit-{host}\"\n if screen does not match expected_config:\n drifted.append(host)\n # Optionally: read scrollback for the full config if it's long\n kill session \"audit-{host}\"\n\n if drifted:\n report: \"Configuration drift detected on: {drifted}\"\n # Inspect each drifted host for details\n\n### Interactive Configuration Tools\n\nSome services have their own interactive configuration tools --\n`mysql_secure_installation`, `dpkg-reconfigure`, `certbot`,\ndatabase migration wizards. These can't be driven by declarative\ntools, but an agent can handle them:\n\n create session \"certbot-{host}\" on server \"{host}\"\n send to \"certbot-{host}\": sudo certbot --nginx\\n\n wait for idle, read screen\n\n # Certbot asks a series of questions\n if \"Enter email address\" in screen:\n send: admin@example.com\\n\n wait for idle, read screen\n\n if \"Terms of Service\" in screen:\n send: A\\n # Agree\n wait for idle, read screen\n\n if \"Which names would you like to activate HTTPS for\" in screen:\n send: 1\\n # Select the first domain\n wait for idle, read screen\n\n # Verify the certificate was issued\n send: sudo certbot certificates\\n\n wait for idle, read screen\n verify certificate is present and valid\n\n kill session \"certbot-{host}\"\n\n## Package Management at Scale\n\nPackage upgrades are one of the most common infrastructure tasks,\nand one of the most prone to interactive surprises.\n\n### The Upgrade Pattern\n\n hosts = [\"web-1\", \"web-2\", \"web-3\", \"db-1\", \"db-2\"]\n\n for host in hosts:\n create session \"upgrade-{host}\" on server \"{host}\"\n\n # Update package lists\n send to \"upgrade-{host}\": sudo apt update\\n\n wait for idle on \"upgrade-{host}\" (timeout 30000ms)\n read screen\n\n if password prompt:\n send: {sudo_password}\\n\n wait for idle, read screen\n\n # Check what would be upgraded (dry run)\n send: apt list --upgradable\\n\n wait for idle, read screen\n record pending_upgrades[host]\n\n # Review: show the operator what each host needs\n report pending upgrades per host\n # At this point, decide whether to proceed\n\n # Execute upgrades\n for host in hosts:\n send to \"upgrade-{host}\": sudo apt upgrade -y\\n\n wait for idle (timeout 60000ms)\n read screen\n\n # Handle the prompts that -y doesn't suppress\n loop until shell prompt returns:\n if \"Configuration file\" in screen:\n # Package wants to overwrite a config file\n # Decide based on whether we maintain custom configs\n send: N\\n # keep current version\n wait for idle, read screen\n\n if \"Daemons using outdated libraries\" in screen:\n send: \\n # accept defaults\n wait for idle, read screen\n\n if \"restart services\" in screen:\n send: y\\n\n wait for idle, read screen\n\n if shell prompt detected:\n break\n\n wait for idle (timeout 5000ms)\n read screen\n\n # Verify\n send: apt list --upgradable\\n\n wait for idle, read screen\n if packages still pending:\n results[host] = \"partial upgrade\"\n else:\n results[host] = \"fully upgraded\"\n\n kill session \"upgrade-{host}\"\n\n### Handling Package Manager Differences\n\nDifferent hosts may use different package managers. Detect and\nadapt:\n\n send to \"detect-{host}\": which apt yum dnf apk 2>\u002Fdev\u002Fnull\\n\n wait for idle, read screen\n\n if \"apt\" in screen:\n update_cmd = \"sudo apt update && sudo apt upgrade -y\"\n elif \"dnf\" in screen:\n update_cmd = \"sudo dnf upgrade -y\"\n elif \"yum\" in screen:\n update_cmd = \"sudo yum update -y\"\n elif \"apk\" in screen:\n update_cmd = \"sudo apk update && sudo apk upgrade\"\n\n send to \"upgrade-{host}\": {update_cmd}\\n\n\n### Specific Package Installation\n\nWhen you need to install a specific package with version pinning:\n\n for host in hosts:\n create session \"install-{host}\" on server \"{host}\", tagged: install\n send to \"install-{host}\": sudo apt install nginx=1.24.0-1~jammy\\n\n wait for idle, read screen\n\n if \"Do you want to continue?\" in screen:\n send: Y\\n\n wait for idle, read screen\n\n if \"unable to locate package\" in screen or \"has no installation candidate\" in screen:\n results[host] = \"package not available\"\n elif \"is already the newest version\" in screen:\n results[host] = \"already installed\"\n else:\n # Verify installation\n send: nginx -v\\n\n wait for idle, read screen\n if \"1.24.0\" in screen:\n results[host] = \"installed\"\n else:\n results[host] = \"unexpected version\"\n\n kill session \"install-{host}\"\n\n## Health Verification\n\nPost-operation checks are what separate a careful deployment from\na reckless one. Always verify after making changes.\n\n### Service Health Check\n\n create session \"health-{host}\" on server \"{host}\"\n\n # Check the service is running\n send to \"health-{host}\": systemctl is-active myapp\\n\n wait for idle, read screen\n service_running = \"active\" in screen\n\n # Check the endpoint responds\n send: curl -sf -o \u002Fdev\u002Fnull -w '%{http_code}' http:\u002F\u002Flocalhost:8080\u002Fhealth\\n\n wait for idle, read screen\n endpoint_healthy = \"200\" in screen\n\n # Check recent logs for errors\n send: journalctl -u myapp --since '5 minutes ago' --no-pager | grep -i error | tail -5\\n\n wait for idle, read screen\n recent_errors = screen contains error lines\n\n # Check resource usage\n send: systemctl show myapp --property=MemoryCurrent,CPUUsageNSec\\n\n wait for idle, read screen\n record resource metrics\n\n kill session \"health-{host}\"\n\n health[host] = {\n running: service_running,\n endpoint: endpoint_healthy,\n errors: recent_errors,\n resources: metrics\n }\n\n### Deep Health Check\n\nFor critical deployments, go beyond surface-level checks:\n\n # Check the application version matches what we deployed\n send: curl -sf http:\u002F\u002Flocalhost:8080\u002Fversion\\n\n wait for idle, read screen\n verify version matches expected\n\n # Check database connectivity\n send: curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth\u002Fdb\\n\n wait for idle, read screen\n\n # Check dependent services\n send: curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth\u002Fredis\\n\n wait for idle, read screen\n\n # Run a smoke test\n send: curl -sf -X POST http:\u002F\u002Flocalhost:8080\u002Fapi\u002Ftest -d '{\"ping\":\"pong\"}'\\n\n wait for idle, read screen\n verify expected response\n\n### Fleet-Wide Health Dashboard\n\nCollect health from all hosts and present a summary:\n\n health_report = {}\n\n for host in hosts:\n create session \"health-{host}\" on server \"{host}\", tagged: health\n send to \"health-{host}\": curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth && echo \"OK\" || echo \"FAIL\"\\n\n\n for host in hosts:\n wait for idle on \"health-{host}\"\n read screen from \"health-{host}\"\n health_report[host] = \"OK\" if \"OK\" in screen else \"FAIL\"\n kill session \"health-{host}\"\n\n healthy_count = count of \"OK\" in health_report\n report: \"{healthy_count}\u002F{total} hosts healthy\"\n if any \"FAIL\":\n report: \"Unhealthy hosts: {list of failed hosts}\"\n\n## Rollback\n\nWhen something fails mid-fleet, you need a plan. The rollback\nstrategy depends on what went wrong and how far you got.\n\n### Pre-Deployment Rollback Preparation\n\nBefore deploying, ensure you can roll back:\n\n for host in deploy_hosts:\n create session \"prep-{host}\" on server \"{host}\"\n\n # Record current version\n send to \"prep-{host}\": readlink \u002Fopt\u002Fmyapp\u002Fcurrent\\n\n wait for idle, read screen\n previous_version[host] = extract version from screen\n\n # Ensure previous release artifacts exist\n send: ls \u002Fopt\u002Fmyapp\u002Freleases\u002F\\n\n wait for idle, read screen\n verify previous version directory exists\n\n kill session \"prep-{host}\"\n\n### Rollback After Failed Deployment\n\nWhen a rolling deploy fails partway through:\n\n # failed_hosts: hosts where deployment failed\n # deployed_hosts: hosts where deployment succeeded\n # remaining_hosts: hosts not yet touched\n\n # Step 1: Fix the failed hosts\n for host in failed_hosts:\n create session \"rollback-{host}\" on server \"{host}\"\n send to \"rollback-{host}\": cd \u002Fopt\u002Fmyapp && .\u002Frollback.sh {previous_version[host]}\\n\n wait for idle, read screen, handle prompts\n verify service is healthy\n kill session \"rollback-{host}\"\n\n # Step 2: Decide about successfully deployed hosts\n # Option A: Roll them back too (full rollback)\n for host in deployed_hosts:\n create session \"rollback-{host}\" on server \"{host}\"\n send to \"rollback-{host}\": cd \u002Fopt\u002Fmyapp && .\u002Frollback.sh {previous_version[host]}\\n\n wait for idle, read screen, handle prompts\n verify service is healthy\n kill session \"rollback-{host}\"\n\n # Option B: Leave them on the new version (partial deploy)\n # Only do this if the new version is working correctly on\n # those hosts and the old\u002Fnew versions are compatible\n\n # Step 3: Remaining hosts were never touched -- no action needed\n\n # Step 4: Verify fleet health\n run fleet-wide health check (see Health Verification)\n\n### Rollback Decision Framework\n\nNot every failure requires a full rollback. Consider:\n\n- **One host failed, rest succeeded**: Investigate the failed\n host. It may be a host-specific issue (disk full, different\n OS version). Fix the host and retry, rather than rolling back\n the entire fleet.\n- **Health checks fail after deploy**: Roll back the current\n batch. Don't proceed to the next batch. Investigate before\n deciding whether to roll back hosts that are already on the\n new version.\n- **Multiple hosts fail the same way**: Likely a problem with\n the release itself. Full rollback, fix the release, try again.\n- **Intermittent failures**: Some hosts fail health checks\n sometimes. This might be a pre-existing issue, not caused by\n the deploy. Compare with pre-deployment health baselines.\n\n## Pitfalls\n\n### Don't Parallelize Destructive Operations Blindly\n\nRunning `apt upgrade` on 50 hosts simultaneously is tempting\nbut dangerous. If the upgrade has an unexpected prompt or failure\nmode, you'll have 50 hosts in an unknown state. Start sequential,\nmove to small batches once you've seen the operation succeed on\nthe first few hosts.\n\n### Handle Prompt Diversity\n\nThe same command produces different prompts on different hosts\ndepending on installed packages, OS version, configuration, and\nstate. Don't assume that because host 1 had no prompts, host 2\nwon't either. Always read the screen and handle what's actually\nthere.\n\n### Clean Up Sessions Aggressively\n\nInfrastructure operations can create dozens of sessions --\ndeploy, health check, rollback, audit sessions for each host.\nKill sessions as soon as they've served their purpose. A session\nis a running process on the backend machine. Leaving 50 orphaned\nsessions across your fleet wastes resources and clutters the\nsession list.\n\n### Don't Forget sudo\n\nMany infrastructure operations require elevated privileges.\nHandle `sudo` password prompts consistently. If you're working\nacross many hosts, consider whether `NOPASSWD` rules for\nspecific commands are appropriate to avoid interactive password\nentry on every host.\n\n### Record What You Do\n\nInfrastructure changes are auditable events. For each operation,\nrecord:\n- What was done on which hosts\n- What the state was before and after\n- Which hosts succeeded and which failed\n- What prompts were encountered and how they were answered\n\nRead scrollback from each session before killing it. This is\nyour audit trail.\n\n### Know When Ansible Is Better\n\nwsh + AI is powerful for interactive, judgment-heavy operations.\nBut Ansible is better for:\n- Fully predictable, idempotent operations at scale\n- Operations that need to be reproduced exactly the same way\n every time (compliance, auditing)\n- Simple fan-out where no per-host judgment is needed\n- Infrastructure-as-code workflows where the desired state\n should be version-controlled\n\nUse wsh for the 20% that requires interaction and judgment.\nUse Ansible for the 80% that doesn't. They complement each\nother.\n\n### Test Your Rollback Before You Need It\n\nThe worst time to discover your rollback procedure doesn't\nwork is during a failed deployment at 2am. Test rollback on\na staging fleet first. Verify that the previous version's\nartifacts exist, that the rollback script works, and that\nthe health checks pass after rolling back.\n\n### Mind the Blast Radius\n\nIf you're deploying across 100 hosts, don't set a batch size\nof 50. A failed batch of 50 means half your fleet is down.\nStart small. Increase batch size only after the first few\nbatches succeed. A common progression: 1, 2, 5, 10, 25, 50.\n",{"data":30,"body":31},{"name":4,"description":6},{"type":32,"children":33},"root",[34,186,193,198,203,208,215,223,252,260,297,303,308,320,325,334,346,365,371,383,390,395,404,409,415,420,429,435,448,490,495,504,509,515,520,526,535,541,574,580,585,594,600,605,611,616,625,631,636,645,651,679,688,694,699,705,714,720,725,734,740,745,754,760,765,771,780,786,791,800,806,811,820,826,831,837,842,851,857,862,871,877,882,925,931,937,949,955,960,966,971,977,998,1004,1009,1032,1037,1043,1048,1071,1076,1082,1087,1093],{"type":35,"tag":36,"props":37,"children":38},"element","blockquote",{},[39,67],{"type":35,"tag":40,"props":41,"children":42},"p",{},[43,50,52,58,60,65],{"type":35,"tag":44,"props":45,"children":46},"strong",{},[47],{"type":48,"value":49},"text","IMPORTANT: EXECUTION CONTEXT",{"type":48,"value":51},"\nThis skill describes ",{"type":35,"tag":53,"props":54,"children":55},"em",{},[56],{"type":48,"value":57},"what to do",{"type":48,"value":59}," — domain patterns and decision-making.\nIt does NOT describe ",{"type":35,"tag":53,"props":61,"children":62},{},[63],{"type":48,"value":64},"how",{"type":48,"value":66}," to call the API.",{"type":35,"tag":68,"props":69,"children":70},"ol",{},[71,121,160],{"type":35,"tag":72,"props":73,"children":74},"li",{},[75,89,91,97,99,105,107,112,114,119],{"type":35,"tag":44,"props":76,"children":77},{},[78,80,87],{"type":48,"value":79},"If you have ",{"type":35,"tag":81,"props":82,"children":84},"code",{"className":83},[],[85],{"type":48,"value":86},"wsh_*",{"type":48,"value":88}," tools",{"type":48,"value":90}," (check your toolkit for ",{"type":35,"tag":81,"props":92,"children":94},{"className":93},[],[95],{"type":48,"value":96},"wsh_send_input",{"type":48,"value":98},",\n",{"type":35,"tag":81,"props":100,"children":102},{"className":101},[],[103],{"type":48,"value":104},"wsh_get_screen",{"type":48,"value":106},", etc.): use them directly. Operation names in this\nskill generally map to tool names (e.g., \"send input\" → ",{"type":35,"tag":81,"props":108,"children":110},{"className":109},[],[111],{"type":48,"value":96},{"type":48,"value":113},").\nWhen in doubt, list your available ",{"type":35,"tag":81,"props":115,"children":117},{"className":116},[],[118],{"type":48,"value":86},{"type":48,"value":120}," tools.",{"type":35,"tag":72,"props":122,"children":123},{},[124,135,137,142,144,150,152,158],{"type":35,"tag":44,"props":125,"children":126},{},[127,129,134],{"type":48,"value":128},"If you do NOT have ",{"type":35,"tag":81,"props":130,"children":132},{"className":131},[],[133],{"type":48,"value":86},{"type":48,"value":88},{"type":48,"value":136},": you are in HTTP\u002Fcurl fallback mode.\n",{"type":35,"tag":44,"props":138,"children":139},{},[140],{"type":48,"value":141},"DO NOT GUESS endpoints or CLI subcommands.",{"type":48,"value":143},"\nLoad the full API reference first: search your workspace for\n",{"type":35,"tag":81,"props":145,"children":147},{"className":146},[],[148],{"type":48,"value":149},"skills\u002Fcore\u002F",{"type":48,"value":151}," and read ",{"type":35,"tag":81,"props":153,"children":155},{"className":154},[],[156],{"type":48,"value":157},"SKILL.md",{"type":48,"value":159},". It contains every endpoint\nwith working curl examples and a bootstrap sequence.",{"type":35,"tag":72,"props":161,"children":162},{},[163,168,170,176,178,184],{"type":35,"tag":44,"props":164,"children":165},{},[166],{"type":48,"value":167},"Quick bootstrap",{"type":48,"value":169},": ",{"type":35,"tag":81,"props":171,"children":173},{"className":172},[],[174],{"type":48,"value":175},"curl -sf --unix-socket ${XDG_RUNTIME_DIR:-\u002Ftmp}\u002Fwsh\u002Fdefault.http.sock http:\u002F\u002Flocalhost\u002Fhealth",{"type":48,"value":177},"\n— if that fails: ",{"type":35,"tag":81,"props":179,"children":181},{"className":180},[],[182],{"type":48,"value":183},"wsh server -L agent-$$ --ephemeral &",{"type":48,"value":185}," and retry.",{"type":35,"tag":187,"props":188,"children":190},"h1",{"id":189},"wshinfrastructure-ops-fleet-management",[191],{"type":48,"value":192},"wsh:infrastructure-ops -- Fleet Management",{"type":35,"tag":40,"props":194,"children":195},{},[196],{"type":48,"value":197},"Ansible, Puppet, and Chef exist because you can't interactively\noperate 50 machines at once. They solve this with declarative\nconfigs and idempotent operations -- you describe the desired state,\nthe tool converges toward it. This works brilliantly for the 80%\nof infrastructure work that's predictable and repeatable.",{"type":35,"tag":40,"props":199,"children":200},{},[201],{"type":48,"value":202},"But 20% of real infrastructure work isn't predictable. Interactive\ninstallers that ask questions nobody anticipated. Package upgrades\nthat present merge conflicts in config files. Services that fail\nin ways that require investigation, not just a restart. Approval\nprompts that need human judgment. Diagnostics that require poking\naround, reading logs, trying things.",{"type":35,"tag":40,"props":204,"children":205},{},[206],{"type":48,"value":207},"wsh changes the equation. An AI agent can sit at every terminal\nsimultaneously -- reading screens, handling prompts, making\ndecisions. This enables an imperative, interactive model for the\nwork that declarative tools can't handle. Not a replacement for\nAnsible. A complement for the cases where Ansible isn't enough.",{"type":35,"tag":209,"props":210,"children":212},"h2",{"id":211},"when-to-use-this",[213],{"type":48,"value":214},"When to Use This",{"type":35,"tag":40,"props":216,"children":217},{},[218],{"type":35,"tag":44,"props":219,"children":220},{},[221],{"type":48,"value":222},"Use infrastructure-ops when:",{"type":35,"tag":224,"props":225,"children":226},"ul",{},[227,232,237,242,247],{"type":35,"tag":72,"props":228,"children":229},{},[230],{"type":48,"value":231},"Operations require interactive input -- installers, prompts,\napproval dialogs, merge conflict resolution",{"type":35,"tag":72,"props":233,"children":234},{},[235],{"type":48,"value":236},"Each host may behave differently and needs per-host judgment",{"type":35,"tag":72,"props":238,"children":239},{},[240],{"type":48,"value":241},"You need to verify results interactively between steps\n(read logs, check endpoints, inspect state)",{"type":35,"tag":72,"props":243,"children":244},{},[245],{"type":48,"value":246},"The operation is exploratory -- diagnosing an issue across\na fleet, where what you do on host N depends on what you\nfound on hosts 1 through N-1",{"type":35,"tag":72,"props":248,"children":249},{},[250],{"type":48,"value":251},"Rolling deployments need health verification with real\ntraffic before proceeding to the next batch",{"type":35,"tag":40,"props":253,"children":254},{},[255],{"type":35,"tag":44,"props":256,"children":257},{},[258],{"type":48,"value":259},"Don't use infrastructure-ops when:",{"type":35,"tag":224,"props":261,"children":262},{},[263,268,273,292],{"type":35,"tag":72,"props":264,"children":265},{},[266],{"type":48,"value":267},"The operation is fully predictable and idempotent -- use\nAnsible, Terraform, or your existing config management",{"type":35,"tag":72,"props":269,"children":270},{},[271],{"type":48,"value":272},"You're provisioning infrastructure from scratch -- use\nTerraform or CloudFormation",{"type":35,"tag":72,"props":274,"children":275},{},[276,278,284,286],{"type":48,"value":277},"The operation is a single command with no interaction --\nuse ",{"type":35,"tag":81,"props":279,"children":281},{"className":280},[],[282],{"type":48,"value":283},"parallel-ssh",{"type":48,"value":285}," or ",{"type":35,"tag":81,"props":287,"children":289},{"className":288},[],[290],{"type":48,"value":291},"ansible -m shell",{"type":35,"tag":72,"props":293,"children":294},{},[295],{"type":48,"value":296},"You don't need per-host decision-making -- fan out with\na simple script instead",{"type":35,"tag":209,"props":298,"children":300},{"id":299},"prerequisites",[301],{"type":48,"value":302},"Prerequisites",{"type":35,"tag":40,"props":304,"children":305},{},[306],{"type":48,"value":307},"You need a federated wsh cluster: a hub server with backends\nregistered for each target machine. Each backend is a wsh server\nrunning on the target host.",{"type":35,"tag":309,"props":310,"children":314},"pre",{"className":311,"code":313,"language":48},[312],"language-text","list servers\n# Expect: hub (local), plus one backend per target host\n",[315],{"type":35,"tag":81,"props":316,"children":318},{"__ignoreMap":317},"",[319],{"type":48,"value":313},{"type":35,"tag":40,"props":321,"children":322},{},[323],{"type":48,"value":324},"If backends aren't registered yet, add them:",{"type":35,"tag":309,"props":326,"children":329},{"className":327,"code":328,"language":48},[312],"add server at address http:\u002F\u002F10.0.1.10:8080\nadd server at address http:\u002F\u002F10.0.1.11:8080\nadd server at address http:\u002F\u002F10.0.1.12:8080\n\n# Wait for all to become healthy\nloop:\n list servers\n if all target servers are healthy: break\n wait briefly\n retry\n",[330],{"type":35,"tag":81,"props":331,"children":332},{"__ignoreMap":317},[333],{"type":48,"value":328},{"type":35,"tag":40,"props":335,"children":336},{},[337,339,344],{"type":48,"value":338},"See the ",{"type":35,"tag":44,"props":340,"children":341},{},[342],{"type":48,"value":343},"wsh:cluster-orchestration",{"type":48,"value":345}," skill for details on server\nregistration, health monitoring, and authentication.",{"type":35,"tag":40,"props":347,"children":348},{},[349,351,356,358,363],{"type":48,"value":350},"You also need the patterns from ",{"type":35,"tag":44,"props":352,"children":353},{},[354],{"type":48,"value":355},"wsh:drive-process",{"type":48,"value":357}," (the\nsend\u002Fwait\u002Fread loop) and ",{"type":35,"tag":44,"props":359,"children":360},{},[361],{"type":48,"value":362},"wsh:multi-session",{"type":48,"value":364}," (parallel session\nmanagement, tagging, fan-out). This skill composes on top of both.",{"type":35,"tag":209,"props":366,"children":368},{"id":367},"core-pattern-same-operation-across-n-hosts",[369],{"type":48,"value":370},"Core Pattern: Same Operation Across N Hosts",{"type":35,"tag":40,"props":372,"children":373},{},[374,376,381],{"type":48,"value":375},"The fundamental infrastructure pattern: run the same operation on\nevery host, but handle each host's response individually. This is\ndifferent from ",{"type":35,"tag":81,"props":377,"children":379},{"className":378},[],[380],{"type":48,"value":283},{"type":48,"value":382}," because you react to what each host\ndoes -- you don't just fire and forget.",{"type":35,"tag":384,"props":385,"children":387},"h3",{"id":386},"sequential-fan-out",[388],{"type":48,"value":389},"Sequential Fan-Out",{"type":35,"tag":40,"props":391,"children":392},{},[393],{"type":48,"value":394},"The safe default. Operate on one host at a time, verify success\nbefore moving to the next:",{"type":35,"tag":309,"props":396,"children":399},{"className":397,"code":398,"language":48},[312],"hosts = [\"web-1\", \"web-2\", \"web-3\", \"web-4\", \"web-5\"]\nresults = {}\n\nfor host in hosts:\n create session \"op-{host}\" on server \"{host}\"\n send to \"op-{host}\": sudo systemctl restart myapp\\n\n wait for idle on \"op-{host}\"\n read screen from \"op-{host}\"\n\n if password prompt detected:\n send to \"op-{host}\": {sudo_password}\\n\n wait for idle on \"op-{host}\"\n read screen from \"op-{host}\"\n\n if success:\n results[host] = \"ok\"\n else:\n results[host] = \"failed\"\n # Decide: continue to next host, or stop?\n\n kill session \"op-{host}\"\n\nreport results\n",[400],{"type":35,"tag":81,"props":401,"children":402},{"__ignoreMap":317},[403],{"type":48,"value":398},{"type":35,"tag":40,"props":405,"children":406},{},[407],{"type":48,"value":408},"Sequential is slower but safer. If the operation breaks the first\nhost, you haven't touched the other four.",{"type":35,"tag":384,"props":410,"children":412},{"id":411},"parallel-fan-out",[413],{"type":48,"value":414},"Parallel Fan-Out",{"type":35,"tag":40,"props":416,"children":417},{},[418],{"type":48,"value":419},"When the operation is safe to run everywhere simultaneously --\nread-only commands, status checks, log collection:",{"type":35,"tag":309,"props":421,"children":424},{"className":422,"code":423,"language":48},[312],"hosts = [\"web-1\", \"web-2\", \"web-3\", \"web-4\", \"web-5\"]\n\n# Launch all at once\nfor host in hosts:\n create session \"check-{host}\" on server \"{host}\", tagged: fleet-check\n send to \"check-{host}\": systemctl status myapp\\n\n\n# Poll for completion\ncompleted = {}\nwhile not all hosts completed:\n wait for idle on sessions tagged \"fleet-check\" (timeout 2000ms)\n # Returns whichever session settled first\n read screen from that session\n record result\n mark as completed\n # Use last_session + last_generation to advance\n\n# Clean up\nfor host in hosts:\n kill session \"check-{host}\"\n\nreport results\n",[425],{"type":35,"tag":81,"props":426,"children":427},{"__ignoreMap":317},[428],{"type":48,"value":423},{"type":35,"tag":384,"props":430,"children":432},{"id":431},"handling-per-host-prompt-diversity",[433],{"type":48,"value":434},"Handling Per-Host Prompt Diversity",{"type":35,"tag":40,"props":436,"children":437},{},[438,440,446],{"type":48,"value":439},"The real reason this pattern exists. When you run ",{"type":35,"tag":81,"props":441,"children":443},{"className":442},[],[444],{"type":48,"value":445},"apt upgrade",{"type":48,"value":447},"\nacross 10 servers, each one may present different prompts:",{"type":35,"tag":224,"props":449,"children":450},{},[451,456,469,480,485],{"type":35,"tag":72,"props":452,"children":453},{},[454],{"type":48,"value":455},"Server 1: no prompts, clean upgrade",{"type":35,"tag":72,"props":457,"children":458},{},[459,461,467],{"type":48,"value":460},"Server 2: \"Configuration file changed, keep or replace? ",{"type":35,"tag":462,"props":463,"children":464},"span",{},[465],{"type":48,"value":466},"Y\u002Fn",{"type":48,"value":468},"\"",{"type":35,"tag":72,"props":470,"children":471},{},[472,474,479],{"type":48,"value":473},"Server 3: \"Service nginx needs to be restarted. Restart now? ",{"type":35,"tag":462,"props":475,"children":476},{},[477],{"type":48,"value":478},"y\u002FN",{"type":48,"value":468},{"type":35,"tag":72,"props":481,"children":482},{},[483],{"type":48,"value":484},"Server 4: \"Pending kernel upgrade. Reboot required.\"",{"type":35,"tag":72,"props":486,"children":487},{},[488],{"type":48,"value":489},"Server 5: dependency conflict requiring manual resolution",{"type":35,"tag":40,"props":491,"children":492},{},[493],{"type":48,"value":494},"A declarative tool must handle all these cases upfront with\nconfiguration flags. An agent reads each screen and responds\nwith judgment:",{"type":35,"tag":309,"props":496,"children":499},{"className":497,"code":498,"language":48},[312],"read screen from \"upgrade-{host}\"\n\nif \"keep or replace\" in screen:\n # Inspect the diff, decide based on whether we've\n # customized this config\n send: d\\n # show diff first\n wait, read\n if config has local customizations:\n send: N\\n # keep ours\n else:\n send: Y\\n # take the new version\n\nif \"restart now\" in screen:\n # Check if this is a load-balanced service that can restart\n send: y\\n\n\nif \"reboot required\" in screen:\n # Note for later -- don't reboot mid-upgrade-sweep\n results[host].needs_reboot = true\n",[500],{"type":35,"tag":81,"props":501,"children":502},{"__ignoreMap":317},[503],{"type":48,"value":498},{"type":35,"tag":40,"props":505,"children":506},{},[507],{"type":48,"value":508},"This per-host judgment is the core value proposition. The agent\nadapts to what each host actually presents.",{"type":35,"tag":209,"props":510,"children":512},{"id":511},"rolling-deployments",[513],{"type":48,"value":514},"Rolling Deployments",{"type":35,"tag":40,"props":516,"children":517},{},[518],{"type":48,"value":519},"Deploy to a fleet incrementally, verifying health between each\nbatch. If something goes wrong, stop before it affects the whole\nfleet.",{"type":35,"tag":384,"props":521,"children":523},{"id":522},"the-rolling-deploy-pattern",[524],{"type":48,"value":525},"The Rolling Deploy Pattern",{"type":35,"tag":309,"props":527,"children":530},{"className":528,"code":529,"language":48},[312],"hosts = [\"web-1\", \"web-2\", \"web-3\", \"web-4\", \"web-5\", \"web-6\"]\nbatch_size = 2\nfailed = []\n\nfor batch in chunk(hosts, batch_size):\n # Deploy this batch in parallel\n for host in batch:\n create session \"deploy-{host}\" on server \"{host}\", tagged: deploy\n send to \"deploy-{host}\": cd \u002Fopt\u002Fmyapp && .\u002Fdeploy.sh v2.1.0\\n\n\n # Wait for all in this batch to finish\n for host in batch:\n wait for idle on \"deploy-{host}\" (timeout 30000ms)\n read screen from \"deploy-{host}\"\n\n if deployment prompt detected:\n handle prompt (see per-host prompt handling)\n\n wait for idle on \"deploy-{host}\"\n read screen from \"deploy-{host}\"\n\n if deployment failed:\n failed.append(host)\n\n # Health check this batch before proceeding\n for host in batch:\n create session \"health-{host}\" on server \"{host}\", tagged: health\n send to \"health-{host}\": curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth\\n\n wait for idle on \"health-{host}\"\n read screen from \"health-{host}\"\n\n if health check failed:\n failed.append(host)\n\n kill session \"health-{host}\"\n\n # Clean up deploy sessions for this batch\n for host in batch:\n kill session \"deploy-{host}\"\n\n # Gate: stop if this batch had failures\n if failed is not empty:\n report: \"Stopping rolling deploy. Failed hosts: {failed}\"\n report: \"Remaining hosts not deployed: {remaining}\"\n break\n\n # Optional: wait for the new version to soak\n wait 30 seconds\n # Re-check health after soak period\n for host in batch:\n create session \"soak-{host}\" on server \"{host}\"\n send to \"soak-{host}\": curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth\\n\n wait for idle, read screen\n if unhealthy:\n failed.append(host)\n kill session \"soak-{host}\"\n\n if failed is not empty:\n report: \"Soak check failed. Stopping.\"\n break\n\nif failed is empty:\n report: \"Rolling deploy complete. All hosts healthy.\"\n",[531],{"type":35,"tag":81,"props":532,"children":533},{"__ignoreMap":317},[534],{"type":48,"value":529},{"type":35,"tag":384,"props":536,"children":538},{"id":537},"choosing-batch-size",[539],{"type":48,"value":540},"Choosing Batch Size",{"type":35,"tag":224,"props":542,"children":543},{},[544,554,564],{"type":35,"tag":72,"props":545,"children":546},{},[547,552],{"type":35,"tag":44,"props":548,"children":549},{},[550],{"type":48,"value":551},"Batch of 1",{"type":48,"value":553},": Safest. Every host is individually verified.\nSlowest. Use for the first deploy of a risky change.",{"type":35,"tag":72,"props":555,"children":556},{},[557,562],{"type":35,"tag":44,"props":558,"children":559},{},[560],{"type":48,"value":561},"Batch of N\u002F3",{"type":48,"value":563},": Good balance. You always have 2\u002F3 of the\nfleet on the old version if something goes wrong.",{"type":35,"tag":72,"props":565,"children":566},{},[567,572],{"type":35,"tag":44,"props":568,"children":569},{},[570],{"type":48,"value":571},"Batch of N-1",{"type":48,"value":573},": Deploy to all but one (the canary). If the\ncanary is still fine on the old version after you've deployed\neverywhere else, you know the issue is with the new version.",{"type":35,"tag":384,"props":575,"children":577},{"id":576},"canary-pattern",[578],{"type":48,"value":579},"Canary Pattern",{"type":35,"tag":40,"props":581,"children":582},{},[583],{"type":48,"value":584},"A special case of rolling deployment: deploy to one host first,\nsoak for longer, then proceed if healthy:",{"type":35,"tag":309,"props":586,"children":589},{"className":587,"code":588,"language":48},[312],"canary = hosts[0]\nrest = hosts[1:]\n\n# Deploy to canary\ncreate session \"deploy-canary\" on server \"{canary}\"\nsend to \"deploy-canary\": .\u002Fdeploy.sh v2.1.0\\n\nwait for idle, read screen, handle prompts\nkill session \"deploy-canary\"\n\n# Extended health check on canary\nfor i in range(5):\n wait 60 seconds\n create session \"canary-check\" on server \"{canary}\"\n send to \"canary-check\": curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth && echo \"HEALTHY\" || echo \"UNHEALTHY\"\\n\n wait for idle, read screen\n if \"UNHEALTHY\" in screen:\n report: \"Canary failed health check. Aborting.\"\n initiate rollback on canary\n stop\n kill session \"canary-check\"\n\n# Canary is healthy after 5 minutes. Deploy the rest.\nproceed with rolling deploy on rest\n",[590],{"type":35,"tag":81,"props":591,"children":592},{"__ignoreMap":317},[593],{"type":48,"value":588},{"type":35,"tag":209,"props":595,"children":597},{"id":596},"configuration-management",[598],{"type":48,"value":599},"Configuration Management",{"type":35,"tag":40,"props":601,"children":602},{},[603],{"type":48,"value":604},"Inspect current configuration, apply changes, and verify the\nresult -- interactively, with the ability to inspect and judge\nat each step.",{"type":35,"tag":384,"props":606,"children":608},{"id":607},"inspect-modify-verify",[609],{"type":48,"value":610},"Inspect-Modify-Verify",{"type":35,"tag":40,"props":612,"children":613},{},[614],{"type":48,"value":615},"The fundamental configuration pattern:",{"type":35,"tag":309,"props":617,"children":620},{"className":618,"code":619,"language":48},[312],"for host in hosts:\n create session \"config-{host}\" on server \"{host}\"\n\n # 1. Inspect current state\n send to \"config-{host}\": cat \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n wait for idle, read screen\n record current_config[host]\n\n # 2. Decide whether to modify\n if config needs updating:\n # Back up first\n send: cp \u002Fetc\u002Fmyapp\u002Fconfig.yaml \u002Fetc\u002Fmyapp\u002Fconfig.yaml.bak\\n\n wait for idle\n\n # Apply the change (using sed, patch, or writing a new file)\n send: sed -i 's\u002Fmax_connections: 100\u002Fmax_connections: 200\u002F' \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n wait for idle\n\n # 3. Verify the change\n send: cat \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n wait for idle, read screen\n if change is correct:\n results[host] = \"updated\"\n else:\n # Restore backup\n send: cp \u002Fetc\u002Fmyapp\u002Fconfig.yaml.bak \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n wait for idle\n results[host] = \"reverted -- change didn't apply correctly\"\n else:\n results[host] = \"no change needed\"\n\n kill session \"config-{host}\"\n",[621],{"type":35,"tag":81,"props":622,"children":623},{"__ignoreMap":317},[624],{"type":48,"value":619},{"type":35,"tag":384,"props":626,"children":628},{"id":627},"configuration-drift-detection",[629],{"type":48,"value":630},"Configuration Drift Detection",{"type":35,"tag":40,"props":632,"children":633},{},[634],{"type":48,"value":635},"Compare configurations across the fleet to find hosts that\nhave drifted from the expected state:",{"type":35,"tag":309,"props":637,"children":640},{"className":638,"code":639,"language":48},[312],"expected_config = \"...\" # known-good configuration\ndrifted = []\n\nfor host in hosts:\n create session \"audit-{host}\" on server \"{host}\", tagged: audit\n send to \"audit-{host}\": cat \u002Fetc\u002Fmyapp\u002Fconfig.yaml\\n\n\nfor host in hosts:\n wait for idle on \"audit-{host}\"\n read screen from \"audit-{host}\"\n if screen does not match expected_config:\n drifted.append(host)\n # Optionally: read scrollback for the full config if it's long\n kill session \"audit-{host}\"\n\nif drifted:\n report: \"Configuration drift detected on: {drifted}\"\n # Inspect each drifted host for details\n",[641],{"type":35,"tag":81,"props":642,"children":643},{"__ignoreMap":317},[644],{"type":48,"value":639},{"type":35,"tag":384,"props":646,"children":648},{"id":647},"interactive-configuration-tools",[649],{"type":48,"value":650},"Interactive Configuration Tools",{"type":35,"tag":40,"props":652,"children":653},{},[654,656,662,664,670,671,677],{"type":48,"value":655},"Some services have their own interactive configuration tools --\n",{"type":35,"tag":81,"props":657,"children":659},{"className":658},[],[660],{"type":48,"value":661},"mysql_secure_installation",{"type":48,"value":663},", ",{"type":35,"tag":81,"props":665,"children":667},{"className":666},[],[668],{"type":48,"value":669},"dpkg-reconfigure",{"type":48,"value":663},{"type":35,"tag":81,"props":672,"children":674},{"className":673},[],[675],{"type":48,"value":676},"certbot",{"type":48,"value":678},",\ndatabase migration wizards. These can't be driven by declarative\ntools, but an agent can handle them:",{"type":35,"tag":309,"props":680,"children":683},{"className":681,"code":682,"language":48},[312],"create session \"certbot-{host}\" on server \"{host}\"\nsend to \"certbot-{host}\": sudo certbot --nginx\\n\nwait for idle, read screen\n\n# Certbot asks a series of questions\nif \"Enter email address\" in screen:\n send: admin@example.com\\n\n wait for idle, read screen\n\nif \"Terms of Service\" in screen:\n send: A\\n # Agree\n wait for idle, read screen\n\nif \"Which names would you like to activate HTTPS for\" in screen:\n send: 1\\n # Select the first domain\n wait for idle, read screen\n\n# Verify the certificate was issued\nsend: sudo certbot certificates\\n\nwait for idle, read screen\nverify certificate is present and valid\n\nkill session \"certbot-{host}\"\n",[684],{"type":35,"tag":81,"props":685,"children":686},{"__ignoreMap":317},[687],{"type":48,"value":682},{"type":35,"tag":209,"props":689,"children":691},{"id":690},"package-management-at-scale",[692],{"type":48,"value":693},"Package Management at Scale",{"type":35,"tag":40,"props":695,"children":696},{},[697],{"type":48,"value":698},"Package upgrades are one of the most common infrastructure tasks,\nand one of the most prone to interactive surprises.",{"type":35,"tag":384,"props":700,"children":702},{"id":701},"the-upgrade-pattern",[703],{"type":48,"value":704},"The Upgrade Pattern",{"type":35,"tag":309,"props":706,"children":709},{"className":707,"code":708,"language":48},[312],"hosts = [\"web-1\", \"web-2\", \"web-3\", \"db-1\", \"db-2\"]\n\nfor host in hosts:\n create session \"upgrade-{host}\" on server \"{host}\"\n\n # Update package lists\n send to \"upgrade-{host}\": sudo apt update\\n\n wait for idle on \"upgrade-{host}\" (timeout 30000ms)\n read screen\n\n if password prompt:\n send: {sudo_password}\\n\n wait for idle, read screen\n\n # Check what would be upgraded (dry run)\n send: apt list --upgradable\\n\n wait for idle, read screen\n record pending_upgrades[host]\n\n# Review: show the operator what each host needs\nreport pending upgrades per host\n# At this point, decide whether to proceed\n\n# Execute upgrades\nfor host in hosts:\n send to \"upgrade-{host}\": sudo apt upgrade -y\\n\n wait for idle (timeout 60000ms)\n read screen\n\n # Handle the prompts that -y doesn't suppress\n loop until shell prompt returns:\n if \"Configuration file\" in screen:\n # Package wants to overwrite a config file\n # Decide based on whether we maintain custom configs\n send: N\\n # keep current version\n wait for idle, read screen\n\n if \"Daemons using outdated libraries\" in screen:\n send: \\n # accept defaults\n wait for idle, read screen\n\n if \"restart services\" in screen:\n send: y\\n\n wait for idle, read screen\n\n if shell prompt detected:\n break\n\n wait for idle (timeout 5000ms)\n read screen\n\n # Verify\n send: apt list --upgradable\\n\n wait for idle, read screen\n if packages still pending:\n results[host] = \"partial upgrade\"\n else:\n results[host] = \"fully upgraded\"\n\n kill session \"upgrade-{host}\"\n",[710],{"type":35,"tag":81,"props":711,"children":712},{"__ignoreMap":317},[713],{"type":48,"value":708},{"type":35,"tag":384,"props":715,"children":717},{"id":716},"handling-package-manager-differences",[718],{"type":48,"value":719},"Handling Package Manager Differences",{"type":35,"tag":40,"props":721,"children":722},{},[723],{"type":48,"value":724},"Different hosts may use different package managers. Detect and\nadapt:",{"type":35,"tag":309,"props":726,"children":729},{"className":727,"code":728,"language":48},[312],"send to \"detect-{host}\": which apt yum dnf apk 2>\u002Fdev\u002Fnull\\n\nwait for idle, read screen\n\nif \"apt\" in screen:\n update_cmd = \"sudo apt update && sudo apt upgrade -y\"\nelif \"dnf\" in screen:\n update_cmd = \"sudo dnf upgrade -y\"\nelif \"yum\" in screen:\n update_cmd = \"sudo yum update -y\"\nelif \"apk\" in screen:\n update_cmd = \"sudo apk update && sudo apk upgrade\"\n\nsend to \"upgrade-{host}\": {update_cmd}\\n\n",[730],{"type":35,"tag":81,"props":731,"children":732},{"__ignoreMap":317},[733],{"type":48,"value":728},{"type":35,"tag":384,"props":735,"children":737},{"id":736},"specific-package-installation",[738],{"type":48,"value":739},"Specific Package Installation",{"type":35,"tag":40,"props":741,"children":742},{},[743],{"type":48,"value":744},"When you need to install a specific package with version pinning:",{"type":35,"tag":309,"props":746,"children":749},{"className":747,"code":748,"language":48},[312],"for host in hosts:\n create session \"install-{host}\" on server \"{host}\", tagged: install\n send to \"install-{host}\": sudo apt install nginx=1.24.0-1~jammy\\n\n wait for idle, read screen\n\n if \"Do you want to continue?\" in screen:\n send: Y\\n\n wait for idle, read screen\n\n if \"unable to locate package\" in screen or \"has no installation candidate\" in screen:\n results[host] = \"package not available\"\n elif \"is already the newest version\" in screen:\n results[host] = \"already installed\"\n else:\n # Verify installation\n send: nginx -v\\n\n wait for idle, read screen\n if \"1.24.0\" in screen:\n results[host] = \"installed\"\n else:\n results[host] = \"unexpected version\"\n\n kill session \"install-{host}\"\n",[750],{"type":35,"tag":81,"props":751,"children":752},{"__ignoreMap":317},[753],{"type":48,"value":748},{"type":35,"tag":209,"props":755,"children":757},{"id":756},"health-verification",[758],{"type":48,"value":759},"Health Verification",{"type":35,"tag":40,"props":761,"children":762},{},[763],{"type":48,"value":764},"Post-operation checks are what separate a careful deployment from\na reckless one. Always verify after making changes.",{"type":35,"tag":384,"props":766,"children":768},{"id":767},"service-health-check",[769],{"type":48,"value":770},"Service Health Check",{"type":35,"tag":309,"props":772,"children":775},{"className":773,"code":774,"language":48},[312],"create session \"health-{host}\" on server \"{host}\"\n\n# Check the service is running\nsend to \"health-{host}\": systemctl is-active myapp\\n\nwait for idle, read screen\nservice_running = \"active\" in screen\n\n# Check the endpoint responds\nsend: curl -sf -o \u002Fdev\u002Fnull -w '%{http_code}' http:\u002F\u002Flocalhost:8080\u002Fhealth\\n\nwait for idle, read screen\nendpoint_healthy = \"200\" in screen\n\n# Check recent logs for errors\nsend: journalctl -u myapp --since '5 minutes ago' --no-pager | grep -i error | tail -5\\n\nwait for idle, read screen\nrecent_errors = screen contains error lines\n\n# Check resource usage\nsend: systemctl show myapp --property=MemoryCurrent,CPUUsageNSec\\n\nwait for idle, read screen\nrecord resource metrics\n\nkill session \"health-{host}\"\n\nhealth[host] = {\n running: service_running,\n endpoint: endpoint_healthy,\n errors: recent_errors,\n resources: metrics\n}\n",[776],{"type":35,"tag":81,"props":777,"children":778},{"__ignoreMap":317},[779],{"type":48,"value":774},{"type":35,"tag":384,"props":781,"children":783},{"id":782},"deep-health-check",[784],{"type":48,"value":785},"Deep Health Check",{"type":35,"tag":40,"props":787,"children":788},{},[789],{"type":48,"value":790},"For critical deployments, go beyond surface-level checks:",{"type":35,"tag":309,"props":792,"children":795},{"className":793,"code":794,"language":48},[312],"# Check the application version matches what we deployed\nsend: curl -sf http:\u002F\u002Flocalhost:8080\u002Fversion\\n\nwait for idle, read screen\nverify version matches expected\n\n# Check database connectivity\nsend: curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth\u002Fdb\\n\nwait for idle, read screen\n\n# Check dependent services\nsend: curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth\u002Fredis\\n\nwait for idle, read screen\n\n# Run a smoke test\nsend: curl -sf -X POST http:\u002F\u002Flocalhost:8080\u002Fapi\u002Ftest -d '{\"ping\":\"pong\"}'\\n\nwait for idle, read screen\nverify expected response\n",[796],{"type":35,"tag":81,"props":797,"children":798},{"__ignoreMap":317},[799],{"type":48,"value":794},{"type":35,"tag":384,"props":801,"children":803},{"id":802},"fleet-wide-health-dashboard",[804],{"type":48,"value":805},"Fleet-Wide Health Dashboard",{"type":35,"tag":40,"props":807,"children":808},{},[809],{"type":48,"value":810},"Collect health from all hosts and present a summary:",{"type":35,"tag":309,"props":812,"children":815},{"className":813,"code":814,"language":48},[312],"health_report = {}\n\nfor host in hosts:\n create session \"health-{host}\" on server \"{host}\", tagged: health\n send to \"health-{host}\": curl -sf http:\u002F\u002Flocalhost:8080\u002Fhealth && echo \"OK\" || echo \"FAIL\"\\n\n\nfor host in hosts:\n wait for idle on \"health-{host}\"\n read screen from \"health-{host}\"\n health_report[host] = \"OK\" if \"OK\" in screen else \"FAIL\"\n kill session \"health-{host}\"\n\nhealthy_count = count of \"OK\" in health_report\nreport: \"{healthy_count}\u002F{total} hosts healthy\"\nif any \"FAIL\":\n report: \"Unhealthy hosts: {list of failed hosts}\"\n",[816],{"type":35,"tag":81,"props":817,"children":818},{"__ignoreMap":317},[819],{"type":48,"value":814},{"type":35,"tag":209,"props":821,"children":823},{"id":822},"rollback",[824],{"type":48,"value":825},"Rollback",{"type":35,"tag":40,"props":827,"children":828},{},[829],{"type":48,"value":830},"When something fails mid-fleet, you need a plan. The rollback\nstrategy depends on what went wrong and how far you got.",{"type":35,"tag":384,"props":832,"children":834},{"id":833},"pre-deployment-rollback-preparation",[835],{"type":48,"value":836},"Pre-Deployment Rollback Preparation",{"type":35,"tag":40,"props":838,"children":839},{},[840],{"type":48,"value":841},"Before deploying, ensure you can roll back:",{"type":35,"tag":309,"props":843,"children":846},{"className":844,"code":845,"language":48},[312],"for host in deploy_hosts:\n create session \"prep-{host}\" on server \"{host}\"\n\n # Record current version\n send to \"prep-{host}\": readlink \u002Fopt\u002Fmyapp\u002Fcurrent\\n\n wait for idle, read screen\n previous_version[host] = extract version from screen\n\n # Ensure previous release artifacts exist\n send: ls \u002Fopt\u002Fmyapp\u002Freleases\u002F\\n\n wait for idle, read screen\n verify previous version directory exists\n\n kill session \"prep-{host}\"\n",[847],{"type":35,"tag":81,"props":848,"children":849},{"__ignoreMap":317},[850],{"type":48,"value":845},{"type":35,"tag":384,"props":852,"children":854},{"id":853},"rollback-after-failed-deployment",[855],{"type":48,"value":856},"Rollback After Failed Deployment",{"type":35,"tag":40,"props":858,"children":859},{},[860],{"type":48,"value":861},"When a rolling deploy fails partway through:",{"type":35,"tag":309,"props":863,"children":866},{"className":864,"code":865,"language":48},[312],"# failed_hosts: hosts where deployment failed\n# deployed_hosts: hosts where deployment succeeded\n# remaining_hosts: hosts not yet touched\n\n# Step 1: Fix the failed hosts\nfor host in failed_hosts:\n create session \"rollback-{host}\" on server \"{host}\"\n send to \"rollback-{host}\": cd \u002Fopt\u002Fmyapp && .\u002Frollback.sh {previous_version[host]}\\n\n wait for idle, read screen, handle prompts\n verify service is healthy\n kill session \"rollback-{host}\"\n\n# Step 2: Decide about successfully deployed hosts\n# Option A: Roll them back too (full rollback)\nfor host in deployed_hosts:\n create session \"rollback-{host}\" on server \"{host}\"\n send to \"rollback-{host}\": cd \u002Fopt\u002Fmyapp && .\u002Frollback.sh {previous_version[host]}\\n\n wait for idle, read screen, handle prompts\n verify service is healthy\n kill session \"rollback-{host}\"\n\n# Option B: Leave them on the new version (partial deploy)\n# Only do this if the new version is working correctly on\n# those hosts and the old\u002Fnew versions are compatible\n\n# Step 3: Remaining hosts were never touched -- no action needed\n\n# Step 4: Verify fleet health\nrun fleet-wide health check (see Health Verification)\n",[867],{"type":35,"tag":81,"props":868,"children":869},{"__ignoreMap":317},[870],{"type":48,"value":865},{"type":35,"tag":384,"props":872,"children":874},{"id":873},"rollback-decision-framework",[875],{"type":48,"value":876},"Rollback Decision Framework",{"type":35,"tag":40,"props":878,"children":879},{},[880],{"type":48,"value":881},"Not every failure requires a full rollback. Consider:",{"type":35,"tag":224,"props":883,"children":884},{},[885,895,905,915],{"type":35,"tag":72,"props":886,"children":887},{},[888,893],{"type":35,"tag":44,"props":889,"children":890},{},[891],{"type":48,"value":892},"One host failed, rest succeeded",{"type":48,"value":894},": Investigate the failed\nhost. It may be a host-specific issue (disk full, different\nOS version). Fix the host and retry, rather than rolling back\nthe entire fleet.",{"type":35,"tag":72,"props":896,"children":897},{},[898,903],{"type":35,"tag":44,"props":899,"children":900},{},[901],{"type":48,"value":902},"Health checks fail after deploy",{"type":48,"value":904},": Roll back the current\nbatch. Don't proceed to the next batch. Investigate before\ndeciding whether to roll back hosts that are already on the\nnew version.",{"type":35,"tag":72,"props":906,"children":907},{},[908,913],{"type":35,"tag":44,"props":909,"children":910},{},[911],{"type":48,"value":912},"Multiple hosts fail the same way",{"type":48,"value":914},": Likely a problem with\nthe release itself. Full rollback, fix the release, try again.",{"type":35,"tag":72,"props":916,"children":917},{},[918,923],{"type":35,"tag":44,"props":919,"children":920},{},[921],{"type":48,"value":922},"Intermittent failures",{"type":48,"value":924},": Some hosts fail health checks\nsometimes. This might be a pre-existing issue, not caused by\nthe deploy. Compare with pre-deployment health baselines.",{"type":35,"tag":209,"props":926,"children":928},{"id":927},"pitfalls",[929],{"type":48,"value":930},"Pitfalls",{"type":35,"tag":384,"props":932,"children":934},{"id":933},"dont-parallelize-destructive-operations-blindly",[935],{"type":48,"value":936},"Don't Parallelize Destructive Operations Blindly",{"type":35,"tag":40,"props":938,"children":939},{},[940,942,947],{"type":48,"value":941},"Running ",{"type":35,"tag":81,"props":943,"children":945},{"className":944},[],[946],{"type":48,"value":445},{"type":48,"value":948}," on 50 hosts simultaneously is tempting\nbut dangerous. If the upgrade has an unexpected prompt or failure\nmode, you'll have 50 hosts in an unknown state. Start sequential,\nmove to small batches once you've seen the operation succeed on\nthe first few hosts.",{"type":35,"tag":384,"props":950,"children":952},{"id":951},"handle-prompt-diversity",[953],{"type":48,"value":954},"Handle Prompt Diversity",{"type":35,"tag":40,"props":956,"children":957},{},[958],{"type":48,"value":959},"The same command produces different prompts on different hosts\ndepending on installed packages, OS version, configuration, and\nstate. Don't assume that because host 1 had no prompts, host 2\nwon't either. Always read the screen and handle what's actually\nthere.",{"type":35,"tag":384,"props":961,"children":963},{"id":962},"clean-up-sessions-aggressively",[964],{"type":48,"value":965},"Clean Up Sessions Aggressively",{"type":35,"tag":40,"props":967,"children":968},{},[969],{"type":48,"value":970},"Infrastructure operations can create dozens of sessions --\ndeploy, health check, rollback, audit sessions for each host.\nKill sessions as soon as they've served their purpose. A session\nis a running process on the backend machine. Leaving 50 orphaned\nsessions across your fleet wastes resources and clutters the\nsession list.",{"type":35,"tag":384,"props":972,"children":974},{"id":973},"dont-forget-sudo",[975],{"type":48,"value":976},"Don't Forget sudo",{"type":35,"tag":40,"props":978,"children":979},{},[980,982,988,990,996],{"type":48,"value":981},"Many infrastructure operations require elevated privileges.\nHandle ",{"type":35,"tag":81,"props":983,"children":985},{"className":984},[],[986],{"type":48,"value":987},"sudo",{"type":48,"value":989}," password prompts consistently. If you're working\nacross many hosts, consider whether ",{"type":35,"tag":81,"props":991,"children":993},{"className":992},[],[994],{"type":48,"value":995},"NOPASSWD",{"type":48,"value":997}," rules for\nspecific commands are appropriate to avoid interactive password\nentry on every host.",{"type":35,"tag":384,"props":999,"children":1001},{"id":1000},"record-what-you-do",[1002],{"type":48,"value":1003},"Record What You Do",{"type":35,"tag":40,"props":1005,"children":1006},{},[1007],{"type":48,"value":1008},"Infrastructure changes are auditable events. For each operation,\nrecord:",{"type":35,"tag":224,"props":1010,"children":1011},{},[1012,1017,1022,1027],{"type":35,"tag":72,"props":1013,"children":1014},{},[1015],{"type":48,"value":1016},"What was done on which hosts",{"type":35,"tag":72,"props":1018,"children":1019},{},[1020],{"type":48,"value":1021},"What the state was before and after",{"type":35,"tag":72,"props":1023,"children":1024},{},[1025],{"type":48,"value":1026},"Which hosts succeeded and which failed",{"type":35,"tag":72,"props":1028,"children":1029},{},[1030],{"type":48,"value":1031},"What prompts were encountered and how they were answered",{"type":35,"tag":40,"props":1033,"children":1034},{},[1035],{"type":48,"value":1036},"Read scrollback from each session before killing it. This is\nyour audit trail.",{"type":35,"tag":384,"props":1038,"children":1040},{"id":1039},"know-when-ansible-is-better",[1041],{"type":48,"value":1042},"Know When Ansible Is Better",{"type":35,"tag":40,"props":1044,"children":1045},{},[1046],{"type":48,"value":1047},"wsh + AI is powerful for interactive, judgment-heavy operations.\nBut Ansible is better for:",{"type":35,"tag":224,"props":1049,"children":1050},{},[1051,1056,1061,1066],{"type":35,"tag":72,"props":1052,"children":1053},{},[1054],{"type":48,"value":1055},"Fully predictable, idempotent operations at scale",{"type":35,"tag":72,"props":1057,"children":1058},{},[1059],{"type":48,"value":1060},"Operations that need to be reproduced exactly the same way\nevery time (compliance, auditing)",{"type":35,"tag":72,"props":1062,"children":1063},{},[1064],{"type":48,"value":1065},"Simple fan-out where no per-host judgment is needed",{"type":35,"tag":72,"props":1067,"children":1068},{},[1069],{"type":48,"value":1070},"Infrastructure-as-code workflows where the desired state\nshould be version-controlled",{"type":35,"tag":40,"props":1072,"children":1073},{},[1074],{"type":48,"value":1075},"Use wsh for the 20% that requires interaction and judgment.\nUse Ansible for the 80% that doesn't. They complement each\nother.",{"type":35,"tag":384,"props":1077,"children":1079},{"id":1078},"test-your-rollback-before-you-need-it",[1080],{"type":48,"value":1081},"Test Your Rollback Before You Need It",{"type":35,"tag":40,"props":1083,"children":1084},{},[1085],{"type":48,"value":1086},"The worst time to discover your rollback procedure doesn't\nwork is during a failed deployment at 2am. Test rollback on\na staging fleet first. Verify that the previous version's\nartifacts exist, that the rollback script works, and that\nthe health checks pass after rolling back.",{"type":35,"tag":384,"props":1088,"children":1090},{"id":1089},"mind-the-blast-radius",[1091],{"type":48,"value":1092},"Mind the Blast Radius",{"type":35,"tag":40,"props":1094,"children":1095},{},[1096],{"type":48,"value":1097},"If you're deploying across 100 hosts, don't set a batch size\nof 50. A failed batch of 50 means half your fleet is down.\nStart small. Increase batch size only after the first few\nbatches succeed. A common progression: 1, 2, 5, 10, 25, 50.",{"items":1099,"total":1258},[1100,1116,1132,1146,1158,1170,1182,1193,1209,1223,1235,1247],{"slug":1101,"name":1101,"fn":1102,"description":1103,"org":1104,"tags":1105,"stars":1113,"repoUrl":1114,"updatedAt":1115},"deepclaw-voice","configure phone calls with Deepgram Voice","Set up phone calling to OpenClaw using Deepgram Voice Agent API",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1106,1109,1110],{"name":1107,"slug":1108,"type":15},"API Development","api-development",{"name":9,"slug":8,"type":15},{"name":1111,"slug":1112,"type":15},"Voice","voice",78,"https:\u002F\u002Fgithub.com\u002Fdeepgram\u002Fdeepclaw","2026-07-12T08:29:25.371332",{"slug":1117,"name":1117,"fn":1118,"description":1119,"org":1120,"tags":1121,"stars":1129,"repoUrl":1130,"updatedAt":1131},"1password","manage secrets with 1Password CLI","Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading\u002Finjecting\u002Frunning secrets via op.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1122,1125,1126],{"name":1123,"slug":1124,"type":15},"Automation","automation",{"name":13,"slug":14,"type":15},{"name":1127,"slug":1128,"type":15},"Security","security",23,"https:\u002F\u002Fgithub.com\u002Fdeepgram\u002Fdglabs-deepclaw","2026-07-12T08:28:49.991939",{"slug":1133,"name":1133,"fn":1134,"description":1135,"org":1136,"tags":1137,"stars":1129,"repoUrl":1130,"updatedAt":1145},"apple-notes","manage Apple Notes on macOS","Manage Apple Notes via the `memo` CLI on macOS (create, view, edit, delete, search, move, and export notes). Use when a user asks OpenClaw to add a note, list notes, search notes, or manage note folders.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1138,1139,1142],{"name":13,"slug":14,"type":15},{"name":1140,"slug":1141,"type":15},"Knowledge Management","knowledge-management",{"name":1143,"slug":1144,"type":15},"macOS","macos","2026-07-12T08:29:01.538106",{"slug":1147,"name":1147,"fn":1148,"description":1149,"org":1150,"tags":1151,"stars":1129,"repoUrl":1130,"updatedAt":1157},"apple-reminders","manage Apple Reminders via CLI","Manage Apple Reminders via the `remindctl` CLI on macOS (list, add, edit, complete, delete). Supports lists, date filters, and JSON\u002Fplain output.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1152,1153,1154],{"name":13,"slug":14,"type":15},{"name":1143,"slug":1144,"type":15},{"name":1155,"slug":1156,"type":15},"Task Management","task-management","2026-07-12T08:29:14.035414",{"slug":1159,"name":1159,"fn":1160,"description":1161,"org":1162,"tags":1163,"stars":1129,"repoUrl":1130,"updatedAt":1169},"bear-notes","manage Bear notes via CLI","Create, search, and manage Bear notes via grizzly CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1164,1165,1166],{"name":13,"slug":14,"type":15},{"name":1140,"slug":1141,"type":15},{"name":1167,"slug":1168,"type":15},"Notes","notes","2026-07-12T08:28:51.246011",{"slug":1171,"name":1171,"fn":1172,"description":1173,"org":1174,"tags":1175,"stars":1129,"repoUrl":1130,"updatedAt":1181},"blogwatcher","monitor blogs and RSS feeds","Monitor blogs and RSS\u002FAtom feeds for updates using the blogwatcher CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1176,1177,1178],{"name":1123,"slug":1124,"type":15},{"name":13,"slug":14,"type":15},{"name":1179,"slug":1180,"type":15},"Monitoring","monitoring","2026-07-12T08:29:02.762321",{"slug":1183,"name":1183,"fn":1184,"description":1185,"org":1186,"tags":1187,"stars":1129,"repoUrl":1130,"updatedAt":1192},"blucli","control BluOS audio playback","BluOS CLI (blu) for discovery, playback, grouping, and volume.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1188,1191],{"name":1189,"slug":1190,"type":15},"Audio","audio",{"name":13,"slug":14,"type":15},"2026-07-12T08:28:21.009637",{"slug":1194,"name":1194,"fn":1195,"description":1196,"org":1197,"tags":1198,"stars":1129,"repoUrl":1130,"updatedAt":1208},"bluebubbles","send and manage iMessages","Use when you need to send or manage iMessages via BlueBubbles (recommended iMessage integration). Calls go through the generic message tool with channel=\"bluebubbles\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1199,1202,1205],{"name":1200,"slug":1201,"type":15},"Communications","communications",{"name":1203,"slug":1204,"type":15},"iMessage","imessage",{"name":1206,"slug":1207,"type":15},"Messaging","messaging","2026-07-12T08:28:57.517914",{"slug":1210,"name":1210,"fn":1211,"description":1212,"org":1213,"tags":1214,"stars":1129,"repoUrl":1130,"updatedAt":1222},"camsnap","capture frames and clips from cameras","Capture frames or clips from RTSP\u002FONVIF cameras.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1215,1216,1219],{"name":1123,"slug":1124,"type":15},{"name":1217,"slug":1218,"type":15},"Camera","camera",{"name":1220,"slug":1221,"type":15},"Media","media","2026-07-12T08:28:28.096134",{"slug":1224,"name":1224,"fn":1225,"description":1226,"org":1227,"tags":1228,"stars":1129,"repoUrl":1130,"updatedAt":1234},"clawhub","manage agent skills with ClawHub","Use the ClawHub CLI to search, install, update, and publish agent skills from clawhub.com. Use when you need to fetch new skills on the fly, sync installed skills to latest or a specific version, or publish new\u002Fupdated skill folders with the npm-installed clawhub CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1229,1232,1233],{"name":1230,"slug":1231,"type":15},"Agents","agents",{"name":1123,"slug":1124,"type":15},{"name":13,"slug":14,"type":15},"2026-07-12T08:28:30.589001",{"slug":1236,"name":1236,"fn":1237,"description":1238,"org":1239,"tags":1240,"stars":1129,"repoUrl":1130,"updatedAt":1246},"coding-agent","run coding agents for programmatic control","Run Codex CLI, Claude Code, OpenCode, or Pi Coding Agent via background process for programmatic control.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1241,1242,1243],{"name":1230,"slug":1231,"type":15},{"name":1123,"slug":1124,"type":15},{"name":1244,"slug":1245,"type":15},"Coding","coding","2026-07-12T08:29:08.6658",{"slug":1248,"name":1248,"fn":1249,"description":1250,"org":1251,"tags":1252,"stars":1129,"repoUrl":1130,"updatedAt":1257},"eightctl","control Eight Sleep pod settings","Control Eight Sleep pods (status, temperature, alarms, schedules).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1253,1254],{"name":1123,"slug":1124,"type":15},{"name":1255,"slug":1256,"type":15},"Hardware","hardware","2026-07-12T08:28:39.322181",73,{"items":1260,"total":1338},[1261,1274,1288,1299,1311,1321,1333],{"slug":1262,"name":1262,"fn":1263,"description":1264,"org":1265,"tags":1266,"stars":19,"repoUrl":20,"updatedAt":1273},"agent-orchestration","orchestrate multiple AI agents","Use when you need to launch and drive other AI agents (Claude Code, Aider, Codex, etc.) through their terminal interfaces via wsh. Examples: \"run multiple Claude Code sessions in parallel on different tasks\", \"feed a task to an AI agent and handle its approval prompts\", \"coordinate several AI agents working on subtasks of a larger project\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1267,1268,1269,1270],{"name":1230,"slug":1231,"type":15},{"name":1123,"slug":1124,"type":15},{"name":13,"slug":14,"type":15},{"name":1271,"slug":1272,"type":15},"Multi-Agent","multi-agent","2026-07-12T08:29:40.028618",{"slug":1275,"name":1275,"fn":1276,"description":1277,"org":1278,"tags":1279,"stars":19,"repoUrl":20,"updatedAt":1287},"cluster-orchestration","orchestrate sessions across federated clusters","Use when you need to manage sessions across multiple wsh servers in a federated cluster. Examples: \"distribute builds across several machines\", \"create sessions on a specific backend\", \"monitor health across a cluster of servers\", \"coordinate work across server boundaries\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1280,1283,1284],{"name":1281,"slug":1282,"type":15},"Architecture","architecture",{"name":1123,"slug":1124,"type":15},{"name":1285,"slug":1286,"type":15},"Engineering","engineering","2026-07-12T08:29:38.810244",{"slug":1289,"name":1289,"fn":1290,"description":1291,"org":1292,"tags":1293,"stars":19,"repoUrl":20,"updatedAt":1298},"core","authenticate and bootstrap wsh terminal operations","REQUIRED before any wsh terminal operation when you do NOT have wsh_* MCP tools. Contains the complete HTTP API reference with working curl examples, bootstrap sequence, and authentication guide. wsh has no CLI subcommands for programmatic use — do NOT run 'wsh \u003Cverb>' commands or guess endpoints. Load this skill first.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1294,1297],{"name":1295,"slug":1296,"type":15},"Authentication","authentication",{"name":13,"slug":14,"type":15},"2026-07-12T08:29:32.614733",{"slug":1300,"name":1300,"fn":1301,"description":1302,"org":1303,"tags":1304,"stars":19,"repoUrl":20,"updatedAt":1310},"core-mcp","bootstrap MCP terminal sessions","REQUIRED before any wsh terminal operation. Contains the complete MCP tool reference and bootstrap sequence for wsh_create_session, wsh_send_input, wsh_get_screen, wsh_send_and_read, wsh_send_keys, and all wsh_* tools. Do NOT guess wsh CLI commands or HTTP endpoints — use MCP tools or load this skill first.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1305,1306,1307],{"name":1123,"slug":1124,"type":15},{"name":13,"slug":14,"type":15},{"name":1308,"slug":1309,"type":15},"MCP","mcp","2026-07-12T08:29:45.869993",{"slug":1312,"name":1312,"fn":1313,"description":1314,"org":1315,"tags":1316,"stars":19,"repoUrl":20,"updatedAt":1320},"drive-process","interact with CLI programs via wsh","Use when you need to drive a CLI program through command-and-response interaction via wsh. Examples: \"run a build command and check the output\", \"interact with an installer that asks questions\", \"execute a sequence of shell commands and handle errors\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1317,1318,1319],{"name":1123,"slug":1124,"type":15},{"name":13,"slug":14,"type":15},{"name":1285,"slug":1286,"type":15},"2026-07-12T08:29:51.485222",{"slug":1322,"name":1322,"fn":1323,"description":1324,"org":1325,"tags":1326,"stars":19,"repoUrl":20,"updatedAt":1332},"generative-ui","build interactive terminal user interfaces","Use when you need to build dynamic, interactive terminal experiences on the fly. Examples: \"create a live dashboard in the terminal\", \"build an interactive file browser\", \"generate a custom TUI for this workflow\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1327,1328,1329],{"name":13,"slug":14,"type":15},{"name":1285,"slug":1286,"type":15},{"name":1330,"slug":1331,"type":15},"Frontend","frontend","2026-07-12T08:29:41.286933",{"slug":4,"name":4,"fn":5,"description":6,"org":1334,"tags":1335,"stars":19,"repoUrl":20,"updatedAt":21},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1336,1337],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},12]