SDAF HA Diagnostics

Action-loop skill. Owns read-first, safe diagnostics for a live or recently failed SDAF HA cluster. It does not choose the topology, move resources, clean up stonith history, or invent a recovery procedure.

For the exact evidence ladder, current-code anchors, and stop-list, read references/cluster-evidence.md.

When to invoke

Trigger on: "diagnose HA cluster", "HANA failover failed", "crm status", "pcs status", "check fencing", "resource placement", "cluster unhealthy", "offline_validation/cib", "HCMT result", "SCS/ERS failover", or "read the current Pacemaker state".

Do NOT trigger on:

• pre-deploy design questions such as SBD vs Azure fence agent, ANF vs AFS, ANGI eligibility, distro choice, or scale-out design — route to sdaf-ha-topology;
• requests to run online failover, crash, fencing, or blocked-network tests — route to sdaf-quality-assurance;
• a broader SDAF run failure whose primary symptom is the stage execution itself — route to sdaf-failure-triage.

Preconditions

Ask for the layer under test (HANA DB HA or SCS/ERS HA), OS family, evidence type (live cluster, captured bundle, or prior test run), and any existing crm / pcs output, QA reports, execution logs, or HCMT bundles. If cluster state was already changed manually, preserve who changed what and when before diagnosing anything else.

Recipe

Step 1 — classify the request before touching the cluster

Pick one lane:

1. Live read-only diagnosis — explain current cluster state from status, placement, and existing logs.
2. Captured artefact review — read an existing offline_validation bundle, QA report, or HCMT result without acting on the running cluster.
3. Framework-run or disruptive validation — route to sdaf-quality-assurance; the online functional tests stop resources, move groups, fence nodes, and crash processes.
4. Topology/design — route to sdaf-ha-topology.
5. Stage/run failure — route to sdaf-failure-triage.

Step 2 — collect read-only evidence first

Use the evidence ladder in references/cluster-evidence.md. Minimum safe evidence is:

• OS-specific status: crm status full on SUSE or pcs status --full on Red Hat;
• current placement evidence such as crm_resource --locate for SCS/ERS;
• existing fencing, SBD, or HANA hook-verification evidence;
• existing QA, offline cib, and HCMT artefacts.

Preserve raw output, host, and timestamp. Do not paraphrase away the cluster's own wording.

Step 3 — map the evidence to the right owner

• Stay in this skill for read-only interpretation of current Pacemaker state, placement, fencing/SBD evidence, hook traces, or existing QA/HCMT artefacts.
• Route to sdaf-quality-assurance to run configuration checks, offline HA validation from captured offline_validation/<host>/cib bundles, or disruptive online HA exercises in an approved maintenance window.
• Route to sdaf-failure-triage for failed SDAF runs or install-phase assertions such as missing fencing SPN details, cluster password, or required clustering scripts.
• Route to sdaf-ha-topology when the question is what the cluster should look like rather than what the current evidence says.

Step 4 — stop before state changes

The current codebase contains state-changing cluster actions in validation and post-provision tasks. This skill is not the place to run them.

The explicit stop-list lives in the reference. From this skill do not execute crm/pcs resource move, stonith_admin --cleanup, role-defined cluster cleanup, StartService, StartSystem ALL, or online functional HA tests. If the user asks for recovery or failover action, stop at evidence collection and routing; do not invent or approve a repair step.

Hard rules

• Read-first only. Start with existing cluster status, logs, and artefacts.
• Documented/current-code only (D19). If the docs or shipped roles do not describe a diagnostic path, say so and stop.
• Do not invent recovery commands, failover steps, or cleanup procedures.
• Keep topology and design separate; route those questions to sdaf-ha-topology.
• Treat disruptive validation as a separate workflow owned by sdaf-quality-assurance.

What this skill does NOT do

• Does not choose SBD vs Azure fence agent, storage, distro, ANGI, or scale-out design.
• Does not build, rebuild, reinstall, or actively fail over the cluster.
• Does not treat HCMT as the first diagnostic step; it only reuses existing HCMT artefacts if they are already present.

See also

• sdaf-ha-topology
• sdaf-quality-assurance
• sdaf-failure-triage
• docs/local/07-10-quality-assurance.md