VSS SOP — Post-Deployment Test Suite

Run this skill after the VSS SOP blueprint has been deployed (see vss-sop-deploy). It validates every layer of the stack, and when a test fails it auto-debugs by collecting container logs and printing actionable fix hints.

Overview

• Verify a fresh SOP deployment is healthy
• Check that ELK is receiving VLM data
• Validate VIOS recording and livestream
• Test the VSS agent: MCP server, LLM endpoint, VLM endpoint, snapshot, video, and report generation
• Troubleshoot a broken deployment

Instructions

Quick Start

cd <bp-repo>
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py \
  --bp-repo . \
  --env-file deployments/sop/.env

Run a single phase:

python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py --phase 1   # Service health only
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py --phase 2   # ELK only
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py --phase 3   # VIOS only
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py --phase 4   # VSS Agent only

Test Phases

Phase 1 — Service Health

Checks that all required Docker containers are running (Up).

Auto-debug: Prints docker logs <container> --tail 30 for any missing or unhealthy container.

Manual fix: Redeploy with docker compose -f compose.yml --env-file sop/.env --profile bp_sop_2d up -d.

Phase 2 — ELK Data Pipeline

1. Elasticsearch reachable — GET localhost:9200/_cluster/health
2. Cluster health — must be green or yellow
3. Indices exist — at least one mdx-vlm-captions-* index
4. VLM messages — mdx-vlm-captions-* index has > 0 documents
5. Kibana dashboard fields — verifies the actual ES mapping has the flat JSON fields the dashboard expects (response, sensor_id, cv_execute_time, vlm_execute_time, chunk_idx, frame_number, @timestamp), and that no protobuf-style nested fields (llm, sensor) are present instead
6. Kibana ndjson fields — queries Kibana's saved objects API to confirm the imported data view's runtime fields reference the correct flat field names (not llm.queries.response.keyword or sensor.id.keyword), and that timeFieldName is @timestamp

Auto-debug: Tails mdx-logstash, mdx-elastic, and mdx-kibana logs.

Manual fix if no VLM data:

# Verify Kafka topic has messages
docker exec mdx-kafka kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic mdx-vlm-captions \
  --from-beginning --max-messages 3 --timeout-ms 10000

# Check Logstash pipeline
docker logs mdx-logstash --tail 50

Manual fix if Kibana dashboard field mismatch:

# Check the actual field structure in Elasticsearch
curl localhost:9200/mdx-vlm-captions-*/_mapping?pretty

# If fields are flat (response, sensor_id) but ndjson uses nested (llm.queries.response):
# Re-import the corrected ndjson
curl -X POST localhost:5601/api/saved_objects/_import?overwrite=true \
  -H 'kbn-xsrf: true' \
  --form file=@deployments/sop/sop-app/kibana-dashboard/sop-kibana-objects.ndjson

Phase 3 — VIOS Recording & Livestream

1. VST reachable — GET localhost:30888/vst/api/v1/sensor/list
2. Sensors registered — at least one sensor (typically sensor_0)
3. Streams available — /sensor/streams returns data
4. Recording — checks /record/status per sensor
5. Livestream — checks /live/streams per sensor

Auto-debug: Tails sensor-ms-sop, rtspserver-ms-1-sop, recorder-ms-1-sop, and sdr-http-recorder-sop logs.

Manual fix if no sensor:

cd <bp-repo>/deployments/sop/sop-app/helper-scripts
nohup ./run_rtsp_test.sh > client.log &

Phase 4 — VSS Agent End-to-End

Auto-debug: Tails vss-agent, vss-va-mcp, and mdx-ds-sop-1 logs for any failing sub-test.

Manual fix for common agent failures:

# Agent won't start — check LLM/VLM config
docker inspect vss-agent --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -E '(LLM|VLM)'

# MCP not healthy — restart
docker restart vss-va-mcp

# NVIDIA API key missing for remote NIM
export NVIDIA_API_KEY="$(cat <bp-repo>/.secret/nvidia_build_api_key.txt)"
docker compose -f compose.yml --env-file sop/.env --profile bp_sop_2d up -d vss-agent

Phase 4.7 "no streams found" on fresh deployment: If the test runs immediately after deploy, the report time window (now-1min to now) may precede the first recording. Wait at least 1 minute after the RTSP stream registers before running Phase 4.

Examples

Pass Criteria — MANDATORY

ALL tests must pass (100%). There is NO acceptable failure rate.

Rules the agent MUST follow:

1. Zero tolerance for failures. Every single test check across all phases MUST pass. A result of 36/38 or 17/19 is a FAIL — not a pass with caveats.
2. Do NOT classify failures as "environmental", "infrastructure", "test harness", or "intermittent". Every failure is a bug until you have fixed it and the test passes on re-run.
3. Do NOT lower the pass threshold. Never say "good enough", "residual", "non-blocking", or "acceptable failure". If a test fails, fix it.
4. Do NOT skip or move on. If a test fails after 3 fix attempts, STOP and ask the user for help. Do NOT proceed to the next phase or declare completion.
5. Do NOT mark a phase as PASS if any check within it failed. Use FAIL until every check is green.
6. Timeout failures are bugs too. If a test times out (e.g., 30s client read-timeout), increase the timeout or fix the underlying latency — do NOT dismiss it as a test configuration issue.
7. Re-run after every fix. After applying a fix, re-run the failing phase to confirm the fix works. A fix is not verified until the test passes.

Completion criteria: The pipeline is complete ONLY when the final report shows 38/38 (or equivalent total) with zero failures across all phases. Any other result means more work is needed.

Workflow for the Agent

Copy this checklist and track progress:

Test Progress:
- [ ] Phase 1: Service health — ALL checks PASS
- [ ] Phase 2: ELK data pipeline — ALL checks PASS
- [ ] Phase 3: VIOS recording & livestream — ALL checks PASS (7/7)
- [ ] Phase 4: VSS agent (MCP, LLM, VLM, snapshot, video, report) — ALL checks PASS (7/7)
- [ ] Auto-debug any failures (do NOT skip — every failure must be fixed)
- [ ] Auto-update vss-sop-build skills/scripts if root cause found
- [ ] Confirm 100% pass rate before reporting to user
- [ ] Report summary to user

Step 1 — Determine <bp-repo>

# Usually the vss-sop checkout
BP_REPO=$(pwd)   # or wherever the user cloned vss-sop

Step 2 — Run the full test suite

cd "$BP_REPO"
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py \
  --bp-repo "$BP_REPO" \
  --env-file "$BP_REPO/deployments/sop/.env"

Error Handling

Auto-Debugging Failures (MANDATORY — do NOT skip)

CRITICAL: Every failing test MUST be fixed. Do NOT classify any failure as "environmental" or "acceptable" and move on.

The script already collects container logs for failures. Additionally:

1. Read the [FAIL] output and the auto-debug hints printed by the script.
2. Apply the suggested fix (restart container, re-export key, re-run RTSP test, etc.).
3. Re-run the failing phase only:

python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py --phase <N>

4. Repeat until ALL tests in that phase pass (100%).
5. If a test still fails after 3 fix attempts, STOP and ask the user for help. Do NOT declare completion or proceed to the next step.
6. Never rationalize a failure. Statements like "this is an environmental issue, not a code defect" are NOT acceptable reasons to skip a fix. Either fix the environment or fix the test.

Auto-Updating the Upstream Skill After Root-Cause Fix

When a test failure is debugged and the root cause is identified, update the corresponding upstream skill so the bug cannot recur. First determine where the root cause lives, then update the correct skill:

Keep every change simple and general — fix the pattern, not just the instance.

4a. Root cause in build/deploy → update vss-sop-build

4b. Root cause in DS-SOP source code → update ds-sop-skills

Procedure (applies to both 4a and 4b):

1. Classify the root cause — determine whether it belongs to build/deploy config (vss-sop-build) or DS-SOP source code (ds-sop-skills).
2. Identify the file — map the root cause to the corresponding source file in the chosen skill's directory.
3. Make the minimal fix — change only what is needed; prefer updating a default value, adding a guard/check, or appending an entry under Error Handling over larger rewrites.
4. Verify consistency:
   • If a vss-sop-build reference was changed, re-run agentic/vss-sop-skills/vss-sop-build/scripts/verify_build.sh to confirm the build still passes.
   • If a ds-sop-skills reference was changed, regenerate the source and rebuild ds-sop:1.0.0 to confirm the fix.
5. Log the change — include in the final report which skill and file was updated and why.

Rule of thumb: if the root cause is in the generated DS-SOP source code (Python logic, pipeline config, API, Dockerfile, etc.), the fix belongs in ds-sop-skills — so the next generation produces correct code. If it is in the VSS blueprint build process (compose wiring, .env defaults, copy scripts, build ordering), the fix belongs in vss-sop-build. If it is environment-specific (wrong IP, expired key), just document it in the report — don't change either skill.

Step 5 — Report results

Present the final summary table to the user. The report MUST show 100% pass rate across all phases.

• If all tests pass: report the green summary.
• If any test still fails: report the failure with full debug context and explicitly state that the pipeline is NOT complete. Do NOT use language suggesting partial success (e.g., "36/38 checks pass" is a failure report, not a success report).

If any vss-sop-build or ds-sop-skills files were updated in Step 4, list them with a one-line summary of each change.

Service Endpoints Reference

Dependencies

The test script requires only requests (installed in most Python environments). No additional pip install needed if you have a standard Python 3.8+ environment.

License

Use of this skill is governed by the Creative Commons Attribution 4.0 International License (CC BY 4.0) and the Apache License, Version 2.0.