Prism Dash Platform

Prism Dash is an autonomous Amazon DocumentDB optimization platform written in Python on top of Dash 2.17 (with Dash Bootstrap Components + Plotly). It connects to a DocumentDB cluster, runs an autonomous agent that analyzes the cluster for performance, cost, reliability, and architectural issues, correlates the findings across modules, and surfaces AI-grounded recommendations through a tabbed UI and a Markdown report.

This is the orchestration-level skill. It describes how the whole system fits together and where each capability lives. Two focused sub-skills handle the conversational and review surfaces:

• documentdb-advisor — the read-only conversational advisor (chat, agentic tool-use, per-slow-query recommendations). See documentdb-advisor/SKILL.md.
• documentdb-well-architected-review — the 6-pillar Well-Architected review (38-check catalog, scoring, exports). See documentdb-well-architected-review/SKILL.md.

When a task is specifically about chat/advice or about a WA review, defer to those sub-skills. Use this skill for cross-cutting questions about the platform.

1. Tech Stack

• UI: Dash 2.17 + Dash Bootstrap Components + Plotly
• Backend: Python 3, PyMongo 4.5 (appname="DocDB-Prism"), boto3
• AI: AWS Bedrock — us.anthropic.claude-sonnet-4-20250514-v1:0 (primary), us.anthropic.claude-haiku-4-5-20251001-v1:0 (fallback)
• Target: Amazon DocumentDB clusters (public or private via SSH tunnel)

2. Architecture — The Autonomous Agent (agent_orchestrator.py)

The agent runs an Observe → Reason → Decide → Act loop on a background daemon thread (started by start_agent(conn_data); stopped by stop_agent()). A second daemon thread (_monitor_activity) polls currentOp for live activity. All shared state lives in the module-level _agent_state dict guarded by a threading.Lock; the UI polls a lightweight summary via get_agent_state() and dcc.Interval.

Fixed fallback order (_FALLBACK_ORDER): cluster_snapshot → well_architected → db_analysis → slow_query → compression → bloat → instance_recommender → storage_recommender.

Safety & prioritisation (driven by prism_config.yaml)

• System databases admin, local, config are always excluded.
• _safety_check() enforces skip_databases and skip_collections (prefix match).
• _build_analysis_scope() ranks candidates: active databases first (from currentOp), then priority_databases, then largest-by-size, truncated to max_databases_to_analyse. Skipped databases and their reasons are recorded in analysis_scope.skipped.
• _iter_databases() paces work with database_tree.delay_between_seconds to keep cluster load gentle and honors the _stop_event.

Cross-module correlation

After modules complete, the agent correlates findings across modules (for example slow query on a collection with a missing/low-cardinality index, or high bloat on a collection with compression disabled) and stores human-readable strings in _agent_state["correlated_insights"]. These feed the report's executive summary and are exposed to the chat advisor as the agent_insights data source.

3. Plugin System (analyzer_base.py)

Analysis modules are designed as plugins implementing the AnalyzerPlugin ABC. Concrete subclasses auto-register at import time via __init_subclass__ into _ANALYZER_REGISTRY (accessed through get_analyzer_registry(), get_analyzer(name), get_execution_order()).

class AnalyzerPlugin(ABC):
    name: str            # Unique ID; matches a key in agent_state["modules"]
    display_name: str    # Human-readable name for the UI
    scope: str           # "database" (per-db) or "cluster" (once)
    dependencies: list   # Analyzer names that must complete first
    priority: int = 50   # Lower runs earlier
    enabled: bool = True # Kill switch

    def run(self, conn_data, progress_callback=None, previous_results=None) -> Any: ...
    def should_skip(self, conn_data, previous_results=None) -> Optional[str]: ...
    def validate_prerequisites(self, conn_data) -> Optional[str]: ...
    def get_report_section(self, result, conn_data) -> Optional[str]: ...

conn_data keys: connection_string, database_name, cluster_id, region, log_group. progress_callback(phase: str, detail: str, pct: int) is the standard progress contract across the platform.

analyzers/__init__.py imports the 7 plugin implementations so they register: ClusterSnapshotAnalyzer, DbAnalysisAnalyzer, SlowQueryAnalyzer, CompressionAnalyzer, WellArchitectedAnalyzer, InstanceRecommenderAnalyzer, StorageRecommenderAnalyzer.

Current execution reality: the orchestrator's _act() keeps _PLUGIN_READY_MODULES empty and runs a legacy hardcoded dispatch for every module, because the UI reads from shared-state dicts (_snap, _wa, _agent_state) that the plugins don't yet fully populate. The plugin registry is fully wired and discovered at import; modules graduate to the plugin path by being added to _PLUGIN_READY_MODULES. Document modules by their behavior, not by assuming the plugin path is active.

To add a new analyzer, follow the prism-analyzer-module skill.

4. Module Catalog (8 agent-state slots)

_agent_state["modules"] has 8 slots. Seven map to registered analyzer plugins; bloat is a slot whose analysis is produced as part of db_analysis (index/collection bloat percentages), not a separate registered plugin.

Each module slot holds {status, result, error, ts} where status ∈ {pending, running, done, skipped}.

Instance recommender (instance_recommender.py)

analyze_workload_statistics() computes mean, std, coefficient of variation (CV), spike frequency (values above mean + 2·std), idle frequency (values < 20%), and P5/P95 from CloudWatch CPU datapoints, then classifies the workload as highly_spiky, moderately_spiky, or sustained. recommend_instance_type() then proposes one of: Serverless (DCU-based, for spiky workloads with high idle + spike frequency), NVMe (db.r6gd.*, when buffer cache hit ratio < 85% and not CPU-bound), downgrade (over-provisioned), or upgrade (under-provisioned). Catalogs: STANDARD_INSTANCES (db.t4g, db.r5, db.r6g, db.r8g) and NVME_INSTANCES (db.r6gd). Burstable db.t* types are never recommended for scale up/down targets.

Storage cost analyzer (storage_cost_analyzer.py)

Pulls live AWS Pricing API data and compares Standard vs I/O-Optimized monthly cost (instance compute + storage + I/O), region-specific. I/O-Optimized is favored when per-operation I/O cost is the dominant component of the bill.

5. Memory System (agent_memory.py)

Persistent, cross-run learning under .prism_cache/{cluster_id}/:

• Change detection: _content_hash() hashes content while ignoring timestamp fields, so unchanged data only refreshes last_seen rather than writing a new version. compare_with_previous() emits human-readable drift strings (engine upgrades, instance-type changes, db count changes, bloat/size growth, prior WA/slow query/index summaries).
• Versioning: _write_versioned() keeps up to 3 versions (name.json, name.v1.json, name.v2.json) and preserves the oldest first_seen.
• Retention: cleanup_old_files() deletes activity/daily files older than 30 days.

6. AI Advisory Stack

Three Bedrock-backed components share the same model pair and degrade gracefully. Detailed behavior lives in the sub-skills; the wiring is:

• bedrock_advisor.py — agentic tool-use advisor. Loads its system prompt from documentdb-well-architected-review/advisor-prompt.md (via _load_skill()) and injects topic-matched reference docs from that directory's references/. Defines 8 tools but _execute_tool() returns an error (MCP disabled), so answers are grounded in injected context. See documentdb-advisor/SKILL.md.
• chat_advisor.py — two-step chat (classify → fetch → answer). Step 1 (Haiku) classifies which data sources are needed; step 2 fetches those sources from agent state and answers with the advisor prompt + matched references. Data sources: collection_stats, index_health, slow_queries, live_activity, cluster_config, well_architected, agent_insights.
• slow_query_recommender.py — per-pattern recommendations (Add index / Rewrite query / Scale compute), grounded in documentdb-advisor/SKILL.md + all of its references/. Background generation, per-database concurrency limit of 5, persistent cache keyed by (cluster_id, pattern_key, stats_digest), lifecycle placeholder → generating → done | failed | unavailable.
• bedrock_parallel.py — parallel per-pattern index suggestions (Haiku) with collection context (existing indexes, doc count), ESR-rule prompting, JSON output.

The same Bedrock pair also powers the agent planner (_reason) and the WA recommendation generation (tabs/well_architected._generate_ai_recommendations via documentdb-well-architected-review/wa-advisor-prompt.md).

7. Code Review Engine (code_review_engine.py)

Scans application source code (not the database) for DocumentDB client best practices. start_code_review(target_dir, output_dir) runs on a background thread: _discover_files() finds source files (JS/TS, Java, Python, Go, C#, Ruby, PHP, R) and config files (docker-compose, .env, serverless.yml, CDK, etc.); _scan_patterns() detects MongoClient usage, connection strings, pool/timeout/retry config, and index creation; _evaluate_checklist() scores 54 checklist items; _write_report() emits a Markdown report with a compliance percentage. Surfaced via tabs/code_review_panel.py.

8. Report Generation (agent_report.py)

generate_report(agent_state, conn_data) produces a Markdown report with: title + metadata, Executive Summary (modules completed, correlated insights, top actions), Agent Reasoning Log (step/module/reasoning table), Analysis Scope (in-scope vs skipped databases), per-module sections (cluster config, database analysis, slow queries, compression, Well-Architected, instance sizing, storage), Live Activity Observations, and Skipped Modules. The Well-Architected review additionally supports a PDF export via wa_pdf.generate_wa_pdf(...).

9. Configuration (prism_config.yaml)

Loaded at runtime through prism_cfg.get_config(). Key sections:

• database_tree.initial_batch (10) — databases fully populated at connect; the rest load in the background.
• database_tree.delay_between_seconds (2) — pacing between background db loads and agent iterations.
• compression_analysis.collections_per_batch (5), delay_between_seconds (3).
• agent_prioritisation.priority_databases (), skip_databases (staging, local, admin, config), max_databases_to_analyse (20), skip_collections (audit_log, tmp_, _archive — prefix match), max_collection_size_gb (50), max_compression_sample_docs (1,000,000).

Always respect these limits; never bypass skip_* lists or analyze system databases.

10. Other Components

• aws_discovery.py — discovers DocumentDB clusters/instances in an account/region.
• ssh_tunnel.py — manages SSH tunnels for private clusters (with orphan reaping).
• tabs/ — self-registering Dash UI tabs (tabs/registry.py); tabs/wa_v2/ is the next-gen WA UI that contributes additional checks (e.g. PERF1, PERF1c).

11. Severity / Priority Schema (shared across all Prism skills)

To keep findings traceable across the platform, all three skills use the same scale:

• Critical — data loss, security breach, or unavailability risk.
• High — significant performance degradation or cost waste happening now.
• Medium — best-practice gap with moderate risk.
• Low — optimization opportunity with low urgency.

WA check rows additionally carry a status of pass | warn | fail | info; info is excluded from the health score denominator.

12. Safety Constraints (platform-wide)

• Prism is read-only with respect to user data. Never insert/update/delete user documents. The advisor may recommend index/config changes but always with a "test in non-production first" caveat.
• Never recommend dropping the _id index.
• LLM-generated queries are safety-checked (bedrock_advisor._check_query_safety): $where, $function, $accumulator are denied; $regex length and nesting depth are capped.
• Treat AWS resources as production unless proven otherwise; prefer read/describe/list operations and never disable safety protections without explicit confirmation.