
Skill
nemo-relay-plugin-observability
configure NeMo Relay observability
Description
Use this skill when choosing or configuring NeMo Relay observability through the built-in plugin, subscribers, or exporters, including raw ATOF events, ATIF trajectories, OpenTelemetry traces, OpenInference export, or custom event handling.
SKILL.md
Configure Observability Plugins
Start with one exporter managed by the built-in Observability plugin. This is the default for reusable process configuration and the best first plugin for most users because it makes Relay's captured activity visible. Choose one proof output before layering additional telemetry destinations.
Use manual subscriber or exporter APIs only when a test, script, or application needs direct control over registration names, collection windows, or flush timing. Both paths consume the same canonical event stream.
Choose The Output
Select the output that best matches the user's immediate inspection target:
- Console or custom event handling Use a manual subscriber for short-lived in-process inspection.
- Raw canonical lifecycle events
Use ATOF JSONL; read
references/atof.md. - Portable execution trajectories
Use ATIF; read
references/atif.md. - General OTLP tracing
Use OpenTelemetry; read
references/opentelemetry.md. - OpenInference-aware backends
Use OpenInference; read
references/openinference.md.
Choose one output first and verify it before adding another. ATOF is the default local proof because it preserves the raw event stream with the least translation. Use synthetic, non-sensitive payloads for the first proof. Add and verify sanitization before exporters receive production payloads, and never display complete event records while validating an exporter.
Embedded Event And Subscriber Model
Use this model when explaining how capture and export relate:
- NeMo Relay emits one canonical event stream from scopes, marks, managed tool calls, managed LLM calls, middleware, and manual lifecycle APIs.
- Subscribers consume events without defining the event model. Multiple subscribers can observe the same stream for logging, export, analytics, or diagnostics.
- Global subscribers remain active process-wide until removed.
- Scope-local subscribers are owned by one active scope and disappear when that scope closes.
- Plugin-installed subscribers are reusable, configuration-driven runtime components.
- Exporter-oriented subscribers preserve raw ATOF or translate the event stream into ATIF, OpenTelemetry, or OpenInference output.
- Event payloads reflect sanitized post-guardrail input and output when calls use managed helpers or manual lifecycle params provide those fields.
- LLM annotations follow the freshness rules:
- Each owning agent scope starts fresh, and a
compactionmark refreshes it. - The first subsequent LLM start retains complete annotation history. Later starts retain system instructions, the latest user message, and every following assistant or tool message.
- When a request codec supplies an annotation, Relay applies the same event-only projection to provider-shaped event input without changing provider execution.
- Each owning agent scope starts fresh, and a
- Event fields include semantic input/output through the ATOF
datafield, typed profile data such asmodel_nameandtool_call_id, and codec-provided annotated LLM request/response data for in-process subscribers and exporters. - First-class skill tools and the requests to read a complete
SKILL.mdautomatically emitskill.loadmarks under the tool span. The payload contains onlyskill_name; metadata records the load source and tool name. Partial reads do not count, and ambiguous slash-command expansions use the separateskill.load.inferredname. The eager mark remains present if tool execution later fails.
Shared Lifecycle
- Create the exporter or subscriber.
- Register it with a unique name before the relevant scoped work.
- Run NeMo Relay-instrumented work inside scopes.
- Flush if deterministic delivery is needed and the binding supports it.
- Deregister it, then shut it down when the process or subsystem is done.
Binding Names
Use the names exported by the selected language binding:
- Python:
nemo_relay.subscribers.register(...),AtofExporter,AtifExporter,OpenTelemetrySubscriber, andOpenInferenceSubscriber - Node.js: root exports
registerSubscriber(...),AtofExporter,AtifExporter,OpenTelemetrySubscriber, andOpenInferenceSubscriber - Rust:
nemo_relay::api::subscriberandnemo_relay::observability::* - Go: source-first wrappers expose equivalent register, exporter, and subscriber lifecycle methods
Load A Reference When
Load only the reference required by the selected output:
- Load
references/atof.mdfor raw JSONL events used in local debugging or offline inspection. - Load
references/atif.mdfor ATIF trajectories. - Load
references/opentelemetry.mdfor OTLP/OpenTelemetry traces. - Load
references/openinference.mdfor OpenInference semantic traces.
Use Another Skill When
Choose another skill when the task belongs to an adjacent workflow:
- Use
nemo-relay-plugin-buildto package subscriber-based export behavior as a reusable plugin. - Use
nemo-relay-get-startedornemo-relay-instrument-callswhen no scope, tool call, or LLM call has been instrumented. - Use
nemo-relay-debug-runtime-integrationto diagnose missing telemetry.
Related Skills
Use these skills for adjacent workflows:
- Instrument application calls with
nemo-relay-instrument-calls. - Add typed wrappers with
nemo-relay-instrument-typed-wrappers. - Package reusable behavior with
nemo-relay-plugin-build. - Diagnose missing events with
nemo-relay-debug-runtime-integration.
More skills from the NeMo-Relay repository
View all 10 skillsnemo-relay-debug-runtime-integration
debug NeMo Relay runtime integrations
Jul 17DebuggingEngineeringMiddlewareNVIDIAnemo-relay-get-started
get started with NeMo Relay
Jul 23CLINVIDIAOnboardingPythonnemo-relay-install
install NeMo Relay for various environments
Jul 17CLIDeploymentNode.jsNVIDIA +2nemo-relay-instrument-calls
instrument LLM calls with NeMo Relay
Jul 17AutomationLLMMiddlewareNVIDIAnemo-relay-instrument-context-isolation
isolate NeMo Relay execution contexts
Jul 17ConcurrencyEngineeringMiddlewareNVIDIAnemo-relay-instrument-typed-wrappers
instrument NeMo Relay typed wrappers
Jul 23API DevelopmentJSONMiddlewareNVIDIA
More from NVIDIA
View publishernemoclaw-user-guide
retrieve NemoClaw documentation and configuration
NemoClaw
Jul 20DocumentationMCPSearchmcore-build-and-dependency
manage Megatron-LM development environments
Megatron-LM
Jul 14ContainersDeploymentPythonmcore-bump-base-image
update NVIDIA PyTorch base images
Megatron-LM
Jul 14CI/CDDeploymentmcore-cicd
manage CI/CD pipelines for Megatron-LM
Megatron-LM
Jul 14CI/CDDeploymentGitHubmcore-create-issue
investigate CI failures and create issues
Megatron-LM
Jul 14DebuggingGitHubTriagemcore-linting-and-formatting
lint and format Megatron-LM code
Megatron-LM
Jul 14Best PracticesCode Analysis