Skill

deepstream-sop

Name: deepstream-sop
Author: NVIDIA

deploy and monitor DeepStream inference services

Published by NVIDIA Updated Jul 14

Covers Observability Deployment NVIDIA FastAPI Debugging

Description

Use this skill when building, deploying, evaluating, debugging, or measuring latency for the DeepStream SOP Inference Microservice — a GPU-accelerated FastAPI service that detects whether operators perform assembly-line steps in order via event boundary detection (GEBD) plus VLM classification. Trigger even if the user does not name it: verify operator step sequence, detect missing or out-of-order SOP steps, score factory/work-cell video for procedure compliance, run VLM-based SOP checking on industrial cameras, or call /v1/chat/completions with a file, RTSP, or Basler camera. Also trigger for its internals: SOPVideoProcessor, DeepStream GEBD model (e.g. DDM) via Triton CAPI, nvds_custom_postprocess, Cosmos Reason 1/2 vLLM, SSE streaming, Kafka NvProto/JSON output, Basler/Pylon camera + emulation, Docker compose, chunk-level latency. Do NOT trigger for generic DeepStream pipelines, object detection/tracking, NIM imports, or video summarization.

SKILL.md

DeepStream SOP Inference Microservice Skill

This skill guides AI coding assistants in building, extending, and debugging the NVIDIA DeepStream SOP (Standard Operating Procedure) Inference Microservice — a GPU-accelerated pipeline for temporal action detection and VLM-based SOP compliance monitoring on industrial video feeds.

Reference repository: https://github.com/NVIDIA/sop-monitoring-blueprints/tree/main/microservices/sop-inference-bpLocal reference code: sop-inference-bp/ directory (from a local clone of the repository)

Models

Model-agnostic at both inference stages — swap via env var (and Triton dir for GEBD).

Stage	Role	Model class	Default	Swap via
Stage 1 (CV)	Per-frame boundary scoring → chunk segmentation	Generic Event Boundary Detection (GEBD)	DDM (MCG-NJU/DDM) via Triton Python backend	Replace `triton_model_repo/<model>/` + `DDM_MODEL_PATH` (§ 5)
Stage 3 (VLM)	Per-chunk action classification	Vision-language model via vLLM	Cosmos Reason 1 7B (Reason 2 also supported)	Set `VLLM_MODEL_PATH` to a different HF ID or local path

"GEBD" = swappable Stage-1 slot; "DDM" = the default architecture (terms used interchangeably).

Chunking is selectable per request (§ 2): default ddm-net uses GEBD; uniform produces fixed-length chunks and bypasses Stage-1 GEBD (§ 3, § 6). DDM temporal window is configurable via FRAMES_PER_SIDE / SEQUENCE_BATCH (§ 4, § 5), with optional TensorRT (§ 5).

Architecture Overview

Runs in a Docker container (nvds-action-sop) alongside a Kafka container. Full diagram: references/sop_architecture.svg.

Data flow through the 4-stage SOPVideoProcessor pipeline (per-request):

Input Sources                    Docker Container: nvds-action-sop
─────────────                    ──────────────────────────────────────────────────
Video Files ──┐                  FastAPI Server (port 8300)
RTSP Streams ─┤── base64/       ├─ /v1/chat/completions → SOPProcessManager
Basler Camera ┘   file/rtsp/       │
                  camera           │ ModelInitializer: VLM first, then DDM dummy pipeline
                                   │ 4 Thread Pools: cv(32), clip(32), vlm(64), vlm_req(64)
                                   │
                                   ▼ SOPVideoProcessor (per-request)
                                   ┌────────────────────────────────────────────────┐
                                   │ Stage 1: DeepStream Pipeline (GPU)             │
                                   │   Source → nvstreammux → tee1                  │
                                   │    ├─[inference] queue1 → nvdspreprocess       │
                                   │    │  → nvinferserver (Triton CAPI + DDM)      │
                                   │    │  → InferOutputTensorParser → score_queue  │
                                   │    ├─[frames]  queue3 → nvvideoconvert         │
                                   │    │  → capsfilter → appsink                   │
                                   │    │  → DecodedFrameRetriever → frame_queue    │
                                   │    └─[RTSP out] queue → convert → H.264 enc    │  (optional, § 18)
                                   │       → rtppay → udpsink → RTSPServer (§ 18)   │  opt-in only
                                   │              │ boundary scores                 │
                                   │              ▼                                 │
                                   │ Stage 2: Clip Post-Process                     │
                                   │   Boundary detection → chunk segmentation      │
                                   │              │ video frames + timestamps        │
                                   │              ▼                                 │
                                   │ Stage 3: VLM Inference                         │
                                   │   Embedded vLLM (Cosmos Reason 1/2)            │
                                   │   Frame sampling at VLM_FPS → classification   │
                                   │              │ action labels                    │
                                   │              ▼                                 │
                                   │ Stage 4: SOP Checker                           │
                                   │   Sequence validation → missing/misordered     │
                                   │              │ chunk results                    │
                                   │              ▼                                 │
                                   │         final_queue                            │
                                   └────────────────────────────────────────────────┘
                                          │
Output                                    ▼
──────                             ┌─────────────────┐
SSE Stream (chat.completion.chunk) │ Kafka Messages   │
Non-streaming (chat.completion)    │ (JSON/Protobuf)  │
Prometheus metrics (/v1/metrics)   └────────┬────────┘
                                            ▼
                                   Docker Container: kafka
                                   (apache/kafka:3.7.0)

Section Index

Each section is a standalone file in references/ — load only what your task needs.

§	File	Responsibility
1	`skill_01_fastapi_endpoints.md`	FastAPI endpoints, server init, Prometheus metrics
2	`skill_02_pydantic_schemas.md`	Request/response Pydantic models (`api_types.py`)
3	`skill_03_deepstream_pipeline.md`	DeepStream pyservicemaker pipeline, tensor parser, dummy pipeline
4	`skill_04_config_templates.md`	nvdspreprocess / nvinferserver config templates + rendering
5	`skill_05_triton_ddm_model.md`	Triton model repo, config.pbtxt, model.py, ddm_net.py
5b	`skill_05b_custom_postprocess.md`	C++ postprocess plugin, Makefile, IOptions API
6	`skill_06_sop_process_manager.md`	SOPProcessManager, SOPVideoProcessor, VLLMInference, Kafka
6b	`skill_06b_sop_checker.md`	SOP sequence and checker compliance: MissingNumberDetector, SopCheckerCache, SopCheckerRequest/Response
7	`skill_07_sse_streaming.md`	SSE generator, stream response formatting, dummy test mode
8	`skill_08_basler_camera.md`	Basler camera support, Pylon SDK, emulation, formats
9	`skill_09_docker_build_deploy.md`	Docker build, deploy, .env configuration
10	`skill_10_test_suite.md`	Test suite coverage, assertions, running tests
11	`skill_11_env_variables.md`	All environment variables reference
12	`skill_12_evaluation_workflow.md`	End-to-end eval workflow: static checks, build, launch, tests, API/camera/Kafka checks, report
13	`skill_13_verification_curl.md`	Verification steps and curl examples
14	`skill_14_implementation_checklist.md`	Implementation checklist: file copy list, generated files, Docker prereqs, verification
15	`skill_15_latency_measurement.md`	TTFC and C2C latency measurement for file input via SSE streaming
16	`skill_16_message_schema.md`	Kafka message schema selection (`JSON` default vs `NvProtoSchema`) and extending messages with custom data
17	`skill_17_camera_latency_measurement.md`	Camera / live-stream chunk_e2e latency measurement using internal pipeline timestamps
18	`skill_18_rtsp_streaming_output.md`	OPT-IN RTSP streaming output: `tee1`-tap re-stream, `RTSPStreamingServer`, `SW_ENCODER` toggle. Generate only when user explicitly requests RTSP

For end-to-end evaluation, read § 12 first; load build/test/curl/latency/camera/Kafka as needed.

§ 18 is opt-in — generate only when the user explicitly requests RTSP output; otherwise skip § 18 and the RTSP_* rules below.

Key Files Map

The full source-to-target file mapping lives in skill_14_implementation_checklist.md:

Files copied verbatim from references/ (non-trivial algorithms — cycle detection, qwen_vl_utils preprocessing, DeepStream IOptions API, protobuf sources) with the rationale per file.
Files copied as adaptable templates (Dockerfile, compose.yaml, Triton config and model.py, ddm_net.py, Pylon emulation config, etc.).
Files generated from skill sections — each annotated with the Critical Rules below that the generation must follow exactly.
Docker build prerequisites and post-build verification checklist.

Config files (nvds_preprocess_template.txt, nvds_inference_template.txt, vlm_prompts.txt) are used as-is from configs/.

When skill_06b is loaded, read configs/actions.json from the project root and run the § 6b-G generation workflow to produce nvds_action_detector/missing_number_detector.py. If configs/actions.json is absent or invalid, fall back to copying the reference file.

Critical Rules

Each rule's full detail lives in the linked skill_NN_*.md reference file.

Tag	Rule summary	Details in
`MANAGER_INIT_IN_MAIN`	`SOPProcessManager` init in `main()` before `uvicorn.run()` — not inside `lifespan()`	`skill_01_fastapi_endpoints.md`
`NAMED_KWARGS`	`create_video_processor()` uses named kwargs; camera args as separate kwargs	`skill_06_sop_process_manager.md`
`LIVE_REQUIRES_STREAM_TRUE`	`stream: true` required for live inputs (RTSP / camera)	`skill_08_basler_camera.md`
`VLM_DISABLED_DISABLES_SOP_CHECKER`	`DISABLE_VLM_INFERENCE=true` auto-disables SOP checker at import	`skill_06_sop_process_manager.md`
`CHUNK_PARAMS_MAX_LENGTH`	`ChunkParams.max_length_sec` = 10s internal; 60s API default	`skill_06_sop_process_manager.md`
`VLM_WARMUP_BEFORE_DDM`	`ModelInitializer`: VLM warmup FIRST, then CV dummy pipeline	`skill_06_sop_process_manager.md`
`VLM_WARMUP_3_FRAMES`	VLM warmup needs 3 frames (`torch.zeros`) — Qwen3VL hangs on < 3	`skill_06_sop_process_manager.md`
`THREAD_POOL_SIZES`	4 thread pools: `cv`(32), `clip`(32), `vlm_inference`(64), `vlm_request`(64)	`skill_06_sop_process_manager.md`
`MEDIA_INFO_PYMEDIAINFO`	Media info via `pymediainfo`; live sources set fps=30/duration=inf directly	`skill_06_sop_process_manager.md`
`CAMERA_EMULATION_PYLON_CAMEMU`	`PYLON_CAMEMU=1` for camera emulation (serial `0815-0000`)	`skill_08_basler_camera.md`
`DEEPSTREAM_LIB_HIDE`	DeepStream lib hide trick: rename lib → lib.tmp during gst-plugin-pylon build	`skill_08_basler_camera.md`
`VLM_REAL_GPU_FRAMES`	VLM uses real GPU frames via `DecodedFrameRetriever`; never `torch.zeros` for inference	`skill_06_sop_process_manager.md`
`BUFFER_RETRIEVER_STATIC_BASE`	`DecodedFrameRetriever` MUST inherit `BufferRetriever` statically via `super().__init__()`; runtime `__class__.__bases__` mutation hangs `pipeline.attach()`	`skill_06_sop_process_manager.md`
`FRAME_RETRIEVER_PRIORITY`	`create_inference_pipeline`: `frame_retriever=` kwarg takes priority over `frame_queue`	`skill_03_deepstream_pipeline.md`
`MUX_ORIGINAL_RESOLUTION`	nvstreammux uses original resolution (not 224); pass `mux_width/mux_height` from `get_media_info()` (probe live RTSP for non-camera inputs; camera path unaffected)	`skill_03_deepstream_pipeline.md`, `skill_06_sop_process_manager.md`
`FILE_URI_NO_DOUBLE_PREFIX`	`create_inference_pipeline` file source: check `file_path.startswith("file://")` before prepending — API passes `file://` URLs directly	`skill_03_deepstream_pipeline.md`
`CLEANUP_ON_DISCONNECT`	Pipeline cleanup on client disconnect via `trigger_stop_processors` in `try/finally`	`skill_07_sse_streaming.md`
`UNIFIED_CLIP_POST_PROCESS`	Unified `clip_post_process()` for file + live; `stop()` puts `None` in `_score_queue`	`skill_06_sop_process_manager.md`
`ABORT_INFLIGHT_VLM`	Abort in-flight VLM requests on `stop()` via `llm.abort(req_id)`	`skill_06_sop_process_manager.md`
`LOGGER_EXPORT_GET_LOGGER`	`ds_logger.py` must export `get_logger`	`skill_06_sop_process_manager.md`
`KAFKA_USE_CREATE_PRODUCER`	Kafka: use `create_producer()` from `messager.py`; no `Messager` class	`skill_06_sop_process_manager.md`
`USER_PROMPT_PRIORITY`	User request text takes priority over `VLM_PROMPT_PATH` file; `{"type":"text"}` in the request overrides the config-file prompt	`skill_06_sop_process_manager.md`
`EVAL_USE_CONFIG_PROMPT`	Eval/latency requests omit request text by default so the VLM uses `VLM_PROMPT_PATH`	`skill_12_evaluation_workflow.md`, `skill_13_verification_curl.md`, `skill_15_latency_measurement.md`, `skill_17_camera_latency_measurement.md`
`CHUNK_SCHEMA_FIELD_NAMES`	Chunk schema: `chunk_idx`, `cv_boundary_score`, `checker_result`; summary `chunk_idx=-1`	`skill_06_sop_process_manager.md`
`SEQUENTIAL_FRAME_DRAIN`	Drain `decoded_frame_queue` (FIFO, shared across chunks) in a SINGLE thread and submit VLM per chunk incrementally; parallel drain steals frames → 0-frame chunks / wrong VLM input	`skill_06_sop_process_manager.md`
`WALL_CLOCK_BEFORE_GPU`	`DecodedFrameRetriever.consume()`: capture `wall_clock_entry = time.time()` BEFORE GPU dlpack; queue 3-tuple `(timestamp, wall_clock_entry, tensor)`	`skill_06_sop_process_manager.md`, `skill_17_camera_latency_measurement.md`
`CHUNK_E2E_PIPELINE_TIMESTAMPS`	Write `pipeline_chunk_end_timestamp` (last frame wall_clock) and `pipeline_vlm_ready_timestamp` (`tm_e2e.now()`) into `chunk_info` for camera latency (§ 17)	`skill_06_sop_process_manager.md`, `skill_17_camera_latency_measurement.md`
`VLM_INFERENCE_REQUIRED_KWARGS`	Every `VLLMInference.inference()` call must pass `video_fps`, `system_prompt`, `max_completion_tokens`	`skill_06_sop_process_manager.md`
`UNIFORM_CHUNKING_BYPASSES_DDM`	`chunking_options.algorithm="uniform"` → fixed-length chunks; `create_inference_pipeline(uniform_chunk=True)` skips DDM but keeps `tee1` fanout; Stage 2 uses `uniform_clip_post_process`	`skill_02_pydantic_schemas.md`, `skill_03_deepstream_pipeline.md`, `skill_06_sop_process_manager.md`
`DDM_TEMPORAL_CONFIGURABLE`	`SLIDING_WINDOWS_SIZE = 2*FRAMES_PER_SIDE + SEQUENCE_BATCH` rendered into preprocess/nvinferserver (no hard-coded 18); Triton `config.pbtxt` sequence dim `-1`	`skill_04_config_templates.md`, `skill_05_triton_ddm_model.md`
`DDM_TRT_OPTIONAL_PATH`	`DDM_TRT_OPTIMIZATION=true` runs DDM via TensorRT (per-thread contexts, fixed batch = SEQUENCE_BATCH); PyTorch fallback; never both. PyTorch is default	`skill_05_triton_ddm_model.md`
`DDM_TRT_STREAM_ORDERING`	`DDMTensorRTEngine.infer()`: `wait_stream(current)` → `execute_async_v3` → `torch.cuda.synchronize(device)` (NOT per-stream). Per-stream sync leaves TRT aux-stream work in flight → gst-CV SIGSEGV (NVBug 6289256)	`skill_05_triton_ddm_model.md`
`METADATA_LICENSE_FROM_FILE`	`/v1/metadata` reads `licenseInfo` from `DS_SOP_LICENSE_PATH` (default `/opt/nvidia/nvds_sop/license.txt`); never hard-code license text	`skill_01_fastapi_endpoints.md`
`CAMERA_EMULATION_FRAMES_RGB`	Pylon emulation PNGs must be explicit 3-channel RGB (matches `Emulation_0815-0000.pfs PixelFormat=RGB8Packed`); generate via `nvvideoconvert ! videoconvert ! "video/x-raw,format=RGB" ! pngenc`	`skill_08_basler_camera.md`
`COMPOSE_ENV_PASSTHROUGH`	`docker compose` only substitutes `${VAR}` references; every runtime env var must be explicitly listed under `environment:` to reach the container.	`skill_09_docker_build_deploy.md`

The four RTSP_* rules below apply only when the optional RTSP streaming-output feature (§ 18) is requested. They do not apply to the default build — skip them if the user did not ask for RTSP output.

| RTSP_OUTPUT_TAPS_TEE1 | RTSP output branch links from the existing tee1 (added after the main inference link) only when rtsp_port is present. | skill_18_rtsp_streaming_output.md | | RTSP_LEAKY_QUEUE_TINY | RTSP branch queue must be leaky=2 + tiny cap (max-size-buffers=2) to prevent backpressure and NVMM pool exhaustion. | skill_18_rtsp_streaming_output.md | | RTSP_KEYINT_MAX_30 | RTSP H.264 encoder must set key-int-max=30 (and B-frames disabled) to allow downstream seeking. | skill_18_rtsp_streaming_output.md | | RTSP_ENCODER_FALLBACK | Select software/hardware H.264 encoder based on SW_ENCODER with MJPEG fallback. | skill_18_rtsp_streaming_output.md |

More skills from the skills repository

View all 285 skills

SkillTagsUpdated

More from NVIDIA

View publisher

SkillTagsUpdated