AnomalyGen Release

Use this skill to build and validate AnomalyGen CUDA 12.8 containers. This is a release/build workflow, not a training or inference workflow.

Run every command from the repo root.

Docker privilege. All docker commands below assume your user is in the docker group (or the daemon is rootless). If your environment requires elevated privileges, prefix sudo to each docker invocation.

Container Modes

There are two image modes:

• Product container: protected runtime for users operating AnomalyGen through an agent. The image sets ANOMALYGEN_PRODUCT_MODE=1, runs as a non-root user, locks production code read-only, and keeps runtime artifacts writable.
• Develop container: writable development environment for developers using an agent to edit code. The image leaves ANOMALYGEN_PRODUCT_MODE unset and keeps production code paths writable.

Each mode is available in two variants:

• Standard: checkpoints are mounted at runtime (thin image, ~10 GB).
• Air-gapped: checkpoints are baked into the image (fat image, ~75 GB+). Use when the target environment cannot reach the network at runtime.

Builds should happen from a normal cloned repo or develop container where ANOMALYGEN_PRODUCT_MODE is unset. Do not build nested images from a product runtime container.

User-facing prompts:

Build anomalygen product container
Build anomalygen develop container
Generate airgapped docker image
Build airgapped product image
Build airgapped develop image

Scope

Allowed:

• Inspect docker/Dockerfile.cuda128 and docker/Dockerfile.cuda128.airgapped and release inputs.
• Build product or develop images (standard or air-gapped) with the helper scripts.
• Auto-download missing checkpoints when building an air-gapped image.
• Validate product image runtime guardrails.
• Validate develop image writability for code development.
• Report image tag, image id, mode, and validation results.

Do not:

• Run AnomalyGen training or SDG generation as part of release.
• Bake secrets, private datasets, or user experiment outputs into the image.
• Build images inside ANOMALYGEN_PRODUCT_MODE=1.
• Use arbitrary Docker commands beyond the build and validation steps unless the user explicitly asks.

Canonical Build Commands

Product container:

bash scripts/build_image.sh --mode product

Equivalent Docker command:

DATE_TAG="$(date -u +%Y%m%d)"
DOCKER_BUILDKIT=1 docker build \
    --target product \
    -f docker/Dockerfile.cuda128 \
    -t "paidf-anomalygen:${DATE_TAG}" \
    .

Develop container:

bash scripts/build_image.sh --mode develop

Use a minute-level tag when multiple builds may happen in one day:

bash scripts/build_image.sh \
    --mode product \
    --tag "$(date -u +%Y%m%d-%H%M)"

Air-Gapped Image

Use when the target environment has no network access and cannot pull checkpoints at runtime. The airgapped build uses docker/Dockerfile.cuda128.airgapped, which bakes all checkpoints into the image layers. The result is self-contained (~75 GB+).

Canonical command

bash scripts/build_airgapped_image.sh \
    --mode product

The script:

1. Checks all required checkpoints in checkpoints/ (the paths the Dockerfile COPYs in):
   • checkpoints/nvidia/Cosmos-Predict2-2B-Text2Image/model.pt
   • checkpoints/nvidia/Cosmos-Predict2-14B-Text2Image/model.pt
   • checkpoints/NVDINOV2/nv_dinov2_classification_model.ckpt
   • checkpoints/nvidia/C-RADIO-V3/model.safetensors
   • checkpoints/sam2/sam2.1_hiera_large.pt
   • checkpoints/facebook/dinov2-large/ (non-empty)
   • checkpoints/google-t5/t5-large/ or checkpoints/google-t5/t5-11b/ (at least one)
2. Auto-downloads any missing checkpoints via scripts/utilities/download_checkpoints.sh. This requires HF_TOKEN to be exported and huggingface-cli in PATH. If you do not want auto-download, pass --skip-download and download manually with the setup skill first.
3. Builds the airgapped image after all checkpoints are confirmed present.

Default image names:

Options:

bash scripts/build_airgapped_image.sh \
    --mode product|develop \
    --tag YYYYMMDD \
    --checkpoint-dir checkpoints \
    --skip-download

Running the air-gapped image

No volume mounts required — checkpoints live inside the image:

docker run --gpus all -it --rm --shm-size=16g \
    paidf-anomalygen-airgapped:<tag> bash

Delivering to an air-gapped host

# on the build host
docker save paidf-anomalygen-airgapped:<tag> | gzip \
    > paidf-anomalygen-airgapped-<tag>.tar.gz

# transfer the .tar.gz to the air-gapped host, then:
docker load < paidf-anomalygen-airgapped-<tag>.tar.gz
docker run --gpus all -it --rm --shm-size=16g \
    paidf-anomalygen-airgapped:<tag> bash

Product Filesystem Policy

The product image should run as a non-root runtime user. Only production implementation paths should be non-writable. Runtime data, artifacts, checkpoints, caches, and logs should remain writable.

Production paths expected to be non-writable in product containers and writable in develop containers:

/workspace/paidf-anomalygen/cosmos_predict2/
/workspace/paidf-anomalygen/scripts/anomaly_gen/
/workspace/paidf-anomalygen/scripts/utilities/
/workspace/paidf-anomalygen/.agents/skills/
/workspace/paidf-anomalygen/README.md
/workspace/paidf-anomalygen/CLAUDE.md
/workspace/paidf-anomalygen/docker/Dockerfile*
/workspace/paidf-anomalygen/requirements*.txt
/workspace/paidf-anomalygen/cosmos-predict2-cuda128.yaml

Runtime paths expected to be writable in both modes:

/workspace/paidf-anomalygen/results/
/workspace/paidf-anomalygen/ag_configs/
/workspace/paidf-anomalygen/ag_inference/
/workspace/paidf-anomalygen/datasets/
/workspace/paidf-anomalygen/checkpoints/
/workspace/paidf-anomalygen/logs/
/workspace/paidf-anomalygen/tmp/
/tmp/
the runtime user's `$HOME` (`/home/anomalygen/` in product images)
the runtime user's `$HOME/.cache` (used by HF, torch.compile, and triton)

In addition to the top-level directories above, the validator probes a set of nested paths that AnomalyGen actually writes to during training, SDG, and refine. Top-level writability alone can miss ownership issues on mounted volumes or pre-created subdirectories, so each of these must also be writable:

# Training output
/workspace/paidf-anomalygen/results/anomaly_gen/_permission_test/checkpoints/model

# SDG output (original and searched buckets, plus per-round refine)
/workspace/paidf-anomalygen/results/_permission_test/original/reconstructed_image
/workspace/paidf-anomalygen/results/_permission_test/original/original_mask
/workspace/paidf-anomalygen/results/_permission_test/original/overlay_image
/workspace/paidf-anomalygen/results/_permission_test/searched/reconstructed_image
/workspace/paidf-anomalygen/results/_permission_test/rounds/round_001/sdg/reconstructed_image

# AMP cache
/workspace/paidf-anomalygen/ag_inference/_permission_test/amp
/workspace/paidf-anomalygen/ag_inference/_permission_test/resized_masks

# torch.compile / triton cache
$HOME/.cache/paidf-anomalygen/torch_inductor
$HOME/.cache/paidf-anomalygen/triton

# Hugging Face cache
$HOME/.cache/huggingface

The validator also probes that SDG_result.csv can be created and written under each of the SDG bucket roots:

/workspace/paidf-anomalygen/results/_permission_test/original/SDG_result.csv
/workspace/paidf-anomalygen/results/_permission_test/searched/SDG_result.csv
/workspace/paidf-anomalygen/results/_permission_test/rounds/round_001/sdg/SDG_result.csv

This SDG_result.csv probe exists so the validator can fail fast on read-only mounts where SDG would later be unable to record its index. It does not mean this release skill runs SDG itself; SDG generation remains out-of-scope for the release workflow (see Scope above) and is handled by the sdg-inference skill.

If a product image does not set ANOMALYGEN_PRODUCT_MODE=1, runs as root, or leaves production code writable, treat the image as not productized.

Pre-Build Checklist

Before building:

1. Verify these files exist:
   • docker/Dockerfile.cuda128
   • cosmos-predict2-cuda128.yaml
   • requirements-conda-cuda128.txt
2. Verify the Docker build context does not intentionally include secrets:
   • .env
   • token files
   • SSH keys
   • private datasets
   • user result folders
3. Decide checkpoint strategy for product images:
   • Thin image: checkpoints mounted or downloaded at runtime.
   • Fat image: checkpoints copied into image. This is large and should be explicitly requested.

Validation

After building a product image:

bash scripts/validate_image_permissions.sh \
    --mode product \
    "paidf-anomalygen:${DATE_TAG}"

After building a develop image:

bash scripts/validate_image_permissions.sh \
    --mode develop \
    "paidf-anomalygen-dev:${DATE_TAG}"

If validation fails, do not call the image ready for its intended mode.

Running the Container

Warning: always pass --shm-size (minimum 16g). PyTorch DataLoader uses /dev/shm for multiprocessing shared memory; the Docker default of 64 MB causes "Bus error" crashes or silent hangs during training and inference. Remind the user of this flag whenever reporting a completed build.

Product container:

TAG="paidf-anomalygen:$(date -u +%Y%m%d)"
REPO="$PWD"
docker run --rm -it --gpus all --shm-size=16g \
    -v "${REPO}/checkpoints:/workspace/paidf-anomalygen/checkpoints" \
    -v "${REPO}/datasets:/workspace/paidf-anomalygen/datasets" \
    -v "${REPO}/results:/workspace/paidf-anomalygen/results" \
    "${TAG}" \
    bash

Develop container:

TAG="paidf-anomalygen-dev:$(date -u +%Y%m%d)"
REPO="$PWD"
docker run --rm -it --gpus all --shm-size=16g \
    -v "${REPO}:/workspace/paidf-anomalygen" \
    "${TAG}" \
    bash

Release Summary

Report:

Image: paidf-anomalygen:<tag>
Mode: product | develop
Image ID: <docker image id>
Dockerfile: docker/Dockerfile.cuda128
ANOMALYGEN_PRODUCT_MODE: set | unset | failed validation
Production code: non-writable | writable | failed validation
Runtime paths: writable | failed validation
Checkpoint strategy: thin | fat | unknown
Notes: <warnings, if any>