
Covers Benchmarking Engineering Testing
Description
Create, mine, and validate new benchmark tasks and task suites.
SKILL.md
Skill: Task Scaffolding & Authoring
Scope
Use this skill when the user asks to:
- Create new benchmark tasks or suites
- Mine tasks from source repositories
- Scaffold task definitions with metadata and structure
- Validate task authoring and compliance
- Update task metadata and verifiers
Canonical Commands
# Scaffold new task from template
python3 scripts/authoring/scaffold_task.py --suite csb_sdlc_debug --task-name my-task-001
# Mine tasks from repo
python3 scripts/authoring/mine_bug_tasks.py --source-repo https://github.com/org/repo
# Validate task structure
python3 scripts/authoring/validate_tasks_preflight.py --task benchmarks/csb/debug/my-task-001
# Create task specifications from ground truth
python3 scripts/infra/build_daytona_registry.py --task benchmarks/csb/debug/my-task-001
# Backfill task metadata
python3 scripts/evaluation/backfill_instruction_artifacts.py --task-id my-task-001
Task Definition Structure
benchmarks/csb/CATEGORY/TASK_ID/
├── task.toml # Metadata: language, difficulty, time_limit_sec
├── instruction.md # Task description and requirements
├── instruction_mcp.md # MCP-specific retrieval hints
├── environment/
│ ├── Dockerfile
│ └── Dockerfile.sg_only # Sourcegraph-only variant
├── tests/
│ ├── test.sh # Main verifier
│ ├── oracle_checks.py # Ground truth validation
│ ├── task_spec.json # Task contract
│ └── ground_truth.json # Reference solution
└── reviewers.json # Metadata: author, reviewer
Task Authoring Steps
- Create — scaffold template with metadata
- Write — instruction.md, environment/Dockerfile
- Implement — test.sh and oracle_checks.py
- Validate —
validate_tasks_preflight.py - Review — ensure compliance before inclusion
Related Skills
/audit— validate task definitions/run— execute new tasks/evaluate— score task results
More skills from the CodeScaleBench repository
View all 9 skillsSkillTagsUpdated
audit
audit repository health and benchmark integrity
Jul 17AuditBenchmarkingQAevaluate
score traces and evaluate benchmark results
Jul 16BenchmarkingData AnalysisEngineeringEvalsinfra
check infrastructure and system dependencies
Jul 16AuditEngineeringInfrastructureMCPnext
plan benchmarking and coverage tasks
Jul 17BenchmarkingPlanningStrategyreport
generate CodeScaleBench evaluation reports
Jul 16AnalyticsBenchmarkingReportingrun
manage CodeScaleBench benchmark runs
Jul 16AutomationBenchmarkingTesting