Skill

doris-debug-import

Name: doris-debug-import
Author: Apache Software Foundation

troubleshoot Apache Doris data import issues

Published by Apache Software Foundation Updated Jul 25

Covers Data Engineering Database Debugging

Description

Use for Doris Stream/Broker/Routine Load issues and Group Commit async_mode WAL pile-up. Tunables: table group_commit_interval_ms / group_commit_data_bytes; BE group_commit_insert_threads, group_commit_wal_path, group_commit_wal_max_disk_limit.

SKILL.md

Import / Group Commit

async_mode WAL semantics

Write WAL → RPC returns → background group commit → delete_wal on success
(group_commit_mgr.cpp). Drain rate = commit finish rate ≠ compaction GB/s.

Knobs (config.cpp)

Config	Default	Role
`group_commit_insert_threads`	10	commit worker pool
`group_commit_relay_wal_threads`	10	WAL replay
`group_commit_wal_path`	under storage_root	WAL disks
`group_commit_wal_max_disk_limit`	10%	WAL quota
table `group_commit_data_bytes`	often 64MB	size flush
table `group_commit_interval_ms`	often 10000	time flush

High MB/s guidance

Split load across BEs (WAL is local to coordinator BE).
wal-du on each BE storage path.
Prefer dedicated NVMe for group_commit_wal_path.
A/B group_commit_data_bytes (128MB vs 512MB) — size threshold dominates when ingest ≫ interval.
sync_mode or client throttle for backpressure.

./scripts/doris-debug wal-du /path/to/be/storage
./scripts/doris-debug log-grep be/log --pack group_commit

More skills from the doris-skills repository

View all 14 skills

SkillTagsUpdated

More from Apache Software Foundation

View publisher

SkillTagsUpdated