Skill

doris-debug-resource-isolation

Name: doris-debug-resource-isolation
Author: Apache Software Foundation

debug Apache Doris resource isolation issues

Published by Apache Software Foundation Updated Jul 25

Covers Performance Operations Debugging

Description

Use for Doris Workload Group / resource tag queue starvation, CPU/memory isolation leaks, and workload policy debugging. Commands: SHOW WORKLOAD GROUPS, EXPLAIN resource.

SKILL.md

Resource Isolation (Workload Group)

Causes

ID	Cause	Evidence	Source anchor
A	Queue starvation	Queries queued in `SHOW WORKLOAD GROUPS`; `QueueTime` growing	`WorkloadGroupMgr.java`
B	CPU oversubscription	One WG consuming >90% CPU despite limits	`CgroupCpuCtl.cpp`
C	Memory leak across groups	MemTracker shows one WG exceeding `memory_limit`	`WorkloadGroupMemMgr.cpp`
D	Tag routing wrong	Query hits wrong BE set; ResourceTag mismatch	`ResourceTag.java`, `Tag.java`
E	WG not active / cgroup missing	`CREATE WORKLOAD GROUP` had no effect	`WorkloadGroupMgr.cpp` cgroup init

10 min triage

-- Workload group status
SHOW WORKLOAD GROUPS\G
-- Check columns: Id, Name, ActiveQueries, QueuedQueries, CpuShares, MemoryLimit

-- Which WG is my query in?
SELECT * FROM information_schema.workload_group_resource_usage;

-- Workload policies (routing rules)
SHOW WORKLOAD POLICY\G

# BE cgroup inspection
cat /sys/fs/cgroup/cpu/doris/<wg_id>/cpu.shares
cat /sys/fs/cgroup/memory/doris/<wg_id>/memory.limit_in_bytes

./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "workload_group"

Cause A — Queue starvation

-- See who's queued
SHOW WORKLOAD GROUPS\G

-- Raise queue concurrency
ALTER WORKLOAD GROUP etl_wg SET (
    "max_concurrency" = "8",
    "max_queue_size" = "100",
    "queue_timeout" = "300"
);

Check be/src/pipeline/task_scheduler.cpp — the pipeline scheduler respects cgroup shares; if a high-share WG starves a low-share one, the low-share WG's queue backs up.

Cause B — CPU oversubscription

-- Create a hard CPU cap
ALTER WORKLOAD GROUP report_wg SET (
    "cpu_share" = "1024",
    "cpu_hard_limit" = "200%"   -- 2 cores max
);

-- Force query into specific WG
SET workload_group = 'report_wg';

cgroup v1 vs v2 behavior differs. Confirm which is active on BE hosts:

mount | grep cgroup
ls /sys/fs/cgroup/cpu/doris/   # cgroup v1
ls /sys/fs/cgroup/doris/       # cgroup v2

Cause D — Tag routing

-- Check BE tags
SHOW BACKENDS\G  -- look at Tag column

-- Check WG tag constraint
SHOW WORKLOAD GROUPS\G  -- look at ResourceTag

If a workload group specifies a tag but no BEs carry that tag, queries queue forever. Fix:

ALTER SYSTEM MODIFY BACKEND "host:9050" SET ("tag.tag.location" = "high_mem");

Enabling cgroup on BE

# be.conf
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris
enable_workload_group = true

Without cgroup configured, cpu_share is a cooperative hint, not a hard guarantee.

Source

be/src/common/config.cpp — enable_workload_group, cgroup paths
be/src/pipeline/task_scheduler.cpp — pipeline scheduling
fe/.../workloadgroup/WorkloadGroupMgr.java — WG lifecycle
fe/.../resource/Tag.java — resource tags
be/src/util/cgroup_util.cpp — cgroup v1/v2 adapter

More skills from the doris-skills repository

View all 14 skills

SkillTagsUpdated

More from Apache Software Foundation

View publisher

SkillTagsUpdated