Apache Software Foundation logo

Skill

doris-debug-tablet

manage Apache Doris tablet and replica health

Covers Operations Database Debugging

Description

Use for Doris tablet/replica health, version skew, clone backlog, disk balance, and tablet repair. Commands: SHOW PROC tablet_health, SHOW TABLET, ADMIN REPAIR.

SKILL.md

Tablet & Replica

Quick scan

SHOW PROC '/cluster_health/tablet_health';
SHOW PROC '/statistic';
SHOW PROC '/tasks';

Causes

IDCauseEvidenceSource anchor
AReplica missing / unhealthySHOW PROC '/cluster_health/tablet_health' Health≠TOTALTabletHealth.java
BVersion skewtablet version lag > 3 behind quorumTabletInvertedIndex.cpp
CClone backlogSHOW PROC '/tasks' CLONE queue depth > 50CloneTask.cpp, BE clone_worker_count
DDisk skewSHOW PROC '/statistic' one disk > 85% while others < 50%DiskBalancer / balance config
ESingle tablet too largetablet size > compaction_tablet_size_thresholdconfig.cpp + compaction

10 min triage

-- 1. Global health snapshot
SHOW PROC '/cluster_health/tablet_health';

-- 2. Per-table replica drilldown
ADMIN SHOW REPLICA STATUS FROM db.table WHERE ReplicaStatus != 'OK';

-- 3. Task queue depth
SHOW PROC '/tasks';
# BE-side tablet count & version lag
./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "tablet|version|clone"
./scripts/doris-debug log-grep be/log --pack versions

# Check specific tablet distribution
curl -s "http://$FE:8030/api/tablets_distribution?db=db_name&table=tbl_name"

Cause A — Replica missing

-- Identify unhealthy replicas
ADMIN SHOW REPLICA STATUS FROM db.table WHERE ReplicaStatus != 'OK';

-- Repair with low priority (production safe)
ADMIN REPAIR TABLE db.table PRIORITY = 'LOW';

-- For urgent single-tablet repair
ADMIN REPAIR TABLE db.table PARTITION (p202401) PRIORITY = 'HIGH';

Check BE heartbeat_service_mgr.cpp — heartbeats from a BE with missing tablets won't self-heal if compaction or clone threads are exhausted.

Cause C — Clone backlog

Clone concurrency: clone_worker_count (default 3 in config.cpp). A high Clone backlog often means:

  1. Tablet migration triggered by balance_load_disk_safe_threshold / storage_high_watermark_usage_percent
  2. Clone threads can't keep up with the repair plan
# be.conf — raise clone workers cautiously
clone_worker_count = 6

Cause D — Disk skew

SHOW PROC '/statistic';          -- per-disk usage
SHOW PROC '/backends';           -- per-BE disk UsedPct
# fe.conf
balance_load_disk_safe_threshold = 0.5
storage_high_watermark_usage_percent = 80

Manual rebalance: ADMIN SET FRONTEND CONFIG ("disable_balance" = "true"); then selective ADMIN REBALANCE DISK.

Source

  • fe/.../master/TabletHealth.java — cluster_health proc
  • fe/.../clone/TabletSchedCtx.java
  • be/src/agent/task_worker_pool.cpp — clone worker pool
  • be/src/common/config.cppclone_worker_count, storage_root_path
  • be/src/olap/tablet_manager.cpp — tablet version tracking

© 2026 YourAI.tools. Every skill from an identity-verified publisher.

Independent catalog. Not affiliated with, endorsed by, or sponsored by Anthropic or any listed publisher. All trademarks belong to their respective owners.