
Skill
doris-debug-tablet
manage Apache Doris tablet and replica health
Description
Use for Doris tablet/replica health, version skew, clone backlog, disk balance, and tablet repair. Commands: SHOW PROC tablet_health, SHOW TABLET, ADMIN REPAIR.
SKILL.md
Tablet & Replica
Quick scan
SHOW PROC '/cluster_health/tablet_health';
SHOW PROC '/statistic';
SHOW PROC '/tasks';
Causes
| ID | Cause | Evidence | Source anchor |
|---|---|---|---|
| A | Replica missing / unhealthy | SHOW PROC '/cluster_health/tablet_health' Health≠TOTAL | TabletHealth.java |
| B | Version skew | tablet version lag > 3 behind quorum | TabletInvertedIndex.cpp |
| C | Clone backlog | SHOW PROC '/tasks' CLONE queue depth > 50 | CloneTask.cpp, BE clone_worker_count |
| D | Disk skew | SHOW PROC '/statistic' one disk > 85% while others < 50% | DiskBalancer / balance config |
| E | Single tablet too large | tablet size > compaction_tablet_size_threshold | config.cpp + compaction |
10 min triage
-- 1. Global health snapshot
SHOW PROC '/cluster_health/tablet_health';
-- 2. Per-table replica drilldown
ADMIN SHOW REPLICA STATUS FROM db.table WHERE ReplicaStatus != 'OK';
-- 3. Task queue depth
SHOW PROC '/tasks';
# BE-side tablet count & version lag
./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "tablet|version|clone"
./scripts/doris-debug log-grep be/log --pack versions
# Check specific tablet distribution
curl -s "http://$FE:8030/api/tablets_distribution?db=db_name&table=tbl_name"
Cause A — Replica missing
-- Identify unhealthy replicas
ADMIN SHOW REPLICA STATUS FROM db.table WHERE ReplicaStatus != 'OK';
-- Repair with low priority (production safe)
ADMIN REPAIR TABLE db.table PRIORITY = 'LOW';
-- For urgent single-tablet repair
ADMIN REPAIR TABLE db.table PARTITION (p202401) PRIORITY = 'HIGH';
Check BE heartbeat_service_mgr.cpp — heartbeats from a BE with missing tablets won't self-heal if compaction or clone threads are exhausted.
Cause C — Clone backlog
Clone concurrency: clone_worker_count (default 3 in config.cpp). A high Clone backlog often means:
- Tablet migration triggered by
balance_load_disk_safe_threshold/storage_high_watermark_usage_percent - Clone threads can't keep up with the repair plan
# be.conf — raise clone workers cautiously
clone_worker_count = 6
Cause D — Disk skew
SHOW PROC '/statistic'; -- per-disk usage
SHOW PROC '/backends'; -- per-BE disk UsedPct
# fe.conf
balance_load_disk_safe_threshold = 0.5
storage_high_watermark_usage_percent = 80
Manual rebalance: ADMIN SET FRONTEND CONFIG ("disable_balance" = "true"); then selective ADMIN REBALANCE DISK.
Source
fe/.../master/TabletHealth.java— cluster_health procfe/.../clone/TabletSchedCtx.javabe/src/agent/task_worker_pool.cpp— clone worker poolbe/src/common/config.cpp—clone_worker_count,storage_root_pathbe/src/olap/tablet_manager.cpp— tablet version tracking
More skills from the doris-skills repository
View all 14 skillsdoris-architecture-advisor
design Apache Doris data architectures
Jul 12ArchitectureData EngineeringData PipelineDatabasedoris-best-practices
optimize Apache Doris table design and clusters
Jul 12DatabaseEngineeringPerformancedoris-debug
diagnose Apache Doris production issues
Jul 25DebuggingMonitoringOperationsdoris-debug-cloud
debug Apache Doris cloud mode issues
Jul 25CloudDebuggingPerformancedoris-debug-compaction
resolve Apache Doris compaction lag
Jul 25DatabaseDebuggingPerformancedoris-debug-data-lake
debug Apache Doris data lake connectivity
Jul 25Data EngineeringDatabaseDebugging
More from Apache Software Foundation
View publisherdatafusion-python
write Apache DataFusion Python code
datafusion-python
Jul 12Data AnalysisPythonSQLbydbql
generate and execute BanyanDB BydbQL queries
skywalking-banyandb
Jul 12AnalyticsDatabaseSQLcompiling
compile and build BanyanDB projects
skywalking-banyandb
Jul 12BuildEngineeringgh-pull-request
create GitHub pull requests for BanyanDB
skywalking-banyandb
Jul 12GitHubPull Requestsvendor-update
update Go and Node.js vendor dependencies
skywalking-banyandb
Jul 12GoNode.jscayenne-cgen
generate Cayenne entity Java classes
cayenne
Jul 12Data ModelingJavaORM