Skill

molt-fetch

Name: molt-fetch
Author: CockroachDB

migrate data to CockroachDB with molt

Published by CockroachDB Updated Jul 25

Covers MySQL Database Migration PostgreSQL

Description

Guide for using molt fetch to migrate data from PostgreSQL, MySQL, Oracle, or MSSQL to CockroachDB. Use when running molt fetch commands, configuring storage backends, handling fetch failures/resumption, or chaining fetch with verify.

SKILL.md

molt fetch

Bulk data migration from source databases (PostgreSQL, MySQL, Oracle, MSSQL) to CockroachDB.

Basic Structure

molt fetch \
  --source "<source-conn>" \
  --target "<crdb-conn>" \
  --bucket-path "s3://bucket/prefix"   # or --direct-copy or --local-path
  [options]

Storage Backends (pick one)

Option	When to use
`--bucket-path "s3://..."`	AWS S3 (also `gs://` for GCS, `azure://` for Azure)
`--direct-copy`	No intermediate storage; fastest for accessible networks
`--local-path "/tmp/molt"` + `--local-path-listen-addr "0.0.0.0:9005"`	CRDB must reach the listen addr

Cloud auth: pass --use-implicit-auth for IAM/ADC/managed identity, or set AWS_ACCESS_KEY_ID/GOOGLE_APPLICATION_CREDENTIALS env vars.

Table Handling (`--table-handling`)

Value	Behavior
`none` (default)	Append to existing tables
`drop-on-target-and-recreate`	Drop + recreate from source schema; enables auto schema creation. Destroys existing target tables and their data, so use only on fresh or empty targets
`truncate-if-exists`	Truncate before loading; errors if table missing

Import Mode

IMPORT INTO (default): Table goes OFFLINE during load. Highest throughput.

COPY FROM (--use-copy): Table stays ONLINE. Use with --direct-copy. Cannot use compression.

# Zero-downtime load
molt fetch --source "..." --target "..." --direct-copy --use-copy

Key Flags

# Filtering
--table-filter "customers|orders"      # POSIX regex for tables to include
--table-exclusion-filter "temp_.*"     # exclude pattern
--schema-filter "public"               # PostgreSQL only

# Performance
--table-concurrency 4                  # parallel tables (default: 4)
--export-concurrency 4                 # export threads (default: 4)
--row-batch-size 100000                # rows per SELECT (default: 100k)

# Schema
--type-map-file "types.json"           # custom type mappings
--transformations-file "transforms.json"  # column exclusions, table aliases

# Logging
--log-file "migration.log"             # or "stdout"
--logging debug                        # info (default), debug, trace
--metrics-listen-addr "0.0.0.0:3030"  # Prometheus scrape endpoint

Source-Specific Prerequisites

MySQL: GTID mode required (gtid_mode=ON, enforce_gtid_consistency=ON). ONLY_FULL_GROUP_BY must be off. Or use --ignore-replication-check.

Oracle: Binary must be built with CGO_ENABLED=1 -tags="cgo source_all". Oracle Instant Client in LD_LIBRARY_PATH.

PostgreSQL: Replication privileges needed, or --ignore-replication-check.

Common Workflows

1. Full migration with schema creation

drop-on-target-and-recreate drops any existing target tables first; run this only against a fresh or empty target.

molt fetch \
  --source "postgresql://<user>:<password>@pg:5432/db" \
  --target "postgresql://root@crdb:26257/db" \
  --bucket-path "s3://mybucket/migration" \
  --table-handling drop-on-target-and-recreate \
  --table-filter "customers|orders|payments" \
  --log-file migration.log

2. Resume after failure

# List available continuation tokens
molt fetch tokens --fetch-id "abc-123" --target "postgresql://root@crdb:26257/db"

# Resume all failed tables
molt fetch \
  --source "..." --target "..." \
  --bucket-path "s3://mybucket/migration" \
  --fetch-id "abc-123" \
  --non-interactive

Error Recovery

Error	Cause	Fix
"GTID-based replication not enabled"	MySQL missing GTID	Enable `gtid_mode=ON` or add `--ignore-replication-check`
"Column mismatch"	Schema diverged	Fix target schema manually or use `--type-map-file`
Silent IMPORT INTO	CockroachDB import running	`SHOW JOBS` on CRDB to check progress
"timestamp in the future"	Docker/Mac clock drift	Sync clocks between hosts

Gotchas

COPY mode: cannot use --compression gzip; must use --compression none (or omit, default is none with copy)
Table is offline during IMPORT INTO — use --use-copy for zero downtime
Schema changes between runs require starting from scratch
--fetch-id continuation tokens live in the target's exceptions table
For MySQL, --ignore-replication-check skips GTID validation but replication-dependent features won't work
After fetch, run molt verify to confirm data integrity

See flags reference for the full flag list.

More skills from the claude-plugin repository

View all 34 skills

SkillTagsUpdated

More from CockroachDB

View publisher

SkillTagsUpdated