Skillversitas

TiDB Data Migration (DM)

DM is a tool for migrating data from MySQL/MariaDB to TiDB. DM supports full migration and incremental replication.

DM Features

  • Full migration (entire databases)
  • Incremental replication (real-time)
  • Shard migration from multiple MySQL instances
  • Database and table filtering
  • Data transformation

DM Architecture

MySQL (Source) ──► DM Worker ──► TiDB (Target)
                        │
                     DM Master
  • DM Master — manages and coordinates the migration
  • DM Worker — executes the migration from the MySQL source

Installing DM

# Using TiUP
tiup install dm

# Install the DM cluster
tiup dm deploy dm-cluster ./dm-topology.yaml

Full Migration

Source Configuration

# source.yaml
source-id: "mysql-source"
from:
  host: "10.0.1.1"
  port: 3306
  user: "root"
  password: ""

Run the Full Migration

dmctl start-task task-full.yaml

Task Configuration

# task-full.yaml
name: "full-migration"
task-mode: "full"
source-id: ["mysql-source"]

target-database:
  host: "10.0.2.1"
  port: 4000
  user: "root"
  password: ""

mysql-instances:
  - source-id: "mysql-source"
    block-allow-list:
      ba-list:
        do-dbs:
          - "ecommerce"

Incremental Replication

Binlog Configuration

# task-incremental.yaml
name: "incremental-replication"
task-mode: "incremental"
source-id: ["mysql-source"]

target-database:
  host: "10.0.2.1"
  port: 4000
  user: "root"
  password: ""

mysql-instances:
  - source-id: "mysql-source"
    binlog-name: "mysql-bin.000001"
    binlog-pos: 4

Sharding Migration

Migrating from multiple MySQL shards to a single TiDB:

# shard-task.yaml
name: "shard-migration"
task-mode: "full"
source-id: ["shard-1", "shard-2", "shard-3"]

target-database:
  host: "10.0.2.1"
  port: 4000

mysql-instances:
  - source-id: "shard-1"
  - source-id: "shard-2"
  - source-id: "shard-3"

shard-group: true

Monitoring

# Show the task status
dmctl query-status

# Show task details
dmctl query-status --task full-migration

# Stop the task
dmctl stop-task full-migration

Best Practices

  • Test the migration in a staging environment first
  • Monitor replication lag during the migration
  • Use filters to ignore tables that are not needed
  • Prepare a rollback plan

In the next chapter, we will look at the TiDB Dashboard.