Skillversitas

TiDB Data CDC

TiCDC (TiDB Change Data Capture) is a tool for replicating data in real time from TiDB to other systems.

What is TiCDC?

TiCDC captures change logs from TiKV and replicates them to other sinks (MySQL, Kafka, Pulsar, etc.). This enables:

  • Real-time replication to MySQL
  • Streaming data to Kafka for event-driven architecture
  • Backup and audit logging

Architecture

TiDB Cluster
    │
    ▼
  TiKV ──► TiCDC ──► MySQL Sink
    │              ──► Kafka Sink
    │              ──► Pulsar Sink
    ▼
  PD (Metadata)

Installing TiCDC

# Using TiUP
tiup install cdc

# Or add it to the cluster
tiup cluster scale-out my-cluster ./cdc-topology.yaml

Example: Replicate to MySQL

-- In TiDB (source)
CREATE TABLE users (
    id INT PRIMARY KEY AUTO_RANDOM,
    name VARCHAR(100),
    email VARCHAR(100)
);

-- In MySQL (target)
CREATE TABLE users (
    id INT PRIMARY KEY,
    name VARCHAR(100),
    email VARCHAR(100)
);

Create a Replication Task

cdc cli changefeed create \
  --pd=http://10.0.1.1:2379 \
  --sink-uri="mysql://root:@10.0.2.1:3306" \
  --changefeed-id="mysql-sink"

Monitoring

# Show the changefeed status
cdc cli changefeed list --pd=http://10.0.1.1:2379

# Show changefeed details
cdc cli changefeed query --pd=http://10.0.1.1:2379 --changefeed-id="mysql-sink"

Example: Replicate to Kafka

cdc cli changefeed create \
  --pd=http://10.0.1.1:2379 \
  --sink-uri="kafka://10.0.3.1:9092?topic=tidb-cdc" \
  --changefeed-id="kafka-sink"

Kafka Message Format

TiCDC sends messages in JSON format:

{
  "id": 1,
  "name": "John",
  "email": "[email protected]"
}

Filter

cdc cli changefeed create \
  --pd=http://10.0.1.1:2379 \
  --sink-uri="mysql://root:@10.0.2.1:3306" \
  --filter="!system.*,-mysql.*" \
  --changefeed-id="filtered-sink"

Protocol

MySQL Protocol

TiCDC uses the MySQL replication protocol. The target must be MySQL compatible.

Avro Protocol

cdc cli changefeed create \
  --pd=http://10.0.1.1:2379 \
  --sink-uri="kafka://10.0.3.1:9092?protocol=avro" \
  --changefeed-id="avro-sink"

Monitoring Metrics

In Grafana, look for the TiCDC panel:

  • Changefeed Count — number of active changefeeds
  • Event Count — number of events processed
  • Replication Lag — replication delay (should be < 1 second)
  • Sink Duration — time to deliver to the sink

In the next chapter, we will look at TiDB Data Migration (DM).