Monitoring (Prometheus & Grafana)

TiDB uses Prometheus for metrics collection and Grafana for visualization. Monitoring is essential for keeping the cluster healthy.

Monitoring Components

Prometheus

Prometheus collects metrics from all TiDB components:

• TiDB Server metrics
• TiKV metrics
• PD metrics
• TiFlash metrics
• TiCDC metrics

Grafana

Grafana displays metrics as visual dashboards:

• Real-time graphs
• Alerting
• Custom dashboards

Accessing Monitoring

Main Dashboards in Grafana

TiDB Overview

Displays an overview of the entire cluster:

• QPS — queries per second
• TPS — transactions per second
• Latency — average response time
• Error Rate — number of errors

TiDB Detail

• SQL statement types
• Connection count
• Memory usage
• GC duration

TiKV Detail

• Region distribution
• Read/Write throughput
• Storage size
• RocksDB metrics

PD Detail

• Cluster status
• Operator count
• Scheduler status

Important Metrics

Performance Metrics

tidb_session_transaction_duration_seconds
-- Transaction duration

tidb_server_handle_query_duration_seconds
-- Query execution duration

tikv_storage_engine_size_bytes
-- Data size in TiKV

Availability Metrics

tidb_server_connections
-- Number of active connections

tidb_server_tokens
-- Token bucket for rate limiting

pd_cluster_status
-- Cluster status (1 = healthy)

Alerting Rules

# alerting-rules.yaml
groups:
  - name: tidb-alerts
    rules:
      - alert: TiDBHighQueryLatency
        expr: histogram_quantile(0.99, rate(tidb_server_handle_query_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High query latency detected"

      - alert: TiKVRegionUnhealthy
        expr: pd_regions_unhealthy > 0
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Unhealthy regions detected"

Monitoring Tips

• Monitor QPS and Latency in real time
• Watch Region Health — no unhealthy Regions
• Monitor Disk Usage — never let the disk fill up
• Set up Alerting for critical metrics
• Use the Slow Query Log to identify slow queries

In the next chapter, we will look at performance tuning in TiDB.