Documentationpgraft Documentation

Monitoring and Observability

Monitoring Fundamentals

  • Cluster health: Track leader election, node connectivity, and quorum status.
  • Replication lag: Monitor Raft log replication and corrective actions.
  • Performance metrics: Heartbeats, elections, message throughput, worker uptime.
  • Resource usage: Disk availability for WAL/logs, memory pressure, snapshot retention.
  • Operational events: Node additions/removals, failovers, configuration changes.

Status Views and Health Functions

Cluster Overview

Primary function for monitoring leadership, quorum, and message counters from any node.

Cluster status snapshot

-- Comprehensive cluster status
SELECT * FROM pgraft_get_cluster_status();

-- Columns include:
-- node_id, current_term, leader_id, state, num_nodes,
-- messages_processed, heartbeats_sent, elections_triggered
Health indicators: All nodes should agree on leader_id and current_term, only one node reports state = 'leader', and elections_triggered should rarely increase.

Node & Worker Visibility

Registered members

-- View cluster membership
SELECT * FROM pgraft_get_nodes();

-- During partitions use Raft source of truth
SELECT pgraft_get_nodes_from_raft();

Leader detection

SELECT pgraft_get_leader() AS leader_id,
       pgraft_is_leader() AS am_i_leader,
       pgraft_get_term() AS current_term;

Worker status

SELECT * FROM pgraft_worker_status;
-- Columns: state, last_tick, uptime_seconds

Log Replication Monitoring

Replication Status

Track follower lag and match indexes to keep quorum healthy under load.

Follower replication health

SELECT node_id,
       match_index,
       commit_index,
       lag_entries,
       state
  FROM pgraft_log_get_replication_status();
Alert guidance: lag_entries > 100 (warning), > 1000 (critical), orstate = 'stalled' requires immediate remediation.

Log & Snapshot Metrics

Log volume

SELECT first_index,
       last_index,
       commit_index,
       applied_index,
       total_entries,
       disk_usage_mb
  FROM pgraft_log_get_stats();

Growth analysis

SELECT total_entries,
       disk_usage_mb,
       (disk_usage_mb::float / NULLIF(total_entries, 0)) * 1024 AS avg_entry_kb
  FROM pgraft_log_get_stats();

Force follower resync

-- Execute on lagging follower
SELECT pgraft_log_sync_with_leader();

Key-Value Store Monitoring

Usage & Capacity

KV statistics

SELECT total_keys,
       total_size_bytes,
       oldest_key_age,
       newest_key_age,
       avg_value_size
  FROM pgraft_kv_get_stats();

KV status

SELECT keys_count,
       values_size_mb,
       compaction_count,
       last_compact_at
  FROM pgraft_kv_status;

Health & Trend Checks

End-to-end KV health

DO $$
DECLARE
  test_key TEXT := 'health_check_' || extract(epoch FROM now());
  test_value TEXT := 'test_value_' || md5(random()::text);
  roundtrip TEXT;
BEGIN
  PERFORM pgraft_kv_put(test_key, test_value);
  SELECT pgraft_kv_get(test_key) INTO roundtrip;
  IF roundtrip != test_value THEN
    RAISE EXCEPTION 'KV health check failed';
  END IF;
  PERFORM pgraft_kv_delete(test_key);
END;
$$;

30-day growth trend

WITH daily_growth AS (
  SELECT date_trunc('day', created_at) AS day,
         COUNT(*) AS keys_added,
         SUM(pg_column_size(value)) AS bytes_added
    FROM pgraft.kv
   WHERE created_at > now() - interval '30 days'
   GROUP BY 1
)
SELECT day,
       keys_added,
       bytes_added / 1024 / 1024 AS mb_added,
       SUM(keys_added) OVER (ORDER BY day) AS cumulative_keys,
       SUM(bytes_added) OVER (ORDER BY day) / 1024 / 1024 AS cumulative_mb
  FROM daily_growth
 ORDER BY day DESC;

Performance & Election Metrics

Message Throughput

Messages per heartbeat

SELECT node_id,
       messages_processed,
       heartbeats_sent,
       messages_processed / GREATEST(heartbeats_sent, 1) AS msg_per_heartbeat
  FROM pgraft_get_cluster_status();

Historical rate (requires sampling)

WITH current AS (
  SELECT *
    FROM pgraft_metrics_history
   ORDER BY recorded_at DESC
   LIMIT 1
),
previous AS (
  SELECT *
    FROM pgraft_metrics_history
   WHERE recorded_at <= now() - interval '1 hour'
   ORDER BY recorded_at DESC
   LIMIT 1
)
SELECT c.node_id,
       (c.messages_processed - p.messages_processed)
         / EXTRACT(EPOCH FROM (c.recorded_at - p.recorded_at)) AS messages_per_second,
       (c.heartbeats_sent - p.heartbeats_sent)
         / EXTRACT(EPOCH FROM (c.recorded_at - p.recorded_at)) AS heartbeats_per_second
  FROM current c
  JOIN previous p USING (node_id);

Election Stability

Election rate

SELECT node_id,
       state,
       elections_triggered,
       current_term,
       elections_triggered::float / GREATEST(current_term, 1) AS elections_per_term
  FROM pgraft_get_cluster_status();
Investigate immediately when: elections_per_term > 2,current_term spikes unexpectedly, or multiple nodes report state = 'candidate'.

Snapshot hygiene

# Inspect snapshot directory for growth and cadence
ls -lh /var/lib/postgresql/pgraft/snapshots/ | tail -10

# Check configured snapshot threshold
psql -c "SHOW pgraft.snapshot_threshold;"

System Monitoring Views

etcd-Compatible Views

Endpoints & health

SELECT * FROM pgraft.member_list;
SELECT * FROM pgraft.endpoint_health;
SELECT * FROM pgraft.cluster_health;
SELECT * FROM pgraft.cluster_info;

Cluster State Summary

Create summary view

CREATE OR REPLACE VIEW pgraft_health_summary AS
SELECT CURRENT_TIMESTAMP AS checked_at,
       COUNT(*) AS total_nodes,
       COUNT(*) FILTER (WHERE state = 'leader') AS num_leaders,
       COUNT(*) FILTER (WHERE state = 'follower') AS num_followers,
       COUNT(*) FILTER (WHERE state = 'candidate') AS num_candidates,
       MAX(term) AS current_term,
       BOOL_AND(leader) AS has_leader
  FROM pgraft_cluster_state;

Query summary

SELECT * FROM pgraft_health_summary;

Alerting & Automation

Database Health Check

Comprehensive PL/pgSQL check

CREATE OR REPLACE FUNCTION pgraft_health_check()
RETURNS TABLE(check_name TEXT, status TEXT, message TEXT, severity TEXT) AS $$
DECLARE
  v_status pgraft_get_cluster_status%ROWTYPE;
BEGIN
  SELECT * INTO v_status FROM pgraft_get_cluster_status() LIMIT 1;

  RETURN QUERY
    SELECT 'leader_election',
           CASE
             WHEN (SELECT COUNT(*) FROM pgraft_cluster_state WHERE state = 'leader') = 0 THEN 'CRITICAL'
             WHEN (SELECT COUNT(*) FROM pgraft_cluster_state WHERE state = 'leader') > 1 THEN 'CRITICAL'
             ELSE 'OK'
           END,
           format('Leader ID %s', v_status.leader_id),
           'info';

  RETURN QUERY
    SELECT 'quorum',
           CASE WHEN v_status.num_nodes < 3 THEN 'WARNING' ELSE 'OK' END,
           format('%s nodes in cluster', v_status.num_nodes),
           CASE WHEN v_status.num_nodes < 3 THEN 'warning' ELSE 'info' END;

  RETURN QUERY
    SELECT 'election_stability',
           CASE WHEN v_status.elections_triggered::float / GREATEST(v_status.current_term, 1) > 2 THEN 'WARNING' ELSE 'OK' END,
           format('%s elections across %s terms', v_status.elections_triggered, v_status.current_term),
           CASE WHEN v_status.elections_triggered::float / GREATEST(v_status.current_term, 1) > 2 THEN 'warning' ELSE 'info' END;

  RETURN QUERY
    SELECT 'replication_lag',
           CASE WHEN EXISTS (SELECT 1 FROM pgraft_log_get_replication_status() WHERE lag_entries > 100) THEN 'WARNING' ELSE 'OK' END,
           CASE WHEN EXISTS (SELECT 1 FROM pgraft_log_get_replication_status() WHERE lag_entries > 100) THEN 'Followers lagging > 100 entries' ELSE 'All followers up to date' END,
           CASE WHEN EXISTS (SELECT 1 FROM pgraft_log_get_replication_status() WHERE lag_entries > 100) THEN 'warning' ELSE 'info' END;
END;
$$ LANGUAGE plpgsql;

Execute health check

SELECT * FROM pgraft_health_check();

Automation Scripts

Prometheus exporter

#!/usr/bin/env bash
cat <<'EOF'
# HELP pgraft_cluster_nodes Total number of nodes in cluster
# TYPE pgraft_cluster_nodes gauge
pgraft_cluster_nodes $(psql -t -c "SELECT num_nodes FROM pgraft_get_cluster_status();")

# HELP pgraft_is_leader Whether this node is the leader (1=yes, 0=no)
# TYPE pgraft_is_leader gauge
pgraft_is_leader $(psql -t -c "SELECT CASE WHEN pgraft_is_leader() THEN 1 ELSE 0 END;")

# HELP pgraft_current_term Current Raft election term
# TYPE pgraft_current_term counter
pgraft_current_term $(psql -t -c "SELECT pgraft_get_term();")

# HELP pgraft_messages_processed Total Raft messages processed
# TYPE pgraft_messages_processed counter
pgraft_messages_processed $(psql -t -c "SELECT messages_processed FROM pgraft_get_cluster_status();")

# HELP pgraft_elections_triggered Total elections triggered
# TYPE pgraft_elections_triggered counter
pgraft_elections_triggered $(psql -t -c "SELECT elections_triggered FROM pgraft_get_cluster_status();")

# HELP pgraft_log_entries Total log entries
# TYPE pgraft_log_entries gauge
pgraft_log_entries $(psql -t -c "SELECT total_entries FROM pgraft_log_get_stats();")

# HELP pgraft_log_disk_mb Log disk usage in megabytes
# TYPE pgraft_log_disk_mb gauge
pgraft_log_disk_mb $(psql -t -c "SELECT disk_usage_mb FROM pgraft_log_get_stats();")

# HELP pgraft_kv_keys Total KV store keys
# TYPE pgraft_kv_keys gauge
pgraft_kv_keys $(psql -t -c "SELECT total_keys FROM pgraft_kv_get_stats();")
EOF

Email alert driver

#!/usr/bin/env bash
ALERT_EMAIL="ops@example.com"
LOG_FILE="/var/log/pgraft-alerts.log"

HEALTH_CHECK=$(psql -t -c "
SELECT string_agg(
  check_name || ': ' || status || ' - ' || message,
  E'\n'
)
FROM pgraft_health_check()
WHERE severity IN ('warning', 'critical');
")

if [[ -n "$HEALTH_CHECK" ]]; then
  echo "$(date --iso-8601=seconds): ALERT - $HEALTH_CHECK" >> "$LOG_FILE"
  printf '%s
' "$HEALTH_CHECK" | mail -s "pgraft Cluster Alert" "$ALERT_EMAIL"
fi

MAX_LAG=$(psql -t -c "SELECT COALESCE(MAX(lag_entries), 0) FROM pgraft_log_get_replication_status();")
if [[ "$MAX_LAG" -gt 1000 ]]; then
  echo "$(date --iso-8601=seconds): ALERT - Replication lag: $MAX_LAG entries" >> "$LOG_FILE"
  printf 'Critical replication lag: %s entries
' "$MAX_LAG" | mail -s "pgraft Replication Lag Alert" "$ALERT_EMAIL"
fi

Dashboards & Visualization

Sample Grafana queries

-- Cluster overview
SELECT num_nodes AS "Total Nodes",
       (SELECT COUNT(*) FROM pgraft_cluster_state WHERE state = 'leader') AS "Leaders",
       (SELECT COUNT(*) FROM pgraft_cluster_state WHERE state = 'follower') AS "Followers",
       current_term AS "Current Term"
  FROM pgraft_get_cluster_status();

-- Replication lag panel
SELECT node_id AS "Node",
       lag_entries AS "Lag (entries)",
       state AS "Status"
  FROM pgraft_log_get_replication_status()
 ORDER BY lag_entries DESC;

-- KV store growth
SELECT total_keys AS "Total Keys",
       total_size_bytes / 1024 / 1024 AS "Size (MB)",
       avg_value_size / 1024 AS "Avg Value (KB)"
  FROM pgraft_kv_get_stats();

Log Analysis & Debugging

PostgreSQL Logs

Tail and filter

sudo tail -f /var/log/postgresql/postgresql-17-main.log | grep -i pgraft
sudo grep -i "pgraft.*error" /var/log/postgresql/postgresql-17-main.log
sudo grep -i "pgraft.*election" /var/log/postgresql/postgresql-17-main.log | tail -20
sudo grep -E "pgraft.*(add_node|remove_node)" /var/log/postgresql/postgresql-17-main.log

Debug Mode

Toggle verbose tracing

SELECT pgraft_set_debug(true);
-- Perform operations requiring deep tracing
SELECT pgraft_set_debug(false);

Inspect debug output

sudo tail -100 /var/log/postgresql/postgresql-17-main.log | grep "DEBUG.*pgraft"

Best Practices Checklist

  • Continuous monitoring: Execute health checks every 1–5 minutes via cron or agents.
  • Historical retention: Store metrics in a time-series database for capacity planning and regression detection.
  • Alert thresholds: Leader absence (critical), replication lag > 100 entries (warning) / > 1000 (critical), rising election rates, node count mismatches.
  • Dashboard coverage: Visualize leadership, replication health, log throughput, and KV growth trends.
  • Log hygiene: Retain PostgreSQL logs for at least seven days and forward them to centralized logging.
  • Capacity planning: Track disk usage for WAL, snapshots, and KV state to avoid emergency pruning.
  • Regular rehearsal: Test failover and replica catch-up quarterly to validate tooling and runbooks.

Next Steps

PreviousTutorial
NextRaft Protocol