Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

Stable 노드를 모니터링하고 일상적인 유지보수 작업을 수행하기 위한 종합 가이드입니다.

모니터링 스택 개요

권장 스택

  • Prometheus: 메트릭 수집
  • Grafana: 시각화 및 대시보드
  • AlertManager: 알림 라우팅 및 관리
  • Node Exporter: 시스템 메트릭
  • Loki: 로그 집계 (선택 사항)

빠른 모니터링 설정

1단계: Prometheus 메트릭 활성화

# Edit ~/.stabled/config/config.toml
[instrumentation]
prometheus = true
prometheus_listen_addr = ":26660"
namespace = "stablebft"

노드 재시작:

sudo systemctl restart ${SERVICE_NAME}

2단계: Prometheus 설치

# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvf prometheus-2.45.0.linux-amd64.tar.gz
sudo mv prometheus-2.45.0.linux-amd64 /opt/prometheus
 
# Create config
sudo tee /opt/prometheus/prometheus.yml > /dev/null <<EOF
global:
  scrape_interval: 15s
  evaluation_interval: 15s
 
scrape_configs:
  - job_name: 'stable-node'
    static_configs:
      - targets: ['localhost:26660']
        labels:
          instance: 'mainnode'
 
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']
EOF
 
# Create systemd service
sudo tee /etc/systemd/system/prometheus.service > /dev/null <<EOF
[Unit]
Description=Prometheus
After=network.target
 
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus \
  --config.file=/opt/prometheus/prometheus.yml \
  --storage.tsdb.path=/opt/prometheus/data
 
[Install]
WantedBy=multi-user.target
EOF
 
# Start Prometheus
sudo useradd -rs /bin/false prometheus
sudo chown -R prometheus:prometheus /opt/prometheus
sudo systemctl enable prometheus
sudo systemctl start prometheus

3단계: Grafana 설치

# Add Grafana repository
sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
 
# Install Grafana
sudo apt-get update
sudo apt-get install grafana
 
# Start Grafana
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
 
# Access at http://your-ip:3000
# Default login: admin/admin

모니터링할 주요 메트릭

노드 상태 메트릭

메트릭설명알림 임계값
up노드 가용성5분간 = 0
stablebft_consensus_height현재 블록 높이5분간 증가 없음
stablebft_consensus_validators활성 검증자해당 없음
stablebft_consensus_rounds합의 라운드> 3
stablebft_consensus_block_interval블록 시간> 10s
stablebft_p2p_peers연결된 피어< 3
stablebft_mempool_size멤풀 크기> 1500
stablebft_mempool_failed_txs실패한 트랜잭션> 100/분

시스템 메트릭

메트릭설명알림 임계값
node_cpu_seconds_totalCPU 사용량5분간 > 80%
node_memory_MemAvailable_bytes사용 가능한 메모리< 10%
node_filesystem_avail_bytes사용 가능한 디스크< 10%
node_network_receive_bytes_total네트워크 RX> 100MB/s
node_disk_io_time_seconds_total디스크 I/O> 80%
node_load15시스템 부하> CPU 코어 수 * 2

Grafana 대시보드 설정

Stable 대시보드 가져오기

{
  "dashboard": {
    "title": "Stable Node Monitoring",
    "panels": [
      {
        "title": "Block Height",
        "targets": [
          {
            "expr": "stablebft_consensus_height{chain_id=\"stabletestnet_2201-1\"}"
          }
        ]
      },
      {
        "title": "Peers",
        "targets": [
          {
            "expr": "stablebft_p2p_peers"
          }
        ]
      },
      {
        "title": "Block Time",
        "targets": [
          {
            "expr": "rate(stablebft_consensus_height[1m]) * 60"
          }
        ]
      },
      {
        "title": "Mempool Size",
        "targets": [
          {
            "expr": "stablebft_mempool_size"
          }
        ]
      }
    ]
  }
}

커스텀 대시보드 가져오기

Grafana UI를 통해 대시보드를 가져옵니다:

# Navigate to Dashboards > Import > Upload JSON file
# Or use Dashboard ID in Grafana's dashboard library

AlertManager 설정

AlertManager 설치

# Download AlertManager
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xvf alertmanager-0.26.0.linux-amd64.tar.gz
sudo mv alertmanager-0.26.0.linux-amd64 /opt/alertmanager
 
# Configure
sudo tee /opt/alertmanager/alertmanager.yml > /dev/null <<EOF
global:
  resolve_timeout: 5m
 
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'team-notifications'
 
receivers:
  - name: 'team-notifications'
    webhook_configs:
      - url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        send_resolved: true
    email_configs:
      - to: 'alerts@yourteam.com'
        from: 'prometheus@yournode.com'
        smarthost: 'smtp.gmail.com:587'
        auth_username: 'your@gmail.com'
        auth_password: 'app-specific-password'
 
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']
EOF
 
# Start AlertManager
sudo systemctl enable alertmanager
sudo systemctl start alertmanager

알림 규칙

# /opt/prometheus/alerts.yml
groups:
  - name: stable_alerts
    rules:
      - alert: NodeDown
        expr: up{job="stable-node"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Node {{ $labels.instance }} is down"
 
      - alert: BlockProductionStopped
        expr: increase(stablebft_consensus_height[5m]) == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Block production stopped"
 
      - alert: LowPeerCount
        expr: stablebft_p2p_peers < 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low peer count: {{ $value }}"
 
      - alert: HighMempool
        expr: stablebft_mempool_size > 1500
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High mempool size: {{ $value }}"
 
      - alert: DiskSpaceLow
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space: {{ $value | humanizePercentage }}"
 
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage: {{ $value }}%"

로그 모니터링

Systemd 로그

# View recent logs
sudo journalctl -u ${SERVICE_NAME} -n 100
 
# Follow logs
sudo journalctl -u ${SERVICE_NAME} -f
 
# Filter by time
sudo journalctl -u ${SERVICE_NAME} --since "1 hour ago"
 
# Export logs
sudo journalctl -u ${SERVICE_NAME} --since today > stable-logs-$(date +%Y%m%d).log

로그 분석 스크립트

#!/bin/bash
# analyze-logs.sh
 
# Count errors in last hour
echo "Errors in last hour:"
sudo journalctl -u ${SERVICE_NAME} --since "1 hour ago" | grep -c ERROR
 
# Show peer connections
echo "Peer connections:"
sudo journalctl -u ${SERVICE_NAME} --since "10 minutes ago" | grep "Peer connection" | tail -10
 
# Check for consensus issues
echo "Consensus rounds:"
sudo journalctl -u ${SERVICE_NAME} --since "30 minutes ago" | grep -E "enterNewRound|Timeout" | tail -20
 
# Memory usage patterns
echo "Memory warnings:"
sudo journalctl -u ${SERVICE_NAME} --since "1 day ago" | grep -i memory

Loki 설정 (선택 사항)

# Install Loki
wget https://github.com/grafana/loki/releases/download/v2.9.0/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
sudo mv loki-linux-amd64 /usr/local/bin/loki
 
# Install Promtail
wget https://github.com/grafana/loki/releases/download/v2.9.0/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail
 
# Configure Promtail
sudo tee /etc/promtail-config.yml > /dev/null <<EOF
server:
  http_listen_port: 9080
 
positions:
  filename: /tmp/positions.yaml
 
clients:
  - url: http://localhost:3100/loki/api/v1/push
 
scrape_configs:
  - job_name: stable
    systemd_journal:
      matches: "_SYSTEMD_UNIT=stabled.service"
      labels:
        job: stable
        host: localhost
EOF
 
# Start services
promtail -config.file=/etc/promtail-config.yml

상태 확인 엔드포인트

HTTP 엔드포인트

# Basic health check
curl -s http://localhost:26657/health
 
# Node status
curl -s http://localhost:26657/status | jq
 
# Net info
curl -s http://localhost:26657/net_info | jq
 
# Consensus state
curl -s http://localhost:26657/consensus_state | jq
 
# Unconfirmed transactions
curl -s http://localhost:26657/num_unconfirmed_txs | jq

상태 확인 스크립트

#!/bin/bash
# health-check.sh
 
set -e
 
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
export SERVICE_NAME="stable"
 
echo "=== Stable Node Health Check ==="
echo
 
# Check if service is running
if systemctl is-active --quiet ${SERVICE_NAME}; then
    echo -e "${GREEN}✓${NC} Service is running"
else
    echo -e "${RED}✗${NC} Service is not running"
    exit 1
fi
 
# Check node sync status
SYNC_STATUS=$(curl -s localhost:26657/status | jq -r '.result.sync_info.catching_up')
if [ "$SYNC_STATUS" = "false" ]; then
    echo -e "${GREEN}✓${NC} Node is synced"
else
    echo -e "${YELLOW}⚠${NC} Node is syncing"
fi
 
# Check peer count
PEERS=$(curl -s localhost:26657/net_info | jq -r '.result.n_peers')
if [ "$PEERS" -ge 3 ]; then
    echo -e "${GREEN}✓${NC} Connected peers: $PEERS"
else
    echo -e "${YELLOW}⚠${NC} Low peer count: $PEERS"
fi
 
# Check disk space
DISK_USAGE=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//')
if [ "$DISK_USAGE" -lt 80 ]; then
    echo -e "${GREEN}✓${NC} Disk usage: ${DISK_USAGE}%"
else
    echo -e "${YELLOW}⚠${NC} High disk usage: ${DISK_USAGE}%"
fi
 
# Check memory
MEM_AVAILABLE=$(free -m | awk 'NR==2 {print $7}')
MEM_TOTAL=$(free -m | awk 'NR==2 {print $2}')
MEM_PERCENT=$((100 - (MEM_AVAILABLE * 100 / MEM_TOTAL)))
if [ "$MEM_PERCENT" -lt 80 ]; then
    echo -e "${GREEN}✓${NC} Memory usage: ${MEM_PERCENT}%"
else
    echo -e "${YELLOW}⚠${NC} High memory usage: ${MEM_PERCENT}%"
fi
 
echo
echo "=== Health Check Complete ==="

유지보수 작업

일일 유지보수

#!/bin/bash
# daily-maintenance.sh
 
# Rotate logs
sudo journalctl --rotate
sudo journalctl --vacuum-time=7d
 
# Clear cache
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
 
# Check for updates
echo "Checking for updates..."
curl -s https://api.github.com/repos/stable-chain/stable/releases/latest | jq -r '.tag_name'
 
# Backup important config files
cp ~/.stabled/config/node_key.json ~/backups/node_key_$(date +%Y%m%d).json
 
# Generate report
echo "Daily report generated: $(date)" > ~/reports/daily_$(date +%Y%m%d).log
curl -s localhost:26657/status | jq >> ~/reports/daily_$(date +%Y%m%d).log

주간 유지보수

#!/bin/bash
# weekly-maintenance.sh
 
# Prune old data
stabled prune
 
# Compact database
stabled compact
 
# Update peer list
wget https://raw.githubusercontent.com/stable-chain/networks/main/testnet/peers.txt
cat peers.txt >> ~/.stabled/config/config.toml
 
# Create snapshot (optional)
./create-snapshot.sh
 
# System updates
sudo apt update
sudo apt upgrade -y
 
# Restart node (during low activity)
sudo systemctl restart ${SERVICE_NAME}

데이터베이스 유지보수

# Check database size
du -sh ~/.stabled/data/
 
# Analyze database
stabled debug db stats ~/.stabled/data

성능 모니터링

리소스 사용량 추적

#!/bin/bash
# track-resources.sh
 
while true; do
    TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
    CPU=$(top -bn1 | grep "stabled" | awk '{print $9}')
    MEM=$(top -bn1 | grep "stabled" | awk '{print $10}')
    IO=$(iostat -x 1 2 | tail -n2 | awk '{print $14}')
 
    echo "$TIMESTAMP,CPU:$CPU,MEM:$MEM,IO:$IO" >> ~/metrics/resources.csv
 
    sleep 60
done

쿼리 성능

# Monitor RPC response times
while true; do
    START=$(date +%s%N)
    curl -s http://localhost:26657/status > /dev/null
    END=$(date +%s%N)
    DIFF=$((($END - $START) / 1000000))
    echo "RPC response time: ${DIFF}ms"
    sleep 5
done

모니터링 모범 사례

  1. 중복 모니터링 설정
    • 외부 모니터링 서비스 사용
    • 크로스 노드 모니터링 구현
    • 데드맨 스위치 알림 설정
  2. 알림 피로 방지
    • 기준치를 기반으로 알림 임계값 조정
    • 알림 그룹화 및 억제 사용
    • 에스컬레이션 정책 구현
  3. 데이터 보존
    • 메트릭을 최소 30일간 유지
    • 중요한 로그 아카이브
    • 모니터링 설정의 정기 백업
  4. 보안
    • 강력한 비밀번호로 Grafana 보호
    • 모든 엔드포인트에 HTTPS 사용
    • prometheus 접근 제한
  5. 문서화
    • 모든 커스텀 메트릭 문서화
    • 알림에 대한 런북 유지
    • 대시보드 설명 최신 상태 유지

다음 단계