Operations

This guide covers monitoring and maintaining HoloMUSH in production. For the metric set, health-check endpoints, extensions, and log-location lookups, see the Monitoring reference.

Health Checks

Both core and gateway expose health endpoints (see Health-check endpoints).

Core (default port 9100):

curl http://localhost:9100/healthz/readiness

Gateway (default port 9101):

curl http://localhost:9101/healthz/readiness

Status Command

Check overall system health:

holomush status

This queries both core and gateway health endpoints via gRPC.

Prometheus Metrics

HoloMUSH exposes Prometheus metrics at /metrics (see the metric catalog). Point your scraper at the core and gateway endpoints:

scrape_configs:
  - job_name: holomush-core
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: holomush-gateway
    static_configs:
      - targets: ["localhost:9101"]

Enable pg_stat_statements

HoloMUSH requires pg_trgm and optionally uses pg_stat_statements (see PostgreSQL extensions). For query performance monitoring, enable pg_stat_statements:

# postgresql.conf
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.track = all

Restart PostgreSQL after configuration changes.

Query Performance Monitoring

The thresholds for the columns below are in Query-performance metrics.

Slowest Queries

Find queries consuming the most time:

SELECT
    calls,
    round(total_exec_time::numeric, 2) as total_ms,
    round(mean_exec_time::numeric, 2) as mean_ms,
    round(stddev_exec_time::numeric, 2) as stddev_ms,
    rows,
    query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

High I/O Queries

Identify queries causing disk reads:

SELECT
    calls,
    shared_blks_hit + shared_blks_read as total_blocks,
    round(100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0), 2) as cache_hit_pct,
    query
FROM pg_stat_statements
WHERE shared_blks_hit + shared_blks_read > 0
ORDER BY shared_blks_read DESC
LIMIT 10;

Most Frequent Queries

Find queries called most often:

SELECT
    calls,
    round(total_exec_time::numeric, 2) as total_ms,
    round(mean_exec_time::numeric, 2) as mean_ms,
    query
FROM pg_stat_statements
ORDER BY calls DESC
LIMIT 10;

Reset Statistics

Clear accumulated statistics:

SELECT pg_stat_statements_reset();

Connection Monitoring

Active Connections

View current database connections:

SELECT
    datname,
    usename,
    application_name,
    client_addr,
    state,
    query_start,
    now() - query_start as query_duration
FROM pg_stat_activity
WHERE datname = 'holomush'
ORDER BY query_start;

Connection Counts

Connections grouped by state:

SELECT state, count(*)
FROM pg_stat_activity
WHERE datname = 'holomush'
GROUP BY state;

JetStream Consumer Health

HoloMUSH uses an embedded NATS JetStream server for real-time event delivery. Check consumer state via the NATS monitoring port (default: disabled; set event_bus.monitor_port in config to enable):

# List stream info (requires nats CLI and monitoring port enabled)
nats stream info EVENTS --server nats://localhost:<monitor_port>

# Check audit projection consumer lag
nats consumer info EVENTS host_audit_projection

The audit_projection_lag_seconds Prometheus metric alerts at > 5s lag. The embedded server does not open a network port by default (DontListen: true).

Index Health

Unused Indexes

Find indexes that may be candidates for removal:

SELECT
    schemaname,
    relname as table_name,
    indexrelname as index_name,
    idx_scan,
    pg_size_pretty(pg_relation_size(indexrelid)) as index_size
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;

Index Usage

Verify indexes are being used:

SELECT
    relname as table_name,
    round(100.0 * idx_scan / nullif(seq_scan + idx_scan, 0), 2) as index_usage_pct,
    seq_scan,
    idx_scan
FROM pg_stat_user_tables
WHERE seq_scan + idx_scan > 0
ORDER BY seq_scan DESC;

Table Statistics

Table Sizes

Monitor table and index sizes:

SELECT
    relname as table_name,
    pg_size_pretty(pg_total_relation_size(relid)) as total_size,
    pg_size_pretty(pg_relation_size(relid)) as table_size,
    pg_size_pretty(pg_indexes_size(relid)) as index_size,
    n_live_tup as live_rows,
    n_dead_tup as dead_rows
FROM pg_stat_user_tables
ORDER BY pg_total_relation_size(relid) DESC;

Tables Needing Vacuum

Identify tables with dead tuples:

SELECT
    relname,
    n_dead_tup,
    last_vacuum,
    last_autovacuum
FROM pg_stat_user_tables
WHERE n_dead_tup > 1000
ORDER BY n_dead_tup DESC;

Backup and Recovery

PostgreSQL Backups

Use pg_dump for logical backups:

pg_dump -U holomush -h localhost holomush > backup.sql

For production, consider:

• Continuous archiving with WAL shipping
• Point-in-time recovery (PITR)
• Streaming replication for high availability

Restore

Restore from a logical backup:

psql -U holomush -h localhost holomush < backup.sql

Log Management

Logs go to stdout per component (see Log locations).

Log Aggregation

For production, aggregate logs with:

• Loki - Grafana’s log aggregation
• Elasticsearch - Full-text search
• CloudWatch - AWS managed logging

Docker Logging

View container logs:

docker compose logs -f core
docker compose logs -f gateway

Troubleshooting

Connection Issues

Symptom: Gateway cannot connect to core.

Check:

1. Core is running: holomush status
2. Network connectivity: nc -zv localhost 9000
3. Firewall rules allow traffic
4. TLS certificates are valid

Database Connection Failed

Symptom: Core fails with database errors.

Check:

1. PostgreSQL is running: pg_isready -h localhost
2. DATABASE_URL is correct
3. User has permissions: \du in psql
4. Database exists: \l in psql

High Memory Usage

Possible causes:

• Too many active sessions
• Large query result sets
• Plugin memory leaks

Investigate:

# Check active sessions
curl http://localhost:9100/metrics | grep sessions_active

# Check PostgreSQL connections
psql -c "SELECT count(*) FROM pg_stat_activity WHERE datname='holomush';"

Slow Queries

Investigate:

1. Enable pg_stat_statements (see above)
2. Check for missing indexes
3. Analyze query plans with EXPLAIN ANALYZE

-- Replace <ulid> with an actual location ULID from your database
EXPLAIN ANALYZE SELECT * FROM events WHERE stream = 'location:<ulid>' ORDER BY id DESC LIMIT 100;

Alias Database-Cache Inconsistency

Symptom: Log message with severity=critical containing “alias rollback failed - database-cache inconsistency”.

Monitoring: Alert on the holomush_alias_rollback_failures_total metric.

# Prometheus alert rule
- alert: AliasRollbackFailure
  expr: increase(holomush_alias_rollback_failures_total[5m]) > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "Alias database-cache inconsistency detected"
    description: "A rollback failure has left aliases in an inconsistent state."

Cause: During alias creation, the database write succeeded but the cache update failed (e.g., circular reference detected). The subsequent rollback (database delete) also failed, leaving the alias in the database but not in the cache.

Impact:

• The alias exists in the database but is not loaded into cache
• On server restart, the alias will be loaded and may cause issues
• The alias may conflict with other aliases or commands

Recovery:

1. Identify the problematic alias from the log message (check alias and player_id fields)

2. For player aliases, delete from database:

   DELETE FROM player_aliases
   WHERE player_id = '<player_id>' AND alias = '<alias_name>';

3. For system aliases, delete from database:

   DELETE FROM system_aliases WHERE alias = '<alias_name>';

4. Verify cleanup:

   -- Check player aliases
   SELECT * FROM player_aliases WHERE alias = '<alias_name>';
   
   -- Check system aliases
   SELECT * FROM system_aliases WHERE alias = '<alias_name>';

5. No restart required - the cache does not contain the alias, so removing it from the database prevents it from being loaded on the next restart.

Prevention: This failure typically occurs when database connectivity is degraded. Monitor database connection health and ensure proper connection pooling configuration.

Next Steps

• Monitoring reference - Metrics, endpoints, extensions, log locations
• Configuration - Adjust server settings
• Installation - Deployment options