Monitoring

SLOzy uses VictoriaMetrics as the metrics backend, Loki for log aggregation, Tempo for distributed tracing, and Alertmanager for alerting. All services run in Docker containers managed by docker-compose.yml.

Docker Compose Services

Service	Port	Description
`victoria-metrics`	8428	Metrics storage (Prometheus-compatible)
`loki`	3100	Log aggregation
`alloy`	—	Log agent, sends Docker logs to Loki (replaces Promtail)
`tempo`	3200 / 4317 / 4318	Distributed tracing backend (OTLP)
`alertmanager`	9093	Alert routing (Telegram, etc.)
`grafana`	3001	Dashboards & visualisation
`node-exporter`	9100	Host metrics (CPU, RAM, disk, network)
`cadvisor`	8083	Container metrics (per-container CPU, memory)

Health Check Endpoints

`GET /health`

Basic liveness probe used by Docker.

json

{
  "status": "healthy",
  "time": "2026-06-11T10:00:00Z"
}

`GET /api/v1/health`

Detailed readiness probe with per-service health status.

json

{
  "status": "ok",
  "timestamp": "2026-06-11T10:00:00Z",
  "services": {
    "cache": true,
    "notifications": true,
    "websocket": true
  }
}

`GET /api/v1/status/public`

Public status endpoint used by the landing page widget. No authentication required.

healthy — SLO has at least one metric meeting its target
violated — SLO has metrics, but none meet the target
no_data — SLO has no metrics yet

json

{
  "status": "ok",
  "uptime": "3d 2h 15m",
  "timestamp": "2026-06-22T08:17:39Z",
  "services": {
    "api": true,
    "database": true,
    "cache": true
  },
  "slos": {
    "total": 5,
    "healthy": 0,
    "violated": 0,
    "no_data": 5
  }
}

VictoriaMetrics (Metrics)

Replaces Prometheus as the metrics backend. API-compatible with Prometheus.

Configuration

Scrape config: victoria-metrics/scrape.yml

yaml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'slozy-web'
    targets: ['slozy-web:8080']
  - job_name: 'node-exporter'
    targets: ['node-exporter:9100']
  - job_name: 'cadvisor'
    targets: ['cadvisor:8080']
  - job_name: 'alertmanager'
    targets: ['alertmanager:9093']
    metrics_path: '/alertmanager/metrics'
  - job_name: 'loki'
    targets: ['loki:3100']
  - job_name: 'tempo'
    targets: ['tempo:3200']
  - job_name: 'victoria-metrics'
    targets: ['victoria-metrics:8428']

HTTP RED Metrics

SLOzy exports standard RED (Rate/Errors/Duration) metrics via the /metrics endpoint:

Metric	Type	Labels	Description
`http_requests_total`	Counter	`method`, `path`, `status`	Total HTTP requests by method, route, and status code
`http_request_duration_seconds`	Histogram	`method`, `path`	Request latency in seconds (standard Prometheus buckets)

These metrics are collected by the built-in middleware (internal/middleware/metrics.go) and registered with Prometheus' default registry. VictoriaMetrics scrapes them automatically via the slozy-web job.

Example PromQL queries:

promql

# Request rate by route
rate(http_requests_total{job="slozy-web"}[5m])

# Error ratio (5xx / total)
sum(rate(http_requests_total{job="slozy-web",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="slozy-web"}[5m]))

# 95th percentile latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="slozy-web"}[5m])) by (le, path))

Alert Rules

Defined in alertmanager/rules/alerts.yml, evaluated by VictoriaMetrics:

Rule	Severity	Condition
`HostHighCpu`	warning	CPU > 80% for 5m
`HostDiskFull`	warning	Disk > 85%
`HostMemoryLow`	warning	RAM > 85%
`SlozyWebDown`	critical	Web unreachable for 1m
`SloBudgetBurnRate`	critical	Burn rate > 1x

Storage

Retention: 30 days
Data path: /victoria-metrics-data (Docker volume victoria_metrics_data)

Loki (Logs)

Configuration

Config: loki/loki-config.yml

Filesystem storage at /loki (Docker volume loki_data)
Retention: 168 hours (7 days) for old samples rejection

Alloy (Log Agent)

Config: alloy/alloy-config.river

Replaces Promtail (EOL March 2026). Collects Docker container logs and ships them to Loki using a stage-based pipeline.

Reads Docker container logs from /var/lib/docker/containers/*/*.log
Sends to http://loki:3100/loki/api/v1/push
Parses JSON log format with timestamp, level, and message extraction

Tempo (Traces)

Configuration

Config: tempo/tempo.yml

Listens on OTLP gRPC (4317) and OTLP HTTP (4318)
Local filesystem storage at /var/tempo (Docker volume tempo_data)
Block retention: 72 hours
Metrics generator enabled: service graphs and span metrics, remote-writes to VictoriaMetrics

Application Integration

SLOzy uses the OpenTelemetry SDK (go.opentelemetry.io/otel) to create and export spans. The tracer is initialized in cmd/slozy-web/main.go — when OTEL_EXPORTER_OTLP_ENDPOINT is set, spans are batch-exported via OTLP HTTP to Tempo:

env

OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4318
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

Alertmanager

Configuration

Config: alertmanager/alertmanager.yml

Default receiver: null (discards all alerts)
To enable Telegram, update bot_token and chat_id in the telegram receiver, then change the route
Alertmanager runs with --web.external-url=/alertmanager — health endpoint at /alertmanager/-/healthy

Two Notification Systems

SLOzy has two independent notification systems — they do not share configuration and serve different purposes.

SLO Notifications (built-in)

Configured via the web interface (Settings → Notification Channels). Used for SLO violation alerts, delivered through the SLOzy backend.

Supported channels: Email, Slack, Telegram, Mattermost, Webhook.

Infrastructure Alerts (Alertmanager)

A separate system for host and container alerts (CPU, disk, RAM, service down). Works through Alertmanager + vmalert → Telegram.

Configuration: alertmanager/alertmanager.yml. The Telegram bot token is stored in alertmanager/telegram_bot_token (not in git). After a fresh deploy, create this file manually:

bash

echo -n "YOUR_BOT_TOKEN" > alertmanager/telegram_bot_token
chmod 644 alertmanager/telegram_bot_token
docker compose up -d --force-recreate alertmanager

⚠️ These systems are independent — adding a channel in the web UI does not configure Alertmanager, and configuring Alertmanager does not affect SLO notifications.

Grafana Dashboards

Grafana runs at https://slozy.net/grafana (admin / from .env).

Pre-provisioned Datasources

Name	Type	URL
VictoriaMetrics	prometheus	`http://victoria-metrics:8428`
Loki	loki	`http://loki:3100`
Tempo	tempo	`http://tempo:3200`

Available Dashboards

Dashboard	Source	Description
Host & Containers	`grafana/dashboards/host-overview.json`	13 panels — host CPU, RAM, disk, load, network, IO + per-container CPU, memory, network

Dashboards are auto-provisioned from grafana/dashboards/.

Infrastructure Monitoring

Exporter	Container	Metrics
node_exporter	`slozy-node-exporter`	Host CPU, memory, disk, network, load
cAdvisor	`slozy-cadvisor`	Per-container CPU, memory, network

Both are pre-configured in docker-compose.yml and auto-discovered by VictoriaMetrics. Their metrics are visualised in the Host & Containers Grafana dashboard.

Cache Statistics

GET /api/v1/cache/statistics

Key metrics:

hit_rate — Target > 0.85
memory_usage_mb — Should stay within limits
hot_entries_count — Frequently accessed entries

SLO Real-Time Metrics

GET /api/v1/prometheus/real-time-metrics

Returns SLO values, error budget, and status for all active SLOs in the user's organization.

Dashboard: Availability Trend

The Availability Trend chart on the Dashboard page shows the average SLI (Service Level Indicator) across all SLOs over time.

How It Works

The frontend calls GET /api/v1/slos/availability-trend?period=24h&step=1h
For each SLO with a Prometheus data source, the backend queries VictoriaMetrics for the actual SLI value (the result of the SLO's metric_query)
Responses are aggregated: avg_sli = mean of all SLO SLI values at each time point
The frontend renders an area chart with the averaged SLI line

What the Numbers Mean

Metric	Meaning
Average across all SLOs: 74.95%	Weighted average of the SLO `target` values across all SLOs for the selected team. Represents the expected performance floor, not the actual measured SLI.
SLOs with data	Number of SLOs that returned metric data from VictoriaMetrics for the period. If all SLOs show 0, check that your Prometheus data source is reachable.
Chart Y-axis	SLI ratio (0.0–1.0), where 1.0 = 100% compliance with the target.

Why "No metric data yet"

This message appears when:

No SLOs have Prometheus data sources configured
The Prometheus queries return no time-series data (e.g., metric names don't exist yet, or the data source is unreachable)
VictoriaMetrics is still starting up and hasn't collected enough data

Once you create SLOs with valid metric_query values that match real Prometheus metrics, the trend chart will populate automatically.

Monitoring ​

Docker Compose Services ​

Health Check Endpoints ​

GET /health ​

GET /api/v1/health ​

GET /api/v1/status/public ​

VictoriaMetrics (Metrics) ​

Configuration ​

HTTP RED Metrics ​

Alert Rules ​

Storage ​

Loki (Logs) ​

Configuration ​

Alloy (Log Agent) ​

Tempo (Traces) ​

Configuration ​

Application Integration ​

Alertmanager ​

Configuration ​

Two Notification Systems ​

SLO Notifications (built-in) ​

Infrastructure Alerts (Alertmanager) ​

Grafana Dashboards ​

Pre-provisioned Datasources ​

Available Dashboards ​

Infrastructure Monitoring ​

Cache Statistics ​

SLO Real-Time Metrics ​

Dashboard: Availability Trend ​

How It Works ​

What the Numbers Mean ​

Why "No metric data yet" ​

Monitoring

Docker Compose Services

Health Check Endpoints

`GET /health`

`GET /api/v1/health`

`GET /api/v1/status/public`

VictoriaMetrics (Metrics)

Configuration

HTTP RED Metrics

Alert Rules

Storage

Loki (Logs)

Configuration

Alloy (Log Agent)

Tempo (Traces)

Configuration

Application Integration

Alertmanager

Configuration

Two Notification Systems

SLO Notifications (built-in)

Infrastructure Alerts (Alertmanager)

Grafana Dashboards

Pre-provisioned Datasources

Available Dashboards

Infrastructure Monitoring

Cache Statistics

SLO Real-Time Metrics

Dashboard: Availability Trend

How It Works

What the Numbers Mean

Why "No metric data yet"