Reliability

SLOzy implements several reliability patterns to ensure stable operation under adverse conditions: circuit breakers to prevent cascading failures, exponential backoff with jitter to avoid thundering herds, and request tracing for latency analysis.

Circuit Breaker

The circuit breaker wraps all external HTTP calls — Prometheus queries, notification delivery, and license phone-home — with a state machine that prevents requests from reaching a failing dependency.

States

State	Meaning	Transition
Closed	Normal operation, requests pass through	→ Open after 5 consecutive failures
Open	Requests fail immediately without calling the dependency	→ Half-Open after 30 seconds
Half-Open	Limited requests allowed to probe recovery	→ Closed after 3 consecutive successes, or back to Open on failure

Configuration

resilience.NewCircuitBreaker(resilience.BreakerConfig{
    Name:            "prometheus-client",
    MaxFailures:     5,
    HalfOpenMaxReqs: 3,
    OpenTimeout:     30 * time.Second,
})

Affected components

Component	File	Circuit
Prometheus queries	internal/prometheus/client.go	prometheus-client

When the circuit is open, the Prometheus client returns ErrBreakerOpen immediately, preventing downstream requests from reaching the database.

Exponential Backoff with Jitter

Retries use exponential backoff with full jitter to spread retry attempts across a time window and avoid thundering herds.

Formula

delay = random(0, min(baseDelay × 2^attempt, maxDelay))

Retry configuration

Component	Max attempts	Base delay	Max delay
Prometheus queries	3	100 ms	5 s
License phone-home	3	1 s	5 s
Notification delivery	3 (configurable)	5 min	5 min

Context cancellation

All retry loops respect context cancellation. If the parent context is cancelled or times out during a backoff sleep, the retry loop exits immediately with ctx.Err(). This prevents retries from blocking application shutdown.

Implementation

func BackoffDuration(attempt int, base time.Duration, max time.Duration) time.Duration {
    exp := 1 << uint(attempt)
    d := time.Duration(float64(base) * float64(exp))
    if d > max { d = max }
    return time.Duration(rand.Int63n(int64(d)))
}

func Retry(ctx context.Context, maxAttempts int, base, max time.Duration, fn func(context.Context) error) error {
    for attempt := 0; attempt < maxAttempts; attempt++ {
        if attempt > 0 {
            delay := BackoffDuration(attempt-1, base, max)
            timer := time.NewTimer(delay)
            select {
            case <-ctx.Done():
                timer.Stop()
                return ctx.Err()
            case <-timer.C:
            }
        }
        if err := fn(ctx); err == nil {
            return nil
        } else {
            lastErr = err
        }
    }
    return lastErr
}

Request Tracing

SLOzy uses OpenTelemetry SDK with OTLP HTTP export to send traces to Grafana Tempo. The tracer is initialised in main.go via resilience.InitTracerProvider().

Configuration

Variable	Default	Description
OTEL_EXPORTER_OTLP_ENDPOINT	—	Tempo OTLP HTTP endpoint (e.g. http://tempo:4318)
OTEL_TRACES_SAMPLER	parentbased_traceidratio	Sampling strategy
OTEL_TRACES_SAMPLER_ARG	0.1	10% of requests sampled

When OTEL_EXPORTER_OTLP_ENDPOINT is set, the app exports spans to Tempo. If unset, tracing is a no-op.

Slow request detection

Requests taking longer than 100 ms are logged as potential performance issues:

[trace] slow request: GET /api/v1/slos took 1.2s (status=200)

Span types

Type	Description
SpanKindServer	Incoming HTTP request (Gin middleware via otelgin)
SpanKindClient	Outbound HTTP call (Prometheus query)

Middleware registration

Registered in cmd/slozy-web/main.go:

app.router.Use(resilience.TracingMiddleware("slozy-web"))

The middleware wraps otelgin.Middleware, which automatically captures HTTP attributes (method, path, status, etc.) and propagates trace context.

Grafana integration

Traces are available in Grafana via the Tempo datasource. Use Explore → Tempo to search by service name (slozy-web), duration, or tags. Span metrics (service graphs, span metrics) are generated by Tempo's metrics_generator and available in VictoriaMetrics.

Related

• Monitoring — Prometheus metrics and health checks
• Dashboards — Grafana dashboards for host, container, and application monitoring