Stable response procedures for the optional Prometheus Operator rules shipped with Jikji. Treat the example thresholds as starting points, then tune them from load tests and observed production traffic.
Install the rules only after the scrape target, external check, notification route, and service ownership labels are configured.
The /metrics endpoint is intentionally unauthenticated and tenant labels are scrubbed by default. Restrict it to the telemetry network and keep press.expose_tenant_metrics: false unless the scrape path has an independent authentication boundary.
Confirm the external /health check and Kubernetes liveness state. Inspect the termination reason, restart count, OOM state, node pressure, and previous log.
If /health works but scraping fails, inspect the Service, NetworkPolicy, Prometheus selector, and TLS or proxy path before restarting the workload.
Break down jikji_http_route_responses_total by normalized route and status, then correlate the first failure with Galley, provider, and deployment events.
Stop an active rollout when the new ReplicaSet owns the increase. Preserve the failing request class without logging credentials or tenant payloads.
Confirm the global p99 increase in jikji_http_request_duration_seconds_bucket against request rate, in-flight work, provider latency, Galley latency, and dispatch saturation. Use normalized route response counters and traces for attribution; the latency histogram deliberately omits tenant and route labels.
The shipped five-second threshold is a starting point, not a contractual SLO. Tune it from deployment load tests and production evidence, and prove recovery with a sustained p99 below the reviewed threshold.
Shedding is a protection mechanism, not a reason to immediately raise limits. Check run duration, provider latency, Galley latency, queue depth, and per-pod CPU and memory.
Scale only an HA deployment backed by shared PostgreSQL or MySQL. On a single SQLite node, adding replicas does not create a supported shared-state topology.
Confirm that both jikji_dispatch_queue_depth and jikji_dispatch_saturated_targets remain above zero. Use the authenticated GET /v1/dispatch/queues view to identify the provider and model target; per-target Prometheus labels exist only when tenant metrics are explicitly enabled on a trusted scrape path.
Check provider latency and errors before increasing concurrency. Raising queue capacity alone delays rejection when downstream capacity remains below arrival rate. Record the target, arrival rate, service rate, mitigation, and recovery evidence.
Check /ready; it must return 503 while Galley cannot be pinged. Confirm database connectivity, TLS validity, pool exhaustion, storage capacity, migration state, and failover status.
Do not repeatedly restart all controllers during a database incident. Keep one evidence-producing path alive and coordinate recovery with the database owner.
Inspect the authenticated operator status backup.error_class, storage capacity, encryption-key availability, and restore-verification dependencies. Do not delete older artifacts until a new restore-verified backup lands.
This alert covers the built-in SQLite scheduler. PostgreSQL and MySQL require database-native backup, PITR, age, and restore-evidence monitoring.
Compare jikji_backup_last_success_timestamp_seconds with the configured interval and confirm the acting node is leader. Inspect bounded attempt state, artifact publication, and restore verification before changing the interval.
Resolve only after a new verified artifact is present and its digest, schema version, measured RPO, and measured RTO are retained outside the workload.
Identify the affected provider and model, then inspect upstream status, credential expiry, rate-limit windows, request size, and breaker state.
Use a reviewed fallback only if data residency, price, context window, tool-use behavior, and tenant policy remain valid. Record the routing change and its rollback condition.
Treat the breaker as evidence of an upstream failure. Verify recovery attempts and upstream health before manually resetting it.
Repeated resets can amplify an otherwise bounded dependency outage. Prefer the configured half-open recovery path and watch success and latency after it admits probes.
Separate TLS handshake failures from certificate or peer authorization denials with the bounded direction, operation, and status labels on jikji_relay_operations_total. Check certificate validity, trust-bundle rollout, clock synchronization, and the configured peer allowlist.
The metric intentionally omits peer keys, node IDs, tenants, addresses, and certificates. Attribute failures with bounded relay audit events and trusted operator logs; never widen the allowlist or disable certificate verification as incident containment.
Inspect upstream connect outcomes in jikji_relay_operations_total, then check the selected service-node record, DNS, network policy, TLS server name, root CA rotation, and upstream listener health. Route-event evidence identifies a selected node only when tenant metrics are explicitly enabled on a protected scrape path.
Do not bypass TLS verification to restore connectivity. Prove recovery with successful upstream connection samples and a bounded relay request before resolving the incident.
Correlate jikji_trace_exporter_health, jikji_trace_exporter_consecutive_failures, and the success/failure series in jikji_trace_exporter_exports_total. Check collector reachability, OTLP protocol and endpoint, TLS trust, credentials, collector capacity, and recent configuration changes without exposing exporter secrets.
Serving health can remain green while telemetry delivery is degraded. Prove recovery with a new successful export, a reset consecutive-failure gauge, and a current jikji_trace_exporter_last_success_timestamp_seconds; do not treat application readiness alone as trace-delivery evidence.
Use the component and module labels to select the owning runbook, logs, and dashboard. Confirm the matching /ready dependency check.
Do not silence a persistent degraded value without an accepted incident or maintenance record, a named owner, and an expiry.
Compare resident memory, Go heap, active streams, in-flight requests, queue depth, and recent configuration changes. Capture a bounded profile before restart when policy allows.
Raise a limit only when the pod and node both retain capacity. Otherwise drain the pod, reduce admitted work, or correct the growth source.
Break down jikji_metric_cardinality_overflow_total by bounded family. A sustained increase means observations are being folded into the other series, so registry memory remains bounded but diagnostic precision is being lost.
Check for unnormalized paths, uncontrolled provider, model, or status values, tenant churn, and probing before raising a cap. Prove recovery with the overflow rate returning to zero.
Jikji does not currently emit a durable latest-verified-backup timestamp. Monitor the database platform's backup and point-in-time recovery system independently.