Skip to main content

Application Observability

Hannibal phase-1 operational telemetry uses the existing self-hosted Prometheus, Grafana, Alertmanager, Loki, and infrastructure exporters. The backend exposes application and Node.js runtime metrics at GET /metrics on the dedicated Docker-internal port 9464. The product HTTP listener on port 3001 does not serve metrics.

This phase does not include OpenTelemetry traces, continuous profiling, frontend browser telemetry, or AI-service metrics.

Backend metrics​

HTTP RED metrics​

MetricTypePurpose
http_requests_totalCounterCompleted requests by method, route template, and status
http_request_duration_secondsHistogramEnd-to-end Express request duration
http_requests_in_progressGaugeRequests currently executing
http_request_aborted_totalCounterConnections closed before completion
http_response_size_bytesHistogramResponse content length when known

HTTP metrics use these bounded labels:

  • service="hannibal-backend"
  • environment — deployment identity from ENVIRONMENT (dev, dev1, dev2, or production), falling back to NODE_ENV only when unset
  • method — GET, POST, PUT, PATCH, DELETE, OPTIONS, HEAD, or OTHER; unsupported methods are coalesced to keep cardinality bounded
  • route — a literal router mount plus the Express route template
  • status — the final HTTP status code emitted by Express

Raw URLs are never labels. Unmatched requests use route="unmatched", and array/regular-expression routes use route="complex-route". User IDs, usernames, order IDs, fixture IDs, provider account identifiers, request bodies, credentials, cookies, and financial values must never be metric labels.

HTTP recording starts before CORS and body parsing, so malformed, oversized, and otherwise early-rejected API requests remain observable. The recorder only accepts /api paths, while scrapes use a separate HTTP server, so operational traffic and static assets do not inflate product request-rate or latency metrics.

Status instrumentation is independent of the response body shape: 201, 429, 503, and every other final status are recorded as sent. An endpoint that embeds an application error in an HTTP 200 response will still be counted as successful HTTP traffic. Such endpoints should be corrected to use the appropriate HTTP status; the metrics layer deliberately does not inspect response bodies because doing so would add overhead and risk exposing private data.

Metrics endpoint network boundary​

The internal GET :9464/metrics listener intentionally has no application-level authentication. Access is enforced by topology: port 9464 is exposed only to the private Docker network and has no host port mapping. Prometheus scrapes strykr-backend:9464, while the product listener on port 3001 returns 404 for /metrics. A reverse-proxy catch-all on the product port therefore cannot publish runtime telemetry.

Prometheus itself is host-bound to loopback on port 9090, remains reachable to Grafana and exporters over the monitoring Docker network, and does not enable the destructive Prometheus admin API. Deployment reloads are performed through the loopback-only lifecycle endpoint after promtool validates the mounted configuration.

Deployment workflows require backend health and the private metrics listener to return 200, then require the public /metrics path to return 404. A separate blackbox probe continuously alerts if either the dev or production public path returns anything other than the required 404, covering proxy changes made outside the application deployment workflow.

The deploy check intentionally requires exactly 404. This proves the metrics route is absent from the public application listener. The continuous blackbox module disables redirects so it evaluates the same direct response. A 401 or 403 would mean the operational endpoint is publicly reachable behind application authentication, while a redirect or 5xx can hide an edge or deployment error; none satisfy the private-network contract.

Continuous Prometheus, blackbox, and Grafana monitoring covers the long-lived dev and production environments. dev1 and dev2 are ephemeral isolated validation stacks and intentionally are not continuous monitoring targets. Their deployment workflows still verify backend health, the private metrics listener, and the exact public 404 privacy contract on every deployment.

If a future deployment cannot preserve this network boundary, add access control at the edge and update the Prometheus scrape configuration in the same change before making the endpoint reachable. Do not expose the endpoint first and rely on a later security change.

Node.js runtime metrics​

prom-client default metrics provide:

  • process_cpu_seconds_total
  • process_resident_memory_bytes
  • nodejs_heap_size_used_bytes and nodejs_heap_size_total_bytes
  • nodejs_eventloop_lag_p50_seconds, p90, and p99
  • nodejs_gc_duration_seconds
  • nodejs_active_handles_total
  • nodejs_active_requests_total
  • process start time, open file descriptors, heap spaces, and Node version

GC histogram bucket series appear after the process observes its first garbage collection event; the metric descriptor is present from startup.

CPU is reported as CPU-seconds. rate(process_cpu_seconds_total[5m]) is the average number of CPU cores consumed by the Node process during five minutes; it is not a whole-host percentage.

Useful PromQL queries​

Request rate:

sum(rate(http_requests_total{service="hannibal-backend"}[5m])) by (route)

5xx ratio:

sum(rate(http_requests_total{service="hannibal-backend",status=~"5.."}[5m]))
/
clamp_min(sum(rate(http_requests_total{service="hannibal-backend"}[5m])), 0.001)

p95 latency by route:

histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket{service="hannibal-backend"}[5m])) by (le, route)
)

Node process CPU cores:

rate(process_cpu_seconds_total{service="hannibal-backend"}[5m])

Resident and heap memory:

process_resident_memory_bytes{service="hannibal-backend"}
nodejs_heap_size_used_bytes{service="hannibal-backend"}

Event-loop delay:

nodejs_eventloop_lag_p99_seconds{service="hannibal-backend"}

Prometheus keeps the configured retention window, so the Grafana time picker or Prometheus HTTP API can evaluate these expressions at a specified historical time. Container CPU attribution remains limited by the current cAdvisor labels; the backend's own process metrics are the authoritative phase-1 Node.js view.

Dashboard and alerts​

Grafana provisions Application Health from infra/monitoring/grafana/dashboards/application-health.json. It shows target health, request rate, 5xx ratio, latency percentiles, route-level performance, process CPU, RSS/heap, event-loop delay, garbage collection, and active work.

Initial alerts cover:

  • any configured Prometheus target down for two minutes, or the expected backend target disappearing from service discovery entirely;
  • repeated 5xx responses and a sustained high 5xx ratio;
  • p95/p99 API latency;
  • sustained Node CPU-core saturation;
  • backend RSS above 1 GiB;
  • event-loop p99 above 100 ms and 1 second;
  • any public dev or production /metrics path returning a status other than the required 404;
  • absence of the continuous public-metrics privacy probe.

The thresholds are initial operational guardrails based on the current single Node.js process and observed production RSS baseline. Review them after 30 days of history rather than treating them as permanent capacity limits.

Viewing dev performance​

After the observability changes are merged and deployed to dev, Grafana automatically discovers the provisioned dashboard within about 30 seconds. Sign in with the existing dev Grafana credentials and open Dashboards → Application Health, select dev, and choose the time range you want to inspect.

Grafana is bound to the server's loopback interface and is not a public admin surface. Open an SSH tunnel from your workstation:

ssh -L 3030:127.0.0.1:3030 strykrdev

Then open http://localhost:3030. The former public http://dev.strykr.io:3030 endpoint is intentionally unavailable. Do not commit or share Grafana credentials.

Deployments explicitly load the server-managed /root/strykr/infra/docker/.env for the monitoring stack and fail before recreating Grafana if its required admin or Timescale datasource variables are missing. The file remains untracked and its values must never be printed in deployment logs.

The same reconciliation starts and verifies Prometheus, Grafana, Blackbox Exporter, and Alertmanager together. A deploy fails if the private probes are not active or if the loopback-only operator services are not ready. Prometheus, Blackbox Exporter, and Alertmanager candidate configurations are validated in pinned tool images before mutation; the config-owning containers are then force-recreated so bind mounts cannot retain stale pre-pull file inodes. Both dev and production call scripts/reconcile-monitoring.sh for this shared contract. The script discovers each operator service's actual loopback host port from Docker before readiness, reload, or query requests, so .env port overrides cannot make deployment verification inspect the wrong listener. Production materializes the helper from the commit that supplied the running deployment workflow, rather than from the requested rollback ref. This keeps the workflow's current verification contract in force while older application revisions are deployed.

The dashboard environment selector is sourced from process_start_time_seconds, an application metric emitted immediately at process startup. It therefore works before the first API request while retaining the backend registry's environment label; Prometheus's synthetic up metric does not inherit labels emitted by the scraped application.

Use Request rate by route to see which APIs are busiest, p95 latency by route to find slow APIs, and the 5xx, CPU, memory, event-loop, garbage collection, and active-work panels to correlate slow or failing APIs with runtime pressure. For a status-by-route breakdown, run this in Grafana Explore:

sum(rate(http_requests_total{service="hannibal-backend",environment="dev"}[5m])) by (route, status)

Validation​

Validation belongs in the Hannibal dev workflow, not on a developer laptop:

  1. Run focused backend unit tests and typecheck in the selected dev workflow.

  2. Validate Prometheus configuration and rules with promtool check config and promtool check rules in the Prometheus image/environment.

  3. Parse all provisioned dashboard JSON.

  4. Deploy to dev and confirm hannibal-backend is up in Prometheus.

  5. Confirm port 9464 has no host mapping, Prometheus can scrape strykr-backend:9464, and the product listener returns 404 for /metrics.

  6. Verify the public negative case directly:

    test "$(curl -sS -o /dev/null -w '%{http_code}' https://dev.strykr.io/metrics)" = "404"

  7. Exercise representative 2xx, 4xx, and 5xx API paths.

  8. Confirm route labels contain templates and never real identifiers.

  9. Confirm Application Health panels populate and the target-down alert is not firing for intentionally uninstrumented frontend or AI services.