Oz > Self-hosting

Self-hosted worker monitoring

Monitor self-hosted Oz workers with OpenTelemetry metrics. Export to Prometheus, OTLP, or console to track worker health, task throughput, and saturation.

The oz-agent-worker daemon exports infrastructure-level metrics over OpenTelemetry, giving your team real-time visibility into worker health, task throughput, and capacity. Combine these metrics with the Oz dashboard for full observability across both the orchestration plane and your self-hosted compute.

Key features

Prometheus scrape — Expose a /metrics endpoint for Prometheus to scrape, with optional PodMonitor support for the Prometheus Operator.
OTLP push — Push metrics to any OpenTelemetry-compatible collector (Grafana Alloy, Datadog Agent, New Relic, etc.).
Standard configuration — Exporter selection uses the standard OpenTelemetry environment variables, so the worker integrates with your existing observability stack without custom configuration.
Pre-seeded series — All metric series appear at startup (before any tasks run), so dashboards and alerts can reference them immediately.

How it works

The worker uses the go.opentelemetry.io/contrib/exporters/autoexport package to select an exporter at runtime based on the OTEL_METRICS_EXPORTER environment variable. Supported values:

prometheus — Starts an in-process HTTP server serving /metrics.
otlp — Pushes metrics over OTLP (HTTP/protobuf by default).
console — Writes metrics to stdout (useful for debugging).
none — Disables metrics export entirely.

When OTEL_METRICS_EXPORTER is unset, the worker defaults to OTLP push targeting OTEL_EXPORTER_OTLP_ENDPOINT (which itself defaults to http://localhost:4318 for http/protobuf or http://localhost:4317 for grpc).

All metrics carry resource attributes (service.name=oz-agent-worker, service.version, worker.id, worker.backend) so each worker process shows up as a distinct series in your monitoring system.

Enable Prometheus scrape

Set these environment variables before starting the worker to expose a Prometheus-compatible /metrics endpoint:

export OTEL_METRICS_EXPORTER=prometheus
export OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0
export OTEL_EXPORTER_PROMETHEUS_PORT=9464
oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"

Verify the endpoint is serving metrics:

curl -s localhost:9464/metrics | grep oz_worker_

Expected outcome: You see oz_worker_connected, oz_worker_tasks_active, and other oz_worker_* metric families in the output.

Enable OTLP push

Set these environment variables to push metrics to an OpenTelemetry collector:

export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability.svc:4318
oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"

The worker pushes metrics at the SDK's default interval. Configure the collector endpoint, protocol, and headers using standard OTLP exporter environment variables.

Helm chart configuration

The Helm chart includes built-in support for metrics. Enable metrics with metrics.enabled=true:

helm install oz-agent-worker ./charts/oz-agent-worker \
  --namespace warp-oz \
  --set worker.workerId=my-worker \
  --set image.tag=VERSION \
  --set metrics.enabled=true

With metrics.enabled=true and the default metrics.exporter=prometheus, the chart adds:

A containerPort: metrics (default 9464) on the worker Deployment.
The OTEL_METRICS_EXPORTER, OTEL_EXPORTER_PROMETHEUS_HOST, and OTEL_EXPORTER_PROMETHEUS_PORT environment variables.
A namespace-scoped Service named <release>-oz-agent-worker-metrics with prometheus.io/scrape annotations.
Optionally, a PodMonitor (metrics.podMonitor.create=true) for clusters using the Prometheus Operator.

Helm values

Core:

metrics.enabled — Enable metrics export. Defaults to false.
metrics.exporter — Exporter type: prometheus (default), otlp, console, or none.
metrics.port — Port for the Prometheus exporter. Defaults to 9464. Ignored for otlp/console.
metrics.extraEnv — Extra environment variables for the worker container (e.g., OTEL_EXPORTER_OTLP_ENDPOINT).

Service (Prometheus scrape):

metrics.service.create — Create a metrics Service. Defaults to true.
metrics.service.type — Service type. Defaults to ClusterIP.
metrics.service.annotations — Annotations on the Service. Defaults include prometheus.io/scrape: "true".

PodMonitor (Prometheus Operator):

metrics.podMonitor.create — Create a PodMonitor. Defaults to false (avoids requiring monitoring.coreos.com CRDs).
metrics.podMonitor.interval — Scrape interval. Defaults to 30s.
metrics.podMonitor.scrapeTimeout — Scrape timeout. Defaults to 10s.
metrics.podMonitor.additionalLabels — Extra labels on the PodMonitor resource.

OTLP push via Helm

To push metrics to an OTLP collector instead of exposing a Prometheus endpoint, set metrics.exporter=otlp and forward the endpoint via metrics.extraEnv:

metrics:
  enabled: true
  exporter: otlp
  extraEnv:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://otel-collector.observability.svc:4318

Metric catalog

All metrics use the oz_worker_ prefix. Each worker process emits a distinct set of series, identified by the resource attributes service.name, service.version, worker.id, and worker.backend.

oz_worker_connected (gauge) — 1 while the worker has an active WebSocket connection to Oz's backend, 0 otherwise.
oz_worker_tasks_active (gauge / UpDownCounter) — Tasks currently executing on this worker.
oz_worker_tasks_max_concurrent (gauge) — Configured concurrency limit (0 means unlimited).
oz_worker_tasks_claimed_total (counter) — Total tasks accepted since process start.
oz_worker_tasks_rejected_total{reason} (counter) — Tasks the worker declined (e.g., reason="at_capacity").
oz_worker_tasks_completed_total{result} (counter) — Completed tasks labeled result="succeeded" or result="failed".
oz_worker_task_duration_seconds{result} (histogram) — Wall-clock task duration on the worker, labeled by result.
oz_worker_websocket_reconnects_total{reason} (counter) — WebSocket reconnect attempts (e.g., reason="dial_failed", reason="remote_close"). Spikes indicate flapping workers.
oz_worker_info{version,backend,worker_id} (gauge, constant 1) — Build and runtime metadata. Useful for joining other series by labels.

Grafana dashboard

A ready-to-import Grafana dashboard is published in the grafana/ directory of the oz-agent-worker repository. It covers all five areas of worker observability — health, throughput, latency, failures, and connectivity — and works with any Prometheus datasource.

Dashboard sections

Worker Health — Connected workers, active workers, tasks active, and fleet saturation gauge.
Task Throughput — Task completion rate by result (succeeded/failed/cancelled), 5-minute success rate, and rejection rate.
Task Duration — p50 / p95 / p99 latency over a sliding 5-minute window.
Failure Analysis — Failure rate by reason (e.g., image_pull, container_oom, unschedulable) and by phase (assignment, backend, cleanup).
Connection Health — WebSocket reconnect rate by reason, and a worker fleet info table showing each connected worker's ID, version, and backend.

The dashboard also includes an Instance variable so you can filter to a specific worker pod or see fleet-wide aggregations.

Importing the dashboard

Enable Prometheus metrics on your workers. See Enable Prometheus scrape above, or set metrics.enabled=true in the Helm chart.
In Grafana, go to Dashboards → Import.
Click Upload JSON file and select oz-agent-worker-overview.json from the grafana/ directory, or paste the raw file URL:
```
https://raw.githubusercontent.com/warpdotdev/oz-agent-worker/main/grafana/oz-agent-worker-overview.json
```
When prompted, choose your Prometheus datasource and click Import.

You can also import via the Grafana API:

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  --data-binary @oz-agent-worker-overview.json \
  http://your-grafana/api/dashboards/import

Prometheus alerting rules

The same grafana/ directory includes alerts.yaml — a set of Prometheus alerting rules covering the most common operational issues:

OzWorkerFleetEmpty (critical) — No workers connected for 5 minutes.
OzWorkerDisconnected (warning) — A specific worker has been disconnected for 2 minutes.
OzWorkerReconnectStorm (warning) — A worker is reconnecting at more than 0.1/s.
OzWorkerFleetSaturated (warning) — Fleet concurrency utilization is above 90%.
OzWorkerTasksRejected (warning) — Workers are actively rejecting tasks.
OzWorkerHighFailureRate (warning) — More than 10% of completed tasks failed.
OzWorkerImagePullFailures (warning) — Container image pulls are failing.
OzWorkerContainerOOM (warning) — Task containers are being OOM-killed.
OzWorkerUnschedulablePods (warning) — Kubernetes task pods cannot be scheduled.

All thresholds are conservative starting points — tune them for your fleet size and SLOs. See grafana/README.md for instructions on loading the rules into vanilla Prometheus or the Prometheus Operator.

Sample PromQL queries

Direct mappings for common operational questions:

Workers available:
```
sum(oz_worker_connected)
```
Workers active (running at least one task):
```
count(oz_worker_tasks_active > 0)
```
Fleet saturation:
```
sum(oz_worker_tasks_active) / sum(oz_worker_tasks_max_concurrent > 0)
```
This ratio is only meaningful when every worker has a non-zero oz_worker_tasks_max_concurrent. Workers configured with 0 (unlimited) are excluded from the denominator, which can make the saturation result look misleadingly high or undefined for fleets that mix bounded and unlimited workers.

Task success rate (5-minute window):

sum(rate(oz_worker_tasks_completed_total{result="succeeded"}[5m]))
/ sum(rate(oz_worker_tasks_completed_total[5m]))

Task duration p95:

histogram_quantile(0.95, sum by (le) (rate(oz_worker_task_duration_seconds_bucket[5m])))

Failure rate:

sum(rate(oz_worker_tasks_completed_total{result="failed"}[5m]))

Reconnect storms (alert threshold):

sum(rate(oz_worker_websocket_reconnects_total[5m])) > 0.1

Disabling metrics

To fully disable metrics export, set OTEL_METRICS_EXPORTER=none:

export OTEL_METRICS_EXPORTER=none
oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"

Or in the Helm chart:

metrics:
  enabled: false

Self-hosting overview — Architecture, decision guide, and Enterprise requirements.
Self-hosted worker reference — CLI flags, config file schema, and metrics environment variables.
Managed: Kubernetes — Helm chart deployment, including metrics values.
Troubleshooting — Diagnostics for metrics issues and other common problems.
Security and networking — Network egress and data boundaries.