Send telemetry

Fonoster services can send traces, metrics and logs over OTLP/HTTP to any OpenTelemetry collector. Telemetry is off by default: nothing is sent until you set an endpoint. Every call’s spans, logs and metrics carry the attribute fonoster.call_ref, so you can follow one call across services.

Before you begin

You need Fonoster installed with Docker Compose. See Install with Docker Compose. You also need a collector that accepts OTLP over HTTP (usually port 4318) and that the containers can reach. If you don’t have one yet, run a local collector first.

Turn on telemetry

Set the endpoint in .env

In your fonoster folder, set your collector’s OTLP/HTTP endpoint in .env:

OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.example.com:4318

Give the base URL only. Each service sends to /v1/traces, /v1/metrics and /v1/logs under it.

Recreate the services

Recreate the containers so they get the new value:

docker compose up -d --force-recreate

Compose passes OTEL_EXPORTER_OTLP_ENDPOINT to the API server, AppBridge, the Roomr node and the hosted Autopilot. Each one reads it through telemetry.otlpEndpoint in its file in config/:

telemetry: { otlpEndpoint: "${OTEL_EXPORTER_OTLP_ENDPOINT:-}" }

An empty value turns telemetry off. To point one service at a different collector, change that line in its config/<service>.example.yaml and run npx -p @fonoster/common fonoster-config init again.

What each service sends

Service	`service.name`	Traces	Metrics
API server	`fonoster-apiserver`	One span per API request	`fonoster.apiserver.grpc.requests` (by method and status), `fonoster.apiserver.records.indexed`
AppBridge	`fonoster-appbridge`	`appbridge.call`, one span per call	`fonoster.appbridge.calls.total`, `fonoster.appbridge.calls.active`
Roomr node	`fonoster-roomr`	None	`fonoster.roomr.calls.incoming` (by outcome), `fonoster.roomr.rooms.active`

AppBridge also sends its logs; the Roomr node’s logs and the API server’s stay in docker compose logs. A call’s trace starts in AppBridge, under the traceparent header of the incoming call when it has one. Metrics are sent every 15 seconds.

Other settings

The services also read these standard OpenTelemetry variables:

Variable	What it does
`OTEL_EXPORTER_OTLP_HEADERS`	Headers sent with every export, as `key=value,key=value` (for example an API key for a hosted backend)
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `..._METRICS_ENDPOINT`, `..._LOGS_ENDPOINT`	Full URL for one signal, used instead of the base endpoint
`OTEL_SERVICE_NAME`	Replaces the `service.name` above
`OTEL_SDK_DISABLED`	`true` turns telemetry off

Compose passes OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_PROTOCOL from .env to each of those services. For a hosted collector that needs an auth header (Grafana Cloud, Honeycomb), set both in .env (and keep the file private, it holds a credential):

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-appbridge-prod-us-east-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 of instance:token>

With Helm, set otel.endpoint and either otel.headers or, to keep the credential out of values, otel.headersSecret.name (and .key, default headers) pointing at a Secret that holds the string. To pass any other variable, add it to the service’s environment in a Compose file of your own (see the next section for how to add one). The hosted Autopilot reports autopilot.session (a child of AppBridge’s call span, so one trace covers the call), autopilot.turn (the caller’s final transcript to the end of the reply) and autopilot.llm.request spans, plus fonoster.autopilot.turns, fonoster.autopilot.llm.requests and fonoster.autopilot.llm.latency_ms (the sum of the time to the first token; divide by requests).

Try it with a local collector

Add a collector

Save this collector config as config/otel-collector.yaml. It prints what it receives:

receivers:
  otlp:
    protocols:
      http: { endpoint: 0.0.0.0:4318 }
exporters:
  debug: { verbosity: detailed }
service:
  pipelines:
    traces: { receivers: [otlp], exporters: [debug] }
    metrics: { receivers: [otlp], exporters: [debug] }
    logs: { receivers: [otlp], exporters: [debug] }

Then save this as compose.telemetry.yaml, next to compose.yaml. It runs the collector on Fonoster’s network:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.111.0
    restart: unless-stopped
    volumes:
      - ./config/otel-collector.yaml:/etc/otelcol-contrib/config.yaml:ro

Point Fonoster at it

In .env, add the file to COMPOSE_FILE and set the endpoint:

COMPOSE_FILE=compose.yaml:compose.telemetry.yaml
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

If COMPOSE_FILE already lists compose.tls.yaml, keep it and add :compose.telemetry.yaml at the end.

Start it and watch

Start the collector, recreate the services, and follow the collector’s output:

docker compose up -d --force-recreate
docker compose logs -f otel-collector

Within 15 seconds, metrics arrive from fonoster-appbridge and fonoster-roomr. Sign in to the dashboard to see API server spans. Make a call to see its appbridge.call span.

Get started

Autopilot

Streams

Telephony

Observe

Workspace

Reference

Self-hosting

Private regions

Hacking Fonoster

Before you begin

Turn on telemetry

What each service sends

Other settings

Try it with a local collector

​Before you begin

​Turn on telemetry

​What each service sends

​Other settings

​Try it with a local collector

Before you begin

Turn on telemetry

What each service sends

Other settings

Try it with a local collector