> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fonoster.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Send telemetry

> Send traces, metrics and logs from Fonoster to your OpenTelemetry collector.

Fonoster services can send traces, metrics and logs over OTLP/HTTP to any OpenTelemetry collector.
Telemetry is off by default: nothing is sent until you set an endpoint. Every call's spans, logs and
metrics carry the attribute `fonoster.call_ref`, so you can follow one call across services.

## Before you begin

You need Fonoster installed with Docker Compose. See
[Install with Docker Compose](https://docs.fonoster.com/self-hosting/install-compose.md). You also need a collector that
accepts OTLP over HTTP (usually port `4318`) and that the containers can reach. If you don't have
one yet, [run a local collector](#try-it-with-a-local-collector) first.

## Turn on telemetry

**1. Set the endpoint in .env**

In your `fonoster` folder, set your collector's OTLP/HTTP endpoint in `.env`:

```bash theme={"system"}
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.example.com:4318
```

Give the base URL only. Each service sends to `/v1/traces`, `/v1/metrics` and `/v1/logs` under
it.

**2. Recreate the services**

Recreate the containers so they get the new value:

```bash theme={"system"}
docker compose up -d --force-recreate
```

Compose passes `OTEL_EXPORTER_OTLP_ENDPOINT` to the API server, AppBridge, the Roomr node
and the hosted Autopilot. Each one reads it through `telemetry.otlpEndpoint` in its file in
`config/`:

```yaml theme={"system"}
telemetry: { otlpEndpoint: "${OTEL_EXPORTER_OTLP_ENDPOINT:-}" }
```

An empty value turns telemetry off. To point one service at a different collector, change that
line in its `config/<service>.example.yaml` and run `npx -p @fonoster/common fonoster-config init`
again.

## What each service sends

| Service | `service.name` | Traces | Metrics |
| - | - | - | - |
| API server | `fonoster-apiserver` | One span per API request | `fonoster.apiserver.grpc.requests` (by method and status), `fonoster.apiserver.records.indexed` |
| AppBridge | `fonoster-appbridge` | `appbridge.call`, one span per call | `fonoster.appbridge.calls.total`, `fonoster.appbridge.calls.active` |
| Roomr node | `fonoster-roomr` | None | `fonoster.roomr.calls.incoming` (by outcome), `fonoster.roomr.rooms.active` |

AppBridge also sends its logs; the Roomr node's logs and the API server's stay in
`docker compose logs`. A call's trace starts in AppBridge, under the `traceparent` header of the
incoming call when it has one.
Metrics are sent every 15 seconds.

## Other settings

The services also read these standard OpenTelemetry variables:

| Variable | What it does |
| - | - |
| `OTEL_EXPORTER_OTLP_HEADERS` | Headers sent with every export, as `key=value,key=value` (for example an API key for a hosted backend) |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `..._METRICS_ENDPOINT`, `..._LOGS_ENDPOINT` | Full URL for one signal, used instead of the base endpoint |
| `OTEL_SERVICE_NAME` | Replaces the `service.name` above |
| `OTEL_SDK_DISABLED` | `true` turns telemetry off |

Compose passes `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS` and
`OTEL_EXPORTER_OTLP_PROTOCOL` from `.env` to each of those services. For a hosted collector that
needs an auth header (Grafana Cloud, Honeycomb), set both in `.env` (and keep the file private, it
holds a credential):

```bash theme={"system"}
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-appbridge-prod-us-east-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 of instance:token>
```

With Helm, set `otel.endpoint` and either `otel.headers` or, to keep the credential out of values,
`otel.headersSecret.name` (and `.key`, default `headers`) pointing at a Secret that holds the string.
To pass any other variable, add it to the service's `environment` in a Compose file of your own (see
the next section for how to add one).

The hosted Autopilot reports `autopilot.session` (a child of AppBridge's call span, so one
trace covers the call), `autopilot.turn` (the caller's final transcript to the end of the reply) and
`autopilot.llm.request` spans, plus `fonoster.autopilot.turns`, `fonoster.autopilot.llm.requests`
and `fonoster.autopilot.llm.latency_ms` (the sum of the time to the first token; divide by requests).

## Try it with a local collector

**1. Add a collector**

Save this collector config as `config/otel-collector.yaml`. It prints what it receives:

```yaml theme={"system"}
receivers:
  otlp:
    protocols:
      http: { endpoint: 0.0.0.0:4318 }
exporters:
  debug: { verbosity: detailed }
service:
  pipelines:
    traces: { receivers: [otlp], exporters: [debug] }
    metrics: { receivers: [otlp], exporters: [debug] }
    logs: { receivers: [otlp], exporters: [debug] }
```

Then save this as `compose.telemetry.yaml`, next to `compose.yaml`. It runs the collector on
Fonoster's network:

```yaml theme={"system"}
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.111.0
    restart: unless-stopped
    volumes:
      - ./config/otel-collector.yaml:/etc/otelcol-contrib/config.yaml:ro
```

**2. Point Fonoster at it**

In `.env`, add the file to `COMPOSE_FILE` and set the endpoint:

```bash theme={"system"}
COMPOSE_FILE=compose.yaml:compose.telemetry.yaml
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
```

If `COMPOSE_FILE` already lists `compose.tls.yaml`, keep it and add
`:compose.telemetry.yaml` at the end.

**3. Start it and watch**

Start the collector, recreate the services, and follow the collector's output:

```bash theme={"system"}
docker compose up -d --force-recreate
docker compose logs -f otel-collector
```

Within 15 seconds, metrics arrive from `fonoster-appbridge` and `fonoster-roomr`. Sign in to
the dashboard to see API server spans. Make a call to see its `appbridge.call` span.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.