# Tracing Internals

Fullsend's distributed tracing system records structured telemetry for every
agent run. This document explains how the tracing implementation works and
how to extend it. It is aimed at contributors modifying the telemetry
package or the span instrumentation in the run command.

For enabling tracing on an installation, see
[How To Emit Traces](../user/how-to-emit-traces.md). For span schemas and
attribute definitions, see the
[Tracing Reference](../infrastructure/distributed-tracing.md). The design
is specified in
[ADR 0050](../../ADRs/0050-distributed-tracing-instrumentation.md).

## How the tracing system is structured

The tracing code is split between two locations:

- **`internal/telemetry/`** - the TracerProvider, exporters, and W3C
  traceparent helpers.
- **`internal/cli/run.go`** - span creation, attribute assignment, and
  trace context propagation to child scripts and the in-sandbox runtime.

The telemetry package owns the provider and exporters. `run.go` owns the
span lifecycle: it decides when spans start and end, and which attributes
they carry.

### Package layout

```
internal/telemetry/
├── telemetry.go          Setup, TracerProvider wiring, env parsing
├── fileexporter.go       Synchronous OTLP JSON file exporter
├── trace.go              W3C traceparent formatting helpers
├── main_test.go
├── telemetry_test.go
├── fileexporter_test.go
├── otlpsink_test.go
└── trace_test.go
```

## TracerProvider setup

`telemetry.Setup(dir, version)` returns a `trace.Tracer` and a cleanup
function. It creates a `TracerProvider` with two span processors:

1. **fileExporter** via `SimpleSpanProcessor`: writes every completed span
   as one OTLP JSON line to `run-telemetry.jsonl`. Calls `f.Sync()` after
   each write so spans survive a process crash.

2. **otlptracehttp** via `BatchSpanProcessor` wrapped in
   `parentSampledProcessor`: exports to a remote backend when an
   `OTEL_EXPORTER_OTLP_*ENDPOINT` env var is set.

If neither exporter can be created (bad directory, SDK disabled), `Setup`
returns a noop tracer. Telemetry failures never affect the run.

```
telemetry.Setup()
│
├── OTEL_SDK_DISABLED check   → noop tracer if "true"
├── os.OpenFile(jsonl)        → noop tracer on error
│
├── SimpleSpanProcessor(fileExporter)           ← always present
│
└── OTEL_EXPORTER_OTLP_*ENDPOINT check
    ├── validateEndpoints()   → stderr warning, skip
    ├── newOTLPExporter()     → stderr warning, skip
    └── parentSampledProcessor(BatchSpanProcessor(otlpExporter))
```

### Why parentSampledProcessor exists

The OTel SDK's `AlwaysSample` sampler records all spans locally, which is
needed for the file exporter. But `AlwaysSample` ignores an upstream
unsampled decision. The `parentSampledProcessor` wraps the OTLP batch
processor and suppresses the entire trace from export when the root span's
remote parent has the W3C sampled flag unset (`-00`). It tracks suppressed
trace IDs in a `sync.Map` so child spans under the same trace are also
dropped.

Result: the file exporter always writes all spans; the OTLP exporter
respects upstream sampling.

### Endpoint validation

`validateEndpoints` rejects non-http(s) URLs and unsupported protocols
before creating the exporter. A malformed endpoint produces a stderr
warning; the SDK's default `localhost:4318` fallback is never used
silently.

## Span lifecycle in run.go

`run.go` creates four span types arranged in a parent-child hierarchy:

```
run (root)
├── sandbox_create    (gen_ai.operation.name=create_agent)
└── agent             (one per iteration; gen_ai.operation.name=invoke_agent)
    └── execute_tool  (one per id-bearing tool call; gen_ai.operation.name=execute_tool)
```

### Root span

Created by `resolveTraceIdentity()`. When an inbound `TRACEPARENT` env var
is present, the W3C propagator extracts a remote span context and the root
span is `SpanKindConsumer`. Otherwise it is `SpanKindInternal`.

Start attributes include: `fullsend.agent`, `fullsend.work_item_id`,
`gen_ai.operation.name`, `gen_ai.agent.name`.

End attributes (set in a deferred cleanup): `exit_code`,
`fullsend.cost_usd`, `fullsend.num_turns`, `fullsend.tool_calls`,
`fullsend.iterations`.

For the full attribute list (including `fullsend.security_trace_id` and
`fullsend.prescript.*`), see the
[Tracing reference attribute table](../infrastructure/distributed-tracing.md#span-attributes).

### sandbox_create span

Started before sandbox creation, ended after bootstrap completes. Carries
`gen_ai.operation.name=create_agent`.

### agent spans

One per iteration (validation loop iterations included). Started before
`Exec()`, ended after output extraction.

The helper functions `agentSpanStartAttrs()` and `agentSpanEndAttrs()`
build the attribute slices. Start attributes: `iteration`,
`gen_ai.operation.name`, `gen_ai.agent.name`. End attributes: `iteration`,
`exit_code`, `gen_ai.system` and `gen_ai.provider.name` (same serving-endpoint
value), model, token counts, `fullsend.cost_usd`, `fullsend.runtime`,
`fullsend.tool_calls`. Multi-provider runtimes resolve the provider from the
effective model via `runtime.GenAISystemFor`; `System()` is only the fallback.

### execute_tool spans

One per id-bearing tool call the runtime reports (Claude Code today), up to
1,024 per iteration, a child of that iteration's agent span, named
`execute_tool <tool name>`. `toolSpanTracker`
(`internal/cli/tool_spans.go`) starts the span when it handles the
`ToolUseEvent` and ends it when it handles the matching `ToolResultEvent`.
`iterationEventHandler` runs the console renderer first, then the tracker,
then the Level 3 collector, so both timestamps are runner-side receipt
instants on one clock, each trailing the sandbox by the pipe latency and the
parser's decode of the line: the start also trails receipt by the renderer's output for
the call (the renderer prints nothing for a result), and neither waits on
the collector's redaction pass over its own event (events are handled one
at a time, so with several calls open an instant can still trail the
collector's pass over an earlier event, such as another call's large
result) — the start is arguments-complete, not execution start. Attributes:
`gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`,
`gen_ai.tool.call.id`; a result flagged `is_error` sets
`error.type=tool_error` and status Error. A result whose stream line
exceeded the parser's 1 MiB bound arrives as `ToolResultEvent{Oversized}`
(the parser salvages the call id from the line's retained prefix) and ends
its span at receipt marked `fullsend.tool.result_oversized`, status Unset
and no `error.type`, since `is_error` was never decoded. A call still open
when the stream ends — the runtime was stopped, or an over-long result
line showed no id in that prefix — is closed as `error.type=unanswered` by
`Finish()`, which `runAgent` calls once `rt.Run` returns, before content
assembly; a
call superseded by a second `tool_use` with the same id is ended the same
way at the reuse; a result with no matching call (its `tool_use` line was skipped) becomes a
near-zero-duration span marked `fullsend.tool.unmatched=true`. Events without
an id — pi and codex emit none — produce no span, so the child count can be
below `fullsend.tool_calls`, which counts every reported call, id or not; a
`server_tool_use` block on an `assistant` line produces no event at all (its
result never arrives as a `tool_result`), so it appears in neither count. The name passes through `security.OutputPipeline()`
— Unicode normalization, then secret redaction, the same pipeline as span
content — and is bounded to 256 bytes before it becomes the attribute; the
span name keeps at most 128 bytes of it. The call id is scanned through the
same pipeline and dropped from the span on any finding — never substituted,
since a masked id could collide with another call's — while the raw bounded
id still keys the open-call map, so correlation is unaffected.
The tracker records at most `maxToolSpansPerIteration` (1,024) spans per
iteration and reports the overflow, which `runAgent` records as
`fullsend.tool_spans.dropped` on the agent span — a burst of agent-controlled
calls must not fill the OTLP batch queue and evict the agent span, which
ends after `Finish()` and content assembly. These spans are metadata: they carry no tool content
and are emitted whether or not the Level 3 gate is on.

### Level 3 content on agent spans

When the content-capture gate is on
(`telemetry.ContentCaptureEnabled()`), `runAgent` constructs one
`contentCollector` per iteration — iteration and agent span are 1:1, so a
run-scoped collector would repeat earlier iterations' content on later
spans — and tees the runtime's normalized event stream to it through
`RunParams.OnEvent`.

**The tee trap:** supplying any `OnEvent` replaces the runtime's default
console renderer (`internal/runtime/claude.go`), so the handler built by
`iterationEventHandler` always calls the renderer first, then the
tool-span tracker, then the collector — the tracker stamps span instants
when it handles an event, so it must not wait on the collector's redaction
pass over that event. The handler is always set — tool
spans are emitted with the gate off — and with the gate off the collector
is nil and inert, so console output stays byte-identical to the default
renderer path.

The collector (`internal/cli/content_collector.go`) coalesces contiguous
text/reasoning deltas, maps tool use to `tool_call` parts and tool
results to `tool_call_response` parts (only the Claude parser emits
`ToolResultEvent` and call ids today — pi and codex emit neither,
[#7414](https://github.com/fullsend-ai/fullsend/issues/7414); the schema's
required result field is `response`), redacts every
part through `security.OutputPipeline()` at assembly (redaction runs
before the size budget — truncating first could split a secret past
recognition), enforces a 256 KiB ordered-suffix budget (the ending survives — the
final answer is what consumers judge) plus an 8 KiB per-tool-result
bound (tail-kept, redacted before the cut, the part marked
`fullsend.truncated`), with exact dropped-byte
accounting across content, tool names, summaries, responses, and part
ids, then holds the marshaled string to `maxEncodedContentBytes`
(255,000 — just under the one size the pilot backend is proven to accept)
by trimming the oldest content again, measured on the encoding itself and
still charged in raw bytes (a separate, earlier boundary — the parser's 1 MiB
stream-line cap — skips oversized lines; an oversized `tool_result` line
still yields an empty part marked `fullsend.truncated`). None of the three
content bounds is a measured backend limit; the constants' comments and
[Size limits](../infrastructure/distributed-tracing.md#content-capture-level-3)
name what blocks raising them. The collector emits
`gen_ai.output.messages` JSON following the GenAI output-messages schema,
including the schema-required `finish_reason` from the iteration outcome. `attachContent` records the
content and its marker attributes on the span before either
`finalizeAgentSpan` path can end it, so failed iterations keep their
content.

**Consumer contract** (for eval scorers and other readers of
`run-telemetry.jsonl`): parse the `gen_ai.output.messages` attribute as
JSON; check `fullsend.content.truncated` / `fullsend.content.dropped_bytes`
before treating content as complete; masked secrets appear as the
redactor's mask tokens and are counted in `fullsend.content.redactions`.
The attribute names and shapes above are the consumption contract — see the
[Tracing reference](../infrastructure/distributed-tracing.md#content-capture-level-3).

## Trace identity and TRACEPARENT propagation

`resolveTraceIdentity()` handles W3C trace context propagation in three
steps:

1. Extracts `TRACEPARENT` and `TRACESTATE` from env via the W3C propagator.
2. Starts the root span (Consumer if remote parent, Internal otherwise).
3. Computes the propagated traceparent with flag preservation: if the
   inbound parent was valid, remote, and unsampled, the outbound
   traceparent keeps the unsampled flag instead of the local
   `AlwaysSample` flag. This prevents child runs from re-advertising as
   sampled when the parent trace opted out.

The resulting `TRACEPARENT` string is passed to pre-scripts and
post-scripts via `childScriptEnv()`. That function strips any inherited
`TRACEPARENT` from `os.Environ()` and `runner_env` before appending
fullsend's own value (issue #2779). That value is the **run-root** span.

A second, complementary path reaches the in-sandbox agent process. Before
every iteration, `runAgent` starts the per-iteration `agent` span and
writes that span's `TRACEPARENT` (same trace ID, agent span ID, flags from
`resolveTraceIdentity`) into `.fullsend/iteration.env` via
`writeIterationEnv`. The sandbox `.env` sources that file last, so the
runtime sees the agent-span parent rather than the run-root value
`childScriptEnv()` gives host-side scripts. An inbound unsampled parent
stays `-00` here too, so a runtime that honours W3C sampling does not
export when the parent trace opted out.

## File exporter output format

`fileExporter` writes OTLP JSON with hex-encoded trace/span IDs (per the
OTLP JSON spec, not base64). Each line is a complete `TracesData` message.
Non-finite floats (NaN, Infinity) are encoded as proto3 JSON strings per
the protobuf spec.

`buildResourceSpans()` groups SDK spans by resource and instrumentation
scope, preserving insertion order. This matches the structure a backend
receives via OTLP/HTTP.

## Prerequisites

- A local clone of the fullsend repo
- Go toolchain (see `go.mod` for the minimum version)
- Podman or Docker, if testing with a local tracing backend

## How to add a new span attribute

1. Add the `attribute.String` / `attribute.Int` / `attribute.Float64` call
   in `run.go` at the appropriate point. Use start attributes for values
   known when the span opens; use end attributes (set before `span.End()`)
   for values computed during the span.

2. Choose the attribute key namespace:
   - `gen_ai.*` for attributes defined by the
     [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
   - `fullsend.*` for project-specific attributes.

3. Update the attribute tables in the
   [Tracing Reference](../infrastructure/distributed-tracing.md).

4. No exporter changes are needed; both exporters pick up new attributes
   automatically.

## How to test

### Unit tests

```bash
go test ./internal/telemetry/...
```

Tests use `t.Setenv` for OTEL env vars and `t.TempDir` for the file
exporter. No external backend is required.

### Test seam for the OTLP exporter

`newOTLPExporter` is a package-level `var` that tests override to spy on
or stub out exporter creation:

```go
orig := newOTLPExporter
defer func() { newOTLPExporter = orig }()
newOTLPExporter = func(_ context.Context) (sdktrace.SpanExporter, error) {
    // spy or stub
}
```

### Testing with a local backend

Start a Jaeger instance and point the exporter at it:

```bash
podman run -d --name jaeger \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/jaeger

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
go run ./cmd/fullsend run triage ...
```

See [Running Agents Locally](../user/running-agents-locally.md) for full
flags. View traces at `http://localhost:16686`.

## Replaying collected traces

`hack/upload-traces.sh` replays `run-telemetry.jsonl` files into any OTLP
backend using `otelcol-contrib`. This is useful for inspecting traces from
CI runs or from another machine without re-running the agent.

```bash
# Replay a single file
hack/upload-traces.sh run/agent-run-123/run-telemetry.jsonl \
  --endpoint http://localhost:4318

# Replay all .jsonl files under a directory
hack/upload-traces.sh run/ --endpoint http://localhost:4318
```

The collector runs continuously and watches for new data; press Ctrl+C
when done.

**Prerequisites:** `otelcol-contrib` >= 0.120.0 on `PATH`
([releases](https://github.com/open-telemetry/opentelemetry-collector-releases/releases)).

**Authentication headers:** `otelcol-contrib` ignores the standard
`OTEL_EXPORTER_OTLP_HEADERS` env var. Edit
`hack/upload-traces-otelcol-config.yaml` and uncomment the `headers:`
block to set auth tokens or routing headers (e.g.
`x-mlflow-experiment-id`).

## Compatible backends

Any OTLP/HTTP-capable backend works. LLM-aware backends recognize the
`gen_ai.*` attributes and surface GenAI dashboards (token cost rollups,
prompt/completion inspection, agent-specific views) without CLI-side
changes.

| Backend | Local quickstart | UI |
|---------|-----------------|-----|
| Jaeger | `podman run -p 16686:16686 -p 4318:4318 jaegertracing/jaeger` | `localhost:16686` |
| Arize Phoenix | `podman run -p 6006:6006 -p 4318:4318 arizephoenix/phoenix` | `localhost:6006` |
| MLflow >= 3.6 | `uvx mlflow server` | `localhost:5000` |
| otel-gui | `podman run -p 4318:4318 ghcr.io/metafab/otel-gui:latest` | `localhost:4318` |

## See also

- [How To Emit Traces](../user/how-to-emit-traces.md): user guide for enabling tracing
- [Tracing Reference](../infrastructure/distributed-tracing.md): span schemas and attribute tables
- [ADR 0050](../../ADRs/0050-distributed-tracing-instrumentation.md): design decision