# Tracing Internals

Fullsend's distributed tracing system records structured telemetry for every
agent run. This document explains how the tracing implementation works and
how to extend it. It is aimed at contributors modifying the telemetry
package or the span instrumentation in the run command.

For enabling tracing on an installation, see
[How To Emit Traces](../user/how-to-emit-traces.md). For span schemas and
attribute definitions, see the
[Tracing Reference](../infrastructure/distributed-tracing.md). The design
is specified in
[ADR 0050](../../ADRs/0050-distributed-tracing-instrumentation.md).

## How the tracing system is structured

The tracing code is split between two locations:

- **`internal/telemetry/`** - the TracerProvider, exporters, and W3C
  traceparent helpers.
- **`internal/cli/run.go`** - span creation, attribute assignment, and
  trace context propagation to child scripts and the in-sandbox runtime.

The telemetry package owns the provider and exporters. `run.go` owns the
span lifecycle: it decides when spans start and end, and which attributes
they carry.

### Package layout

```
internal/telemetry/
├── telemetry.go          Setup, TracerProvider wiring, env parsing
├── fileexporter.go       Synchronous OTLP JSON file exporter
├── trace.go              W3C traceparent formatting helpers
├── main_test.go
├── telemetry_test.go
├── fileexporter_test.go
├── otlpsink_test.go
└── trace_test.go
```

## TracerProvider setup

`telemetry.Setup(dir, version)` returns a `trace.Tracer` and a cleanup
function. It creates a `TracerProvider` with two span processors:

1. **fileExporter** via `SimpleSpanProcessor`: writes every completed span
   as one OTLP JSON line to `run-telemetry.jsonl`. Calls `f.Sync()` after
   each write so spans survive a process crash.

2. **otlptracehttp** via `BatchSpanProcessor` wrapped in
   `parentSampledProcessor`: exports to a remote backend when an
   `OTEL_EXPORTER_OTLP_*ENDPOINT` env var is set.

If neither exporter can be created (bad directory, SDK disabled), `Setup`
returns a noop tracer. Telemetry failures never affect the run.

```
telemetry.Setup()
│
├── InstallOTELRedactingLogger()                ← redact OTel SDK internal logs (prevent header leaks)
├── OTEL_SDK_DISABLED check   → noop tracer if "true"
├── os.OpenFile(jsonl)        → noop tracer on error
│
├── SimpleSpanProcessor(fileExporter)           ← always present
│
└── OTEL_EXPORTER_OTLP_*ENDPOINT check
    ├── validateEndpoints()     → stderr warning, skip
    ├── ValidateOTLPHeaders()   → stderr warning, skip
    ├── newOTLPExporter()       → stderr warning, skip
    └── parentSampledProcessor(BatchSpanProcessor(otlpExporter))
```

### Why parentSampledProcessor exists

The OTel SDK's `AlwaysSample` sampler records all spans locally, which is
needed for the file exporter. But `AlwaysSample` ignores an upstream
unsampled decision. The `parentSampledProcessor` wraps the OTLP batch
processor and suppresses the entire trace from export when the root span's
remote parent has the W3C sampled flag unset (`-00`). It tracks suppressed
trace IDs in a `sync.Map` so child spans under the same trace are also
dropped.

Result: the file exporter always writes all spans; the OTLP exporter
respects upstream sampling.

### Endpoint and header validation

`validateEndpoints` checks all non-empty endpoint variables
(`OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`),
rejecting non-http(s) URLs and unsupported protocols. Error messages never
echo the endpoint value (which may carry credentials in userinfo or the query
string): they name the env var and the problem, and include the parsed scheme
only when it is a well-known non-http(s) scheme such as `grpc` or `ftp`.

`ValidateOTLPHeaders` checks `OTEL_EXPORTER_OTLP_HEADERS` and
`OTEL_EXPORTER_OTLP_TRACES_HEADERS` to ensure headers strictly follow
OTel `key=value` syntax:
- Every non-empty comma-separated entry must contain an `=` separator.
- Header keys must consist exclusively of RFC 7230 token characters.
- Values must have valid URL percent-encoding.

Malformed headers produce a value-free warning on stderr and skip remote
OTLP export without failing the run. `ValidateOTLPHeaders` is enforced in `Setup`
and `fullsend eval-measure`.

Additionally, `InstallOTELRedactingLogger` installs a redacting `logr.LogSink`
for SDK internal logging (`internal/global`). It preserves error messages on
stderr while dropping `keysAndValues` and redacting `*url.Error` details so
secrets are never echoed. Installed at CLI startup (`PersistentPreRun`),
`telemetry.Setup`, and shared exporter constructors.

## Span lifecycle in run.go

`run.go` creates four span types arranged in a parent-child hierarchy:

```
run (root)
├── sandbox_create    (gen_ai.operation.name=create_agent)
└── agent             (one per iteration; gen_ai.operation.name=invoke_agent)
    └── execute_tool  (one per id-bearing tool call; gen_ai.operation.name=execute_tool)
```

### Root span

Created by `resolveTraceIdentity()`. When an inbound `TRACEPARENT` env var
is present, the W3C propagator extracts a remote span context and the root
span is `SpanKindConsumer`. Otherwise it is `SpanKindInternal`.

Start attributes include: `fullsend.agent`, `fullsend.work_item_id`,
`gen_ai.operation.name`, `gen_ai.agent.name`.

End attributes (set in a deferred cleanup): `exit_code`,
`fullsend.cost_usd`, `fullsend.num_turns`, `fullsend.tool_calls`,
`fullsend.iterations`.

For the full attribute list (including `fullsend.security_trace_id` and
`fullsend.prescript.*`), see the
[Tracing reference attribute table](../infrastructure/distributed-tracing.md#span-attributes).

### sandbox_create span

Started before sandbox creation, ended after bootstrap completes. Carries
`gen_ai.operation.name=create_agent`.

### agent spans

One per iteration (validation loop iterations included). Started before
`Exec()`, ended after output extraction.

The helper functions `agentSpanStartAttrs()` and `agentSpanEndAttrs()`
build the attribute slices. Start attributes: `iteration`,
`gen_ai.operation.name`, `gen_ai.agent.name`. End attributes: `iteration`,
`exit_code`, `gen_ai.system` and `gen_ai.provider.name` (same serving-endpoint
value), model, token counts, `fullsend.cost_usd`, `fullsend.runtime`,
`fullsend.tool_calls`. Multi-provider runtimes resolve the provider from the
effective model via `runtime.GenAISystemFor`; `System()` is only the fallback.

### execute_tool spans

One per id-bearing tool call the runtime reports (Claude Code today), up to
1,024 per iteration, a child of that iteration's agent span, named
`execute_tool <tool name>`. `toolSpanTracker`
(`internal/cli/tool_spans.go`) starts the span when it handles the
`ToolUseEvent` and ends it when it handles the matching `ToolResultEvent`.
`iterationEventHandler` runs the console renderer first, then the tracker,
then the Level 3 collector, so both timestamps are runner-side receipt
instants on one clock, each trailing the sandbox by the pipe latency and the
parser's decode of the line: the start also trails receipt by the renderer's output for
the call (the renderer prints nothing for a result), and neither waits on
the collector's redaction pass over its own event (events are handled one
at a time, so with several calls open an instant can still trail the
collector's pass over an earlier event, such as another call's large
result) — the start is arguments-complete, not execution start. Attributes:
`gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`,
`gen_ai.tool.call.id`; a result flagged `is_error` sets
`error.type=tool_error` and status Error. A result whose stream line
exceeded the parser's 1 MiB bound arrives as `ToolResultEvent{Oversized}`
(the parser salvages the call id from the line's retained prefix) and ends
its span at receipt marked `fullsend.tool.result_oversized`, status Unset
and no `error.type`, since `is_error` was never decoded. A call still open
when the stream ends — the runtime was stopped, or an over-long result
line showed no id in that prefix — is closed as `error.type=unanswered` by
`Finish()`, which `runAgent` calls once `rt.Run` returns, before content
assembly; a
call superseded by a second `tool_use` with the same id is ended the same
way at the reuse; a result with no matching call (its `tool_use` line was skipped) becomes a
near-zero-duration span marked `fullsend.tool.unmatched=true`. Events without
an id — pi and codex emit none — produce no span, so the child count can be
below `fullsend.tool_calls`, which counts every reported call, id or not; a
`server_tool_use` block on an `assistant` line produces no event at all (its
result never arrives as a `tool_result`), so it appears in neither count. The name passes through `redactText`
— the runner env literal pass on both sides of `security.OutputPipeline()`,
the redaction span content gets — and is bounded to 256 bytes before it becomes the attribute; the
span name keeps at most 128 bytes of it. The call id goes through the
same `redactText` and is dropped from the span on any finding — never substituted,
since a masked id could collide with another call's — while the raw bounded
id still keys the open-call map, so correlation is unaffected.
The tracker records at most `maxToolSpansPerIteration` (1,024) spans per
iteration and reports the overflow, which `runAgent` records as
`fullsend.tool_spans.dropped` on the agent span — a burst of agent-controlled
calls must not fill the OTLP batch queue and evict the agent span, which
ends after `Finish()` and content assembly. These spans are metadata: they carry no tool content
and are emitted whether or not the Level 3 gate is on.

### Level 3 content on agent spans

When the content-capture gate is on
(`telemetry.ContentCaptureEnabled()`), `runAgent` constructs one
`contentCollector` per iteration — iteration and agent span are 1:1, so a
run-scoped collector would repeat earlier iterations' content on later
spans — and tees the runtime's normalized event stream to it through
`RunParams.OnEvent`.

**The tee trap:** supplying any `OnEvent` replaces the runtime's default
console renderer (`internal/runtime/claude.go`), so the handler built by
`iterationEventHandler` always calls the renderer first, then the
tool-span tracker, then the collector — the tracker stamps span instants
when it handles an event, so it must not wait on the collector's redaction
pass over that event. The handler is always set — tool
spans are emitted with the gate off — and with the gate off the collector
is nil and inert, so console output stays byte-identical to the default
renderer path.

The collector (`internal/cli/content_collector.go`) coalesces contiguous
text/reasoning deltas, maps tool use to `tool_call` parts and tool
results to `tool_call_response` parts (only the Claude parser emits
`ToolResultEvent`, call ids and `ToolUseEvent.Arguments` today — pi and
codex emit none of them,
[#7414](https://github.com/fullsend-ai/fullsend/issues/7414); the schema's
required result field is `response`), redacts every
part — `security.OutputPipeline()`, with `replaceEnvSecrets` for the values
of sensitive runner env keys and of `providerOnlyKeys`, in one pass, on both
sides of it (`redactText`; a string the normalizer stripped an escape
sequence or tag characters from is masked `***` whole — a colour code
ends at the next letter, so it can take a token's first letter, and a
title code's payload is text no pattern sees, so nothing cut from such a
string can be trusted, and no scan of it as written can stand in; so is a
string holding invalid UTF-8, since the span and attribute writers would
join the halves of a value such a byte splits, or the replacement
character a lossy decode leaves in its place, which splits a value the
same way; last, every URL's userinfo, query and fragment are masked,
`maskURLs`, on text the normalizer has folded and the patterns masked) —
at assembly (redaction runs
before the size budget — truncating first could split a secret past
recognition), enforces a 256 KiB ordered-suffix budget (the ending survives — the
final answer is what consumers judge) plus an 8 KiB per-tool-result
bound (tail-kept, redacted before the cut, the part marked
`fullsend.truncated`) and an 8 KiB per-call arguments bound (over it the
arguments are dropped whole and the part marked, since a cut object is not
JSON), with exact dropped-byte
accounting across content, tool names, summaries, responses, part
ids, and arguments dropped whole (charged as the kept encoding when the
bound drops them, otherwise as the redacted text they were scanned as; a
member the recorded list leaves out is not charged), then holds the marshaled string to `maxEncodedContentBytes`
(255,000 — just under the one size the pilot backend is proven to accept)
by trimming the oldest content again, measured on the encoding itself and
still charged in raw bytes (a separate, earlier boundary — the parser's 1 MiB
stream-line cap — skips oversized lines; an oversized `tool_result` line
still yields an empty part marked `fullsend.truncated`). None of the four
content bounds is a measured backend limit; the constants' comments and
[Size limits](../infrastructure/distributed-tracing.md#content-capture-level-3)
name what blocks raising them. The collector emits
`gen_ai.output.messages` JSON following the GenAI output-messages schema,
including the schema-required `finish_reason` from the iteration outcome. `attachContent` records the
content and its marker attributes on the span before either
`finalizeAgentSpan` path can end it, so failed iterations keep their
content.

**Guarantee and limits.** For every string the record or an `execute_tool`
span exports, no 12-byte run of a known secret's body appears, read as raw
bytes, after the normalizer, or after decoding the record's JSON with every
string walked; a known secret is one a `SecretRedactor` pattern matches,
a runner env or provider-only env value the literal pass holds, or a
URL's userinfo, query or fragment.
`TestRedaction_NoSecretSurvivesAnyWrapping` (`internal/cli`) states it and
checks it over every surface (text, names, summaries, arguments on each
path, results, the input message, span names and ids, and a Claude call
with the summary the parser derives from it) and every wrapping a review
has constructed so far — terminal escapes before, around and inside a
secret, repeated and paired secrets, zero-width, bidi, tag, NUL and
variation-selector characters inside it, a URL's delimiters in
compatibility forms, separators escaped for the shell or JSON, a URL
percent-encoded whole or in part (its separator's colon or slashes, its
`@`, `?` or `#`), a
password quoted for the shell, a URL nested in another's path, an
invalid byte or the
replacement character inside it, an invalid byte after it with the
secret straddling the parser's summary cut, compatibility forms, and
noise, inside the record and across the feedback cut, in compatibility
forms or plain with 12 bytes before the cut.
Four limits are named there and here, and nowhere else: an env value
written in compatibility (fullwidth) forms, since the normalizer rewrites
it and the literal pass matches only what the env has; a secret split
across two parts or events; a secret that matches no pattern and is in no
env, among them a token in a URL's path, in a URL holding a double quote
that ends it (one that could close a JSON string, so a URL inside JSON
ends with its string), what follows the quote and userinfo whose `@`
follows it; a summary a pi or codex parser cut before the collector saw it, since
that cut can split a value the parser's own scan did not see joined (a
Claude summary is cut from a member the collector scans whole). A finding
inside the guarantee is a defect; one on this list is a documented limit.

`toolArguments` records the members of a call's input that
`recordedArguments` names for the tool — paths, patterns, commands, modes
and bounds; the reference lists them tool by tool — and nothing else: a
file body, an edit, a prompt, a notebook source or a todo list, any member
whose value is an object or an array, and the whole input of a tool the
list does not name (every `mcp__` tool) is dropped: scanned as text so a
secret in it still counts, then left out, neither charged nor marked,
since the list is the documented policy and the markers mean loss. A
`Bash` `command` is kept whole, so a heredoc or an inline script records
what it writes. A URL's userinfo, query and fragment are masked in every
recorded string (`maskURLs`, above), with a finding that costs a call its
summary. The guarantee is one sentence: the record holds these members of these tools,
each redacted as text, and nothing else of any call's input. The input is decoded
(one JSON object, white space at most after it) and each kept string, and
each kept number as its digits, goes through the text
pipeline on its own, since the redactor's patterns are written for plain
text and over serialised JSON miss an assignment that opens a string or
follows an escaped newline, and a value behind escaped quotes; the result
is encoded again, so key order, spacing and escapes are the encoder's. A
string the normalizer stripped from is masked `***` whole, as every
string is (above). Discarded content is redacted
first, here as everywhere: a dropped member is scanned as text — the
decoded string, or the compact encoding of an object or an array — so a
secret in a file body still counts, and costs the summary as a secret
found anywhere in the arguments does; so do arguments that could not be
judged at all — stripped, or holding invalid UTF-8 or the replacement
character — since the summary was cut from a member no scan saw whole.
Arguments that are not one JSON
object are scanned as text and dropped whole, charged; so are kept
members whose encoding exceeds `maxToolArgumentsBytes`, charged as
encoded. A call the stream reports without a name carries no arguments.
This happens once, when the event is handled; eviction and `Result` do
not rescan it. Arguments are kept only under a tool's exact name, plain
words no scan touches, so a name that redacts to nothing never had any.

The input message does not come from the stream. When `runAgent` composes
a retry prompt (`buildFeedbackPrompt`, under `feedback_mode: append`),
`attachInput` redacts it with the collector's pipeline — `redactFeedback`
ran before the feedback was sanitized and framed, and sanitizing can join
a token that scan saw split, so the recorded copy is pattern-scanned
again; a prompt the composition altered in a way that pass cannot see —
sanitized, or cut at `maxFeedbackBytes` while holding compatibility
characters or a URL `maskURLs` would mask, since the cut can take the `@`
that marks a password (`promptHidesAlterations`) — is recorded `***` whole; the
prompt the agent is sent is not changed, and the mask the first
scan leaves for a connection-string password of ten or more bytes matches
its own pattern and counts a finding —
and sets `gen_ai.input.messages` on the span
before the runtime starts, so each finalize path carries it. Its findings
join the iteration's, and its encoded size is charged against
`maxEncodedContentBytes`: the two content attributes of one span together
stay within the proven size.

**Consumer contract** (for eval scorers and other readers of
`run-telemetry.jsonl`): parse the `gen_ai.output.messages` attribute as
JSON; check `fullsend.content.truncated` / `fullsend.content.dropped_bytes`
before treating content as complete; masked secrets appear as the
redactor's mask tokens, or as `[REDACTED:<key>]` for a runner env or
provider-only value, and are counted in `fullsend.content.redactions`;
what was masked or rewritten for its shape — the normalizer's folds,
removals, strippings and invalid UTF-8, a retry prompt altered upstream,
a URL's userinfo, query or fragment — is counted apart in
`fullsend.content.sanitizations`.
A `tool_call` part's `arguments`, when present, is a JSON value (an object
for the tools seen so far); a `tool_call` part marked `fullsend.truncated`
had arguments that were dropped whole — on that part type the marker
never means a partial value, as it does on a tool result. `gen_ai.input.messages` is absent unless
the iteration is a retry that carried validation feedback — absence is
the normal case, not a gap. Masks `redactFeedback` left in the feedback
(`abcd...`, `***`, `[REDACTED:<ENV_KEY>]`) are in `gen_ai.input.messages`
too, but that function discards its findings, so they are not counted in
`fullsend.content.redactions`; the connection-string mask described above
is the known exception.
The attribute names and shapes above are the consumption contract — see the
[Tracing reference](../infrastructure/distributed-tracing.md#content-capture-level-3).

## Trace identity and TRACEPARENT propagation

`resolveTraceIdentity()` handles W3C trace context propagation in three
steps:

1. Extracts `TRACEPARENT` and `TRACESTATE` from env via the W3C propagator.
2. Starts the root span (Consumer if remote parent, Internal otherwise).
3. Computes the propagated traceparent with flag preservation: if the
   inbound parent was valid, remote, and unsampled, the outbound
   traceparent keeps the unsampled flag instead of the local
   `AlwaysSample` flag. This prevents child runs from re-advertising as
   sampled when the parent trace opted out.

The resulting `TRACEPARENT` string is passed to pre-scripts and
post-scripts via `childScriptEnv()`. That function strips any inherited
`TRACEPARENT` from `os.Environ()` and `runner_env` before appending
fullsend's own value (issue #2779). That value is the **run-root** span.

A second, complementary path reaches the in-sandbox agent process. Before
every iteration, `runAgent` starts the per-iteration `agent` span and
writes that span's `TRACEPARENT` (same trace ID, agent span ID, flags from
`resolveTraceIdentity`) into `.fullsend/iteration.env` via
`writeIterationEnv`. The sandbox `.env` sources that file last, so the
runtime sees the agent-span parent rather than the run-root value
`childScriptEnv()` gives host-side scripts. An inbound unsampled parent
stays `-00` here too, so a runtime that honours W3C sampling does not
export when the parent trace opted out.

## File exporter output format

`fileExporter` writes OTLP JSON with hex-encoded trace/span IDs (per the
OTLP JSON spec, not base64). Each line is a complete `TracesData` message.
Non-finite floats (NaN, Infinity) are encoded as proto3 JSON strings per
the protobuf spec.

`buildResourceSpans()` groups SDK spans by resource and instrumentation
scope, preserving insertion order. This matches the structure a backend
receives via OTLP/HTTP.

## Prerequisites

- A local clone of the fullsend repo
- Go toolchain (see `go.mod` for the minimum version)
- Podman or Docker, if testing with a local tracing backend

## How to add a new span attribute

1. Add the `attribute.String` / `attribute.Int` / `attribute.Float64` call
   in `run.go` at the appropriate point. Use start attributes for values
   known when the span opens; use end attributes (set before `span.End()`)
   for values computed during the span.

2. Choose the attribute key namespace:
   - `gen_ai.*` for attributes defined by the
     [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
   - `fullsend.*` for project-specific attributes.

3. Update the attribute tables in the
   [Tracing Reference](../infrastructure/distributed-tracing.md).

4. No exporter changes are needed; both exporters pick up new attributes
   automatically.

## How to test

### Unit tests

```bash
go test ./internal/telemetry/...
```

Tests use `t.Setenv` for OTEL env vars and `t.TempDir` for the file
exporter. No external backend is required.

### Test seam for the OTLP exporter

`newOTLPExporter` is a package-level `var` that tests override to spy on
or stub out exporter creation:

```go
orig := newOTLPExporter
defer func() { newOTLPExporter = orig }()
newOTLPExporter = func(_ context.Context) (sdktrace.SpanExporter, error) {
    // spy or stub
}
```

### Testing with a local backend

Start a Jaeger instance and point the exporter at it:

```bash
podman run -d --name jaeger \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/jaeger

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
go run ./cmd/fullsend run triage ...
```

See [Running Agents Locally](../user/running-agents-locally.md) for full
flags. View traces at `http://localhost:16686`.

## Replaying collected traces

`hack/upload-traces.sh` replays `run-telemetry.jsonl` files into any OTLP
backend using `otelcol-contrib`. This is useful for inspecting traces from
CI runs or from another machine without re-running the agent.

```bash
# Replay a single file
hack/upload-traces.sh run/agent-run-123/run-telemetry.jsonl \
  --endpoint http://localhost:4318

# Replay all .jsonl files under a directory
hack/upload-traces.sh run/ --endpoint http://localhost:4318
```

The collector runs continuously and watches for new data; press Ctrl+C
when done.

**Prerequisites:** `otelcol-contrib` >= 0.120.0 on `PATH`
([releases](https://github.com/open-telemetry/opentelemetry-collector-releases/releases)).

**Authentication headers:** `otelcol-contrib` ignores the standard
`OTEL_EXPORTER_OTLP_HEADERS` env var. Edit
`hack/upload-traces-otelcol-config.yaml` and uncomment the `headers:`
block to set auth tokens or routing headers (e.g.
`x-mlflow-experiment-id`).

## Compatible backends

Any OTLP/HTTP-capable backend works. LLM-aware backends recognize the
`gen_ai.*` attributes and surface GenAI dashboards (token cost rollups,
prompt/completion inspection, agent-specific views) without CLI-side
changes.

| Backend | Local quickstart | UI |
|---------|-----------------|-----|
| Jaeger | `podman run -p 16686:16686 -p 4318:4318 jaegertracing/jaeger` | `localhost:16686` |
| Arize Phoenix | `podman run -p 6006:6006 -p 4318:4318 arizephoenix/phoenix` | `localhost:6006` |
| MLflow >= 3.6 | `uvx mlflow server` | `localhost:5000` |
| otel-gui | `podman run -p 4318:4318 ghcr.io/metafab/otel-gui:latest` | `localhost:4318` |

## See also

- [How To Emit Traces](../user/how-to-emit-traces.md): user guide for enabling tracing
- [Tracing Reference](../infrastructure/distributed-tracing.md): span schemas and attribute tables
- [ADR 0050](../../ADRs/0050-distributed-tracing-instrumentation.md): design decision