---
title: "66. Behaviour tests with Gherkin and pluggable drivers"
status: Accepted
relates_to:
  - agent-infrastructure
  - agent-architecture
topics:
  - e2e
  - testing
  - behaviour-tests
---

# 66. Behaviour tests with Gherkin and pluggable drivers

Date: 2026-06-07

## Status

Accepted

## Context

Fullsend needs end-to-end tests that validate **deterministic platform behaviour** — dispatch routing, harness loading, schema validation, post-scripts, token scoping, sandbox policy, and SCM mutations — without depending on LLM output. This is distinct from admin install e2e ([ADR 0040](0040-org-pool-for-parallel-e2e-tests.md)) and from LLM/instruction testing ([testing-agents.md](../problems/testing-agents.md)).

Runtime selection is shared with production via `defaults.runtime` in org `config.yaml` ([runtimes.md](../runtimes.md)). Harness definitions remain as in [ADR 0024](0024-harness-definitions.md). Per-repo install mode ([ADR 0033](0033-per-repo-installation-mode.md)) is the behaviour v1 default; per-org install is deferred.

## Decision

- Add **behaviour tests** under `e2e/behaviour/` using **godog** and portable Gherkin feature files.
- Exercise **real SCM + real CI** through **driver interfaces** (`scm.Driver`, `ci.Driver`, `install.Driver`); v1 implementations target GitHub and GitHub Actions.
- Substitute inference with a **dummy runtime** (`runtime: dummy` in per-repo config, or `defaults.runtime: dummy` for per-org) that executes scripted operations in the real OpenShell sandbox and emits `behaviour-results.json`.
- Select backends via **runner env** (`BEHAVIOUR_SCM`, `BEHAVIOUR_CI`, `BEHAVIOUR_INSTALL_MODE`); feature files stay install-mode agnostic. v1 runs **per-repo** against the halfsend org pool; the suite provisions fullsend via `fullsend github setup` rather than requiring pre-installed orgs.
- Use **compatibility tags** (`@skip:*`, `@requires:*`) to filter scenarios for future backends; tags do not select configuration.

## Consequences

- Behaviour tests can pass while prompt quality regresses; LLM evals remain necessary for instruction coverage.
- Behaviour orgs are provisioned at suite start with `--runtime dummy`; production orgs must not use dummy unintentionally.
- **Note (2026-07, #5439 / PR #5489):** Numbered behaviour pool repos (`test-repo-NN`) are lazily created and installed on first scenario use via the ensurer internal to `install.Driver`; suite-start provisioning still applies to the shared admin/`test-repo` install path where used.
- Adding GitLab or Tekton requires new drivers and runner env values, not feature file rewrites.
- Dummy runtime op vocabulary stays minimal; new ops require runtime + docs updates when scenarios need them.
- Behaviour tests depend on live external infrastructure: GitHub API, GitHub Actions runners, GCP WIF/mint, and the shared halfsend org pool. Transient outages, API rate limits, or pool org state corruption can fail the suite; CI distinguishes infrastructure failures from regressions via workflow logs and artifact inspection, but there is no offline fallback.
- Behaviour tests share the halfsend org pool and lock mechanism with admin e2e tests (`e2e.yml` runs both jobs). Lock hold time scales with scenario count; pool size was doubled to absorb the additional load and can be increased again if contention appears.

> **Note (2026-07):** Shared live-test infrastructure (org pool, CLI runner, cleanup) lives in `pkg/e2etest/`; the Gherkin framework lives in `pkg/behaviourtest/`. In-repo runners remain under `e2e/behaviour/` and `e2e/admin/`.
>
> **Note (2026-09):** Live-test infrastructure moved from `pkg/e2etest/` to `internal/e2etest/` so it is not part of the public module surface ([#7279](https://github.com/fullsend-ai/fullsend/issues/7279)).

> **Note (2026-09):** [ADR 0116](0116-dummy-playback-runtime.md) adds the dummy-playback runtime for multi-agent sequential scenarios. The dummy runtime (this ADR) continues to handle single-agent sandbox verification.