E2E Testing

Guide for running and debugging fullsend behaviour and playback tests locally and in CI.

Related ADRs: 0040 (org pool), 0060 (cross-org mint), 0009 (pull_request_target security model for shims; e2e uses a separate gate pattern documented below).

Historical ADRs 0010 (browser session) and 0039 (2FA) are superseded for CI by cross-org mint auth (#2155); local runs no longer use Playwright or stored sessions.

Prerequisites

Before running e2e locally or in CI:

1. Pool orgs (halfsend-01 … halfsend-12) provisioned per Pool org provisioning below
2. Mint deployed with e2e role enrolled and the requesting repository fullsend-ai/fullsend enrolled in PER_REPO_WIF_REPOS (or the mint explicitly in public mode, PER_REPO_WIF_REPOS=*)
3. CI only: pool orgs with FULLSEND_FOREIGN_E2E_REPOS authorizing fullsend-ai/fullsend
4. Local only: gh auth login (or GH_TOKEN / GITHUB_TOKEN) with admin access on pool orgs

Local runs

The default local environment is dev. Both suites deploy a temporary Cloudflare preview mint; GitHub authentication alone is not sufficient.

1. Authenticate as an admin on the pool orgs (gh auth login --web, or export GH_TOKEN).

2. Load the following environment variables from your approved secret store before running either suite. Use test credentials, not production credentials; never commit them or paste them into logs.

   Environment variables	Required values
   TEST_FULLSEND_PEM, TEST_TRIAGE_PEM, TEST_CODER_PEM, TEST_REVIEW_PEM, TEST_RETRO_PEM, TEST_PRIORITIZE_PEM	Private-key contents, not paths, for the installed test GitHub Apps. The driver materializes temporary PEM files for mint deployment and removes them afterward.
   CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN	Test Cloudflare account and token authorized to deploy preview mint Workers. Locally use these names, not the CI secret names TEST_CLOUDFLARE_*.
   E2E_GCP_PROJECT_ID	Test inference project. Authenticate locally with GCP permissions to inspect/provision inference WIF for the pool repos; see inference setup.
   TEST_ACTOR_WRITE_PAT, TEST_ACTOR_TRIAGE_PAT, TEST_ACTOR_OUTSIDER_PAT	Human-like test-actor PATs used by SCM steps. Playback uses the triage PAT to create its tracking issue.

   The default app set is fullsend-test; the PEMs must match those installed Apps. Set BEHAVIOUR_APP_SET only when using a different test app set. Overriding FULLSEND_MINT_URL does not disable preview mint deployment. If no PEMs are loaded, initialization fails with PEMDir is required (no PEMs materialized).

3. Run tests (uses gh auth token, GH_TOKEN, or GITHUB_TOKEN):

make behaviour-test   # Gherkin behaviour suite
make playback-test    # @playback scenarios on the dummy-playback runtime

The obsolete admin install/uninstall suite and make e2e-test target have been removed. CI keeps both the regular behaviour and playback suites.

To run playback manually, select E2E Tests → Run workflow, choose the branch, and leave Run the playback behaviour suite enabled. From the CLI:

gh workflow run e2e.yml --repo fullsend-ai/fullsend \
  --ref YOUR_BRANCH -f run_playback=true

Manual runs use the dev org pool and run playback only. Normal CI runs both suites, concurrently in dev and serialized against the single stage org.

Optional environment variables:

Variable	Purpose
GH_TOKEN / GITHUB_TOKEN	Override token source for local runs
FULLSEND_MINT_URL	Override mint endpoint (default: hosted public mint, same as fullsend admin --mint-url)
E2E_LOCK_TIMEOUT	Max wait for a free pool org (default 10m)
E2E_GCP_PROJECT_ID	GCP project for inference setup (github setup --inference-project)

Behaviour tests use the pool orgs (for ENVIRONMENT=dev) and install fullsend per-repo via fullsend github setup. When ENVIRONMENT=stage, the suite uses the halfsend org with a durable CF Worker mint instead of a pool org. See behaviour-testing.md and behaviour-drivers.md.

Tests acquire an exclusive lock on one org from the pool (halfsend-01 … halfsend-12 for DEV, or halfsend for STAGE) — see ADR 0040.

Shared pool, CLI, and cleanup helpers used by the behaviour and playback suites live in internal/e2etest/.

CI runs

In GitHub Actions, tests mint a cross-org installation token via the mint service:

1. Workflow requests a GHA OIDC token (id-token: write)
2. mintclient.MintToken POSTs to {FULLSEND_MINT_URL or hosted default}/v1/token with {role: "e2e", level: "write", target_org: "<pool org>", repos: ["*"]} (repos: ["*"] is required for installation-wide cross-org access; level: "write" requests full write permissions)
3. Mint verifies the caller against FULLSEND_FOREIGN_E2E_REPOS on the target org (ADR 0060)

Required repository secrets:

Secret	Purpose
E2E_GCP_WIF_PROVIDER	GCP WIF provider (inference / auxiliary GCP access)
E2E_GCP_SERVICE_ACCOUNT	GCP service account for WIF
E2E_GCP_PROJECT_ID	GCP project ID for inference secrets (github setup --inference-project)
TEST_CLOUDFLARE_ACCOUNT_ID	Cloudflare account ID for CF mint behaviour-test deploys (mapped to env CLOUDFLARE_ACCOUNT_ID in the behaviour job)
TEST_CLOUDFLARE_API_TOKEN	Test-only Cloudflare API token for Wrangler against Worker mint-test (DEV) and stage-mint (STAGE) (mapped to env CLOUDFLARE_API_TOKEN; distinct from site-deploy CLOUDFLARE_*)
TEST_ACTOR_WRITE_PAT	Classic PAT for the write-level human-like test actor (fstest-write); exposed to the behaviour job under the same env name
TEST_ACTOR_TRIAGE_PAT	Classic PAT for the triage-level human-like test actor (fstest-triage); exposed to the behaviour job under the same env name
TEST_ACTOR_OUTSIDER_PAT	Classic PAT for the outsider (no org write) human-like test actor (fstest-outsider); exposed to the behaviour job under the same env name

Mint URL uses the hosted public endpoint by default (same as fullsend admin --mint-url). Override with org/repo variable FULLSEND_MINT_URL if needed; no separate e2e secret.

Behaviour job GitHub Environments

The behaviour job in e2e.yml binds to GitHub Environments dev (authorized pull requests and the merge queue) and stage (push to main). It skips workflow_dispatch and other triggers. The job sets ENVIRONMENT to the same value for the suite (dev or stage). GitHub auto-creates those environments on first use.

After the environments exist, restrict stage to main:

1. Open the repository Settings → Environments → stage.
2. Under Deployment branches and tags, choose Selected branches and tags.
3. Add a branch rule for main.

Cloudflare Worker mint BT credentials

The behaviour job wires TEST_CLOUDFLARE_* into Wrangler’s standard CLOUDFLARE_ACCOUNT_ID / CLOUDFLARE_API_TOKEN env names so CF mint BT (#5109) can upload versions of Worker mint-test. These secrets must not reuse the production site-deploy CLOUDFLARE_ACCOUNT_ID / CLOUDFLARE_API_TOKEN used by site-deploy.yml (Worker site).

Prefer wrangler versions upload --name=mint-test --preview-alias=… so runs use preview URLs (<alias>-mint-test.<subdomain>.workers.dev) rather than inventing new Worker names or relying on the production mint-test.…workers.dev route (which may stay disabled). Cloudflare Account API tokens cannot currently attach Workers Scripts permissions under a Specified-Workers-only policy; operators use a dedicated Workers Edit token (for example fullsend-ai/fullsend-mint-test) that is separate from site-deploy credentials and intended only for this test path.

STAGE environment: The STAGE driver (NewRepoPoolCFMintStage) deploys a separate durable Worker stage-mint at stage-mint.fullsend.sh instead of using preview aliases on mint-test. The TEST_CLOUDFLARE_API_TOKEN must have permissions to manage the stage-mint Worker in addition to mint-test. If the Cloudflare account uses a Specified-Workers-only token policy, operators need a distinct Workers Edit token (for example fullsend-ai/fullsend-stage-mint) that covers the stage-mint Worker.

Behaviour tests and per-repo mint enrollment

Behaviour tests install fullsend in per-repo mode (fullsend github setup). Triage workflows mint same-org triage tokens from vendored reusable workflows; that requires per-repo mint enrollment (PER_REPO_WIF_REPOS). The install driver does not run mint enroll — pool org behaviour repos must be enrolled once by a GCP admin on the hosted mint project.

Behaviour tests allocate numbered halfsend-NN/test-repo-01 … test-repo-12 names via the unified install.Driver; these repos are lazily created and installed on demand (see behaviour-testing.md). Pre-provisioning numbered repos in the pool org is no longer required — mint enrollment for those names is still pre-provisioned so it is not on the critical path. Enroll base names only — do not enroll *-fork names (forks are ephemeral PR sources and mint against the enrolled base repo). GitHub repositories need not exist yet — enroll is a mint allowlist / WIF-provider update only.

Inference (E2E_GCP_PROJECT_ID) and mint (it-gcp-konflux-dev-fullsend for the hosted mint) may be different GCP projects. The behaviour install driver resolves the repo-scoped WIF provider using CI credentials on the inference project and passes it to github setup. It runs fullsend inference status <org>/test-repo-NN first and runs fullsend inference provision only when the provider is not healthy, one provision at a time. The provider is cached per repo name for the rest of the run. E2E_GCP_WIF_PROVIDER authenticates the CI job itself; it is not written to pool org repos.

The CI service account needs inference-provision IAM on E2E_GCP_PROJECT_ID:

IAM role	Purpose
roles/iam.workloadIdentityPoolAdmin	Create/update repo-scoped inference WIF providers
roles/resourcemanager.projectIamAdmin	Grant roles/aiplatform.user to repo WIF principals

One-time enrollment for all pool orgs (idempotent). Enroll the singular test-repo (used by the driver today) and the behaviour pool test-repo-01 … test-repo-12 (pre-provisioned for planned parallelization):

export GCP_PROJECT=it-gcp-konflux-dev-fullsend
for i in $(seq -w 1 12); do
  go run ./cmd/fullsend mint enroll "halfsend-${i}/test-repo" \
    --project="$GCP_PROJECT" --region=us-central1
  for j in $(seq -w 1 12); do
    go run ./cmd/fullsend mint enroll "halfsend-${i}/test-repo-${j}" \
      --project="$GCP_PROJECT" --region=us-central1
  done
done

One-time enrollment for the STAGE org (halfsend). The STAGE driver uses the same repo pool pattern (test-repo-01 … test-repo-12) within the halfsend org:

export GCP_PROJECT=it-gcp-konflux-dev-fullsend
for j in $(seq -w 1 12); do
  go run ./cmd/fullsend mint enroll "halfsend/test-repo-${j}" \
    --project="$GCP_PROJECT" --region=us-central1
done

Re-run enrollment when adding a new pool org or after mint infrastructure changes that drop PER_REPO_WIF_REPOS entries. See mint-administration.md for required operator IAM.

Pool org provisioning

Each pool org must be provisioned before e2e can use it:

1. Org exists with botsend as owner
2. test-repo and e2e-lock repos (lock created at runtime)
3. Test actor permissions granted (see Test actor permissions below)
4. All role apps installed, including fullsend-ai-e2e with Repository → Variables: Read and write (actions_variables) and Organization → Variables: Read and write (organization_actions_variables)
5. FULLSEND_FOREIGN_E2E_REPOS includes fullsend-ai/fullsend with org-wide visibility (visibility: all)
6. Mint enrolled: requesting repository fullsend-ai/fullsend in PER_REPO_WIF_REPOS (or public mode), e2e in ROLE_APP_IDS, e2e app PEM enrolled. The pool org is a foreign target, so its access is governed by the FULLSEND_FOREIGN_E2E_REPOS grant and App installation above rather than by requesting-repository enrollment

Use the idempotent setup script. Numeric arguments become halfsend-NN; a full org name (for example halfsend for STAGE) is used as-is:

MINT_PROJECT=... MINT_FUNCTION=... hack/setup-new-e2e-org.sh 07
MINT_PROJECT=... MINT_FUNCTION=... hack/setup-new-e2e-org.sh halfsend

Verify foreign authorization:

go run ./cmd/fullsend admin foreign list --org halfsend-01
# expect e2e → fullsend-ai/fullsend

Existing pool orgs (halfsend-01 … halfsend-12) need a one-time operator pass: install the e2e app (if missing) and run:

go run ./cmd/fullsend admin foreign allow --org halfsend-NN --role e2e --caller fullsend-ai/fullsend

For repo-level grants (scoped to a specific target repo rather than the entire org), add --repo:

go run ./cmd/fullsend admin foreign allow --org halfsend-NN --role e2e --caller fullsend-ai/fullsend --repo target-repo
go run ./cmd/fullsend admin foreign list --org halfsend-NN --repo target-repo

See ADR 0083 for details on repo-level foreign grants.

Test actor permissions

Pool orgs grant three test actor accounts specific access levels for e2e testing of permission-sensitive behaviour:

Actor	Org membership	Organization role	Effective repo permission
fstest-write	member	all-repository write	write on every org repo
fstest-triage	member	all-repository triage	triage on every org repo
fstest-outsider	none	none	public read only

Elevated access uses organization-level all-repository roles, not per-repo collaborator grants or team membership. Direct collaborator grants vanish when the behaviour suite deletes and recreates a pool repo and re-adding them creates pending invitations. Org-level roles survive that delete/recreate cycle and apply to any future test-repo* name, including numbered pool slots that do not exist yet.

The behaviour suite verifies org membership once per org at ensure time. It does not call AddCollaborator, and it does not re-verify the all-repository role itself at runtime — that would require the e2e App installation on every pool org to hold the organization_custom_roles permission solely to call the organization-roles API. The all-repository role is verified once, at setup time, by hack/setup-new-e2e-org.sh (which runs with an org-admin gh session, not the e2e App). Missing membership fails with a message to run hack/setup-new-e2e-org.sh. The outsider must remain outside the organization and must not receive an all-repository role.

The setup script (hack/setup-new-e2e-org.sh) creates or verifies this model idempotently on halfsend-NN and on the STAGE org halfsend. To auto-accept org membership invitations, pass the actor PATs as environment variables:

TEST_ACTOR_WRITE_PAT=ghp_... TEST_ACTOR_TRIAGE_PAT=ghp_... \
  MINT_PROJECT=... MINT_FUNCTION=... hack/setup-new-e2e-org.sh 07

Without the PAT variables, the script pauses for manual acceptance.

Accounts and PATs are managed under #6024. PATs are stored as repository secrets TEST_ACTOR_WRITE_PAT, TEST_ACTOR_TRIAGE_PAT, and TEST_ACTOR_OUTSIDER_PAT on fullsend-ai/fullsend.

Inference gateway test gateway

The inference gateway behaviour tests (#8280, ADR 0137) call a durable agentgateway v1.6.0 instance on Cloud Run in the E2E GCP project. The gateway accepts GitHub Actions OIDC tokens from the pool repositories only. hack/setup-e2e-inference-gateway.sh creates it, or adopts an existing one with the same names, and keeps it in sync. The gateway also answers under the path prefix /e2e-prefix (PATH_PREFIX in the script, #8303): a path-prefix route with no policies of its own strips the prefix and hands the request back to the same listener, so the prefixed URL gets the same authentication, authorization and rate limits as the bare origin. The api-key scenario uses the prefixed URL; the oidc scenarios use the bare origin. The route went live from the #8303 pull request before it merged, so until then a run of the script from main shows the routes: block as a frozen-path difference. The scenarios are gated on @requires:capability: tags. Wiring it into the behaviour tests lists the capabilities CI declares and how to run the rest.

Who runs it, and when

An E2E project admin runs the script from a workstation with gcloud (authenticated), skopeo, jq, curl and gh (authenticated, to read this repository's workflow runs for the busy check). Run it again after any of these changes:

• Pool change: a new pool org, or a different DefaultPoolSize (pkg/behaviourtest/drivers/install/driver.go). Edit POOL_ORGS or POOL_REPOS at the top of the script.
• agentgateway bump: change AGW_VERSION.
• Model change: edit VERTEX_MODELS, plus vertex_params for a model-as-a-service model.

The operator needs these roles on the project:

IAM role	Purpose
roles/serviceusage.serviceUsageViewer	Check that the required APIs are enabled
roles/run.admin	Create and update the Cloud Run service and allow unauthenticated invocation
roles/secretmanager.admin	Create the config and stub key secrets, add versions, grant per-secret access
roles/artifactregistry.admin	Create the repository, copy the agentgateway image into it and read its digest
roles/iam.serviceAccountAdmin	Create the runtime service account
roles/iam.serviceAccountUser	Deploy the service as the runtime service account
roles/resourcemanager.projectIamAdmin	Read the project IAM policy on every run, and add or remove the roles/aiplatform.user grant

Check it, then run it

The project comes from --project or E2E_GCP_PROJECT_ID, and the region defaults to us-east5. The durable gateway serves the Vertex models, so pass --with-vertex on every run. A run with neither --with-vertex nor --without-vertex stops rather than remove them.

1. Preview with --dry-run. It only reads, prints what it would change, and exits 3 if anything would change:

   hack/setup-e2e-inference-gateway.sh --project "$E2E_GCP_PROJECT_ID" --with-vertex --dry-run

   Output against the durable gateway, with the project ID, URL, revision and config hash replaced by example values. "No changes" means a real run would be a no-op:

   ==> Dry run (read only) for the e2e inference gateway in project example-project (us-east5)
   
   ==> Checking required APIs...
       OK: required APIs are enabled
   
   ==> Change control...
       OK: config matches version 27, which the serving revision started with
   
   ==> Artifact Registry repository fullsend-e2e-gateway...
       OK: repository fullsend-e2e-gateway exists
   
   ==> agentgateway v1.6.0 image...
       upstream ghcr.io/agentgateway/agentgateway:v1.6.0 is sha256:9d3e6044ddcdc0878b1787f77bd401252b95e22684203fb5e874c4c42d2ed90c
       OK: us-east5-docker.pkg.dev/example-project/fullsend-e2e-gateway/agentgateway:v1.6.0 matches upstream
   
   ==> Service account fullsend-e2e-gateway@example-project.iam.gserviceaccount.com...
       OK: service account exists
       OK: service account has roles/aiplatform.user
   
   ==> Secrets...
       rendered config sha256 <sha256 of the rendered config>
       OK: secret fullsend-e2e-gateway-config is up to date
       OK: service account can read secret fullsend-e2e-gateway-config
       OK: secret fullsend-e2e-gateway-stub-upstream-key is up to date
       OK: service account can read secret fullsend-e2e-gateway-stub-upstream-key
   
   ==> Cloud Run service fullsend-e2e-gateway...
       OK: service is up to date
       OK: all traffic goes to the latest revision, with no tags
   
   ==> Verifying...
       service Ready condition: True, revision fullsend-e2e-gateway-00009-abc
       no token: GET /v1/models -> HTTP 401 text/plain
         body: authentication failure: no bearer token found
       x-api-key only: GET /v1/models -> HTTP 401 text/plain
         body: authentication failure: no bearer token found
       non-JWT bearer: GET /v1/models -> HTTP 401 text/plain
         body: authentication failure: the token header is malformed: Error(InvalidToken)
   
   ==> No changes: everything was already in place.
   
   ==> Behaviour test settings (set the URL as a repository secret; it embeds
       the project number, so never commit it):
   
       E2E_INFERENCE_GATEWAY_URL=https://fullsend-e2e-gateway-abc123-ul.a.run.app
       E2E_INFERENCE_GATEWAY_AUDIENCE=fullsend-e2e-gateway
   
       A positive check needs a GitHub Actions OIDC token for a pool
       repository; the inference-gateway behaviour test covers it.

2. Apply the changes:

   hack/setup-e2e-inference-gateway.sh --project "$E2E_GCP_PROJECT_ID" --with-vertex

   Each step prints OK: for what is already in place and CHANGED: for what it did. A config change adds one config secret version and rolls one Cloud Run revision. The new version is live for any instance that reads the mounted file as soon as it is added; the new revision makes sure the gateway restarts on it. A serving revision older than the latest secret version is always rolled, which also finishes a rollout an earlier run left incomplete. Any other difference in the service's settings triggers a redeploy, and the script names the setting that differed. Traffic pinned to an older revision, or an older revision reachable through a tag, is moved back to the latest one.

   Not executed: this real run has not happened yet. The durable gateway was deployed by hand with the same names and settings, and the dry run above shows the script adopts it unchanged.

The 401s must be text/plain from agentgateway. An HTML 401 comes from Cloud Run's invoker check, which means --no-invoker-iam-check is missing, and the script fails. The probes need no token, so a positive check is left to the behaviour test, which mints a GitHub Actions OIDC token for a pool repository.

The api-key behaviour test (ADR 0138) needs a gateway key for the echo model only. Pass its hash, never the key, as ECHO_KEY_HASH=sha256:<64 hex> on every run; a run without it removes the key. REAL_KEY_HASH does the same for a time-boxed key used in one local run, and is left unset afterwards. That key may call one real model: claude-haiku-5-5 by default, or gpt-oss-120b with REAL_KEY_MODEL=gpt-oss-120b. With a key, this test gateway serves both modes, so its OIDC check is permissive: a request with no credential passes authentication and each model then refuses it with a 403. That is a test-gateway arrangement, not a recommendation. An OIDC-only gateway keeps strict jwtAuth, and a gateway that does mix the modes must have authorization rules that deny anonymous requests. The probe therefore expects a 401 (api key authentication failure: ...) for a bearer that is neither a valid token nor a configured key, instead of the three probes above.

Change control

The behaviour job of e2e.yml calls this gateway on every run, and the merge queue requires that job. A bad deploy, or a deploy in the middle of a run, fails the queue. release.yml does not call the gateway, but the script also waits for release runs, so nothing changes while a release is cut. So a run that changes the gateway goes through three checks. A run that changes nothing skips them and is still a no-op.

1. Frozen-path diff. Before it changes anything, the script renders the new config and compares it with the live config: the config secret version that the revision getting the traffic started with. A revision pinned to a version number serves that version. The service mounts the secret's latest version, so otherwise that is the newest version created before the revision was, and the post-deploy check of the run that rolled the revision verified it. A version added later is a pending rollout: an instance that restarts may load it, so the script warns about it, but it is not the baseline. If the traffic is split across revisions, the script stops before it changes anything. The comparison masks only the changes that are allowed:

   • extra allowlist repositories after the pool list, such as temporary repositories added to the live config by hand;
   • the time-boxed REAL_KEY_HASH entry, when it may call exactly one real model, and its allow rule on that model;
   • the tokens-per-minute cap, only with --allow-tpm-change.

   Any other difference refuses the deploy with exit 4 and prints the masked diff. That covers the echo and echo-denied models, the pool list, the jwtAuth and apiKey modes, the echo key hash, the requests-per-minute cap, the Vertex models, and the real key's rule on echo or echo-denied. To make such a change, pass --allow-frozen-change. Use it only outside a release window, and tell the maintainers first. --dry-run runs this check too.

2. Busy check. Before its first change of any kind, including a pending rollout, an image or IAM change and --delete, the script refuses with exit 6 while a merge_group run of e2e.yml or a release.yml run is not completed. It asks fullsend-ai/fullsend with gh for each unfinished status, so a run behind a page of completed runs still counts. A failed lookup also stops the run, because an unknown state is not an idle gateway. --force-busy skips the check. Use it only for an outage fix. --dry-run skips it, because it changes nothing.

3. Post-deploy check. After any new revision serves traffic, the script calls claude-haiku-5-5, gemini-3.8-flash, gpt-oss-120b and echo, each on /v1/chat/completions, /v1/messages and /v1/responses, with no credential. All 12 calls must return 403. Then one call with a wrong API key must return 401. On a gateway without --with-vertex, only echo is called. On a gateway without a key hash, the anonymous calls must return 401, because strict jwtAuth refuses them before any model does. If a call gets any other status, or the new revision has no URL to call, the script prints the config secret version that was serving before the run and the command that restores it, then exits 5. If the run removed the Vertex grant, the command grants it again first. If that version is disabled, the script stops before it changes anything: enable it, or roll a revision onto an enabled version, to choose a rollback baseline first.

Exit codes:

Exit	Meaning
0	Done, or nothing to do
1	An error, or a verification probe failed
2	REAL_KEY_MODEL names a model the real key may not call
3	--dry-run found pending changes
4	The config change touches a frozen part of the config
5	The post-deploy check failed. Roll back.
6	A merge queue or release run is using the gateway

Rolling back. The post-deploy failure prints one command line, with the values filled in. It copies the previous version into a new latest version, then rolls a revision onto it:

gcloud secrets versions access <previous> --secret=fullsend-e2e-gateway-config --project="$E2E_GCP_PROJECT_ID" > fullsend-e2e-gateway-config-v<previous>.yaml \
  && gcloud secrets versions add fullsend-e2e-gateway-config --project="$E2E_GCP_PROJECT_ID" --data-file=fullsend-e2e-gateway-config-v<previous>.yaml \
  && gcloud run services update fullsend-e2e-gateway --project="$E2E_GCP_PROJECT_ID" --region=us-east5 \
       --update-secrets=/etc/agw-config/config.yaml=fullsend-e2e-gateway-config:latest,/etc/agw-upstream/key=fullsend-e2e-gateway-stub-upstream-key:latest

The restored config is the latest version again, so the service still mounts :latest and the script sees no drift in the service settings. Fix the change in the script before you re-run it. The next run compares the fixed config with the restored version and goes through the three checks again.

What it manages

Every resource is named fullsend-e2e-gateway*. Those names are reserved for this script: it adopts resources that already have them and touches nothing else. What it creates carries the marker purpose=fullsend-e2e-gateway: a label, or the description of the service account. Adopted resources keep whatever they had.

Resource	Name	Settings
Artifact Registry repository	fullsend-e2e-gateway	Holds ghcr.io/agentgateway/agentgateway:v1.6.0, copied unchanged with skopeo copy --all --preserve-digests. Every run checks that the copy's digest matches upstream, and refuses to continue if it does not.
Service account	fullsend-e2e-gateway@<project>.iam.gserviceaccount.com	Runtime identity. With --with-vertex, it holds roles/aiplatform.user.
Secret	fullsend-e2e-gateway-config	The generated agentgateway config.
Secret	fullsend-e2e-gateway-stub-upstream-key	The stub upstream key, a fixed non-secret value: e2e-stub-upstream-key.
Cloud Run service	fullsend-e2e-gateway	--allow-unauthenticated --no-invoker-iam-check, 1 vCPU / 512 MiB, at most 1 instance, label purpose=fullsend-e2e-gateway. Deployed by the verified image digest; every run checks that the serving revision runs that image. Both secrets are mounted as files at :latest.

The service account gets roles/secretmanager.secretAccessor on each secret, not on the project.

The generated config (print it with --print-config):

• validates tokens with jwtAuth. The issuer is https://token.actions.githubusercontent.com, its JWKS is fetched from GitHub, and the only accepted audience is fullsend-e2e-gateway. The caller's token is never forwarded upstream; that is agentgateway's default. The mode depends on whether a key hash is set:

  • No key hash: jwtAuth runs in strict mode, and a request without a valid token gets a 401.
  • ECHO_KEY_HASH or REAL_KEY_HASH set: jwtAuth runs in permissive mode, followed by an apiKey policy in optional mode. A valid JWT is checked and stripped first, so it never reaches the key check. A bearer that is not a JWT goes on to the key check, and a wrong key gets a 401. Each key carries its own allowedModels: [echo] for the echo key, and the one model REAL_KEY_MODEL names for the real key. A request with no credential passes both checks, and each model's authorisation rules then deny it with a 403. The durable gateway has ECHO_KEY_HASH only, and no real key.

  This mixed setup exists so that one test gateway can serve both the oidc and api-key auth modes. Do not copy it to production. A production OIDC gateway should keep the strict jwtAuth recommended by ADR 0137.

• authorises each model with an exact-match list of jwt.repository values. The list covers test-repo-01 to test-repo-12 in halfsend-01 to halfsend-12 and in the STAGE org halfsend: 156 repositories. It matches names, so keep the pool orgs registered: whoever owns a pool org name can call the gateway. A model that a configured key may call also admits that key's apiKey.purpose.

• strips x-api-key from every request it sends upstream.

• caps each real model at 60 requests and 200,000 tokens per minute for each kind of caller: the pool shares one budget per model, each configured key has its own, and every other caller shares one separate budget. So no caller can use up the behaviour tests' budget, not even with requests the model refuses. echo and echo-denied allow pool and key callers 60 requests a second; every other caller stays in its one shared bucket. Tokens are counted after each response, including cached input, so the token budget is a throttle rather than a hard ceiling: requests already in flight when it runs out still finish, and only later ones are refused. The budgets are held in the single instance and start again with each new revision. On the durable gateway, a refused request gets a 429 whose body is rate limit exceeded; the limit appears only in its x-ratelimit-limit and retry-after headers. During a flood of unauthenticated requests, a request with no credential can receive that 429 instead of a 403. The script's tests check the rendered rules but do not evaluate them; the separation between callers was checked by sending requests to a running gateway as the pool, as a key, and without credentials.

• serves these models:

  Model	Allowed for	Upstream
  claude-haiku-5-5, gemini-3.8-flash (--with-vertex only)	The pool, plus the real key for claude-haiku-5-5 when REAL_KEY_HASH is set and REAL_KEY_MODEL is left at its default	Vertex AI, location global, as the runtime service account
  gpt-oss-120b (--with-vertex only)	The pool, plus the real key when REAL_KEY_HASH is set and REAL_KEY_MODEL=gpt-oss-120b	The Vertex AI model-as-a-service model openai/gpt-oss-120b-maas, location global, as the runtime service account. It serves Responses clients such as Codex on /v1/responses, which the gateway translates to Chat Completions. Gemini 3 tool loops through Responses translation aren't supported on the test gateway, and the Claude models take no Responses requests. The gateway drops assistant messages that have neither content nor tool calls from requests to this model, because its chat template rejects them: a test-gateway requirement, not something fullsend needs from every gateway.
  echo	The pool, plus the echo key when ECHO_KEY_HASH is set	A header-echo listener in the same container. The gateway sends it the stub key. Its answer reports whether that key, and not the caller's credential, arrived, so the custody check can assert it.
  echo-denied	Only fullsend-e2e-gateway-outside/not-a-pool-repo, which is outside the pool	The same echo listener. A behaviour test calls it from a pool repository and expects 403.

• turns off agentgateway's admin, stats and readiness listeners (adminAddr, statsAddr and readinessAddr set to "off"), which would otherwise bind ports 15000, 15020 and 15021. Nothing uses them: Cloud Run's startup probe is its default TCP check on port 8080, and nothing scrapes metrics.

• writes one access log line per request to Cloud Run Logging. The line carries jwt.sub, the GitHub OIDC subject, which names the calling repository and its ref or event, alongside src.addr, the method, host, path, status and duration. It records no header values and no request or response bodies.

Wiring it into the behaviour tests

The gateway URL embeds the E2E project number, so it is never committed. This script changes no workflow files; the wiring is:

1. Set two repository-level secrets, like the E2E_GCP_* secrets:

   • E2E_INFERENCE_GATEWAY_URL: the URL the script prints.
   • E2E_INFERENCE_GATEWAY_TEST_KEY: the plaintext of the echo-only key whose hash is passed as ECHO_KEY_HASH. The gateway authorises it for the echo model only, so it reaches no real model. The suite registers it for redaction. The api-key scenario sets it as the enrolled pool repository's FULLSEND_INFERENCE_GATEWAY_API_KEY secret for the scenario and deletes it afterwards, so the pool needs no standing key.

   E2E_INFERENCE_GATEWAY_AUDIENCE defaults to fullsend-e2e-gateway and needs setting only if it changes.

2. e2e.yml passes both secrets in the env of the behaviour job's "Run behaviour tests" step, next to the E2E_GCP_* secrets, and to the artifact redaction step. The gateway scenarios skip when these variables are unset. Pull requests that the gate job does not authorize never run the behaviour job, so they receive neither secret. Authorized pull requests, including forks, do receive them, and the behaviour job runs the pull request's head code with those secrets in its environment. Authorizing a pull request therefore lets its head code read both values, as it can already read the E2E_GCP_* secrets. The echo-only key scope limits what the key can call, not whether it can be read. To keep fork pull requests away from these secrets, add an explicit condition in the base workflow.

CI declares only the gateway capabilities that reach no real model. They are part of the BEHAVIOUR_CAPABILITIES default in the Makefile, which make behaviour-test and the behaviour job use:

Capability	Scenarios	Declared in CI
inference-gateway	features/runtime/inference-gateway.feature: one dummy-runtime scenario covering the placeholder, credential custody on echo, egress scope and the echo-denied refusal	Yes
inference-gateway-api-key	features/runtime/inference-gateway-api-key.feature (dummy runtime, echo-only key)	Yes
runtime-pi-gateway	features/runtime/pi-gateway.feature (pi through the gateway to a real model, claude-haiku-5-5)	No: costs a real model run
runtime-codex-gateway	features/runtime/codex-gateway.feature (codex through the gateway to a real model, gpt-oss-120b, on Responses)	No: costs a real model run
runtime-claude-gateway	features/runtime/claude-gateway.feature (Claude Code through the gateway to a real model, claude-haiku-5-5, in the oidc mode)	No: costs a real model run
inference-gateway-reseed	The token-rotation scenario in features/runtime/inference-gateway.feature, which also needs inference-gateway	No: holds the sandbox for a 330 s wait

To run the undeclared scenarios on demand, set E2E_INFERENCE_GATEWAY_URL and pass the full capability list. This replaces the Makefile default rather than adding to it:

BEHAVIOUR_CAPABILITIES=runtime-pi,inference-gateway,inference-gateway-api-key,runtime-pi-gateway,runtime-codex-gateway,runtime-claude-gateway,inference-gateway-reseed \
  make behaviour-test

To run them in CI, add them to the BEHAVIOUR_CAPABILITIES default in the Makefile on a pull request. The behaviour job reads the Makefile from the pull request.

Removing it

hack/setup-e2e-inference-gateway.sh --project "$E2E_GCP_PROJECT_ID" --delete

--delete asks you to type the project ID (--yes skips the prompt). It then removes the resources above, plus the roles/aiplatform.user grant. It checks first and stops without deleting anything when:

• a resource lacks the purpose=fullsend-e2e-gateway marker, so the script may not have created it. The durable gateway was deployed by hand and adopted, so removing it needs --include-unlabelled.
• a service of that name runs in a region other than --region, or the service listing reports an unreachable region. The secrets and the service account are global, so a mistyped region must not remove them from under a running gateway.

Not executed: --delete has not been run against the durable gateway, which stays up.

CI authorization

Pull requests trigger e2e via pull_request_target in .github/workflows/e2e.yml so fork PRs can use repository secrets. Because that exposes credentials to untrusted code, a gate job runs first (see workflow comments for why it is a separate job).

Who runs automatically

E2E tests run without maintainer action when the PR author is an org/repo member or collaborator (author_association of OWNER, MEMBER, or COLLABORATOR on the base repo). The gate uses the frozen github.event.pull_request.author_association from the workflow event — not a live REST lookup — because GITHUB_TOKEN lacks read:org and cannot see org membership for members with private visibility. (Note: agent dispatch paths use the collaborator permission API instead, which does not have this limitation — see ADR 0054.)

Who needs ok-to-test

External contributors and fork PR authors must have a maintainer with write access apply the ok-to-test label after the latest push. A label from anyone else (for example a triage-role user) is removed and does not authorize the run. The label must be created once in GitHub repo settings (Settings → Labels).

Stale labels

If new commits are pushed after ok-to-test was applied, the label is removed automatically and e2e is skipped until a maintainer re-applies it after reviewing the latest changes. Freshness compares the label timestamp against the frozen PR updated_at from the workflow event (PR_UPDATED_AT); the live API fallback may over-reject when non-push activity bumped updated_at. Applying the label triggers the E2E ok-to-test / Functional ok-to-test caller workflows, which workflow_call into the main suites.

The main E2E Tests and Functional Tests workflows do not subscribe to labeled events. Only opened / synchronize / reopened cancel in-progress work in the per-PR concurrency group (code changed). Label events are authorization only: they never cancel an in-progress suite. If a suite is already running when ok-to-test is applied, GitHub may queue a second run behind it (no expression-only “skip if busy”); that is accepted.

Other labels (for example ready-for-review, requires-manual-review, or component/*) do not authorize e2e. They may start the thin ok-to-test caller with a skipped run job (GitHub cannot filter by label name at on:), but they do not start skipped checks under the E2E Tests / Functional Tests workflow names, and they never enter the suite concurrency group.

Blocked runs

When the gate runs and denies authorization, a sticky PR comment (marker <!-- e2e-gate -->) explains why and what to do. That is distinct from a non-ok-to-test label event, where the thin caller skips without invoking the gate. Re-run the workflow or add/re-apply ok-to-test as appropriate.

CI architecture

1. PR open/sync — E2E Tests / Functional Tests run gate then suite jobs (trusted authors authorized immediately)
2. ok-to-test label — thin E2E ok-to-test / Functional ok-to-test workflows call the same suites via workflow_call (fork / external path)
3. Gate — authorize the PR author or a fresh ok-to-test label (base checkout only; never checks out PR head)
4. Behaviour / playback — checkout PR head SHA, authenticate to GCP via WIF, mint cross-org tokens per pool org, make behaviour-test (or make playback-test for the playback job)

Pushes to main, merge queue, and workflow_dispatch skip the gate and run e2e directly.

Test GitHub Apps

The fullsend-test-* apps are test-only GitHub Apps owned by fullsend-ai, separate from the production fullsend-ai-* app set. They exist for temporary and test mints, including the Cloudflare Worker mint BT chain. There is no e2e test app. The fix role shares the coder test app and PEM (same as production).

Warning: Do not enroll these apps on the production community mint. They are strictly for test infrastructure.

Role / app / secret inventory

Role	App Slug	App ID	Repository Secret
fullsend	fullsend-test-fullsend	4312984	TEST_FULLSEND_PEM
triage	fullsend-test-triage	4312988	TEST_TRIAGE_PEM
coder	fullsend-test-coder	4312994	TEST_CODER_PEM
review	fullsend-test-review	4313005	TEST_REVIEW_PEM
retro	fullsend-test-retro	4313010	TEST_RETRO_PEM
prioritize	fullsend-test-prioritize	4313012	TEST_PRIORITIZE_PEM

PEM private keys are stored as repository secrets on fullsend-ai/fullsend. The fix role reuses the coder app (fullsend-test-coder) and TEST_CODER_PEM.

Installation targets

All six apps are installed with access to all repositories on halfsend-01 through halfsend-12 and on halfsend (used by the STAGE driver).

App permission scopes

Each app is registered with the minimum explicit permissions its corresponding role requires. Tokens minted for these apps are downscoped by the mint to the canonical role permissions (see internal/mintcore/github.go).

metadata:read is implicitly granted to all GitHub App installations and is not listed below. The canonical role permissions in internal/mintcore/github.go include it for token downscoping completeness, but it is not an explicit registration setting.

Role	Permissions
fullsend	actions:write, actions_variables:read, administration:write, checks:read, contents:write, issues:read, members:read, organization_projects:read, pull_requests:write, workflows:write
triage	contents:read, issues:write
coder	checks:read, contents:write, issues:write, packages:read, pull_requests:write
fix	Reuses the coder app and PEM; the fix stage mints the coder role and therefore uses the coder permission set.
review	checks:read, contents:read, issues:write, pull_requests:write
retro	actions:read, contents:read, issues:write, pull_requests:write
prioritize	contents:read, issues:write, organization_projects:write

The fix row is a hosted-mint dispatch alias: fix workflows mint the coder role and reuse fullsend-test-coder and TEST_CODER_PEM; it is not a separate App registration in that setup. Per-repo setup may still request the literal fix role and create a role-specific App for that repository.

Operator notes

Permission rollout for the coder App: When adding a permission such as packages:read, update the fullsend-test-coder App registration first. Then have the installation owner for each existing pool org (halfsend-01 through halfsend-12) Accept the pending permission update. New pool-org installs receive the permission during installation; existing installs may otherwise continue using the mint's rollout warning path.

PEM rotation: Generate a new private key on the app's settings page (https://github.com/apps/<slug>/settings), then update the corresponding TEST_*_PEM repository secret on fullsend-ai/fullsend. If the app is enrolled on a test mint, update the PEM secret there as well.

Cloudflare test token rotation: Create a new Account API token with Workers Edit (or the Edit Cloudflare Workers template), store it as TEST_CLOUDFLARE_API_TOKEN, and keep TEST_CLOUDFLARE_ACCOUNT_ID aligned with the account that hosts Workers mint-test and stage-mint. The token must cover both Workers. Do not put the new value into site-deploy CLOUDFLARE_API_TOKEN.

Installing on a new pool org: Install each app via its public install URL:

# For each app slug in the inventory table above:
# https://github.com/apps/<slug>/installations/new
# Select the target pool org and grant access to "All repositories".

Using App IDs in ROLE_APP_IDS for temporary mints: When configuring a temporary or test mint (e.g., the CF Worker mint), set ROLE_APP_IDS using the App IDs from the inventory table:

{
  "fullsend": "4312984",
  "triage": "4312988",
  "coder": "4312994",
  "review": "4313005",
  "retro": "4313010",
  "prioritize": "4313012"
}

Point the mint's PEM directory (or Secret Manager entries) at the corresponding TEST_*_PEM keys.