# Operations

Day-2 administration for fullsend per-repo installations: configuration updates, workflow syncing, uninstall, and standalone commands for split-responsibility workflows.

## Prerequisites

- **fullsend CLI** installed (see [Getting Started](../getting-started/))

The remaining prerequisites are forge-specific:

**GitHub:**

- **GitHub access** — repository admin for the target repository
- **`gh` CLI** authenticated with the required OAuth scopes (see [OAuth scope reference](../infrastructure/advanced-setup.md#oauth-scope-reference))

**GitLab:** none of the GitHub-specific prerequisites above apply — GitLab
does not use `gh`. See [Configuring GitLab § Prerequisites](configuring-gitlab.md#prerequisites)
for the GitLab access token and permissions needed for the day-2 tasks
documented below.

## Updating configuration values

### GitHub

Update individual secrets or variables without re-running full setup:

```bash
fullsend github set "$OWNER/$REPO" FULLSEND_GCP_PROJECT_ID new-gcp-project
fullsend github set "$OWNER/$REPO" FULLSEND_GCP_REGION global
```

| Key | Storage Type | Description | Example value |
|-----|-------------|-------------|---------------|
| `FULLSEND_GCP_REGION` | Repo variable | GCP region for Agent Platform inference | `global` |
| `FULLSEND_OPENAI_AUDIENCE` | Repo variable | OpenAI WIF audience; use with the two identifiers below | `openai` |
| `FULLSEND_OPENAI_IDENTITY_PROVIDER_ID` | Repo variable | OpenAI WIF identity-provider ID | `idp-example` |
| `FULLSEND_OPENAI_SERVICE_ACCOUNT_ID` | Repo variable | OpenAI WIF service-account ID | `sa-example` |
| `FULLSEND_REVIEW_CLIENT_ID` | Repo variable | OAuth client ID of the review agent's GitHub App (best-effort, auto-set by installer) | `Iv23li1nIorNLIQy6NWK` |
| `FULLSEND_GCP_PROJECT_ID` | Repo secret | GCP project ID where Agent Platform is enabled | `my-gcp-project` |
| `FULLSEND_GCP_WIF_PROVIDER` | Repo secret | Full WIF provider resource name for OIDC authentication | `projects/123456789/locations/global/...` |
| `FULLSEND_OPENAI_API_KEY` | Repo secret | OpenAI API key for repos whose `inference.auth` is `openai-api-key`, written by `repos install --openai-api-key` (exported as `OPENAI_API_KEY`; unused when the WIF trio is set) | `sk-...` |
| `FULLSEND_INFERENCE_GATEWAY_API_KEY` | Repo secret | Inference gateway API key for repos whose `inference.gateway.auth` is `api-key` (ADR 0138); set by the repository admin (`fullsend github set`), not by setup. A long-lived secret: prefer `auth: oidc` | — |
| `FULLSEND_GITHUB_RUNNER_SECRETS` | Repo or org secret (optional) | JSON object of name → value for host-side scripts, read through each harness's `env.runner` ([ADR 0136](../../ADRs/0136-runner-secrets-through-one-stored-secret.md)). Set with `gh secret set`, not `fullsend github set`; see [Pass secrets to host-side scripts](../user/building-custom-agents.md#pass-secrets-to-host-side-scripts-github) | `{"JIRA_API_TOKEN":"..."}` |

Repositories selecting `inference.auth: openai-wif` need no inference secrets unless
Vertex sub-agents require GCP credentials. See [OpenAI Workload Identity](../infrastructure/openai-workload-identity.md)
for identifier sources and precedence. These three WIF variables are user-owned;
edit them in GitHub's repository Actions variables settings rather than using `fullsend github set`.

`FULLSEND_APP_SET` (GitHub App set prefix; apps named `{app-set}-{role}`,
defaulting to `fullsend-ai`) is not one of the keys `fullsend github set`
accepts. It is set by `fullsend github setup --app-set` or, for manifest-managed
repositories, from `app_set` in `repos.yaml`, and reruns preserve a custom value
and repair drift or absence. To change it after the fact, run
`fullsend repos set github.app_set <prefix>` (or set it on the repository entry)
and then `fullsend repos install`, or use `fullsend github setup --app-set`.

### GitLab

For initial GitLab setup, see [Configuring GitLab](configuring-gitlab.md). Secrets are checked for presence only: converge cannot read secret values back to compare them, so it never reports drift in `FULLSEND_GCP_PROJECT_ID` or `FULLSEND_GCP_WIF_PROVIDER`. Instead, whenever a project is supplied, by `--vertex-project` or by an `inference.project` declared in the managed configuration, every install rewrites both secrets from the supplied values. To change the project or WIF provider, update the declared `inference.project` / `inference.wif_provider` in `repos.yaml` (or pass the new value with `--vertex-project` / `--vertex-wif-provider` when nothing is declared) and run `fullsend repos install`. When neither supplies a project, existing secrets are reused unchanged.

`FULLSEND_GCP_REGION` is a plain variable. To change the region through
converge, declare it in `repos.yaml`, for example
`fullsend repos set 'gitlab.repos["<group/project>"].config.inference.region' us-east5`,
then run `fullsend repos install`. A declared region is managed: install
repairs a value that differs and `repos status` reports the drift. Without
a declared region, install keeps the existing variable as it is (so a
region edited by hand in Settings → CI/CD → Variables stays) and writes
`global` only where the variable is missing and Vertex credentials are
being written. See
[Effective infrastructure values](../../cli/repos.md#effective-infrastructure-values).

| Key | Storage Type | Description | Example value |
|-----|-------------|-------------|---------------|
| `FULLSEND_FORGE_TOKEN` | CI/CD secret | Legacy shared token from a repository installed before the role-only rollout; no runtime path uses it, and no automated path removes it — clean it up manually | (masked) |
| `FULLSEND_GCP_REGION` | CI/CD variable | GCP region for Agent Platform inference | `us-central1` |
| `FULLSEND_GCP_PROJECT_ID` | CI/CD secret | GCP project ID for inference | `my-gcp-project` |
| `FULLSEND_GCP_WIF_PROVIDER` | CI/CD secret | WIF provider resource name for inference | `projects/123456789/locations/global/...` |
| `FULLSEND_DISPATCH_SECRET` | CI/CD secret | HMAC secret for dispatch variables and poll-state documents; auto-provisioned by `repos install` | (generated) |
| `FULLSEND_TRIGGER_TOKEN` | CI/CD secret | GitLab pipeline trigger token for the webhook fast-path dispatcher. Masked and protected; never logged. Provisioned when the fast-path is enabled. | (masked) |
| `FULLSEND_WEBHOOK_SECRET` | CI/CD secret | GitLab project-webhook secret (`X-Gitlab-Token`) for the webhook fast-path. Masked and protected; never logged. Provisioned when the fast-path is enabled. | (masked) |
| `FULLSEND_GITLAB_ROLE_REGISTRY` | CI/CD variable (protected, unmasked) | Administrator role registry (JSON references and policy, not secret values); empty means built-in roles only. Written by `repos install --gitlab-role-registry`. | `{"roles":[]}` |
| `FULLSEND_GITLAB_ROLE_ROTATION` | CI/CD variable (protected, unmasked) | Per-role rotation state (lock, token IDs, expiry dates, phase). Never stores token values. Written by `repos install` during rotation. | `{"version":2,"roles":{}}` |
| `FULLSEND_GITLAB_POLLER_GENERATIONS` | CI/CD variable (protected, unmasked) | Poller identity-generation state (account IDs and handoff phase; never token values). Written by `repos install` when it creates or rotates the webhook trigger through a fresh Poller generation. An absent variable is an empty state; a present but blank one fails closed because recorded state may have been lost. After the manual cleanup a quarantine message describes, an administrator removes only the `pending` generation and keeps `version`, `current_user_id`, `retiring_user_id` and `abandoned_user_ids`. | `{"version":1}` |
| `FULLSEND_GITLAB_TRIGGER_ACCOUNT` | CI/CD variable (protected, unmasked) | Ledger for the trigger-only GitLab service account (lifecycle phase, nonce, branch, and resource IDs; never token values or private keys). Written by `repos install` and removed by `repos uninstall` only after every resource it records is gone. Do not edit or blank it by hand: a blank or malformed ledger fails closed. | `{"version":1}` |
| `FULLSEND_GITLAB_TRIGGER_ACCOUNT_TOKEN` | CI/CD secret (protected, masked) | Pipeline trigger token owned by the trigger-only service account. Written by `repos install` only after GitLab confirms the variable is masked and protected; no pipeline reads it until the dispatch route is activated. Removed before its token is revoked, and by `repos uninstall`. | (masked) |
| `FULLSEND_GITLAB_POLLER_TOKEN` / `FULLSEND_GITLAB_ANALYST_TOKEN` / `FULLSEND_GITLAB_CODER_TOKEN` | CI/CD secret | Built-in role PATs provisioned by `repos install`; the selected role secret is required for every runtime job | (masked) |
| `FULLSEND_OPENAI_API_KEY` | CI/CD variable (masked) | OpenAI API key for projects whose `inference.auth` is `openai-api-key`, written by `repos install --openai-api-key`. The job maps it to `OPENAI_API_KEY`; an unprefixed `OPENAI_API_KEY` CI/CD variable is no longer used (see [upgrade steps](../../cli/repos.md#gitlab-fullsend_openai_api_key-replaces-openai_api_key-breaking)) | `sk-...` |
| `FULLSEND_INFERENCE_GATEWAY_API_KEY` | CI/CD variable (masked) | Inference gateway API key for projects whose `inference.gateway.auth` is `api-key` (ADR 0138); the job passes it to the runner under its own name. Set by the project admin, not by setup | — |

`repos install` provisions and rotates the registered role credentials directly. There is no migration gate and no shared-token runtime path. Neither install nor uninstall removes a leftover `FULLSEND_FORGE_TOKEN` from a repository installed before the role-only rollout — clean that up manually.

### Per-repo configuration in `repos.yaml`

For repositories managed through `repos.yaml` (every GitLab install, and GitHub fleets), `.fullsend/config.yaml` is managed: change per-repo config values — including `runtime` and `allowed_remote_resources` — in `defaults.config`, `github.config`, `gitlab.config`, or the repository entry's `config`, then run `fullsend repos install`. `repos install` has no configuration flags (no `--runtime`, `--allowed-remote-resources`, `--roles`, `--vertex-region`, or the other removed flags listed in [`fullsend repos`](../../cli/repos.md#removed-configuration-flags-breaking)); edit the manifest by hand or with `fullsend repos set` (for example `fullsend repos set defaults.config.runtime claude`). The former sibling keys (`defaults.runtime`, `defaults.allowed_remote_resources`, and their repository-entry equivalents) are rejected with the replacement path. See [Repo Management — Managed configuration](repo-management.md#managed-configuration).

## Syncing workflow templates

After upgrading the fullsend CLI, re-run `github setup` to update the workflow file for a single repo:

```bash
fullsend github setup "$OWNER/$REPO" \
  --inference-project "<GCP_PROJECT>" \
  --inference-wif-provider "<WIF_PROVIDER>"
```

For manifest-managed installations (including GitLab repos), use `repos install` to converge all repos (including workflow ref upgrades):

```bash
fullsend repos install -f repos.yaml
```

This is idempotent — it provisions new repos, repairs missing or drifted components (workflow, thin callers, variables, secrets, pipeline schedules, GitLab poller protected-ref pipeline access — a disabled GitLab schedule is reported as drift and reactivated only when `--reactivate-schedules` is passed; the GitLab webhook fast path — a trigger token, webhook secret, and project webhook, provisioned only once the dispatcher template defines a runnable dispatcher job that admits trigger pipelines and its scripts are on the default branch, the default branch is protected and is the project's only protected-branch rule (the trigger token is project-wide, so other protected refs are not isolated), no other root job or include is admitted to trigger pipelines (user-owned jobs are never rewritten: add rules that reject `$CI_PIPELINE_SOURCE == "trigger"` to them), and `no_one_allowed` is verified — if default-branch protection disappears, or any of the last three stops holding or cannot be verified, converge revokes the Fullsend-managed trigger tokens and deletes the Fullsend-owned webhook instead of leaving the credential live, with credential-variable protection drift and pending superseded-token revocations repaired and hooks Fullsend does not own left untouched; GitLab pipeline-variable override-role inspection — typed pipeline-input dispatch activation fails closed unless the project already has `ci_pipeline_variables_minimum_override_role=no_one_allowed`; runnable typed templates are never delivered before that setting is verified, even with `FULLSEND_GITLAB_PIPELINE_VAR_RESTRICTION=enforced`; legacy upgrades require maintenance-window preparation; repos still pinned to the legacy variable-based wrapper are exempt from the requirement — see [GitLab Role-Credential Contract](../../contributing/gitlab-role-credentials.md#how-a-job-selects-its-credential)), repairs scaffold content drift (including structural rewrites of `.gitlab/ci/fullsend-pipeline.yml` at an unchanged template ref, and removal of a leftover `.gitlab/ci/fullsend-dispatch.yml` from installs predating #7707), refreshes a declared configuration preset (`.fullsend/config.base.yaml`, after checking the replacement does not implicitly relax the effective configuration; with no preset an existing base file stops that repo before any write), rewrites a drifted managed `.fullsend/config.yaml`, removes the Fullsend-owned assets an earlier vendored install left behind once a GitHub repo resolves `vendor: false` (reported as `vendored-assets` drift; see [Turning vendoring off](../../cli/repos.md#turning-vendoring-off)), and upgrades workflow refs. Variables with manifest-specified values (e.g. mint URL, GCP region, review app client ID) are checked for value drift; secrets and runtime-mutated variables are checked for presence only. For GitLab repos, converge also migrates any leftover legacy poll-state CI/CD variables (from installs predating #7380) into the HMAC-signed poll-state branches and deletes them, so they stop being seeded or reported as orphans. Converge also migrates the root `.gitlab-ci.yml`: an obsolete `merge_request_event` workflow rule left over from installs predating the removal of native MR dispatch (#7322) is stripped automatically, without disturbing any other fullsend or user-owned entries in the file. This automatic migration applies only to repos where fullsend owns the `workflow:` block (fresh installs, identified by the fullsend-generated `workflow.name`); repos enrolled by merging fullsend rules into a pre-existing `workflow:` block have no such ownership marker and are left untouched — remove the leftover `merge_request_event` rule from those manually, as described in the uninstall steps below. Converge separately strips a leftover empty `dispatch` stage from `stages:`, left over from installs predating the removal of the empty dispatch stage (#7337). This migration is gated differently: it applies whenever the file has the fullsend pipeline include and at least one current fullsend stage (`poll` or `agent`) already in `stages:`, and it additionally scans every job definition in the file — including ones reached only via `extends:` or the YAML merge key (`<<:`) — for a still-live reference to `dispatch`, leaving the stage in place if any job depends on it. Converge also backfills a missing `CI_DEBUG_TRACE` deny-before-admit workflow rule (ADR 0125) into `workflow:rules` for repos where fullsend owns the `workflow:` block; as with the other root-file migrations above, repos enrolled by merging fullsend rules into a pre-existing `workflow:` block have no ownership marker, so converge leaves them untouched — add that deny rule to those repos manually. Every repository's `.fullsend/config.yaml` is managed; an existing one without the ownership marker needs adoption and is left untouched (install stops before any write).

## Uninstalling

### Per-repo teardown

To remove fullsend from a single repository:

**GitHub repos:**

1. Delete `.github/workflows/fullsend.yaml`, `.github/workflows/prioritize.yml`, and repo-level secrets/variables
2. Run `fullsend inference deprovision "$OWNER/$REPO"` to remove WIF access
3. Remove the `FULLSEND_MINT_URL` repository variable (if set) — no separate unenrollment is needed for the hosted community mint

**GitLab repos:**

1. Run `fullsend repos uninstall "$PROJECT_PATH"` (the full `group/subgroup/project` path) to open a PR that removes fullsend entries from `.gitlab-ci.yml` and deletes `.gitlab/ci/fullsend-pipeline.yml`, `.gitlab/ci/fullsend-agent.yml`, `.gitlab/ci/fullsend-poll.yml`, `.gitlab/ci/fullsend-dispatcher.yml`, leftover `.gitlab/ci/fullsend-dispatch.yml` from installs predating #7707, `.gitlab/ci/scripts/trust-ci-server-ca.sh`, `.gitlab/ci/scripts/pin-ci-job-identity.sh`, `.gitlab/ci/scripts/select-gitlab-role-token.sh`, `.gitlab/ci/scripts/install-fullsend-cli.sh`, `.gitlab/ci/scripts/run-poll-job.sh`, `.gitlab/ci/scripts/run-dispatcher-job.sh`, `.gitlab/ci/scripts/run-agent-job.sh`, `.gitlab/ci/scripts/checkout-mr-source.sh`, and `.fullsend/config.yaml` (pass `--direct` to push those file changes to the default branch). Before the scaffold is removed, the Fullsend-owned webhook fast-path project webhook is deleted and its managed pipeline trigger token is revoked (hooks Fullsend does not own are left untouched); if that fails, nothing else is removed and uninstall can be retried. After the webhook fast path is removed and before role cleanup, uninstall removes the trigger-only service account (its published token variable, trigger token, protected-branch push entry, deploy key, and the account), under the project lease; if that fails, or the Poller identity cannot be verified, nothing else is removed, `FULLSEND_GITLAB_TRIGGER_ACCOUNT` is kept as the cleanup record, and uninstall can be retried (see [Trigger-only service account recovery](configuring-gitlab.md#trigger-only-service-account)). Before any of that, uninstall verifies the `FULLSEND_GITLAB_ROLE_ROTATION` ownership state and stops before deleting anything if it is unreadable or malformed. That variable is removed last, after every other cleanup step succeeds; if it records administrator-supplied service accounts, it is kept as an exclusions-only document so a later install does not treat them as Fullsend's. Variables, secrets (including `FULLSEND_TRIGGER_TOKEN` and `FULLSEND_WEBHOOK_SECRET`), and the `fullsend-poll-state-slash` / `fullsend-poll-state-events` branches are deleted immediately via the API, except `FULLSEND_GITLAB_POLLER_GENERATIONS`, which is deleted only after every Poller account it records was verified and deleted, once all other cleanup steps succeed. Uninstall keeps it when cleanup is incomplete or ambiguous (a failed account deletion, a recorded account that is administrator-owned, a pending account creation with no recorded account ID, or an unreadable document), so the quarantine and retirement obligations it records survive until an administrator has reconciled them (see the variable table above). If you prefer manual removal: delete `.gitlab/ci/fullsend-*.yml`, `.gitlab/ci/scripts/*.sh`, and `.fullsend/config.yaml`, then edit `.gitlab-ci.yml` to remove the fullsend pipeline include entry, the fullsend stages (`dispatch`, `poll`, `agent`), the fullsend workflow rules (the `CI_DEBUG_TRACE` deny-before-admit rule, `schedule`, `api`, `trigger`, and `merge_request_event` on installs from before native MR dispatch was removed), and the `auto_cancel` block if fullsend added it. Only delete `.gitlab-ci.yml` entirely if it contains no non-fullsend configuration. Also delete the two poll-state branches if they are still present.

> **Note:** During install, fullsend sets `workflow.auto_cancel.on_new_commit: none` when an existing, non-empty `.gitlab-ci.yml` already has a `workflow:` block and that key is missing; it does not overwrite an existing value. If that existing block has no `rules:` key, fullsend also adds a `CI_DEBUG_TRACE` deny-before-admit rule (`when: never`) followed by the protected-ref `schedule`/`api` rules, which can stop ordinary push pipelines unless the block already has a matching rule; add a catch-all/push rule (or an explicit `when: always` rule) before installing, or remove the name-only block so fullsend can leave `workflow:` absent. If an existing, non-empty file has no `workflow:` block, fullsend leaves it absent so push-triggered pipelines are not disrupted. For a missing or empty `.gitlab-ci.yml`, fullsend writes a fullsend-owned `workflow:` block with the `CI_DEBUG_TRACE` deny rule and protected-ref `schedule`/`api` rules; later push jobs added to that file likewise need additional `workflow.rules` or an explicit `when: always` rule, or GitLab will skip them. Repos with `on_new_commit: interruptible` (or other non-`none` values) may experience agent pipeline cancellations because fullsend requires `on_new_commit: none` for reliable agent runs. If you see unexpected pipeline cancellations, set `on_new_commit: none` in your `.gitlab-ci.yml` workflow block.

2. Delete all CI/CD variables prefixed with `FULLSEND_` (including `FULLSEND_OPENAI_API_KEY`). `repos uninstall` already removes every Fullsend-managed inference variable (`FULLSEND_GCP_PROJECT_ID`, `FULLSEND_GCP_WIF_PROVIDER`, `FULLSEND_GCP_REGION`, and `FULLSEND_OPENAI_API_KEY`) whatever the project's `inference.auth` is, including leftovers from an earlier method, so this step only matters for manual teardown or variables it did not manage. If you still have a legacy unprefixed `OPENAI_API_KEY` from the old static-key route, delete it yourself too if you want it gone — fullsend never created it (it is a plain CI/CD variable, not `FULLSEND_`-prefixed) and does not delete it as part of uninstall
3. Confirm that this repo's role project access tokens — `fullsend-poller`, `fullsend-analyst`, `fullsend-coder`, and any `fullsend-role-*` tokens — were revoked. `repos uninstall` revokes them automatically when it can list project access tokens, and it also deletes the verified Fullsend-managed project service accounts that issued them (administrator-supplied accounts and their tokens are left in place); if uninstall failed on token revocation, retry it, then delete any leftover project service accounts from Project information → Members (their personal access tokens do not appear under Settings → Access Tokens) and revoke any remaining project access tokens from Settings → Access Tokens. A 403 from the token-list API fails uninstall closed rather than being treated as "nothing to revoke" — GitLab returns the identical error for plan-tier feature gating, group-level PAT disablement, and insufficient token permissions, so silently continuing risks leaving tokens live after a reported-successful uninstall. Only a confirmed-gone project (404) is treated as an empty inventory. If the project access token API is genuinely and permanently unavailable for this project (for example, a plan tier that doesn't offer it), retrying will not converge — finish the teardown by revoking any remaining tokens by hand from Settings → Access Tokens and deleting any leftover project service accounts from Project information → Members, then run `fullsend repos uninstall "$PROJECT_PATH" --manifest-only` to drop the manifest entry once cleanup is confirmed. If this repo was installed before the role-only rollout and still has a `fullsend-bot` token, `repos uninstall` does **not** revoke it — revoke it by hand from Settings → Access Tokens
4. If you enrolled role credentials with personal PATs (`--gitlab-role-token` — see [Configuring GitLab § Free-tier role-token enrollment](configuring-gitlab.md#free-tier-role-token-enrollment) and [Role identities and GitLab Free](configuring-gitlab.md#role-identities-and-gitlab-free)), also revoke those PATs on the accounts that issued them. Deleting the CI/CD variables in step 2 does not revoke the underlying PATs.
5. Delete fullsend pipeline schedules (`fullsend slash poll` and `fullsend event poll`)
6. If you provisioned the shared `gitlab-oidc` WIF provider (see [Configuring GitLab § Inference Setup](configuring-gitlab.md#inference-setup)), revoke this repo's trust — `fullsend inference deprovision` does **not** cover `gitlab-oidc`; it only removes GitHub-style per-repo providers. Deleting the `FULLSEND_GCP_WIF_PROVIDER` CI/CD variable in step 2 does not revoke the underlying GCP IAM trust. What to do next depends on which install recipe you used:
   * **Default single-repo, or [multiple specific projects](configuring-gitlab.md#authorizing-multiple-specific-projects-alternative)** (a per-repo `attribute.project_path` principalSet): remove this repo's IAM binding — `gcloud projects remove-iam-policy-binding "$GCP_PROJECT" --role="roles/aiplatform.user" --member="principalSet://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/fullsend-inference/attribute.project_path/$PROJECT_PATH"`. This alone fully revokes `roles/aiplatform.user` for the repo. For hygiene, also drop this repo's clause from the shared `--attribute-condition` (`assertion.project_path == '...'`, and `assertion.project_id == '...'` if present) — or narrow/delete the condition and provider entirely if no other GitLab repo shares them — so a stale CI job elsewhere can't even request a token against the provider.
   * **[Group or group tree](configuring-gitlab.md#authorizing-a-group-or-group-tree-alternative)** (a shared `attribute.namespace_path` principalSet): there is no per-repo IAM member to remove. A single repo cannot be carved out of group-wide trust while sibling repos in the namespace still need it — removing the `attribute.namespace_path/$GROUP_PATH` principalSet or narrowing the `--attribute-condition` revokes Vertex AI access for every project in the namespace, not just the one being uninstalled. To retire trust for just this repo, first migrate the namespace to the [multiple specific projects recipe](configuring-gitlab.md#authorizing-multiple-specific-projects-alternative) (one principalSet per remaining repo), then remove this repo's binding as described above.
   * **Direct `principal://.../subject/<value>` IAM member** (not the documented principalSet path): remove the binding matching whichever value the provider's `google.subject` mapping currently uses — the GitLab numeric project ID if you've migrated to the mapping in [Configuring GitLab § Inference Setup](configuring-gitlab.md#inference-setup), or the OIDC `sub` claim if you haven't migrated yet. If you're not sure which mapping is live, remove both. The documented `principalSet://.../attribute.project_path/...` and `principalSet://.../attribute.namespace_path/...` bindings do not change.

If you manage your own self-hosted mint, run `fullsend mint unenroll "$OWNER/$REPO"` to remove the repo from the mint's allowlist. See the [standalone commands](#standalone-commands) table for details.

## Standalone commands

For organizations that separate GCP and GitHub responsibilities across teams, fullsend provides standalone commands that let each team run only the steps they own:

| Role | Command | What it does |
|------|---------|-------------|
| GCP Admin (Inference) | `fullsend inference provision <org\|owner/repo>` | Create WIF pool/provider and grant Agent Platform access (idempotent — safe to re-run for new orgs) |
| GCP Admin (Inference) | `fullsend inference deprovision <org\|owner/repo>` | Remove org or repo from WIF |
| GCP Admin (Inference) | `fullsend inference status <org\|owner/repo>` | Check WIF health, print config values |
| Repo Maintainer (OpenAI) | `fullsend inference openai request <owner/repo>[,...]` | Generate the provider/mapping request for an OpenAI organization admin (GPT on pi or codex) |
| Repo Maintainer (OpenAI) | `fullsend inference openai import [reply.json]` | Record the admin's reply in `config.yaml`, or set the repository variables |
| Repo Maintainer (OpenAI) | `fullsend inference openai status <owner/repo>` | Check the OpenAI WIF identifiers, and the exchange when run inside Actions |
| Repo Maintainer (Gateway) | `fullsend inference gateway status <owner/repo>` | Check the `inference.gateway` block, its source and its auth mode. In the default `oidc` mode it also checks the gateway's response to the job's OIDC token when run inside Actions. In the `api-key` mode it warns about the long-lived secret, reports whether `FULLSEND_INFERENCE_GATEWAY_API_KEY` is set, and stops at the config checks |
| GitHub Maintainer | `fullsend github setup <owner/repo>` | Configure a GitHub repo (no GCP needed) |
| GitHub Maintainer | `fullsend github set <owner/repo> <key> <value>` | Update a single config value (secret or variable) |
| GCP Admin (Mint) | `fullsend mint deploy` | Deploy the token mint Cloud Function |
| GCP Admin (Mint) | `fullsend mint delete` | Tear down mint infrastructure (inverse of deploy) |
| GCP Admin (Mint) | `fullsend mint add-role <role>` | Register a role PEM and app ID on the mint |
| GCP Admin (Mint) | `fullsend mint remove-role <role>` | Remove a role from the mint (deletes PEM secret by default) |
| GCP Admin (Mint) | `fullsend mint enroll <owner/repo>` | Register a repo in the mint (does not grant Agent Platform access — use `inference provision`) |
| GCP Admin (Mint) | `fullsend mint unenroll <owner/repo>` | Remove a repo from the mint |
| GCP Admin (Mint) | `fullsend mint status` | Inspect mint state and PEM health |

| Platform Admin | `fullsend repos install [repos...]` | Converge repos to desired state: provision new, repair component drift (workflow, thin callers, variables, secrets, pipeline schedules, GitLab poller protected-ref pipeline access, GitLab pipeline-variable override-role inspection — typed jobs require verified `no_one_allowed` before template delivery; legacy upgrades need maintenance-window preparation; legacy variable-based wrappers are exempt), repair scaffold content drift, refresh a declared configuration preset, rewrite a drifted managed `.fullsend/config.yaml`, remove stale vendored assets when a GitHub repo resolves `vendor: false` ([details](../../cli/repos.md#turning-vendoring-off)), upgrade refs |
| Platform Admin | `fullsend repos uninstall <repos...>` | Tear down fullsend from repos and remove from manifest |
| Fleet Admin | `fullsend repos status` | Compare manifest against actual per-repo state: detect missing or drifted components, ref drift, scaffold content drift, declared configuration-preset drift, an existing `.fullsend/config.base.yaml` with no resolved preset, managed `.fullsend/config.yaml` drift and markerless-overlay adoption (both layers are checked before first install too), `vendored-assets` drift (stale assets from an earlier vendored install, see [Turning vendoring off](../../cli/repos.md#turning-vendoring-off)), and GitLab `protected-ref-pipeline` drift |
| Fleet Admin | `fullsend repos set <path> <value>` | Set or remove (`--unset <path>`) any field in the manifest: defaults, forge sections, repository entries, and nested `config` fields. Replaces the removed `repos set-default` |

| Developer | `fullsend agent new <name>` | Generate a complete custom agent and register it |
| Developer | `fullsend agent add <url-or-path>` | Register an agent in config (URL auto-pinned to commit SHA) |
| Developer | `fullsend agent list` | List registered agents and their sources |
| Developer | `fullsend agent set <name>` | Set an agent's runtime, model or effort |
| Developer | `fullsend agent update <name> [sha]` | Re-pin a URL agent or a local harness `base:` URL to a new commit SHA |
| Developer | `fullsend agent remove <name>` | Unregister an agent from config |

The typical handoff for self-managed mints: a GCP admin runs `mint deploy` + `mint enroll` + `inference provision`, then passes the mint URL and WIF provider resource name to a GitHub maintainer who runs `github setup --mint-url=... --inference-wif-provider=...`. For the hosted community mint, enrollment is automatic — install the shared Apps and use the CLI defaults.

### Per-command IAM role breakdown

When using the split-responsibility workflow, each standalone command requires a subset of IAM roles. Use this table to request only what you need.

| IAM Role | `inference provision` | `inference deprovision` | `inference status` | `mint deploy` | `mint delete` | `mint add-role` | `mint remove-role` | `mint enroll` | `mint unenroll` | `mint status` |
|----------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| `roles/iam.workloadIdentityPoolAdmin` | x | x | | x | x | | | x | x | |
| `roles/resourcemanager.projectIamAdmin` | x | | | \* | | | | | | |
| `roles/iam.serviceAccountAdmin` | | | | x | x | | | | | |
| `roles/secretmanager.admin` | | | | \* | x | \*\* | \*\*\* | | | |
| `roles/cloudfunctions.developer` | | | | x | x | | | | | |
| `roles/cloudfunctions.viewer` | | | | | | x | x | x | x | x‡ |
| `roles/run.admin` | | | | x | | x | x | x | x | |
| `roles/iam.workloadIdentityPoolViewer` | | | x† | | | | | | | |
| `roles/secretmanager.viewer` | | | | | | § | | | | x‡ |

\* `roles/resourcemanager.projectIamAdmin` and `roles/secretmanager.admin` are required for `mint deploy` only when using `--pem-dir` (first-time bootstrap). Standard deploys without `--pem-dir` do not need these roles.

\*\* `roles/secretmanager.admin` is required for `mint add-role` when uploading a new PEM (`--pem` or browser mode). When using `--use-existing-pem-secret`, only `roles/secretmanager.viewer` is required (see §).

\*\*\* `roles/secretmanager.admin` is required for `mint remove-role` unless `--keep-pem` is passed (default deletes the PEM secret).

§ `roles/secretmanager.viewer` is required for `mint add-role` when using `--use-existing-pem-secret` (checks that the PEM secret exists).

† All commands that call GCP APIs also require `resourcemanager.projects.get` (typically available via `roles/browser` or any project-level viewer role). This is only notable for `inference status` where it is not covered by the other listed roles.

‡ GCP viewer roles for `mint status` are only required when using `--project` (GCP-based) mode. The API-based mode (`--mint-url` / `FULLSEND_MINT_URL`) requires only valid GitHub credentials and no GCP IAM roles.

Enrollment (org- or repo-scoped) does not grant IAM bindings — Vertex AI access is provisioned separately via `inference provision`.

Required GCP APIs also differ by command group:

```bash
# Inference commands (inference provision/deprovision/status):
gcloud services enable \
  iam.googleapis.com \
  cloudresourcemanager.googleapis.com \
  aiplatform.googleapis.com \
  --project="$GCP_PROJECT"

# Mint commands (mint deploy/enroll/unenroll/status):
gcloud services enable \
  iam.googleapis.com \
  cloudresourcemanager.googleapis.com \
  cloudfunctions.googleapis.com \
  run.googleapis.com \
  secretmanager.googleapis.com \
  iamcredentials.googleapis.com \
  --project="$GCP_PROJECT"
```

> **Note:** `iamcredentials.googleapis.com` is a runtime dependency — the deployed mint Cloud Function uses it for WIF token exchange, not the CLI itself. It must be enabled before `mint deploy`.

## Status notifications

See [Status Notifications](../user/customizing-agents.md#status-notifications) for configuring start/completion comments and reactions.

The composite action accepts five optional inputs for status notifications:

| Input | Description |
|-------|-------------|
| `run-url` | URL of the CI/CD run shown in the status comment |
| `status-repo` | Repository (`owner/repo`) to post status comments on |
| `status-number` | Issue or PR number for status comments |
| `status-comment-id` | ID of the comment that triggered a slash-command run; when set, reactions target that comment instead of the issue/PR |
| `mint-url` | URL of the token mint service used to obtain fresh tokens for posting comments |

All reusable workflows pass these inputs automatically.

### GitLab CI

On GitLab CI, the agent reads status notification context from standard CI/CD environment variables:

| Variable | Description |
|----------|-------------|
| `FULLSEND_FORGE_TOKEN` | Legacy shared credential; no longer selected by runtime or CI routing. Neither `repos install` nor `repos uninstall` removes a leftover copy from a repository installed before the role-only rollout — clean it up manually. The agent job exports `GITLAB_TOKEN` from `FULLSEND_JOB_TOKEN`, resolved by `.gitlab/ci/scripts/select-gitlab-role-token.sh`, which always resolves the registered role credential and fails closed if that secret is missing. See [gitlab-role-credentials.md](../../contributing/gitlab-role-credentials.md). |
| `CI_SERVER_URL` | GitLab instance URL (set automatically by GitLab CI). Fallback when `FULLSEND_GITLAB_URL` and `GITLAB_API_URL` are unset. |
| `CI_COMMIT_SHA` | Commit SHA shown in the status comment. |
| `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | Preferred over `CI_COMMIT_SHA` in merge request pipelines. |
| `CI_PIPELINE_ID` | Used as the run ID for status comment markers. |
| `CI_MERGE_REQUEST_IID` | When set, status comments target the merge request notes API instead of issues. |
| `FULLSEND_GITLAB_URL` | Override for `GITLAB_API_URL` and `CI_SERVER_URL` (e.g., for self-hosted instances). |
| `FULLSEND_NOTE_TARGET` | Set to `merge_requests` to force MR note targeting when `CI_MERGE_REQUEST_IID` is unavailable (e.g., child pipelines, scheduled jobs). |
| `CI_SERVER_TLS_CA_FILE` | GitLab Runner predefined path to a job-local PEM CA bundle when `tls-ca-file` is set. Consumed by poll/agent jobs and the GitLab Go client. See [Private CA](#private-ca-self-hosted-gitlab). |

For a generated scaffold, `repos install` provisions the protected
role-credential secrets (`FULLSEND_GITLAB_POLLER_TOKEN`,
`FULLSEND_GITLAB_ANALYST_TOKEN`, `FULLSEND_GITLAB_CODER_TOKEN`, or a
registered custom role's secret); the agent job exports `GITLAB_TOKEN`
from `FULLSEND_JOB_TOKEN`, resolved by `select-gitlab-role-token.sh` from
the registered role credential — failing closed if that secret is
missing. There is unconditionally no shared-token fallback for status
notifications. If wiring fullsend into GitLab CI manually, provision the
matching role-credential secret instead. See
[Configuring GitLab](configuring-gitlab.md#verifying-the-installation).

## Private CA (self-hosted GitLab)

Self-hosted GitLab instances that terminate TLS with a corporate or private CA need that CA in two **separate** places. A path that exists in the CI job container is not automatically present on a sandbox host.

Fullsend never disables TLS verification (`GIT_SSL_NO_VERIFY`, `curl -k`, or Go `InsecureSkipVerify`). Untrusted certificates are rejected. Installations that do not set a custom CA continue to use the public trust store.

### Job containers (poll and agent)

GitLab Runner injects `CI_SERVER_TLS_CA_FILE` when `tls-ca-file` is set in the runner `config.toml`. That file is a job-local PEM bundle. Generated poll and agent jobs source `.gitlab/ci/scripts/install-fullsend-cli.sh`, which in turn sources `.gitlab/ci/scripts/trust-ci-server-ca.sh` before their first GitLab network operation, and the GitLab Go client also loads the same variable, so curl, git, and `fullsend` all trust the CA without per-tool configuration. Public CAs stay in the pool: the extra PEM is appended, not used as a replacement.

Administrator contract:

1. Install the corporate CA on the **runner** (the process that talks to GitLab and starts jobs), not only on nodes that happen to run other workloads.
2. Point the runner at that bundle with [`tls-ca-file`](https://docs.gitlab.com/runner/configuration/tls-self-signed/) so GitLab Runner both verifies the GitLab server and sets `CI_SERVER_TLS_CA_FILE` in the job.
3. Keep the generated `.gitlab/ci/fullsend-*.yml` templates (re-run `repos install` to converge). Do not unset or override `CI_SERVER_TLS_CA_FILE` as a pipeline variable.

On the Kubernetes executor, `tls-ca-file` is still the contract. How the PEM gets onto the runner (host bind, ConfigMap volume, cluster-wide proxy CA) is an infrastructure choice; OpenShift-specific injection notes live with [#7406](https://github.com/fullsend-ai/fullsend/issues/7406). A volume mount of the CA into the job pod is not a substitute for `tls-ca-file` unless GitLab Runner also sets `CI_SERVER_TLS_CA_FILE`.

If `CI_SERVER_TLS_CA_FILE` is set but the file is missing, unreadable, or not a PEM certificate bundle, the job and the Go client fail with a diagnostic naming that variable. Leave it unset on public-CA instances (including gitlab.com).

Local `fullsend poll` / `fullsend run --forge gitlab` against a private-CA instance can set `CI_SERVER_TLS_CA_FILE` to a readable PEM path; the GitLab client will append it to the system pool.

### Sandbox hosts

OpenShell sandboxes do **not** inherit `CI_SERVER_TLS_CA_FILE`. That path is job-local and must not be treated as available inside the sandbox or on a remote gateway host. Agent-controlled TLS environment overrides (`SSL_CERT_FILE`, `SSL_CERT_DIR`, `CURL_CA_BUNDLE`, `NODE_EXTRA_CA_CERTS`) stay blocked.

The OpenShell supervisor reads a fixed list of system CA paths in the sandbox container to build upstream trust (GitLab, registries, inference) and the bundle handed to sandboxed processes. Administrators provision that trust on the **sandbox host**, independently of the job container:

- **fullsend GitLab Runner VMs** (`hack/gitlab-runner-vm/`): `setup.sh` installs the host CA (`install_ca_certs`) and an OCI `createRuntime` hook (`install_ca_hook`) that copies the host trust bundle into every container rootfs before PID 1 starts. That hook is specific to the Podman custom executor on those VMs. The Kubernetes job executor does not use it.
- **Other sandbox hosts** (including a gateway used by the Kubernetes executor): install the corporate CA in the host trust store (and any equivalent OCI hook or image) so the supervisor can verify GitLab. Do not copy a job-container path into the sandbox configuration.

A successful sandboxed agent run against the private-CA GitLab instance is the end-to-end check: poll jobs reach `/user`, agent jobs reach the GitLab API, and git/curl inside the sandbox reach GitLab through the supervisor's upstream trust.

## See Also

- [Getting Started](../getting-started/) — Standard per-repo installation
- [Advanced setup](../infrastructure/advanced-setup.md) — Alternative installation paths, setup flags, custom app sets
- [Mint service administration](../infrastructure/mint-administration.md) — Deploying and managing the token mint
- [Infrastructure Reference](../infrastructure/infrastructure-reference.md) — Token mint, WIF, and secrets deployment details
- [CLI Internals](../dev/cli-internals.md) — Command structure and implementation details