Gateway ops — deploy, migrate, seed

Live

The hosted gateway at api.tenkeybridge.com went live 2026-07-09 (Fly.io sjc + Neon aws-us-west-2), proven end-to-end against a real QuickBooks Enterprise company file. This page documents the deploy flow that stood it up.

The gateway is a single Node/TypeScript process (Fly.io + Neon Postgres) that terminates the QBO-compatible REST API, brokers the agent-plane WebSocket, and runs the OAuth2 code-flow clone. This page covers standing one up: first deploy, DNS + TLS, database migrations, and seeding the first tenant.

One-time: create the Fly app

From the repo root:

cd apps/gateway
fly launch --no-deploy

This reads apps/gateway/fly.toml (app name tenkeybridge-gateway, region sjc — the nearest live region to the Neon database in aws-us-west-2; Fly deprecated den and sea in mid-2026 — min_machines_running = 1, auto_stop_machines = "off" — the agent's persistent WebSocket connection needs a machine that's always warm) and creates the Fly app without deploying yet.

Secrets

DATABASE_URL is never committed to fly.toml — it's set as a Fly secret so it isn't visible in the build config or flyctl config show:

fly secrets set DATABASE_URL="postgres://<user>:<pass>@<neon-host>/<db>?sslmode=require"

Use the pooled Neon connection string (PgBouncer, port 6543) — the gateway holds one long-lived pg.Pool per process, and Fly may run more than one machine.

Auth secrets (#91 P1)

BETTER_AUTH_SECRET signs session cookies and tokens. Generate 32+ bytes and set it once — rotating it invalidates every session, so treat it as permanent unless it leaks.

fly secrets set --app tenkeybridge-gateway \
  BETTER_AUTH_SECRET="$(openssl rand -base64 32)" \
  RESEND_API_KEY="<your Resend API key>"

RESEND_API_KEY sends the magic-link email; src/config.ts requires both of these (and AUTH_EMAIL_FROM, below) — the gateway refuses to boot without them, the same as it already refuses to boot without DATABASE_URL.

Google and GitHub sign-in are optional — set both halves of a pair or set neither:

fly secrets set --app tenkeybridge-gateway \
  GOOGLE_CLIENT_ID="..." GOOGLE_CLIENT_SECRET="..." \
  GITHUB_CLIENT_ID="..." GITHUB_CLIENT_SECRET="..."

src/config.ts's socialProvider() treats an id with no matching secret (or vice versa) as absent rather than erroring, so a half-configured pair just means the portal doesn't offer that sign-in button — the gateway still boots and the other provider (or magic-link email) still works. Nobody is blocked on setting these up; add them whenever the Google/GitHub OAuth apps exist.

Non-secret values (PORTAL_BASE_URL, AUTH_COOKIE_DOMAIN, AUTH_EMAIL_FROM, GATEWAY_BASE_URL) live in fly.toml [env], not as secrets — they're already there as of #91.

Admin gate secret (#91 final review)

ADMIN_GATE_USER / ADMIN_GATE_PASSWORD close the entire /admin/v1 surface (sign-in, sign-up, org CRUD, the api-key endpoints, and the read endpoints) behind HTTP Basic auth — the human partner's call, made because that surface has no self-serve gate of its own until the P4 portal ships. Both are optional as a pair: unset entirely, src/admin/adminGate.ts skips the gate (this is what keeps local dev, CI, and the test suite green); set only one and loadConfig() throws at boot, since a half-configured gate is a deployment mistake, not a safely-degraded state.

fly secrets set --app tenkeybridge-gateway \
  ADMIN_GATE_USER="..." \
  ADMIN_GATE_PASSWORD="$(openssl rand -base64 24)"

The password is a Fly secret, never a fly.toml [env] entry — that's exactly the mistake this gate exists to prevent one layer up. Remove both once the P4 portal ships its own account-signup/auth flow — leaving them set past that point just adds a second, forgotten front door.

A valid x-api-key bypasses this gate on its own — no -u Basic flag needed — everywhere on /admin/v1 except in front of /admin/v1/auth/* (sign-in, sign-up, OAuth callbacks, magic links), which stays Basic-gated unconditionally: an API key must never be a way into the sign-up surface this gate exists to hide. Once you hold an org-scoped key, every other admin call looks like:

curl -s https://api.tenkeybridge.com/admin/v1/realms?orgId=$ORG_ID \
  -H "x-api-key: $TKB_API_KEY"

See Admin API for the full endpoint list and response shapes.

OAuth provider callback URLs — register these with each provider (Better Auth is mounted at /admin/v1/auth, see src/auth/auth.ts's basePath):

Provider	Authorized redirect URI
Google	https://api.tenkeybridge.com/admin/v1/auth/callback/google
GitHub	https://api.tenkeybridge.com/admin/v1/auth/callback/github

AUTH_COOKIE_DOMAIN=.tenkeybridge.com is what lets portal.tenkeybridge.com read a session cookie set by api.tenkeybridge.com. Leave it unset locally — browsers reject a Domain attribute on localhost.

Running db:migrate or seed from a shell (not on Fly) needs the same required vars in that shell's environment, not just DATABASE_URL — loadConfig() validates the full config eagerly before either command touches the database. For a one-off migration or seed run, a throwaway 32+ byte BETTER_AUTH_SECRET and any non-empty RESEND_API_KEY / AUTH_EMAIL_FROM are enough; neither command sends email or opens a session.

Billing secrets (#92)

Billing (Stripe subscriptions, metered per company file — see the billing guide) is opt-in: src/config.ts's loadBilling() treats STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, and STRIPE_PRICE_PRODUCTION as all-or-nothing. Leave all three unset and the gateway boots exactly as before #92 (no plan enforcement, no /billing/v1 route). Set only one or two and loadConfig() throws at boot — a half-configured Stripe integration is a deployment mistake, the same idiom as the admin gate pair above.

fly secrets set --app tenkeybridge-gateway \
  STRIPE_SECRET_KEY="sk_test_..." \
  STRIPE_WEBHOOK_SECRET="whsec_..." \
  STRIPE_PRICE_PRODUCTION="price_..."

Use Stripe test-mode keys (sk_test_...) until launch. Erik sets the live (sk_live_...) keys when the product actually goes live — do not swap in live keys unprompted.

BILLING_ENFORCE controls whether a non-entitled org's requests are actually blocked:

fly secrets set --app tenkeybridge-gateway BILLING_ENFORCE="false"

• Unset (or empty) defaults to true once the three Stripe vars above are all set — enforcement is on by default, not an opt-in on top of opt-in.
• Only the exact lowercase literals "true" and "false" are recognized. Any other value ("0", "False", "no", …) is silently treated as false — it does not error, and it does not mean "true" — so always set it to exactly one of those two strings.
• BILLING_ENFORCE is ignored (effectively false) whenever billing itself isn't configured — there's nothing to enforce.

Register the webhook endpoint in the Stripe Dashboard (or stripe listen/stripe trigger for local testing) pointed at:

https://api.tenkeybridge.com/billing/v1/webhook

Subscribe it to exactly these five event types — src/billing/webhook.ts only handles these (everything else is a no-op "ignored" branch):

• checkout.session.completed
• customer.subscription.updated
• customer.subscription.deleted
• invoice.payment_failed
• invoice.paid

Copy the endpoint's signing secret (whsec_...) from the Stripe Dashboard into STRIPE_WEBHOOK_SECRET above — that's what stripe.webhooks.constructEvent verifies deliveries against.

Rollout: deploy with BILLING_ENFORCE=false first. In shadow mode every non-entitled request still succeeds (200, not 402), but the entitlement check still runs and logs a warn — { orgId, realmId, reason }, message "billing: organization is not entitled" — for every one it would have blocked. Watch that line in Axiom for a few days to confirm it's only firing for orgs you expect (test/trial accounts, not real customers), then flip:

fly secrets set --app tenkeybridge-gateway BILLING_ENFORCE="true"

Pre-deploy checklist (#91 P1) — completed 2026-08-04

The P1 cutover is done: secrets set, migration rehearsed on a Neon branch, 0002 and seed backfill-org run against production, gateway deployed (v29), and 0003 (#135, org_id NOT NULL) shipped behind it. Kept below as the record of what a schema-plus-data cutover on this gateway involves.

1. Set secrets — BETTER_AUTH_SECRET, RESEND_API_KEY (and ADMIN_GATE_USER/ADMIN_GATE_PASSWORD if the gate isn't already configured) — see §Secrets above.
2. Rehearse the migration on a Neon branch — §Required pre-deploy step, below. Do not skip straight to production.
3. db:migrate against production — §Run migrations against Neon.
4. seed backfill-org against production — §The P1 ownership backfill.
5. Deploy — fly deploy, below.

The ordering mattered because P1's SELECTs (verifyAdminKey, getOAuthClient, verifyClientSecret) name org_id, and the gateway refuses to boot without BETTER_AUTH_SECRET / RESEND_API_KEY — deploying before migrating would have taken the live realm's OAuth surface down, or crash-looped the whole service including the agent WebSocket. The same rule holds for any future migration that adds a column existing code reads: migrate first, deploy second.

For ordinary deploys, fly.toml's release_command = "pnpm db:migrate" (runs after build, before the new image serves traffic) handles migrations on its own; re-running an applied migration is a no-op. It never runs a backfill or any other seed command — those stay by hand.

Deploy

fly deploy

This builds apps/gateway/Dockerfile from the repo root (multi-stage, node:24-slim, pnpm install --prod — the gateway runs via tsx directly, no compile step), runs release_command (pnpm db:migrate, [deploy] in fly.toml) against the production database, and only then rolls the new image out to traffic.

Health & readiness

The gateway exposes two check endpoints, both excluded from the request logging described in Logs below (Fly polls them continuously, and logging every poll would drown real traffic):

• GET /healthz — liveness only: "the HTTP listener is up." Always 200 { ok: true } as long as the process is running and accepting connections. It does not touch the database.
• GET /readyz (#94) — readiness: "the gateway can actually serve traffic." Runs SELECT 1 against the database with a 1.5s timeout and returns 200 { ok: true, db: "ok" } on success or 503 { ok: false, db: "error" } if the query fails or times out. The timeout races the query with a timer via Promise.race — the timer is always cleared and is unref()d, so a hung DB connection can never hang this endpoint or leave a dangling handle behind. On failure only a short, static reason is logged at debug level — the underlying driver error can carry connection-string material, so it's never logged at info/warn or included in the response body.

Sustained outage: what /readyz does not cover

The 1.5s timeout guarantees the response to the poller never hangs, but it only gives up client-side — it does not cancel the in-flight SELECT 1 or the connection attempt underneath it. During a genuine sustained Postgres outage, each 30s poll that lands on a hung query or a stuck connection attempt can leave that query/connection abandoned rather than freed, and over enough polls this can pressure the gateway's own connection pool (the same pool real REST traffic uses) even though every individual /readyz response still comes back in ~1.5s. A per-query statement_timeout was investigated as a fix (#94 review) but isn't a cheap addition here: the gateway's Db handle is deliberately driver-agnostic (real Postgres in production, PGlite in tests, both behind the same execute() call — see src/store/db.ts), and PGlite's driver neither supports the multi-statement SET LOCAL statement_timeout; SELECT 1 trick node-postgres allows nor actually enforces statement_timeout cancellation when driven the way that would require. Making it real-Postgres-only would mean a second, untested code path solely for production. Follow-up, not yet built: a small, separate connection (or 1-connection pool) dedicated to /readyz, with its own connectionTimeoutMillis/statement_timeout set at construction time. That configuration would live only on this separate connection — the shared app pool real REST/OAuth traffic uses stays untouched — so a hung readiness check could never accumulate against production traffic's own connections either.

fly.toml wires both into [http_service.checks]:

[[http_service.checks]]
  method = "get"
  path = "/healthz"
  interval = "15s"
  timeout = "2s"

[[http_service.checks]]
  method = "get"
  path = "/readyz"
  interval = "30s"
  timeout = "3s"
  grace_period = "10s"

/readyz is checked less often and given a longer timeout and a startup grace period than /healthz — a DB blip shouldn't trip a restart as eagerly as the process being fully unresponsive would.

Paired with the checks is a restart policy:

[[restart]]
  policy = "always"
  max_retries = 10

A machine that fails its checks (crashes, or reports not-ready repeatedly) is restarted automatically rather than left down until someone notices fly logs or an alert. max_retries = 10 caps the restart loop so a consistently-broken deploy (e.g. bad DATABASE_URL) fails loud instead of churning forever.

Single-machine constraint

The gateway currently runs as exactly one Fly machine (min_machines_running = 1, auto_stop_machines = "off"), and that is a hard constraint, not a cost-saving default: AgentHub (apps/gateway/src/hub/agentHub.ts) keeps every connected edge agent's WebSocket in an in-memory map, keyed by realm, that exists only on the machine that accepted that agent's upgrade. If a second machine were added, a REST request for a realm whose agent dialed into the other machine would 503 AGENT_OFFLINE even though that agent is online — just on the wrong box. So min_machines_running must never be raised above 1 as it stands today. (A restart under the policy above is safe — it replaces the one machine, it doesn't add a second one alongside it.)

Upgrade path, if the gateway ever needs to scale past one machine:

1. fly-replay by realm — keep a realm→machine-id lookup (e.g. in Postgres or Fly's own machine metadata) and have any machine that receives a REST request for a realm it doesn't hold an agent socket for respond with a Fly-Replay header pointing at the machine that does. Fly re-routes the request there. Lowest-effort option; keeps AgentHub's in-memory design as-is.
2. A shared broker — move agent socket state (or the requests/responses themselves) out of process into something all machines can reach, e.g. Redis pub/sub or a similar message bus, so any machine can serve any realm regardless of which one holds the actual WebSocket. More work, but removes the single-machine constraint entirely rather than routing around it.

Neither is built. This section exists so a future scale-up doesn't rediscover the constraint the hard way by silently raising min_machines_running and shipping intermittent AGENT_OFFLINEs.

DNS + TLS

Point api.tenkeybridge.com at the Fly app with a CNAME in Vercel DNS:

api.tenkeybridge.com.  CNAME  tenkeybridge-gateway.fly.dev.

Then request the certificate on the Fly side:

fly certs add api.tenkeybridge.com
fly certs show api.tenkeybridge.com   # poll until status is "Ready"

Fly's edge proxy forwards the WebSocket upgrade for /agent over the same internal_port as the REST traffic — no separate listener or extra Fly config is needed for the agent plane.

Run migrations against Neon

Migrations are drizzle-kit generated SQL, checked into apps/gateway/drizzle/. Apply them with the migrate CLI, pointed at the real (non-pooled, for DDL) Neon URL:

DATABASE_URL="postgres://<user>:<pass>@<neon-host>/<db>?sslmode=require" \
  pnpm --filter @tenkeybridge/gateway db:migrate

To regenerate migrations after a schema change:

cd apps/gateway && pnpm db:generate

schema.ts (Drizzle table definitions) and applySchema in src/store/db.ts (hand-written idempotent DDL used by the test suite's PGlite instances) are the same schema expressed twice — keep them in agreement and re-run the gateway test suite after any change to either.

Required pre-deploy step: rehearse on a Neon branch first

db:migrate runs drizzle-orm/node-postgres/migrator, which wraps every pending migration in a single transaction. That path has only ever been exercised against PGlite in tests — PGlite doesn't go through the real Postgres migrator, so nothing in the test suite proves this migrator works against real Postgres. Before running db:migrate against production, branch Neon's production database (§Staging, below), point DATABASE_URL at the branch, and run the full sequence — db:migrate then seed backfill-org — there first. Confirm the branch ends with organization/user/member tables present, realms.org_id/oauth_clients.org_id populated for the existing rows, and org_id still nullable on both, then tear the branch down. Do not treat this as optional or skip straight to production — it is the only place the production data path gets exercised before it runs for real.

The gateway test suite cannot be pointed at the branch — every test config hardcodes pglite://memory, which is the whole reason this rehearsal exists. Run pnpm --filter @tenkeybridge/gateway test as a normal regression check, but to exercise the migrated database, boot the gateway itself against the branch (DATABASE_URL=<branch url> pnpm --filter @tenkeybridge/gateway dev) and drive the OAuth flow end to end: POST /oauth2/v1/authorize → POST /oauth2/v1/tokens → an authenticated /v3/company/... call. That covers getOAuthClient and verifyClientSecret — the two SELECTs this branch changed to name org_id. A 503 AGENT_OFFLINE on the REST call is the expected pass: it means auth cleared and only the agent is absent.

Rehearsed 2026-08-03 on branch p1-migrator-rehearsal (a copy of production: 1 realm, 3 oauth_clients). 0002 applied cleanly in one transaction, adding Better Auth's eight tables and a nullable org_id; backfill-org attached 1 realm + 3 clients and was a verified no-op on a second run. Credential columns (admin_key_hash — since dropped by 0006 — secret_hash, redirect_uris) hashed identically to production before and after, and the full OAuth flow succeeded against the migrated branch. Branch torn down.

One-time: the P1 ownership backfill (#91)

realms and oauth_clients gained an owning organization. 0002 adds Better Auth's eight tables plus a nullable org_id column on both — it does not, and as shipped on this branch cannot, make org_id NOT NULL in the same migration run. drizzle-orm's Postgres migrator applies all pending migrations in one transaction (node_modules/drizzle-orm/pg-core/dialect.js), so a SET NOT NULL migrated in the same batch as the ADD COLUMN would fail on production's existing ownerless row and roll the ADD COLUMN back with it — leaving no org_id column at all and an unrunnable backfill. Tightening org_id to NOT NULL was therefore deferred to its own later migration and deploy, tracked in #135.

Both halves are now done. This section is kept as the record of a sequence that is finished, not a runbook to re-run: the backfill ran against production on 2026-08-04 (1 realm, 3 oauth_clients, 0 rows left NULL), and 0003 — the two-statement SET NOT NULL — shipped after it. A fresh database gets both from db:migrate in order and needs no backfill at all, because every write path has required org_id since P1.

So there were two steps here, not three, and no schema change happened between them:

export NEON_URL="postgres://...neon.tech/tenkeybridge?sslmode=require"   # DIRECT url, not pooled

# 1. Better Auth's eight tables + a nullable org_id on realms/oauth_clients
DATABASE_URL=$NEON_URL pnpm --filter @tenkeybridge/gateway db:migrate

# 2. Create the owning org + owner account and attach every ownerless row.
#    Idempotent: safe to re-run.
DATABASE_URL=$NEON_URL pnpm --filter @tenkeybridge/gateway seed backfill-org \
  --org-name "ExampleCo" --org-slug exampleco \
  --owner-email owner@exampleco.example --owner-name "Pat Owner"

The backfill (src/store/backfill.ts) never recreates a realm and never touches admin_key_hash (dropped by 0006), secret_hash, or redirect_uris — the credentials that work before it work after it. It matches the org on slug and the user on email, and only attaches rows where org_id is still NULL, so a realm that already belongs to someone is never moved — re-running it after a partial failure, or after it has already succeeded, is a no-op on anything already attached.

The owner account it creates has a verified email and no linked provider. Signing in with Google or GitHub at that same address links to it rather than forking a second user, so the first portal login lands on the org that owns the existing realm.

Seed the first tenant

Use the seed CLI (apps/gateway/src/seed.ts, exposed as pnpm gateway:seed from the repo root — there is no root-level pnpm seed, that name only exists as a package script inside apps/gateway) against the same DATABASE_URL.

Realms and OAuth clients are owned by an organization as of #91 P1 — create-realm and create-client both now require --org <orgId>. If no organization exists yet, seed backfill-org (above) creates one and prints its orgId; otherwise look the id up in the organization table.

# 1. Create a realm (tenant) under an org — prints realmId
DATABASE_URL=$NEON_URL pnpm --filter @tenkeybridge/gateway seed create-realm \
  --name "Acme Co" --org "$ORG_ID"

# 2. Issue an agent token for that realm — the edge agent's appsettings.json needs this
DATABASE_URL=$NEON_URL pnpm --filter @tenkeybridge/gateway seed issue-agent-token --realm $REALM_ID

# 3. Register an OAuth client for whatever app will call the REST API, also org-owned
DATABASE_URL=$NEON_URL pnpm --filter @tenkeybridge/gateway seed create-client \
  --name "My App" --org "$ORG_ID" --redirect https://myapp.example.com/oauth/callback

Each command prints its secret (agentToken, client_secret) once — store it immediately, it is not retrievable later (only the hash is persisted).

create-realm no longer takes or prints an admin key — the adminKey/consent-form credential (P1) was fully retired in #91 P2/P3. There is nothing to pass and nothing to store beyond realmId.

gateway:seed is now the internal / break-glass path only — the self-serve Admin API (/admin/v1, P2) does everything above (create a realm, issue an agent token, register a client) for a normal customer, gated by org membership instead of shell access to this database. Use seed when you need to provision something before an organization/owner account exists to call the admin API with, or when debugging directly against the database. seed backfill-org remains the only way to create an organization from scratch until the portal (P4) ships its own signup flow.

Run the OAuth2 code-flow (authorize with realm_id → consent → redirect with code → token exchange) against /oauth2/v1/authorize and /oauth2/v1/tokens to get an ACCESS_TOKEN scoped to that realm — see Authentication.

Milestone proof

With an edge agent connected for the realm and a valid access token, this is the end-to-end proof the gateway is live and bridging real QuickBooks data:

curl -s -H "Authorization: Bearer $ACCESS_TOKEN" \
  https://api.tenkeybridge.com/v3/company/$REALM_ID/customer/$CUSTOMER_ID | jq .

Logs

Every HTTP request — REST or OAuth, success or failure, even an early auth-middleware rejection — produces exactly one structured pino line (/healthz and /readyz are deliberately excluded: Fly polls both continuously and the noise would drown real traffic — see Health & readiness). This is what "check the Fly logs for that request" should actually show:

{"level":40,"method":"GET","path":"/v3/company/:realmId/:entity/:id","status":404,"durationMs":3,"realmId":"4562...","entity":"widget","msg":"gateway http request"}

{"level":40,"method":"GET","path":"/v3/company/:realmId/query","status":400,"durationMs":2,"realmId":"4562...","query":"SELECT * FROM Customer MAXRESULTS -5","errorCode":"UNSUPPORTED_QUERY","msg":"gateway http request"}

{"level":30,"method":"POST","path":"/oauth2/v1/tokens","status":200,"durationMs":41,"clientId":"tkbcl_...","grantType":"refresh_token","msg":"gateway http request"}

Fields: method, path (the route template, e.g. :entity/:id, not the raw URL — so log lines group cleanly), status, durationMs, and realmId/entity when the route carries them. A REST query request additionally logs the raw query text (the thing you need to diagnose an UNSUPPORTED_QUERY 422/400 without guessing), and any 4xx/5xx logs an errorCode pulled from the response body's fault/error code. OAuth requests log grantType/responseType/clientId when known. Level is info for 2xx/3xx, warn for 4xx, error for 5xx.

Hard constraint: never a request/response body, an Authorization header, a token, an auth code, or a client secret — fly logs is safe to paste into an issue.

Durable sink: Axiom

fly logs is lossy — Fly's NATS-based log pipeline silently drops lines under normal operation (observed live, #53), and keeps no history. The source of truth is Axiom: when the AXIOM_TOKEN + AXIOM_DATASET secrets are set, the gateway ships every log line directly to Axiom over HTTPS (via @axiomhq/pino), bypassing Fly's pipeline. stdout stays wired in parallel, so fly logs still works for casual live-tailing — just never treat a missing line there as evidence the request didn't happen; query Axiom instead.

• Where: app.axiom.co, org empire-innovations, dataset tkb-gateway (free tier: 30-day retention, 500 GB/mo — orders of magnitude above gateway volume).

• Setup: create an ingest-only API token scoped to the dataset (Axiom → Settings → API tokens), then:

  fly secrets import -a tenkeybridge-gateway
  # paste, then Ctrl-D:
  #   AXIOM_TOKEN=xaat-...
  #   AXIOM_DATASET=tkb-gateway

• Rotation: revoke the token in Axiom, create a new one, re-run the import. With either var unset the gateway logs to stdout only (local dev, tests, and CI never touch Axiom).

• Querying: the msg field is gateway http request; filter on path, status, realmId, or errorCode. Example APL:

  ['tkb-gateway'] | where path contains "companyinfo" | sort by _time desc

Alerting

Five Axiom monitors are specified — APL, thresholds, and an email notifier to erik@empireinnovators.com — covering 5xx rate, agent-offline spikes, unhandled errors, gateway silence, and rate-limit storms. Full spec, exact queries, field-name verification against the pino call sites, and the MCP calls to create them: apps/gateway/ops/axiom-monitors.md (repo path, not a published docs page). Not yet created — creation is blocked on Axiom MCP re-authentication; that file has a "How to create" section a follow-up session can run verbatim once /mcp is re-authed.

Staging

Staging E2E runs against a Neon branch database seeded with throwaway tenant data — not a second Fly app. Branch Neon's main database, point a local or preview gateway process at the branch's connection string via DATABASE_URL, run migrations and seed as above, and tear the branch down when done.

Backups & restore

Neon backup posture (point-in-time recovery)

Neon protects the database continuously via PITR (it retains the write-ahead log for a window, not periodic snapshots) — any point inside that window is restorable, not just a nightly checkpoint. The gateway project's configured retention window is UNVERIFIED as of 2026-08-28 — this session had no signed-in Neon console session and no neonctl/API key available on the build machine, so the actual value was not read. Check it before relying on it: Neon console → project tenkeybridge → Settings → History retention (neon projects get once authenticated shows the same value).

Neon's plan defaults, at time of writing:

Plan	History retention
Free	6 hours
Launch	7 days

If the actual configured value is under 24 hours, upgrading the project to Launch is a launch prerequisite — a bad migration or an unqualified DELETE discovered the next morning would otherwise be unrecoverable. This is Erik's call, not an autonomous purchase — do not upgrade the plan without his sign-off; just report what the console shows.

Point-in-time branch restore

The normal way to inspect, or recover, a past state without touching production: branch the database as of a timestamp.

Console: project tenkeybridge → Branches → Create branch → parent main → Time → pick the point in time → Create. Connect with psql using the branch's own connection string from the console.

neon (binary neon; neonctl is an alias for it — either name runs the same commands):

neon branches create --project-id <project-id> \
  --name drill-2026-08-28 --parent 2026-08-28T12:00:00Z

--parent accepts a branch name/id, a timestamp, or an LSN — a bare RFC 3339 timestamp defaults to branching off main — see neon branches create --help if this syntax has shifted since.

Then, against the branch's own connection string (never the production one):

SELECT count(*) FROM realms;
SELECT max(created_at) FROM "session";

Both should come back sane — a realm count matching production, a session row timestamped near the point you picked. If either errors or looks wrong, the chosen point predates a write you needed. Tear the branch down when done:

neon branches delete --project-id <project-id> <branch-id>

See apps/gateway/ops/restore-drill.md for the dated record of drills run against this recipe.

Manual dump/restore (belt-and-suspenders, off-Neon)

PITR covers "restore to a point in time" as long as the Neon project itself exists; a manual dump is the fallback if the project is ever lost, misconfigured, or you want an offline copy outside Neon entirely.

pg_dump "$NEON_URL" --no-owner --format=custom -f tkb-$(date -u +%F).dump

Restore into a fresh Neon branch — never into main — to inspect or verify before doing anything destructive with it:

pg_restore --no-owner -d "$BRANCH_URL" tkb-2026-08-28.dump

Use the direct (non-pooled) Neon URL for both, same as migrations (§Run migrations against Neon). pg_dump/pg_restore are not installed on the Mac by default — brew install libpq gets a matching client (it's keg-only, so add it to PATH), or run the dump from the Fly gateway machine (fly ssh console -a tenkeybridge-gateway) instead — the runtime image doesn't ship Postgres client tools either, so that's the same install either place.

Where dumps live: Erik's encrypted local disk, and nowhere else. Never commit a dump to the repo, attach one to a GitHub issue/PR/Actions artifact, or upload it anywhere off Erik's own machine — it's a full copy of every tenant's data.

Uptime & status

Erik to-do — nothing below has been created yet. This session's permission policy blocked Better Stack account/monitor creation, so this section is the exact spec to execute, not a record of what exists.

Create a Better Stack account (free tier) under erik@empireinnovators.com if one doesn't already exist, then add these HTTP monitors:

Monitor	URL	Notes
Gateway liveness	https://api.tenkeybridge.com/healthz
Gateway readiness	https://api.tenkeybridge.com/readyz	add only after this PR (#94) deploys — the route doesn't exist in production before then
Docs site	https://docs.tenkeybridge.com/
Marketing site	https://tenkeybridge.com/

• Interval: the free-tier minimum (30s at time of writing — use whatever the plan actually offers if that's changed since).
• Alerts: both email and phone (SMS/call) to Erik on every monitor — the free tier supports both.
• Status page: a public status page covering all four monitors. The default Better Stack subdomain is fine to start. A custom status.tenkeybridge.com is optional — if set up, add the CNAME Better Stack provides as a Vercel DNS record (same pattern as the api CNAME in §DNS + TLS above) and note it here once done.

Status page URL: PENDING (Erik) Monitor ids: PENDING (Erik)

Ops checklist

• [ ] Monthly restore drill — run the point-in-time branch restore above (or follow the recipe in apps/gateway/ops/restore-drill.md directly), verify both queries look sane, delete the branch, and record the result as a new dated entry in apps/gateway/ops/restore-drill.md.

Updating legal pages

The public legal pages (Terms, Privacy, DPA, Subprocessors) live in apps/marketing, driven by apps/marketing/lib/legal.ts. To publish a change:

1. Bump LEGAL.effectiveDate in apps/marketing/lib/legal.ts.
2. Add a dated entry to apps/marketing/content/legal/CHANGELOG.md (newest first).
3. Redeploy marketing: vercel deploy --prod --cwd apps/marketing.

The trademark notice and legal-page links are also hardcoded in apps/gateway/src/portal/layout.ts and apps/docs/.vitepress/config.ts — if the wording itself changes, update those two alongside legal.ts.