Deployments¶
Per-service mechanics, names and status commands. How code reaches production in general (the stage branch, the
v* release, the approval gate, rollback) is on Branching and Releases; read that first.
Wherever this page says "the repo's production reviewers", it means the members of that repo's production GitHub
environment, who are listed there.
Core API¶
How It Deploys¶
Core API has two deploy paths:
- Stage: push to the
stagebranch. Pushing tomaindeploys nothing. - Production: publish a GitHub Release on a
v*tag onmain(see Production below).
Stage:
- Quality checks runs on the
stagebranch (lint, typecheck, tests) - On success, Deploy to stage triggers (a
workflow_runof Quality checks, filtered tostage): - Authenticates to AWS via OIDC (role:
gh-oidc-deploy-core-api) - Builds the Docker image and checks PDF rendering inside it
- Tags with both the commit SHA and
latest - Pushes to ECR (
258618559895.dkr.ecr.eu-central-1.amazonaws.com/core-api) - Points the service at the newest task-definition revision and forces a new deployment on
stage-euc1-core-ecs-cluster. The stage task definition is not SHA-pinned: the running image is:latest(IS-135).
flowchart LR
A[Push to stage] --> B[Quality checks]
B -->|pass| C[Deploy to stage]
C --> D[Build image]
D --> E[Push to ECR]
E --> F[ECS redeploy]
As with every workflow_run deploy, the run is filed under main in the Actions UI, not under stage; see
Where did my stage deploy go?.
Infrastructure¶
| Component | Value |
|---|---|
| ECR repo | core-api |
| ECS cluster | stage-euc1-core-ecs-cluster |
| ECS service | core-api |
| Task definition | stage-euc1-core-core-api |
| Resources | 512 CPU / 512 MB memory |
| Port | 3000 |
| Health check | GET /health |
| URL | https://core-api.stage.cookiehub.net |
| Logs | CloudWatch /ecs/stage-euc1-core-core-api |
Checking Deployment Status¶
# Watch the GitHub Actions run
gh run list --repo cookiehub-com/core-api --limit 5
# Check ECS service status
aws ecs describe-services \
--cluster stage-euc1-core-ecs-cluster \
--services core-api \
--query 'services[0].{desired:desiredCount,running:runningCount,status:status}' \
--region eu-central-1
# Check which image is running
aws ecs describe-task-definition \
--task-definition stage-euc1-core-core-api \
--query 'taskDefinition.containerDefinitions[0].image' \
--region eu-central-1
Viewing Logs¶
# Tail recent logs
aws logs tail /ecs/stage-euc1-core-core-api \
--follow \
--region eu-central-1
# Search for errors in the last hour
aws logs filter-log-events \
--log-group-name /ecs/stage-euc1-core-core-api \
--start-time $(date -v-1H +%s000) \
--filter-pattern "error" \
--region eu-central-1
Rollback¶
Stage is disposable: the simplest rollback is to force-push the known-good commit to stage. For production
see Production below. To roll stage back to a previous image without a rebuild:
# 1. Find the commit SHA of the version you want
gh run list --repo cookiehub-com/core-api --limit 10
# 2. Re-tag that image as latest
ROLLBACK_SHA="<commit-sha>"
REPO="258618559895.dkr.ecr.eu-central-1.amazonaws.com/core-api"
aws ecr get-login-password --region eu-central-1 | \
docker login --username AWS --password-stdin 258618559895.dkr.ecr.eu-central-1.amazonaws.com
docker pull $REPO:$ROLLBACK_SHA
docker tag $REPO:$ROLLBACK_SHA $REPO:latest
docker push $REPO:latest
# 3. Force new deployment
aws ecs update-service \
--cluster stage-euc1-core-ecs-cluster \
--service core-api \
--force-new-deployment \
--region eu-central-1
Manual Deploy¶
If CI is broken and you need to deploy manually:
# Authenticate to ECR
aws ecr get-login-password --region eu-central-1 | \
docker login --username AWS --password-stdin 258618559895.dkr.ecr.eu-central-1.amazonaws.com
# Build and push
docker build -t core-api:latest .
docker tag core-api:latest 258618559895.dkr.ecr.eu-central-1.amazonaws.com/core-api:latest
docker push 258618559895.dkr.ecr.eu-central-1.amazonaws.com/core-api:latest
# Trigger redeploy
aws ecs update-service \
--cluster stage-euc1-core-ecs-cluster \
--service core-api \
--force-new-deployment \
--region eu-central-1
Production¶
deploy-prod.yml runs on release: published and declares environment: production. It waits for a reviewer to
approve (the ancestor guard has not run yet at that point), then checks out the tag, verifies the tag is an ancestor of
main and builds the image, pushes :<sha>, :latest and
:<tag>, and registers a new revision of prod-euc1-core-core-api pinned to the commit SHA
(render-task-definition plus deploy-task-definition), then waits for the service to be stable. The production
cluster is prod-euc1-core-ecs-cluster, the role comes from the AWS_DEPLOY_ROLE_ARN_PROD secret.
core-api is not live in production
The production ECS service is held at 0 tasks until go-live (IS-92, CH-96); the live core API is still the
Laravel app. A release registers a revision and updates the service, but no task starts and no traffic is served.
Both repos are pre-production: main needs 0 approvals until the first production release (see
Branching and Releases).
Rollback is the ECS service procedure: re-run the earlier release's run from
the Actions UI (a rebuild; works for 30 days after the original run, after which cut a new patch release from a revert
on main), or point the service at an earlier revision (break-glass until IS-135 proves it).
Required Access¶
- GitHub: Write access to
cookiehub-com/core-api(to push tostageand to create releases) - Production deploys: membership of the repo's
productionenvironment reviewers (and not being the person who cut the release) - AWS: IAM user in the stage account with ECS/ECR permissions (only needed for rollback or manual deploy)
Core (Laravel Dashboard)¶
How It Deploys¶
Core deploys to stage on push to stage branch via SSM:
- Quality checks workflow runs (Pint, PHPStan, Pest, asset build)
- On success, Deploy to stage workflow triggers:
- Authenticates to AWS via OIDC (role:
gh-oidc-deploy-core) - Sends SSM command to the dash EC2 instance
- SSM runs:
git stash, fetch andreset --hard origin/stage,composer install,npm ci && npm run build,artisan migrate --force,config:clear,route:clear,view:clear, then an Apache reload - Waits for completion, then health checks the site
flowchart LR
A[Push to stage] --> B[Quality checks]
B -->|pass| C[Deploy to stage]
C --> D[SSM SendCommand]
D --> E[git pull + build]
E --> F[Health check]
Infrastructure¶
| Component | Value |
|---|---|
| Server | stage-euc1-core-dash (t3.small, Ubuntu 24.04) |
| Deploy via | AWS SSM SendCommand |
| URL | https://dash.stage.cookiehub.net |
| Runtime | PHP 8.3, Node 18, Apache |
| Code path | /home/cookiehub/dash |
Rollback¶
# SSM into the instance
aws ssm start-session --target <instance-id> --region eu-central-1
# Revert to previous commit
cd /home/cookiehub/dash
git log --oneline -5 # find the good commit
git checkout <commit-sha>
composer install --no-dev --optimize-autoloader
php artisan config:clear && php artisan route:clear
sudo systemctl reload apache2
Production¶
Core ships by a v* tag on main: publish a GitHub Release and deploy-prod.yml runs. There is no prod branch and
no version file; the tag is the version (CH-831), and the deploy writes it to storage/app/VERSION.
- It waits at the
productionenvironment gate for one of the repo's production reviewers (never the person who cut the release). The ancestor guard has not run yet when they approve. - The workflow verifies the tag is an ancestor of
main. - It assumes the role in
AWS_DEPLOY_ROLE_ARN_PROD(the role trusts both the tag-ref and theenvironment:productionsubject) and sends an SSM command to the production EC2 box. - On the box, as the
cookiehubuser, in the existing checkout at/home/cookiehub/core:git stash,git fetch --tags,git checkoutandgit reset --hardto the tag, thencomposer install,npm ci,npm run build,php artisan migrate --force,config:clear,route:clear,view:clear, a Bugsnag deploy marker, and an Apache reload. - It health-checks
https://dash.cookiehub.comand posts to the prod deploys Teams channel.
config:clear, never config:cache
About 160 env() calls in application code return null under a config cache. Neither the deploy nor any manual
step on the box may run config:cache (or route:cache).
The update is in place by design: public/ on the box holds per-domain widget bundles that exist nowhere else and are
gitignored so reset --hard leaves them alone. Do not replace the deploy with a clean checkout or an artifact
download. The health check only proves Cloudflare answers; it is not proof the dashboard works, so verify the release
itself. migrate --force runs forward only, which is why a rollback is not a full restore (see
Rolling back core).
Rebuilding cs150p (production)¶
Production Core runs on a single hand-managed EC2 box (cs150p), not in Terraform. Its deploy
(deploy-prod.yml) updates the checkout in place with git reset --hard and deliberately leaves
gitignored paths under public/ untouched, because several release-time artifacts exist only on
that disk. A from-scratch rebuild of the box must restore them:
resources/dist/<ver>/— every 2.9+ widget version's build, written at release time.CodeCompile29inlines it into every domain's compiled bundle and silently skips a missing version, so without this a rebuilt box compiles bundles that are quietly missing widget code.public/c2/css/<ver>.cssand<ver>-ng.css— the shared versioned stylesheets, also written at release time.public/c2/css/additionally holds per-domain custom CSS, which is a second, unrelated thing under the same path — don't assume everything there is version-keyed.public/cdn/<cdn>/— per-domain custom-CDN bundles.
Restoring dist and the shared stylesheets: for each widget version still in use, re-running the
release registration (POST /api/widget/v1/releases) re-syncs both from S3. Under the immutable
builds rule this may refuse to re-register a version already released — if so, copy the files
directly from S3 instead: widget-releases/<ver>/ for resources/dist/<ver>/, and the CDN bucket's
c2/css/ prefix for the stylesheets.
Restoring custom-CDN bundles: run widget:republish-current, or publish the affected domains
individually.
Everything else gitignored under public/ (c2-preview/, client/<code>/<lang>.json) is
self-healing: each domain rewrites them on its next settings save, and (while
CORE_API_DOMAINS_ENABLED is off, as in prod) regenerates them on its next Settings-page visit.
That regeneration marks the domain unpublished, so expect a wave of spurious "pending changes"
banners after a rebuild; a save or publish clears each one. Nothing to restore by hand.
See Immutable builds for why re-registering an already-released version is refused.
Required Access¶
- GitHub: Write access to
cookiehub-com/core(to push tostageand to create releases) - AWS: IAM user with SSM access in the stage account (only for stage rollback)
- VPN: Required for direct SSH access as alternative
Admin Portal¶
How It Deploys¶
One workflow, Deploy (.github/workflows/deploy.yml in cookiehub-com/admin). The trigger picks the environment:
- Stage: push to
main. Admin has nostagebranch; this is legacy rather than a decision, and moving it to the stage-branch model is tracked in IS-133. It also means admin's stage is not disposable the way other repos' is. - Production: publish a GitHub Release on a
v*tag (release: published). Thedeploy-productionjob declaresenvironment: production, so it waits for one of the repo's production reviewers; a baregit pushof a tag triggers nothing. Admin differs from the other repos: its ancestor guard, and its image push to the production ECR, run inbuild-pushbefore the gate.
flowchart LR
A[Push to main] --> T[stage settings]
R[Publish v* release] --> P[production settings]
T --> B[Build image]
P --> B
B --> W[Prod only: ancestor guard<br/>in build-push]
W --> E[Push to ECR<br/>sha, latest, tag]
E --> G[Prod only: production gate]
E --> D[Read live task definition]
G --> D
D --> S[Swap image, register revision]
S --> U[Update service, wait for stable]
Each environment has its own deploy role. The stage role trusts refs/heads/main only, so the stage path deliberately
has no environment: (adding one would change the OIDC subject and break stage). The production role trusts both the
tag ref and environment:production, which is what lets the production job carry the gate (infra#45). That is why
the roll is two jobs and the settings are prefixed variables rather than one job with a dynamic environment.
Every deploy registers a new task definition revision from the live one, changing only the image. Environment,
secrets, ports and logging stay as Terraform set them — and Terraform must not revert the image on its next apply, so
the service tracks the newest ACTIVE task-definition revision with the max() pattern (not ignore_changes).
Infrastructure¶
| Stage | Production | |
|---|---|---|
| AWS account | 258618559895 |
759286286879 |
| ECR repo | admin-portal |
admin-portal |
| ECS cluster | stage-euc1-core-ecs-cluster |
prod-euc1-core-ecs-cluster |
| ECS service | admin-portal |
admin-portal |
| Task definition | stage-euc1-core-admin-portal |
prod-euc1-core-admin-portal |
| Container | admin-portal |
admin-portal |
| OIDC role | gh-oidc-deploy-admin-portal |
gh-oidc-deploy-admin-portal |
| Role trusts | refs/heads/main |
refs/tags/v*, environment:production |
| Port | 3000 | 3000 |
| Health check | GET /api/health |
GET /api/health |
| SSM params | /core/stage/admin-portal/* |
/core/prod/admin-portal/* |
| Image tags | <sha>, latest |
<sha>, latest, <tag> |
Repository Settings¶
The workflow reads one set of GitHub repository variables per environment. None of them is a secret — a role ARN
and a repository name identify things, and the role's trust policy is what guards it. <ENV> is STAGE or PROD.
| Variable | Stage | Production |
|---|---|---|
<ENV>_AWS_ROLE_ARN |
arn:aws:iam::258618559895:role/gh-oidc-deploy-admin-portal |
arn:aws:iam::759286286879:role/gh-oidc-deploy-admin-portal |
<ENV>_ECR_REPOSITORY |
admin-portal (the ECR URL is accepted too) |
admin-portal |
<ENV>_ECS_CLUSTER |
stage-euc1-core-ecs-cluster |
prod-euc1-core-ecs-cluster |
<ENV>_ECS_SERVICE |
admin-portal |
admin-portal |
<ENV>_ECS_TASK_FAMILY |
stage-euc1-core-admin-portal |
prod-euc1-core-admin-portal |
Optional: <ENV>_ECS_CONTAINER_NAME (default admin-portal) and <ENV>_AWS_REGION (default eu-central-1). The
role ARN and the ECR URL are the infra outputs admin_portal_gh_deploy_role_arn and
admin_portal_ecr_repository_url. A missing required variable fails the run with a message naming it, after the image
has been pushed — so a tag pushed before the PROD_… set exists is a red run, not a silent no-op.
Deploying to Production¶
- Make sure the commit has been through stage; it deploys there on merge to
main. - Follow Cutting a release: version bump merged, release note written,
then publish the release from
main:
gh release create v1.2.0 --repo cookiehub-com/admin --target <sha of main> --title v1.2.0 --notes-file release-note.md
- A reviewer approves the
productiondeployment, then the service rolls. Watch it:gh run list --repo cookiehub-com/admin --limit 3
Admin is the one repo whose workflow also has workflow_dispatch: gh workflow run deploy.yml --repo cookiehub-com/admin --ref main
redeploys stage, and --ref v1.2.0 re-runs a production release (still behind the gate). Unlike a UI re-run, this does not depend on the
original run being under 30 days old.
Checking Deployment Status¶
# Watch the GitHub Actions run
gh run list --repo cookiehub-com/admin --limit 5
# Stage: service state and the revision it is on
aws ecs describe-services \
--cluster stage-euc1-core-ecs-cluster \
--services admin-portal \
--query 'services[0].{desired:desiredCount,running:runningCount,status:status,taskDefinition:taskDefinition}' \
--region eu-central-1
# Which image that revision runs (ask the service, not the family — a rollback may leave the service
# on an older revision than the newest one)
aws ecs describe-task-definition \
--task-definition "$(aws ecs describe-services --cluster stage-euc1-core-ecs-cluster --services admin-portal \
--query 'services[0].taskDefinition' --output text --region eu-central-1)" \
--query 'taskDefinition.containerDefinitions[0].image' \
--region eu-central-1
# Production: the same with --cluster prod-euc1-core-ecs-cluster and --profile prod-admin
Viewing Logs¶
The log group is in the task definition, so ask it rather than guess (by convention it is
/ecs/<task definition family>):
aws ecs describe-task-definition \
--task-definition stage-euc1-core-admin-portal \
--query 'taskDefinition.containerDefinitions[0].logConfiguration.options' \
--region eu-central-1
aws logs tail /ecs/stage-euc1-core-admin-portal --follow --region eu-central-1
Rollback¶
Every deploy is a task definition revision, so rolling back is pointing the service at an earlier one. No image is re-tagged.
# Revisions, newest first (the service's current one is in describe-services above)
aws ecs list-task-definitions \
--family-prefix stage-euc1-core-admin-portal \
--sort DESC --max-items 5 \
--region eu-central-1
# Point the service at the one you want
aws ecs update-service \
--cluster stage-euc1-core-ecs-cluster \
--service admin-portal \
--task-definition stage-euc1-core-admin-portal:<revision> \
--region eu-central-1
Production: the same against prod-euc1-core-ecs-cluster / prod-euc1-core-admin-portal with --profile prod-admin.
Re-running an earlier release is the other route — gh workflow run deploy.yml --repo cookiehub-com/admin --ref v1.1.0
rebuilds that tag and registers a new revision from it; the revision route above is the one that restores an image
that already ran. See Rolling back.
Manual Deploy¶
deploy/ecr-push.sh in the repo builds for linux/amd64 and pushes <sha> and latest. It refuses a dirty tree and
does not create the repository (Terraform's). Then either run the workflow, or do what it does by hand:
aws login # or AWS_PROFILE=…
deploy/ecr-push.sh # ECR_REPOSITORY=admin-portal, AWS_REGION=eu-central-1 by default
gh workflow run deploy.yml --repo cookiehub-com/admin --ref main
# Without Actions: register a revision with the new image and update the service
CLUSTER=stage-euc1-core-ecs-cluster; FAMILY=stage-euc1-core-admin-portal
IMAGE="258618559895.dkr.ecr.eu-central-1.amazonaws.com/admin-portal:<tag>"
aws ecs describe-task-definition --task-definition "$FAMILY" --query taskDefinition --region eu-central-1 \
| jq --arg img "$IMAGE" '
(.containerDefinitions[] | select(.name == "admin-portal") | .image) = $img
| del(.taskDefinitionArn, .revision, .status, .requiresAttributes,
.compatibilities, .registeredAt, .registeredBy)' > /tmp/td.json
ARN=$(aws ecs register-task-definition --cli-input-json file:///tmp/td.json \
--query taskDefinition.taskDefinitionArn --output text --region eu-central-1)
aws ecs update-service --cluster "$CLUSTER" --service admin-portal --task-definition "$ARN" --region eu-central-1
Required Access¶
- GitHub: Write access to
cookiehub-com/admin— push tomainfor stage, create releases for production - AWS stage: IAM user in the stage account (
258618559895) with ECS/ECR permissions (rollback and manual deploy only) - AWS prod:
prod-adminprofile withDevOpsAdminRoleassumption (rollback only) - SSM: the parameters under
/core/{env}/admin-portal/must hold real values before the first task starts — see Secrets
Vault API¶
How It Deploys¶
Vault API has two deploy paths:
- Stage: Push to
stagebranch → Quality checks pass → Deploy to stage - Production: Create a GitHub Release → Deploy to production
flowchart LR
A[Push to stage] --> B[Quality checks]
B -->|pass| C[Deploy to stage]
C --> D[Build image]
D --> E[Push to ECR]
E --> F[ECS redeploy]
flowchart LR
A[Create Release] --> R[Await reviewer approval]
R --> G{Tag on main?}
G -->|no| X[Fail]
G -->|yes| C[Build image]
C --> D[Push to ECR<br/>sha, latest, tag]
D --> S[Swap image into live task definition,<br/>register revision]
S --> U[Update service, wait for stable]
The production job is gated on the production GitHub environment, so it waits for one of the repo's production
reviewers to approve the deployment; GitHub stores the approver, the timestamp and the SHA in the
environment's deployment history. The task definition is pinned to the commit SHA, never
:latest. See
Branching and Releases.
Infrastructure¶
| Stage | Production | |
|---|---|---|
| AWS account | 258618559895 |
759286286879 |
| ECR repo | vault-api |
vault-api |
| ECS cluster | stage-euc1-vault-ecs-cluster |
prod-euc1-vault-ecs-cluster |
| ECS service | vault-api |
vault-api |
| Task definition | stage-euc1-vault-vault-api |
prod-euc1-vault-vault-api |
| Container | vault-api |
vault-api |
| OIDC role | gh-oidc-deploy-vault-api |
gh-oidc-deploy-vault-api |
| Role trusts | refs/heads/main |
refs/tags/v*, environment:production |
| GitHub secret | AWS_DEPLOY_ROLE_ARN |
AWS_DEPLOY_ROLE_ARN_PROD |
| URL | Internal ALB (stage VPC) | vault-api.internal.cookiehub.net |
| Port | 3000 | 3000 |
| Health check | GET /health |
GET /health |
| SSM params | /vault/stage/vault-api/* |
/vault/prod/vault-api/* |
| Logs | CloudWatch /ecs/stage-euc1-vault-vault-api |
CloudWatch /ecs/prod-euc1-vault-vault-api |
| Image tags | <sha>, latest |
<sha>, latest, <tag> (the revision uses <sha>) |
Deploying to Production¶
- Merge the change to
mainby reviewed PR, and bump the version there.stageis not on the path tomain. - Cut the release from
main:
gh release create v1.4.0 --repo cookiehub-com/vault-api --target <sha of main> --title v1.4.0 --notes-file release-note.md
The note is the text from the Linear release; see Cutting a release for the Linear steps that go with the cut.
- The deploy workflow fires, waits for a required reviewer to approve the
productiondeployment, and only then checks that the tag is an ancestor ofmain(the guard has not run when you approve). Approve it at the run or from the repo's Environments page.
The workflow refuses any tag that is not on main, so a tag cut from a feature branch fails before
it builds anything.
Checking Deployment Status¶
# Watch the GitHub Actions run
gh run list --repo cookiehub-com/vault-api --limit 5
# Check stage
aws ecs describe-services \
--cluster stage-euc1-vault-ecs-cluster \
--services vault-api \
--query 'services[0].{desired:desiredCount,running:runningCount,status:status}' \
--region eu-central-1
# Check prod
aws ecs describe-services \
--cluster prod-euc1-vault-ecs-cluster \
--services vault-api \
--query 'services[0].{desired:desiredCount,running:runningCount,status:status,taskDefinition:taskDefinition}' \
--region eu-central-1 \
--profile prod-admin
# Which image prod is actually running (ask the service, not the family — a rollback can leave the
# service on an older revision than the newest one)
aws ecs describe-task-definition \
--task-definition "$(aws ecs describe-services --cluster prod-euc1-vault-ecs-cluster --services vault-api \
--query 'services[0].taskDefinition' --output text --region eu-central-1 --profile prod-admin)" \
--query 'taskDefinition.containerDefinitions[0].image' \
--region eu-central-1 \
--profile prod-admin
Rollback¶
Production¶
Re-run the previous release's deploy run from the Actions UI (this works for 30 days after the original run; after that
cut a new patch release from a revert on main, or use the revision route below, which is break-glass until IS-135
proves it):
The workflow registers a revision pinned to the commit-SHA image tag, but a re-run rebuilds the image from source
and overwrites that :<sha> tag (ECR repositories are mutable), so it is a redeploy of the same commit, not of the
image that ran before. It needs a reviewer approval, like any production deploy — see
Break-glass if nobody is available.
To restore the image that actually ran, point the service at the earlier task-definition revision. Terraform tracks the newest ACTIVE revision, so nothing reverts this on the next apply:
aws ecs list-task-definitions --family-prefix prod-euc1-vault-vault-api \
--sort DESC --max-items 5 --region eu-central-1 --profile prod-admin
aws ecs update-service --cluster prod-euc1-vault-ecs-cluster --service vault-api \
--task-definition prod-euc1-vault-vault-api:<older revision> --region eu-central-1 --profile prod-admin
Do not re-tag :latest, which is not what the service runs. What is left to make rollback fully rebuild-free is
tracked in IS-135.
Stage¶
Push the known-good commit to stage (force-push is fine — stage is disposable) and let the stage
deploy run.
Required Access¶
- GitHub: Write access to
cookiehub-com/vault-api(push to stage, create releases) - Production deploys: membership of the repo's
productionenvironment reviewers - AWS stage: IAM user in stage account (
258618559895) with ECS/ECR permissions - AWS prod:
prod-adminprofile withDevOpsAdminRoleassumption (for inspection)
Edge API¶
How It Deploys¶
Edge API has two deploy paths:
- Stage: Push to
stagebranch → Quality checks pass → Deploy to stage - Production: Create a GitHub Release → Deploy to production, behind the
productionenvironment gate (see Edge API in production below)
flowchart LR
A[Push to stage] --> B[Quality checks]
B -->|pass| C[Deploy to stage]
C --> D[Build image]
D --> E[Push to ECR]
E --> F[ECS redeploy]
Infrastructure¶
| Stage | |
|---|---|
| AWS account | 258618559895 |
| ECR repo | edge-api |
| ECS cluster | stage-euc1-edge-ecs-cluster |
| ECS service | edge-api |
| Task definition | stage-euc1-edge-edge-api |
| OIDC role | gh-oidc-deploy-edge-api |
| GitHub secret | AWS_DEPLOY_ROLE_ARN |
| CloudFront | E1Q7WZ429TOWKP (forwards geo headers to ALB) |
| URL | https://edge-api.stage.cookiehub.net |
| Port | 3000 |
| Health check | GET /health |
| Readiness check | GET /ready (SQS + ClickHouse) |
| SSM params | /edge/stage/edge-api/* |
| SQS queue | stage-euc1-edge-consent (shared with Lambda) |
| Logs | CloudWatch /ecs/stage-euc1-edge-edge-api |
Endpoints¶
| Method | Path | Description |
|---|---|---|
| GET | /health | Liveness check |
| GET | /ready | Readiness (SQS + ClickHouse) |
| GET | /docs | Interactive API documentation |
| v3 (new) | ||
| POST | /v3/session/register | Create edge_token + session_proof |
| POST | /v3/events | Ingest widget event → SQS |
| GET | /v3/consent/state/:edge_token | Consent state lookup from CH |
| POST | /v3/geo/resolve | Resolve region from geo headers |
| v1 (legacy compat) | ||
| GET | /v1/region | Geo resolution (cookiehub-regions header, returns {r,c,s}) |
| POST | /v1/log | Consent log (raw payload → SQS envelope) |
Checking Deployment Status¶
# Watch the GitHub Actions run
gh run list --repo cookiehub-com/edge-api --limit 5
# Check ECS service status
aws ecs describe-services \
--cluster stage-euc1-edge-ecs-cluster \
--services edge-api \
--query 'services[0].{desired:desiredCount,running:runningCount,status:status}' \
--region eu-central-1
# Verify geo headers are working
curl -s -X POST https://edge-api.stage.cookiehub.net/v3/geo/resolve \
-H "Content-Type: application/json" \
-d '{"regions":["EU","C3"]}'
Rollback¶
Stage only (the stage service runs :latest):
ROLLBACK_SHA="<commit-sha>"
REPO="258618559895.dkr.ecr.eu-central-1.amazonaws.com/edge-api"
aws ecr get-login-password --region eu-central-1 | \
docker login --username AWS --password-stdin 258618559895.dkr.ecr.eu-central-1.amazonaws.com
docker pull $REPO:$ROLLBACK_SHA
docker tag $REPO:$ROLLBACK_SHA $REPO:latest
docker push $REPO:latest
aws ecs update-service \
--cluster stage-euc1-edge-ecs-cluster \
--service edge-api \
--force-new-deployment \
--region eu-central-1
Edge API in production¶
deploy-prod.yml runs on release: published and declares environment: production. After a reviewer approves (the
ancestor guard has not run yet), it checks out the tag, verifies it is an ancestor of main, assumes the role in
AWS_DEPLOY_ROLE_ARN_PROD, builds the
image, pushes :<tag> and :latest to the production ECR repository, and runs update-service
--force-new-deployment against the service's existing task-definition revision on prod-euc1-edge-ecs-cluster
(service edge-api, family prod-euc1-edge-edge-api, 2 tasks, fronted by edge.cookiehub.net).
Not SHA-pinned
Unlike vault-api, core-api and admin, the edge-api production workflow does not register a new revision, so
the running image tracks :latest. A task restart or scale-out pulls whatever :latest is at that moment.
Pinning edge-api's workflow is part of IS-135 (the Terraform side was done under IS-102).
Rollback is the ECS service procedure. Because the revision never changes,
re-running an earlier release rebuilds that tag and re-points :latest; pointing the service at an earlier revision
will not help while every revision references :latest. The re-run works for 30 days after the original run. Until
edge-api is pinned, "roll back" means cutting a fixed patch release (a revert on main, through the gate).
Required Access¶
- GitHub: Write access to
cookiehub-com/edge-api(push to stage, create releases) - Production deploys: membership of the repo's
productionenvironment reviewers - AWS stage: IAM user in stage account (
258618559895) with ECS/ECR permissions
Atrax¶
Atrax has two deployable images, in two different regions, each with its own repo and pipeline:
| Repo | Image | ECR | Cluster | Task definition family (prod) |
|---|---|---|---|---|
atrax-api |
atrax-api |
<acct>.dkr.ecr.eu-central-1.amazonaws.com/atrax-api |
{env}-euc1-core-ecs-cluster (shared with core-api) |
prod-euc1-atrax-atrax-api |
atrax |
atrax-node |
<acct>.dkr.ecr.eu-west-1.amazonaws.com/atrax-node |
{env}-euw1-atrax-ecs-cluster |
prod-euw1-atrax-atrax-node |
How they deploy¶
Both repos have the same two paths:
- Stage: push to the
stagebranch → Quality checks pass →deploy-stage.ymlbuilds the image, pushes:<sha>and:latest, and registers a task-definition revision pinned to the SHA.atrax-apihas nostagebranch on the remote today; stage is disposable, so this is harmless, and the branch is created by the first push. - Production: publish a GitHub Release on a
v*tag →deploy-prod.ymlwaits at theproductionenvironment gate, then verifies the tag is onmain(the guard has not run when the reviewer approves), then builds and pushes and registers a revision pinned to the SHA (render-task-definitionplusdeploy-task-definition), waiting for the service to be stable.atraxpushes:<sha>,:latestand:<tag>;atrax-apipushes:<sha>and:latestonly (no:<tag>). The production role comes fromAWS_DEPLOY_ROLE_ARN_PROD.
The production environment of both repos has three required reviewers (the other repos have two); either way,
the person who cut the release cannot approve it. atrax still has a prod branch, left over from the old model; it
deploys nothing and is being retired (IS-129).
Rollback is the ECS service procedure: re-running an earlier release rebuilds
it (within 30 days of the original run; after that cut a new patch release from a revert on main), and pointing the
service at an earlier revision restores the image that ran (break-glass until IS-135 proves it).
Manual deploy¶
Break-glass only
The pipelines above are the way to deploy. A manual docker push of :latest no longer deploys anything, because
the task definition is pinned to a SHA tag; do not use it. If CI is broken, register a revision with the image you
want and update the service, as in the Admin Portal manual deploy, and follow
Break-glass.
force-new-deployment is still how a running task picks up changed SSM values — the ECS agent only reads secrets at task launch:
aws ecs update-service --cluster {env}-euc1-core-ecs-cluster --service atrax-api \
--force-new-deployment --region eu-central-1
aws ecs update-service --cluster {env}-euw1-atrax-ecs-cluster --service atrax-node \
--force-new-deployment --region eu-west-1
Logs¶
# atrax-api
aws logs tail /ecs/{env}-euc1-atrax-atrax-api --follow --region eu-central-1
# atrax-node
aws logs tail /ecs/{env}-euw1-atrax-atrax-node --follow --region eu-west-1
Vault Ingest¶
vault-ingest runs the Vault ETL workers: seven ECS services built from one image, on the vault cluster
({env}-euc1-vault-ecs-cluster). It has the same two paths as the other service repos:
- Stage: push to the
stagebranch → Quality checks pass →deploy-stage.yml. - Production: publish a GitHub Release on a
v*tag →deploy-prod.yml. The job declaresenvironment: production, verifies the tag is onmain, assumesAWS_DEPLOY_ROLE_ARN_PROD, builds the image, pushes:<sha>,:latestand:<tag>, checks the GeoLite database is in the image, and then.github/scripts/deploy-services.shregisters a SHA-pinned revision for each of the seven services (clickhouse-session-ingest,bunny-api-logs,cloudfront-log-index, and the threeconsent-sqs-ingestinstances andconsent-clickhouse-ingest) before updating them all and waiting for stability.
Rollback is the ECS service procedure, but note it is per service: there
are seven services and seven task-definition families, so pointing services at earlier revisions has to be done for
each one that the release changed. A workflow re-run (within 30 days of the original run; after that a new patch
release from a revert on main) rebuilds the image and re-registers all seven.
The Vault API is documented above.