Post-deploy smoke checklist¶
What to check after a production deploy, per service. Run it after every release. It is the "Post-release verification" box of the Linear Release issue, and the step that decides whether the release stays or goes to Rolling back.
Facts on this page come from each repo's production workflow, the Terraform in infra, and the
running endpoints, read on 12 November 2026. Where a fact could not be established read-only, the page
says to confirm and names what to look at.
Required access¶
- AWS: read-only access to the production account (
ecs:Describe*,ecr:DescribeImages,ssm:SendCommandon the core instance for thecoreversion read,s3:GetObjectonwidget-releases/*). Needed only for the version commands. - Secrets: none. Every request on this page is a plain
GET. - VPN: the Client VPN, for
vault-apionly (internal hostname). - GitHub: read access to the Actions runs of the repo being released.
Purpose and when to run it¶
A green workflow run proves the pipeline finished. It does not prove the release is serving. The checks here take a few minutes per service and answer one question: is the new version up and answering?
Run them:
- after every production deploy, once the Teams card says the deploy succeeded;
- after a rollback, to prove the old version is serving again;
- on the Release issue, as the post-release verification box. Tick it only when every check for the service has passed.
The cutter runs the checks. Someone from the production environment reviewer pool may repeat them. If a check fails, do not close the Release issue. Go to the rollback decision point in the service's section.
How each section is laid out¶
Every service has the same five parts:
- What deploys it. The workflow and its trigger.
- Teams card. The card the workflow posts to the prod deploys channel.
- Checks. Two or three requests or commands that prove the release is serving, with the expected result.
- Version. Where the running version shows, and the command to read it.
- Rollback decision point. Which result sends you to Rolling back.
A "healthy minute" part follows only where Terraform declares a log group or an alarm.
Summary¶
| Service | Workflow | Health or smoke URL | Version source |
|---|---|---|---|
| core | core deploy-prod.yml |
https://dash.cookiehub.com/login |
storage/app/VERSION on the box; Bugsnag app version |
| core-api | core-api deploy-prod.yml |
/client/health (hostname to confirm) |
ECS task definition image :<sha> |
| core-portal | core-portal deploy-prod.yml |
/health (no production service yet) |
ECS task definition image :<sha> |
| edge-api | edge-api deploy-prod.yml |
https://edge.cookiehub.net/health and /ready |
ECS deployment record; ECR tag :<tag> |
| vault-api | vault-api deploy-prod.yml |
https://vault-api.internal.cookiehub.net/health (VPN) |
ECS task definition image :<sha> |
| vault-ingest | vault-ingest deploy-prod.yml |
none (workers) | ECS task definition image :<sha>, seven services |
| atrax | atrax deploy-prod.yml |
none (workers) | ECS task definition image :<sha> |
| atrax-api | atrax-api deploy-prod.yml |
https://atrax-api.cookiehub.net/health |
ECS task definition image :<sha> |
| admin | admin deploy.yml |
https://cha.cookiehub.net/ (SSO redirect) |
ECS task definition image :<sha> |
| docs | docs deploy-prod.yml |
https://docs.cookiehub.com/ |
Vercel deployment commit |
| cookiechecker | cookiechecker deploy-prod.yml |
https://www.cookiechecker.com/ |
Vercel deployment commit |
| widget | widget release.yml |
https://cdn.cookiehub.eu/c2/<code>.js |
manifest.json in S3; version string in the bundle |
Reading the version of an ECS service¶
Every ECS service uses the same two commands. Production AWS access is required. These are read-only. Use the cluster, service and region from the service's section.
TD=$(aws ecs describe-services --cluster <cluster> --services <service> --region <region> \
--query 'services[0].taskDefinition' --output text)
echo "$TD" # ...:task-definition/<family>:<revision>
aws ecs describe-task-definition --task-definition "$TD" --region <region> \
--query 'taskDefinition.containerDefinitions[0].image' --output text
For a SHA-pinned service the image ends in :<commit sha>. Compare it with the release commit:
The two must match. The revision number must also be higher than the one before the deploy. To see whether the rollout finished:
aws ecs describe-services --cluster <cluster> --services <service> --region <region> \
--query 'services[0].{desired:desiredCount,running:runningCount,deployments:deployments[].{state:rolloutState,td:taskDefinition,created:createdAt}}'
Expect one deployment with state COMPLETED and running equal to desired. A second, IN_PROGRESS
or FAILED deployment means the rollout has not settled.
Which services are SHA-pinned is in the per-repo status table.
edge-api is not: its running image tracks :latest, so its revision number does not change on a
deploy. For edge-api read the deployment record instead (see its section).
A missing Teams card¶
Success and failure cards are steps of the deploy job. If the job never starts, no card is posted.
That happens when the approval is rejected, or when the release gate fails first (core-api,
core-portal). No card after the expected time is a signal in itself. Open the workflow run from the
Actions tab of the repo.
admin and docs post no card at all (see their sections).
All cards go to the prod deploys Teams channel, through the TEAMS_WEBHOOK_DEPLOY_PROD secret. Every card
has the same shape: a title line, a Version fact that links to the GitHub Release, a By fact with
the person who published the release, and a View deploy logs button that opens the workflow run. The
title is <repo> → <env> released with a green tick on success and <repo> → <env> release failed
with a red cross on failure.
core¶
The legacy Laravel dashboard on the production EC2 box.
What deploys it. deploy-prod.yml in core, on release: published. After the reviewer approves,
it checks the tag is on main, then updates the existing checkout at /home/cookiehub/core in place
over SSM: checkout of the tag, composer install, npm run build, php artisan migrate --force,
config:clear, then a reload of Apache. Its last step is a curl of https://dash.cookiehub.com that
accepts any status from 200 to 399. Cloudflare fronts the box, so that step proves Cloudflare answers.
It does not prove the dashboard works.
Teams card. Success: ✅ core → prod released. Failure: ❌ core → prod release failed.
Checks.
GET https://dash.cookiehub.com/answers302withLocation: https://dash.cookiehub.com/login.GET https://dash.cookiehub.com/loginanswers200with the HTML login page.- Sign in with a staff account and open the domain list. Expect the page, with no error banner. The production smoke account is to confirm: the stage account is in Stage Operations; check whether a production equivalent exists before using a real customer login.
Version. The deploy writes the release tag into storage/app/VERSION before anything else runs.
The app reads it through AppHelper::resolveAppVersion() and reports it to Bugsnag as the app version
(CH-831). Read it over SSM:
aws ssm send-command --instance-ids i-05301a95c43178747 --region eu-central-1 \
--document-name AWS-RunShellScript \
--parameters 'commands=["cat /home/cookiehub/core/storage/app/VERSION"]' \
--query 'Command.CommandId' --output text
aws ssm get-command-invocation --instance-id i-05301a95c43178747 --region eu-central-1 \
--command-id <CommandId from above> --query 'StandardOutputContent' --output text
Expect the tag you cut, for example v3.11.0. In Bugsnag, new production events carry the same tag as
their app version. The deploy also sends a Bugsnag deploy notification, best-effort, so the release
should appear in the project's timeline. A missing timeline entry is not a failure: the step ends with
|| true.
Rollback decision point. Go to
Rolling back, the core section when:
- the SSM command failed or timed out. The box may hold the new code with a half-run build or migration;
/loginanswers anything but200, or the staff login fails;VERSIONshows the old tag after a green run.
A core rollback restores PHP code and dashboard assets only. It does not revert migrations. Read that
section before deciding. If the dash hostname is being served by core-portal at the time, the
rollback for a bad dashboard is the listener rule in
Dashboard Cutover Rollback, not a core redeploy.
A healthy minute. Terraform declares one alarm for the box: cs150p - CPU over 80%
(environments/prod/eu-central-1/monitoring.tf). It should stay in OK. Terraform declares no CloudWatch
log group for core. Where Laravel logs are read in production is to confirm.
core-api¶
The TypeScript service that replaces the legacy client API.
What deploys it. deploy-prod.yml in core-api, on release: published. A gate job refuses a
commit that is not on main or has no green quality run. The deploy job then waits for the
reviewer, builds the image, pushes :<sha>, :latest and :<tag>, and registers a new task-definition
revision pinned to the SHA. It waits for the service to be stable. A manual dispatch runs the gate
alone and deploys nothing.
Teams card. ✅ core-api → prod released or ❌ core-api → prod release failed. A failing gate
posts nothing.
State today. Terraform holds the production service at desired_count = 0
(environments/prod/eu-central-1/core.tf). A deploy registers a revision, but no task runs until the
count is raised. Check runningCount first. The checks below apply once the service is live.
Checks.
GET /client/healthanswers200with{"status":"ok","gateway":{...}}. A503with{"status":"error","error":"database_unavailable",...}means the database is unreachable.- The ECS rollout is
COMPLETEDwithrunningequal todesired(commands above). - The image on the running revision ends in the release commit SHA.
The public hostname is to confirm. Terraform declares core-api.cookiehub.net
(core.tf). The workflow's environment URL is https://api.cookiehub.net, a shared name that also
carries the DSAR API. On 12 November 2026 GET https://api.cookiehub.net/client/health answered
404, which fits a service at zero tasks. Confirm which name the ALB rule serves before the first
live release.
Version. Cluster prod-euc1-core-ecs-cluster, service core-api, family
prod-euc1-core-core-api, region eu-central-1. Use the ECS commands above.
Rollback decision point. Go to
Rolling back, the ECS service section, when /client/health
is not 200 for two minutes after the rollout completes, or the rollout ends FAILED.
A healthy minute.
- Log group
/ecs/prod-euc1-core-core-api. - Alarms
core-api - CPU over <n>%andcore-api - Memory over <n>%. - The public ALB alarm
prod-euc1-core-public-alb/core-api - Unhealthy hosts >= <n>.
core-portal¶
The React dashboard.
What deploys it. deploy-prod.yml in core-portal, on release: published, with the same
gate-then-deploy shape as core-api. Its header says that until the production service exists, a
published release passes the gate and fails at the AWS step without changing anything.
Teams card. ✅ core-portal → prod released or ❌ core-portal → prod release failed.
State today. The production environment declares no core-portal service in Terraform on main.
The workflow names cluster prod-euc1-core-ecs-cluster, service core-portal and family
prod-euc1-core-portal. Treat these as to confirm once the service is created. The environment
has no URL yet because the soft launch uses a private host.
Checks, once live.
GET /healthanswers200with{"status":"ok"}.- The ECS rollout is
COMPLETEDwithrunningequal todesired. - Sign in and load the dashboard home. Expect the page and no redirect loop.
The hostname for the first check is to confirm (private soft-launch host, then dash.cookiehub.com
after the flip).
Version. The ECS commands above, with the names the workflow declares.
Rollback decision point. If core-portal is serving the dash hostname, a failed check sends you to
Dashboard Cutover Rollback (revert the listener rule, Laravel stays
deployed). A failed ECS rollout with no traffic on it goes to
Rolling back.
edge-api¶
The consent and region ingestion service.
What deploys it. deploy-prod.yml in edge-api, on release: published. It builds, pushes :<tag>
and :latest, then runs update-service --force-new-deployment against the existing task definition.
It is not SHA-pinned.
Teams card. ✅ edge-api → prod released or ❌ edge-api → prod release failed.
Checks.
GET https://edge.cookiehub.net/healthanswers200with{"status":"ok"}.GET https://edge.cookiehub.net/readyanswers200with{"status":"ok","sqs":true,"clickhouse":true}. A503with"status":"degraded"names the dependency that is down.- The rollout finished: the deployment record shows
COMPLETEDandrunningequalsdesired(2).
Version. Cluster prod-euc1-edge-ecs-cluster, service edge-api, family
prod-euc1-edge-edge-api, region eu-central-1. The revision does not change on a deploy, so read the
deployment record for the rollout time, and confirm the image was pushed with the tag:
aws ecr describe-images --repository-name edge-api --region eu-central-1 \
--image-ids imageTag=v1.4.0 --query 'imageDetails[0].{pushed:imagePushedAt,tags:imageTags}'
Expect the tag you cut. The running image is :latest, so a push of a newer image by someone else
would also move it.
Rollback decision point. Go to Rolling back when /health
fails, /ready stays 503 for two minutes, or either alarm below fires. Edge ingestion is high volume,
so do not wait.
A healthy minute.
- Log group
/ecs/prod-euc1-edge-edge-api. - Alarms
edge-api - 5xx over 25/minandedge-api - Unhealthy hosts >= 1 (1 min)stay in OK. - Also
edge-api - CPU over <n>%,edge-api - Memory over <n>%and the public ALBedge-apitarget-group alarms.
vault-api¶
The analytics and query service. Internal only.
What deploys it. deploy-prod.yml in vault-api, on release: published. It registers a
SHA-pinned revision and waits for the service to be stable.
Teams card. ✅ vault-api → prod released or ❌ vault-api → prod release failed.
Checks. The hostname is on the internal zone, so connect to the VPN first.
GET https://vault-api.internal.cookiehub.net/healthanswers200with{"status":"ok"}. The health route needs no API key. A503with{"status":"error","detail":"clickhouse_unreachable"}means ClickHouse is unreachable.- The rollout is
COMPLETEDwithrunningequal todesired(2). - The image on the running revision ends in the release commit SHA.
Version. Cluster prod-euc1-vault-ecs-cluster, service vault-api, family
prod-euc1-vault-vault-api, region eu-central-1.
Rollback decision point. Go to Rolling back when /health
is not 200 two minutes after the rollout completes.
A healthy minute.
- Log group
/ecs/prod-euc1-vault-vault-api. - Alarms
vault-api - CPU over <n>%andvault-api - Memory over <n>%. - The internal ALB alarm
prod-euc1-core-internal-alb/vault-api - Unhealthy hosts >= <n>.
vault-ingest¶
Seven ECS services deployed from one image. They have no HTTP endpoint.
What deploys it. deploy-prod.yml in vault-ingest, on release: published. One script registers a
SHA-pinned revision for each service and updates them in turn. No migrations run: ClickHouse DDL is
applied by hand.
Teams card. ✅ vault-ingest → prod released or ❌ vault-ingest → prod release failed.
Checks.
- All seven services show
COMPLETEDandrunningequal todesired:
aws ecs describe-services --cluster prod-euc1-vault-ecs-cluster --region eu-central-1 \
--services clickhouse-session-ingest bunny-api-logs cloudfront-log-index \
consent-sqs-ingest-aws consent-sqs-ingest-eu consent-sqs-ingest-edge consent-clickhouse-ingest \
--query 'services[].{name:serviceName,run:runningCount,want:desiredCount,td:taskDefinition}'
- Every service's image ends in the same release commit SHA (ECS commands above, one per service).
- No error lines in the first minutes of the log groups below.
Version. Cluster prod-euc1-vault-ecs-cluster, families prod-euc1-vault-<name>, region
eu-central-1.
Rollback decision point. Go to Rolling back when any of the seven fails to reach its desired count, or the image SHA differs between services (a half-finished deploy).
A healthy minute.
- Log groups, one per service, under
/ecs/prod-euc1-vault-:ingest,bunny-api-logs,cloudfront-log-index,consent-sqs-ingest-aws,consent-sqs-ingest-eu,consent-sqs-ingest-edge,consent-clickhouse-ingest. - A CPU and a memory alarm per service, named
<service> - CPU over <n>%and<service> - Memory over <n>%. - The edge consent queue alarms from the
sqs-queuemonitoring module, andConsent log messages below threshold (aws network).
atrax¶
The crawler workers, in eu-west-1. There is no load balancer.
What deploys it. deploy-prod.yml in atrax, on release: published. SHA-pinned.
Teams card. ✅ atrax → prod released or ❌ atrax → prod release failed.
Checks.
- The rollout is
COMPLETEDwithrunningequal todesired(4 in Terraform). - The image on the running revision ends in the release commit SHA.
- A scan started from
https://www.cookiechecker.com/finishes and shows a report. The scan flow crossesatraxandatrax-api. The exact scan to use is to confirm.
Expect a short gap while it rolls. The service deploys with minimum healthy 0, so all old tasks stop before new ones start. A scan running during the deploy can fail. Do not read that as a bad release.
Version. Cluster prod-euw1-atrax-ecs-cluster, service atrax-node, family
prod-euw1-atrax-atrax-node, region eu-west-1.
Rollback decision point. Go to Rolling back when the
service does not reach its desired count, or fresh scans fail after the rollout is COMPLETED.
A healthy minute.
- Log group
/ecs/prod-euw1-atrax-atrax-node. - The
eu-west-1cluster and instance alarms, which route to thePagerDuty-Atraxtopic.
atrax-api¶
The control API for the crawler.
What deploys it. deploy-prod.yml in atrax-api, on release: published. SHA-pinned.
Teams card. ✅ atrax-api → prod released or ❌ atrax-api → prod release failed.
Checks.
GET https://atrax-api.cookiehub.net/healthanswers200with{"status":"ok"}.GET https://atrax-api.cookiehub.net/answers200with{"message":"Atrax API"}.- The rollout is
COMPLETEDand the image ends in the release commit SHA.
Version. Cluster prod-euc1-core-ecs-cluster, service atrax-api, family
prod-euc1-atrax-atrax-api, region eu-central-1.
Rollback decision point. Go to Rolling back when /health
is not 200 two minutes after the rollout completes.
A healthy minute.
- Log group
/ecs/prod-euc1-atrax-atrax-api. - Terraform declares no service alarm for
atrax-api. Only theprod-euc1-core-ecs-clusteralarms cover it.
admin¶
The admin portal, behind Entra single sign-on.
What deploys it. deploy.yml in admin, on release: published (the job deploy-production).
The same file deploys stage on a push to main. A workflow_dispatch on a tag ref redeploys
production from that tag. SHA-pinned.
Teams card. None. The workflow has no notify step. Whether that is intended is to confirm. Watch the Actions run instead.
Checks.
GET https://cha.cookiehub.net/answers302with aLocationonlogin.microsoftonline.com. The load balancer does the sign-in redirect, so this proves the listener rule and the host are up. It does not prove the app is.- Sign in with SSO and load the home page. Expect the page and your name.
- The rollout is
COMPLETEDand the image ends in the release commit SHA.
The app's own probe is /api/health, which answers 200 with {"status":"ok"}. The load balancer
checks it. An unauthenticated request from outside is redirected to sign-in, so read it through the
target group state, not with curl.
Version. Cluster prod-euc1-core-ecs-cluster, service admin-portal, family
prod-euc1-core-admin-portal, region eu-central-1.
Rollback decision point. Go to Rolling back when sign-in
fails, or the target-group alarm below fires. admin is the one service you can roll back with
gh workflow run deploy.yml --ref <older tag>.
A healthy minute.
- Log group
/ecs/prod-euc1-core-admin-portal. - Alarms
admin-portal - CPU over <n>%,admin-portal - Memory over <n>%and the public ALBcha - Unhealthy hosts >= <n>.
docs¶
The public documentation site on Vercel.
What deploys it. deploy-prod.yml in docs, on release: published. A quality job runs first,
then the gated deploy job builds and deploys with the Vercel CLI. Vercel's own Git builds are off.
Teams card. None. The workflow has no notify step. Whether that is intended is to confirm.
Checks.
GET https://docs.cookiehub.com/answers200with the HTML home page.- Open a page changed in this release. Expect the new content.
- In the Vercel project, the newest production deployment is Ready and built from the release commit. The Vercel project name is to confirm.
Version. The commit of the production deployment in Vercel. There is no version string on the site.
Rollback decision point. Go to
Rolling back, the Vercel section, when the home page is not
200 or the changed page is wrong.
cookiechecker¶
The public cookie checker site on Vercel.
What deploys it. deploy-prod.yml in cookiechecker, on release: published. A quality job, a gated
deploy job, and a separate notify job.
Teams card. ✅ cookiechecker → production released or
❌ cookiechecker → production release failed. This is the one card that says production, not prod.
Checks.
GET https://www.cookiechecker.com/answers200with the HTML home page. The bare domain answers308to thewwwhost.- Start a scan of a public site. Expect live progress and then a report.
- In the Vercel project, the newest production deployment is Ready and built from the release commit. The Vercel project name is to confirm.
Version. The commit of the production deployment in Vercel. There is no version string on the site.
Rollback decision point. Go to
Rolling back, the Vercel section, when the home page is not
200 or a scan fails to produce a report.
widget¶
The consent widget. Production is a static publish to S3 plus a version registered with core. No
service restarts.
What deploys it. release.yml in widget, on release: published. Three jobs: build, a gated
deploy, and notify. The deploy checks the publish guard, syncs dist/ to
s3://cookiehub-release/widget-releases/<version>/, uploads manifest.json last, and registers the
version with core (POST /api/widget/v1/releases, 201 for a new version).
Teams card. ✅ widget → prod released or ❌ widget → prod release failed. The notify job always
runs, so a failure upstream still posts a card.
What a release changes. The registered version is not visible until someone flips it in the dashboard, and a domain only serves a new version after it is upgraded and republished. So right after a release, the bundles customers load are still the old version. That is expected. Checks 1 and 2 prove the publish. Check 3 proves nothing regressed.
Checks.
- The manifest exists and names the release commit:
Expect {"version":"2.10.0","sha":"<release commit>","run_id":"...","published_at":"..."}. The
version has no v.
- The workflow's Register version step logged
Version registration successful (HTTP 201)(200on a same-commit retry). A409fails the job. - A known sandbox domain still serves.
GET https://cdn.cookiehub.eu/c2/54cd6cf3.js(the page atsandbox.cookiehub.com/inline) answers200withcontent-type: text/javascript. The bundle starts withwindow.__cookiehuband contains'version': 'x.y.z'. Also check2bfd65da(the/inline-tcfpage). Both pages load the bundle fromcdn.cookiehub.eu, as the sandbox repo shows.
Expect the version those domains were on before the release. After you upgrade and republish a sandbox domain from the dashboard, expect the new version string here.
Version. The tag, minus the v, is the version. It shows in manifest.json and in the bundle's
'version' field.
Rollback decision point. The widget has no deploy rollback by design. If the manifest is missing,
the sha differs, or registration failed, do not re-run: a published build is never replaced, so cut the next
build number. If a domain serves a bad bundle, follow
Rolling back, the widget section and repoint the domains to
the older widget_version_id.
A healthy minute. Terraform declares no log group or alarm for the widget path.
Related¶
- Branching and Releases — the release model, the gate and rolling back
- Dashboard Cutover Rollback — reverting the
dashlistener rule - Deployments — per-service mechanics, names and status commands