Skip to content

Post-deploy smoke checklist

What to check after a production deploy, per service. Run it after every release. It is the "Post-release verification" box of the Linear Release issue, and the step that decides whether the release stays or goes to Rolling back.

Facts on this page come from each repo's production workflow, the Terraform in infra, and the running endpoints, read on 12 November 2026. Where a fact could not be established read-only, the page says to confirm and names what to look at.

Required access

  • AWS: read-only access to the production account (ecs:Describe*, ecr:DescribeImages, ssm:SendCommand on the core instance for the core version read, s3:GetObject on widget-releases/*). Needed only for the version commands.
  • Secrets: none. Every request on this page is a plain GET.
  • VPN: the Client VPN, for vault-api only (internal hostname).
  • GitHub: read access to the Actions runs of the repo being released.

Purpose and when to run it

A green workflow run proves the pipeline finished. It does not prove the release is serving. The checks here take a few minutes per service and answer one question: is the new version up and answering?

Run them:

  • after every production deploy, once the Teams card says the deploy succeeded;
  • after a rollback, to prove the old version is serving again;
  • on the Release issue, as the post-release verification box. Tick it only when every check for the service has passed.

The cutter runs the checks. Someone from the production environment reviewer pool may repeat them. If a check fails, do not close the Release issue. Go to the rollback decision point in the service's section.

How each section is laid out

Every service has the same five parts:

  1. What deploys it. The workflow and its trigger.
  2. Teams card. The card the workflow posts to the prod deploys channel.
  3. Checks. Two or three requests or commands that prove the release is serving, with the expected result.
  4. Version. Where the running version shows, and the command to read it.
  5. Rollback decision point. Which result sends you to Rolling back.

A "healthy minute" part follows only where Terraform declares a log group or an alarm.

Summary

Service Workflow Health or smoke URL Version source
core core deploy-prod.yml https://dash.cookiehub.com/login storage/app/VERSION on the box; Bugsnag app version
core-api core-api deploy-prod.yml /client/health (hostname to confirm) ECS task definition image :<sha>
core-portal core-portal deploy-prod.yml /health (no production service yet) ECS task definition image :<sha>
edge-api edge-api deploy-prod.yml https://edge.cookiehub.net/health and /ready ECS deployment record; ECR tag :<tag>
vault-api vault-api deploy-prod.yml https://vault-api.internal.cookiehub.net/health (VPN) ECS task definition image :<sha>
vault-ingest vault-ingest deploy-prod.yml none (workers) ECS task definition image :<sha>, seven services
atrax atrax deploy-prod.yml none (workers) ECS task definition image :<sha>
atrax-api atrax-api deploy-prod.yml https://atrax-api.cookiehub.net/health ECS task definition image :<sha>
admin admin deploy.yml https://cha.cookiehub.net/ (SSO redirect) ECS task definition image :<sha>
docs docs deploy-prod.yml https://docs.cookiehub.com/ Vercel deployment commit
cookiechecker cookiechecker deploy-prod.yml https://www.cookiechecker.com/ Vercel deployment commit
widget widget release.yml https://cdn.cookiehub.eu/c2/<code>.js manifest.json in S3; version string in the bundle

Reading the version of an ECS service

Every ECS service uses the same two commands. Production AWS access is required. These are read-only. Use the cluster, service and region from the service's section.

TD=$(aws ecs describe-services --cluster <cluster> --services <service> --region <region> \
  --query 'services[0].taskDefinition' --output text)
echo "$TD"   # ...:task-definition/<family>:<revision>

aws ecs describe-task-definition --task-definition "$TD" --region <region> \
  --query 'taskDefinition.containerDefinitions[0].image' --output text

For a SHA-pinned service the image ends in :<commit sha>. Compare it with the release commit:

git rev-parse v1.4.0^{commit}

The two must match. The revision number must also be higher than the one before the deploy. To see whether the rollout finished:

aws ecs describe-services --cluster <cluster> --services <service> --region <region> \
  --query 'services[0].{desired:desiredCount,running:runningCount,deployments:deployments[].{state:rolloutState,td:taskDefinition,created:createdAt}}'

Expect one deployment with state COMPLETED and running equal to desired. A second, IN_PROGRESS or FAILED deployment means the rollout has not settled.

Which services are SHA-pinned is in the per-repo status table. edge-api is not: its running image tracks :latest, so its revision number does not change on a deploy. For edge-api read the deployment record instead (see its section).

A missing Teams card

Success and failure cards are steps of the deploy job. If the job never starts, no card is posted. That happens when the approval is rejected, or when the release gate fails first (core-api, core-portal). No card after the expected time is a signal in itself. Open the workflow run from the Actions tab of the repo.

admin and docs post no card at all (see their sections).

All cards go to the prod deploys Teams channel, through the TEAMS_WEBHOOK_DEPLOY_PROD secret. Every card has the same shape: a title line, a Version fact that links to the GitHub Release, a By fact with the person who published the release, and a View deploy logs button that opens the workflow run. The title is <repo> → <env> released with a green tick on success and <repo> → <env> release failed with a red cross on failure.

core

The legacy Laravel dashboard on the production EC2 box.

What deploys it. deploy-prod.yml in core, on release: published. After the reviewer approves, it checks the tag is on main, then updates the existing checkout at /home/cookiehub/core in place over SSM: checkout of the tag, composer install, npm run build, php artisan migrate --force, config:clear, then a reload of Apache. Its last step is a curl of https://dash.cookiehub.com that accepts any status from 200 to 399. Cloudflare fronts the box, so that step proves Cloudflare answers. It does not prove the dashboard works.

Teams card. Success: ✅ core → prod released. Failure: ❌ core → prod release failed.

Checks.

  1. GET https://dash.cookiehub.com/ answers 302 with Location: https://dash.cookiehub.com/login.
  2. GET https://dash.cookiehub.com/login answers 200 with the HTML login page.
  3. Sign in with a staff account and open the domain list. Expect the page, with no error banner. The production smoke account is to confirm: the stage account is in Stage Operations; check whether a production equivalent exists before using a real customer login.

Version. The deploy writes the release tag into storage/app/VERSION before anything else runs. The app reads it through AppHelper::resolveAppVersion() and reports it to Bugsnag as the app version (CH-831). Read it over SSM:

aws ssm send-command --instance-ids i-05301a95c43178747 --region eu-central-1 \
  --document-name AWS-RunShellScript \
  --parameters 'commands=["cat /home/cookiehub/core/storage/app/VERSION"]' \
  --query 'Command.CommandId' --output text

aws ssm get-command-invocation --instance-id i-05301a95c43178747 --region eu-central-1 \
  --command-id <CommandId from above> --query 'StandardOutputContent' --output text

Expect the tag you cut, for example v3.11.0. In Bugsnag, new production events carry the same tag as their app version. The deploy also sends a Bugsnag deploy notification, best-effort, so the release should appear in the project's timeline. A missing timeline entry is not a failure: the step ends with || true.

Rollback decision point. Go to Rolling back, the core section when:

  • the SSM command failed or timed out. The box may hold the new code with a half-run build or migration;
  • /login answers anything but 200, or the staff login fails;
  • VERSION shows the old tag after a green run.

A core rollback restores PHP code and dashboard assets only. It does not revert migrations. Read that section before deciding. If the dash hostname is being served by core-portal at the time, the rollback for a bad dashboard is the listener rule in Dashboard Cutover Rollback, not a core redeploy.

A healthy minute. Terraform declares one alarm for the box: cs150p - CPU over 80% (environments/prod/eu-central-1/monitoring.tf). It should stay in OK. Terraform declares no CloudWatch log group for core. Where Laravel logs are read in production is to confirm.

core-api

The TypeScript service that replaces the legacy client API.

What deploys it. deploy-prod.yml in core-api, on release: published. A gate job refuses a commit that is not on main or has no green quality run. The deploy job then waits for the reviewer, builds the image, pushes :<sha>, :latest and :<tag>, and registers a new task-definition revision pinned to the SHA. It waits for the service to be stable. A manual dispatch runs the gate alone and deploys nothing.

Teams card. ✅ core-api → prod released or ❌ core-api → prod release failed. A failing gate posts nothing.

State today. Terraform holds the production service at desired_count = 0 (environments/prod/eu-central-1/core.tf). A deploy registers a revision, but no task runs until the count is raised. Check runningCount first. The checks below apply once the service is live.

Checks.

  1. GET /client/health answers 200 with {"status":"ok","gateway":{...}}. A 503 with {"status":"error","error":"database_unavailable",...} means the database is unreachable.
  2. The ECS rollout is COMPLETED with running equal to desired (commands above).
  3. The image on the running revision ends in the release commit SHA.

The public hostname is to confirm. Terraform declares core-api.cookiehub.net (core.tf). The workflow's environment URL is https://api.cookiehub.net, a shared name that also carries the DSAR API. On 12 November 2026 GET https://api.cookiehub.net/client/health answered 404, which fits a service at zero tasks. Confirm which name the ALB rule serves before the first live release.

Version. Cluster prod-euc1-core-ecs-cluster, service core-api, family prod-euc1-core-core-api, region eu-central-1. Use the ECS commands above.

Rollback decision point. Go to Rolling back, the ECS service section, when /client/health is not 200 for two minutes after the rollout completes, or the rollout ends FAILED.

A healthy minute.

  • Log group /ecs/prod-euc1-core-core-api.
  • Alarms core-api - CPU over <n>% and core-api - Memory over <n>%.
  • The public ALB alarm prod-euc1-core-public-alb/core-api - Unhealthy hosts >= <n>.

core-portal

The React dashboard.

What deploys it. deploy-prod.yml in core-portal, on release: published, with the same gate-then-deploy shape as core-api. Its header says that until the production service exists, a published release passes the gate and fails at the AWS step without changing anything.

Teams card. ✅ core-portal → prod released or ❌ core-portal → prod release failed.

State today. The production environment declares no core-portal service in Terraform on main. The workflow names cluster prod-euc1-core-ecs-cluster, service core-portal and family prod-euc1-core-portal. Treat these as to confirm once the service is created. The environment has no URL yet because the soft launch uses a private host.

Checks, once live.

  1. GET /health answers 200 with {"status":"ok"}.
  2. The ECS rollout is COMPLETED with running equal to desired.
  3. Sign in and load the dashboard home. Expect the page and no redirect loop.

The hostname for the first check is to confirm (private soft-launch host, then dash.cookiehub.com after the flip).

Version. The ECS commands above, with the names the workflow declares.

Rollback decision point. If core-portal is serving the dash hostname, a failed check sends you to Dashboard Cutover Rollback (revert the listener rule, Laravel stays deployed). A failed ECS rollout with no traffic on it goes to Rolling back.

edge-api

The consent and region ingestion service.

What deploys it. deploy-prod.yml in edge-api, on release: published. It builds, pushes :<tag> and :latest, then runs update-service --force-new-deployment against the existing task definition. It is not SHA-pinned.

Teams card. ✅ edge-api → prod released or ❌ edge-api → prod release failed.

Checks.

  1. GET https://edge.cookiehub.net/health answers 200 with {"status":"ok"}.
  2. GET https://edge.cookiehub.net/ready answers 200 with {"status":"ok","sqs":true,"clickhouse":true}. A 503 with "status":"degraded" names the dependency that is down.
  3. The rollout finished: the deployment record shows COMPLETED and running equals desired (2).

Version. Cluster prod-euc1-edge-ecs-cluster, service edge-api, family prod-euc1-edge-edge-api, region eu-central-1. The revision does not change on a deploy, so read the deployment record for the rollout time, and confirm the image was pushed with the tag:

aws ecr describe-images --repository-name edge-api --region eu-central-1 \
  --image-ids imageTag=v1.4.0 --query 'imageDetails[0].{pushed:imagePushedAt,tags:imageTags}'

Expect the tag you cut. The running image is :latest, so a push of a newer image by someone else would also move it.

Rollback decision point. Go to Rolling back when /health fails, /ready stays 503 for two minutes, or either alarm below fires. Edge ingestion is high volume, so do not wait.

A healthy minute.

  • Log group /ecs/prod-euc1-edge-edge-api.
  • Alarms edge-api - 5xx over 25/min and edge-api - Unhealthy hosts >= 1 (1 min) stay in OK.
  • Also edge-api - CPU over <n>%, edge-api - Memory over <n>% and the public ALB edge-api target-group alarms.

vault-api

The analytics and query service. Internal only.

What deploys it. deploy-prod.yml in vault-api, on release: published. It registers a SHA-pinned revision and waits for the service to be stable.

Teams card. ✅ vault-api → prod released or ❌ vault-api → prod release failed.

Checks. The hostname is on the internal zone, so connect to the VPN first.

  1. GET https://vault-api.internal.cookiehub.net/health answers 200 with {"status":"ok"}. The health route needs no API key. A 503 with {"status":"error","detail":"clickhouse_unreachable"} means ClickHouse is unreachable.
  2. The rollout is COMPLETED with running equal to desired (2).
  3. The image on the running revision ends in the release commit SHA.

Version. Cluster prod-euc1-vault-ecs-cluster, service vault-api, family prod-euc1-vault-vault-api, region eu-central-1.

Rollback decision point. Go to Rolling back when /health is not 200 two minutes after the rollout completes.

A healthy minute.

  • Log group /ecs/prod-euc1-vault-vault-api.
  • Alarms vault-api - CPU over <n>% and vault-api - Memory over <n>%.
  • The internal ALB alarm prod-euc1-core-internal-alb/vault-api - Unhealthy hosts >= <n>.

vault-ingest

Seven ECS services deployed from one image. They have no HTTP endpoint.

What deploys it. deploy-prod.yml in vault-ingest, on release: published. One script registers a SHA-pinned revision for each service and updates them in turn. No migrations run: ClickHouse DDL is applied by hand.

Teams card. ✅ vault-ingest → prod released or ❌ vault-ingest → prod release failed.

Checks.

  1. All seven services show COMPLETED and running equal to desired:
aws ecs describe-services --cluster prod-euc1-vault-ecs-cluster --region eu-central-1 \
  --services clickhouse-session-ingest bunny-api-logs cloudfront-log-index \
    consent-sqs-ingest-aws consent-sqs-ingest-eu consent-sqs-ingest-edge consent-clickhouse-ingest \
  --query 'services[].{name:serviceName,run:runningCount,want:desiredCount,td:taskDefinition}'
  1. Every service's image ends in the same release commit SHA (ECS commands above, one per service).
  2. No error lines in the first minutes of the log groups below.

Version. Cluster prod-euc1-vault-ecs-cluster, families prod-euc1-vault-<name>, region eu-central-1.

Rollback decision point. Go to Rolling back when any of the seven fails to reach its desired count, or the image SHA differs between services (a half-finished deploy).

A healthy minute.

  • Log groups, one per service, under /ecs/prod-euc1-vault-: ingest, bunny-api-logs, cloudfront-log-index, consent-sqs-ingest-aws, consent-sqs-ingest-eu, consent-sqs-ingest-edge, consent-clickhouse-ingest.
  • A CPU and a memory alarm per service, named <service> - CPU over <n>% and <service> - Memory over <n>%.
  • The edge consent queue alarms from the sqs-queue monitoring module, and Consent log messages below threshold (aws network).

atrax

The crawler workers, in eu-west-1. There is no load balancer.

What deploys it. deploy-prod.yml in atrax, on release: published. SHA-pinned.

Teams card. ✅ atrax → prod released or ❌ atrax → prod release failed.

Checks.

  1. The rollout is COMPLETED with running equal to desired (4 in Terraform).
  2. The image on the running revision ends in the release commit SHA.
  3. A scan started from https://www.cookiechecker.com/ finishes and shows a report. The scan flow crosses atrax and atrax-api. The exact scan to use is to confirm.

Expect a short gap while it rolls. The service deploys with minimum healthy 0, so all old tasks stop before new ones start. A scan running during the deploy can fail. Do not read that as a bad release.

Version. Cluster prod-euw1-atrax-ecs-cluster, service atrax-node, family prod-euw1-atrax-atrax-node, region eu-west-1.

Rollback decision point. Go to Rolling back when the service does not reach its desired count, or fresh scans fail after the rollout is COMPLETED.

A healthy minute.

  • Log group /ecs/prod-euw1-atrax-atrax-node.
  • The eu-west-1 cluster and instance alarms, which route to the PagerDuty-Atrax topic.

atrax-api

The control API for the crawler.

What deploys it. deploy-prod.yml in atrax-api, on release: published. SHA-pinned.

Teams card. ✅ atrax-api → prod released or ❌ atrax-api → prod release failed.

Checks.

  1. GET https://atrax-api.cookiehub.net/health answers 200 with {"status":"ok"}.
  2. GET https://atrax-api.cookiehub.net/ answers 200 with {"message":"Atrax API"}.
  3. The rollout is COMPLETED and the image ends in the release commit SHA.

Version. Cluster prod-euc1-core-ecs-cluster, service atrax-api, family prod-euc1-atrax-atrax-api, region eu-central-1.

Rollback decision point. Go to Rolling back when /health is not 200 two minutes after the rollout completes.

A healthy minute.

  • Log group /ecs/prod-euc1-atrax-atrax-api.
  • Terraform declares no service alarm for atrax-api. Only the prod-euc1-core-ecs-cluster alarms cover it.

admin

The admin portal, behind Entra single sign-on.

What deploys it. deploy.yml in admin, on release: published (the job deploy-production). The same file deploys stage on a push to main. A workflow_dispatch on a tag ref redeploys production from that tag. SHA-pinned.

Teams card. None. The workflow has no notify step. Whether that is intended is to confirm. Watch the Actions run instead.

Checks.

  1. GET https://cha.cookiehub.net/ answers 302 with a Location on login.microsoftonline.com. The load balancer does the sign-in redirect, so this proves the listener rule and the host are up. It does not prove the app is.
  2. Sign in with SSO and load the home page. Expect the page and your name.
  3. The rollout is COMPLETED and the image ends in the release commit SHA.

The app's own probe is /api/health, which answers 200 with {"status":"ok"}. The load balancer checks it. An unauthenticated request from outside is redirected to sign-in, so read it through the target group state, not with curl.

Version. Cluster prod-euc1-core-ecs-cluster, service admin-portal, family prod-euc1-core-admin-portal, region eu-central-1.

Rollback decision point. Go to Rolling back when sign-in fails, or the target-group alarm below fires. admin is the one service you can roll back with gh workflow run deploy.yml --ref <older tag>.

A healthy minute.

  • Log group /ecs/prod-euc1-core-admin-portal.
  • Alarms admin-portal - CPU over <n>%, admin-portal - Memory over <n>% and the public ALB cha - Unhealthy hosts >= <n>.

docs

The public documentation site on Vercel.

What deploys it. deploy-prod.yml in docs, on release: published. A quality job runs first, then the gated deploy job builds and deploys with the Vercel CLI. Vercel's own Git builds are off.

Teams card. None. The workflow has no notify step. Whether that is intended is to confirm.

Checks.

  1. GET https://docs.cookiehub.com/ answers 200 with the HTML home page.
  2. Open a page changed in this release. Expect the new content.
  3. In the Vercel project, the newest production deployment is Ready and built from the release commit. The Vercel project name is to confirm.

Version. The commit of the production deployment in Vercel. There is no version string on the site.

Rollback decision point. Go to Rolling back, the Vercel section, when the home page is not 200 or the changed page is wrong.

cookiechecker

The public cookie checker site on Vercel.

What deploys it. deploy-prod.yml in cookiechecker, on release: published. A quality job, a gated deploy job, and a separate notify job.

Teams card. ✅ cookiechecker → production released or ❌ cookiechecker → production release failed. This is the one card that says production, not prod.

Checks.

  1. GET https://www.cookiechecker.com/ answers 200 with the HTML home page. The bare domain answers 308 to the www host.
  2. Start a scan of a public site. Expect live progress and then a report.
  3. In the Vercel project, the newest production deployment is Ready and built from the release commit. The Vercel project name is to confirm.

Version. The commit of the production deployment in Vercel. There is no version string on the site.

Rollback decision point. Go to Rolling back, the Vercel section, when the home page is not 200 or a scan fails to produce a report.

widget

The consent widget. Production is a static publish to S3 plus a version registered with core. No service restarts.

What deploys it. release.yml in widget, on release: published. Three jobs: build, a gated deploy, and notify. The deploy checks the publish guard, syncs dist/ to s3://cookiehub-release/widget-releases/<version>/, uploads manifest.json last, and registers the version with core (POST /api/widget/v1/releases, 201 for a new version).

Teams card. ✅ widget → prod released or ❌ widget → prod release failed. The notify job always runs, so a failure upstream still posts a card.

What a release changes. The registered version is not visible until someone flips it in the dashboard, and a domain only serves a new version after it is upgraded and republished. So right after a release, the bundles customers load are still the old version. That is expected. Checks 1 and 2 prove the publish. Check 3 proves nothing regressed.

Checks.

  1. The manifest exists and names the release commit:
aws s3 cp s3://cookiehub-release/widget-releases/2.10.0/manifest.json -

Expect {"version":"2.10.0","sha":"<release commit>","run_id":"...","published_at":"..."}. The version has no v.

  1. The workflow's Register version step logged Version registration successful (HTTP 201) (200 on a same-commit retry). A 409 fails the job.
  2. A known sandbox domain still serves. GET https://cdn.cookiehub.eu/c2/54cd6cf3.js (the page at sandbox.cookiehub.com/inline) answers 200 with content-type: text/javascript. The bundle starts with window.__cookiehub and contains 'version': 'x.y.z'. Also check 2bfd65da (the /inline-tcf page). Both pages load the bundle from cdn.cookiehub.eu, as the sandbox repo shows.
curl -s https://cdn.cookiehub.eu/c2/54cd6cf3.js | grep -m1 "'version'"

Expect the version those domains were on before the release. After you upgrade and republish a sandbox domain from the dashboard, expect the new version string here.

Version. The tag, minus the v, is the version. It shows in manifest.json and in the bundle's 'version' field.

Rollback decision point. The widget has no deploy rollback by design. If the manifest is missing, the sha differs, or registration failed, do not re-run: a published build is never replaced, so cut the next build number. If a domain serves a bad bundle, follow Rolling back, the widget section and repoint the domains to the older widget_version_id.

A healthy minute. Terraform declares no log group or alarm for the widget path.