Skip to content

Incident response

What to do when production is broken: who hears about it, how bad it is, what to look at in the first ten minutes, how to roll back, when to use break-glass, who tells customers, and how to learn from it afterwards.

Facts on this page come from the Terraform in infra, the production deploy workflows and the other runbooks, read on 12 November 2026. Where the team has not agreed a rule, the page says to decide. Where a fact could not be established read-only, it says to confirm. Nothing here is an invented rota.

Required access

Reading this page needs nothing. Acting on it needs:

  • AWS: read-only access to the production account (759286286879): ecs:Describe*, ecs:List*, logs:FilterLogEvents, logs:StartLiveTail or logs:GetLogEvents, cloudwatch:DescribeAlarms, elasticloadbalancing:Describe*, cloudfront:Get*, ssm:SendCommand and ssm:GetCommandInvocation on the core instance. Region eu-central-1, except atrax in eu-west-1.
  • Rollback and break-glass: ecs:UpdateService and ecs:RegisterTaskDefinition on the service, and for break-glass a GitHub organisation admin. Both are covered in Rollback and Break-glass.
  • Secrets / SSM: none for the checks on this page. Every request is a plain GET.
  • VPN: the Client VPN, for vault-api only (internal hostname). See VPN Access.
  • GitHub: read access to the Actions runs of the affected repo, and the Teams channels listed below.

1. Who is paged and how

There is no pager tool and no on-call rota. Be plain about this. What exists today:

What Where it goes Source
Production deploy result cards The prod deploys Teams channel (TEAMS_WEBHOOK_DEPLOY_PROD) The deploy workflows
Stage deploy result cards The stage deploys Teams channel (TEAMS_WEBHOOK_DEPLOY_STAGE) The deploy workflows
CI failure cards The CI Teams channel (TEAMS_WEBHOOK_CI) Scheduled infra drift plan, and the scheduled-job failure notifier
Scheduled-job failures The CI Teams channel, through a notifier Lambda modules/core/scheduled-jobs (IS-200), declared on stage only so far
CloudWatch alarms, production eu-central-1 The SNS topic named PagerDuty environments/prod/eu-central-1/monitoring.tf
CloudWatch alarms, production eu-west-1 (atrax) The SNS topic named PagerDuty-Atrax environments/prod/eu-west-1/monitoring.tf

Three things follow.

  1. Alarms do not post to Teams. They publish to two SNS topics that were created outside Terraform. Terraform references them by name only, so who or what is subscribed is to confirm in the SNS console. The names suggest a paging service. Do not assume a person's phone rings.
  2. Teams cards are not alerts. The deploy cards say a release finished or failed. The CI channel gets failures of scheduled work. None of them means a customer-facing check is red.
  3. A missing card is a signal. If a release was expected and no card arrived, open the Actions run. See A missing Teams card.

To decide:

  • Who is on call, in what hours, and how the rota is published.
  • Who is reached first when an alarm fires, and by what channel (phone, Teams, both).
  • Whether alarms should post to a Teams channel, and which one.
  • Who takes over if the first person does not answer, and after how long.

Until those are decided, whoever sees an alarm or a failed card owns the first response and posts in the Teams channel they saw it in (see Communication).

2. Severity levels

Pick the level from what customers and their visitors experience, not from how the problem feels. If unsure, pick the higher level and lower it later.

Level Meaning Services that sit in it Response
S1 Customer-facing outage of the widget CDN, or consent writes are failing cdn.cookiehub.eu (Bunny pull zone over the public cookiehub-cdn bucket); edge-api and its CloudFront distribution (edge.cookiehub.net, region-eu.cookiehub.net, consent-eu.cookiehub.net) Acknowledge: to decide. Update cadence: to decide. Target recovery: to decide
S2 Dashboard or API degraded core (dashboard on the EC2 box, dash.cookiehub.com); core-api and core-portal once live; admin-portal; atrax-api; vault-api; dsar-api once live Acknowledge: to decide. Update cadence: to decide. Target recovery: to decide
S3 A single feature is broken, the rest works atrax crawls and the cookiechecker scan; vault-ingest workers (analytics lag, not loss, while the queues hold); scheduled jobs; SES sending; docs and cookiechecker sites Acknowledge: to decide. Fix: next working day unless to decide says otherwise
S4 Cosmetic Copy, layout and styling anywhere No incident. File a Linear issue

Notes on placing a problem:

  • Why edge-api is S1. It receives the consent events the widget sends. Edge ingestion is high volume, so waiting costs data. The widget keeps showing its banner while edge-api is down, but the consent log stops growing.
  • A widget that loads is not a widget that works. Customer domains serve bundles pinned to a widget_version_id. A fresh release does not change them until a domain is upgraded and republished. A bad bundle on one domain is S3 for that domain, and S1 only if the bundle path itself is down.
  • core-api and core-portal are not serving traffic today. The production Terraform holds core-api at desired_count = 0 and declares no core-portal service. Today the dash hostname is the legacy Laravel box. Re-rate when the flip happens.
  • Data loss and security issues are S1 or S2 whatever the user-visible effect. Who decides, and whether a security incident has its own path, is to decide.

3. The first ten minutes

The aim is to answer three questions: what changed, what is failing, and is it getting worse? Do not fix anything yet. Read first.

Every incident

  1. Say you are on it. Post one line in the Teams channel where the problem was reported: what you see and that you own the first response. Open a Linear issue on the owning team right away, even if empty. Its identifier is the incident's record for break-glass and the review.
  2. What changed last? Open the prod deploys Teams channel. Read the latest card for the affected repo: the version, who published it, and the View deploy logs link. A release in the last hour is the first suspect. If there is none, check whether infra was applied (ask the person who applies).
  3. Check the alarms. List the ones in ALARM state:
aws cloudwatch describe-alarms --state-value ALARM --region eu-central-1 \
  --query 'MetricAlarms[].{name:AlarmName,since:StateUpdatedTimestamp,reason:StateReason}'
aws cloudwatch describe-alarms --state-value ALARM --region eu-west-1 \
  --query 'MetricAlarms[].{name:AlarmName,since:StateUpdatedTimestamp,reason:StateReason}'
  1. Run the service's health URL from the table below, or the check in the post-deploy checklist.
  2. Decide the severity and say it in the channel.

Which service is it

All names are from the production Terraform and the post-deploy checklist. Region is eu-central-1 unless noted.

Service Cluster Service Log group Health URL
edge-api prod-euc1-edge-ecs-cluster edge-api /ecs/prod-euc1-edge-edge-api https://edge.cookiehub.net/health and /ready
core-api prod-euc1-core-ecs-cluster core-api /ecs/prod-euc1-core-core-api /client/health (hostname to confirm; service is at zero tasks)
admin-portal prod-euc1-core-ecs-cluster admin-portal /ecs/prod-euc1-core-admin-portal https://cha.cookiehub.net/ redirects to sign-in
atrax-api prod-euc1-core-ecs-cluster atrax-api /ecs/prod-euc1-atrax-atrax-api https://atrax-api.cookiehub.net/health
vault-api prod-euc1-vault-ecs-cluster vault-api /ecs/prod-euc1-vault-vault-api https://vault-api.internal.cookiehub.net/health (VPN)
atrax (eu-west-1) prod-euw1-atrax-ecs-cluster atrax-node /ecs/prod-euw1-atrax-atrax-node none

vault-ingest runs seven services in prod-euc1-vault-ecs-cluster, with no HTTP endpoint. Their names and log groups are in the post-deploy checklist. dsar-api is declared in the core cluster at zero tasks. core-portal has no production service yet.

An ECS service

Replace the placeholders with the row above.

# 1. Is it running, and did a rollout settle?
aws ecs describe-services --cluster <cluster> --services <service> --region <region> \
  --query 'services[0].{desired:desiredCount,running:runningCount,pending:pendingCount,td:taskDefinition,deployments:deployments[].{state:rolloutState,td:taskDefinition,created:createdAt}}'

# 2. Why did tasks stop or fail to start? Newest events first.
aws ecs describe-services --cluster <cluster> --services <service> --region <region> \
  --query 'services[0].events[:10].[createdAt,message]' --output text

# 3. Application logs. Follow live, or read the last 30 minutes.
aws logs tail <log-group> --region <region> --since 30m --follow
aws logs tail <log-group> --region <region> --since 30m --filter-pattern '?ERROR ?error ?Error'

Read the result like this:

  • running below desired with repeated "unable to place" or "stopped" events: tasks are crashing or the cluster has no capacity. The core cluster is EC2-backed, so capacity is a real cause.
  • A second deployment that is IN_PROGRESS or FAILED: a rollout is stuck. That points at the last release.
  • Running at the right count but the health URL fails: read the log group, then check the load balancer target health:
aws elbv2 describe-target-health --region eu-central-1 \
  --target-group-arn <arn from `aws elbv2 describe-target-groups --names <name>`>

For the image the service is running, see Reading the version of an ECS service.

  1. GET https://edge.cookiehub.net/health and /ready. A 503 with "status":"degraded" names the dependency that is down: sqs or clickhouse.
  2. Run the ECS commands above with prod-euc1-edge-ecs-cluster and edge-api. Desired count is 2.
  3. Look at the two alarms edge-api - 5xx over 25/min and edge-api - Unhealthy hosts >= 1 (1 min).
  4. CloudFront sits in front. Check the distribution is enabled and deployed:
aws cloudfront list-distributions \
  --query "DistributionList.Items[?Comment=='edge.cookiehub.net — edge-api'].{id:Id,status:Status,enabled:Enabled}"
  1. edge-api runs against :latest. A new image pushed to the ECR repository edge-api moves the running image at the next task start, whatever was released.

core on the EC2 box (S2)

The instance is cs150p, i-05301a95c43178747, in eu-central-1. It is not managed by Terraform and Cloudflare fronts it, not a load balancer. Reach it over SSM SendCommand, as the deploy workflow does. Do not open an interactive shell unless you must.

  1. GET https://dash.cookiehub.com/login should answer 200. If it answers a Cloudflare error, the problem is between Cloudflare and the box. If it answers 5xx from the app, read the box.
  2. Read the box. Each call needs two commands: send, then fetch the output.
CMD=$(aws ssm send-command --instance-ids i-05301a95c43178747 --region eu-central-1 \
  --document-name AWS-RunShellScript \
  --parameters 'commands=["uptime","free -m","df -h /","systemctl is-active apache2","tail -n 50 /home/cookiehub/core/storage/logs/laravel.log"]' \
  --query 'Command.CommandId' --output text)
aws ssm get-command-invocation --instance-id i-05301a95c43178747 --region eu-central-1 \
  --command-id "$CMD" --query 'StandardOutputContent' --output text

The Laravel log path is the framework default under the checkout the deploy updates (/home/cookiehub/core). Whether production logs somewhere else is to confirm. Terraform declares no CloudWatch log group for core. 3. Check the one Terraform alarm for the box, cs150p - CPU over 80%. 4. Read which tag is deployed, with the VERSION command in the post-deploy checklist. If it shows the tag that was just released and the login is failing, the release is the first suspect.

The CDN and the widget (S1)

Two different things carry the name CDN.

  • Widget bundles on cdn.cookiehub.eu. Production is a Bunny pull zone that pulls from the public cookiehub-cdn S3 bucket. It is not CloudFront. Terraform declares no log group or alarm for this path. Check:
curl -sI https://cdn.cookiehub.eu/c2/54cd6cf3.js | head -5
curl -s https://cdn.cookiehub.eu/c2/54cd6cf3.js | grep -m1 "'version'"

Expect 200 and content-type: text/javascript. 54cd6cf3 is the sandbox domain used in the post-deploy checklist. If it fails, check whether the bucket object exists and whether the pull zone is serving. The Bunny dashboard login is to confirm as a requirement. - edge.cookiehub.net on CloudFront. See the section above. This is the consent-write path.

A widget bundle that serves but is wrong is a different problem. See the widget rollback: repoint the domain to an older widget_version_id and republish.

Other services

vault-ingest, atrax and vault-api follow the ECS commands above. docs and cookiechecker are on Vercel: check the newest production deployment in the Vercel project.

4. Rollback

Roll back when a release caused the incident and a fix forward is slower than going back. The per-service mechanics are in Rolling back. This page does not repeat them. In short:

What Rollback Notes
ECS service (core-api, edge-api, vault-api, vault-ingest, atrax, atrax-api, admin-portal) aws ecs update-service --cluster <cluster> --service <service> --task-definition <family>:<older revision> --region <region> Break-glass: no approval gate, only CloudTrail records it. A rollback is not yet proven end to end (IS-135): ECR repositories are mutable, so the image behind an old revision may have been overwritten. Verify after.
ECS service, through the pipeline Re-run the earlier release's deploy run from the Actions UI (admin also accepts gh workflow run deploy.yml --ref <older tag>) Needs a reviewer approval. A re-run rebuilds the image. Works for 30 days after the original run. Slower than the command above.
edge-api As above, but the running image tracks :latest, so a task-definition revision does not pin it Re-run the earlier release's deploy to push the old build as :latest.
core Re-run the earlier release's deploy-prod.yml run on the previous tag, within 30 days, and approve it Restores PHP code and dashboard assets only. It does not revert migrations, compiled widget bundles or CDN objects. Old code meets a newer schema.
Dashboard cutover (core-portal serving dash) The listener-rule change in Dashboard Cutover Rollback One load-balancer change. Laravel stays deployed. Do not improvise beyond that page.
Widget Repoint domains to the older widget_version_id and republish There is no deploy rollback by design.
docs, cookiechecker Promote the previous deployment in Vercel, or re-run the earlier deploy

Find the older revision:

aws ecs list-task-definitions --family-prefix <family> --sort DESC --max-items 5 --region <region>

After any rollback, run the service's section of the post-deploy checklist to prove the old version is serving, and keep the incident's Linear issue open.

A reviewer approval is the normal gate on a re-run. If waiting for one is worse than the risk, that is break-glass.

5. Break-glass

Break-glass is for an urgent production change when the normal approval cannot happen in time. The wording below is from Branching and Releases.

It should be possible, but not easy. There is no standing bypass. Every production environment is created with can_admins_bypass: false, so using break-glass means an admin deliberately flipping that boolean — which the org audit log records.

Operations Security Procedure 3.1.4 permits an urgent change to skip peer review only if it is peer-reviewed after the fact. So the bypass is not finished when the deploy lands: it is finished when the other approver has reviewed it and recorded that on the incident's Linear issue.

How to use it:

  1. Prefer the smaller step. A task-definition rollback needs AWS access and no environment bypass. Flip can_admins_bypass only if a gated workflow must run and no second reviewer can approve it.
  2. Say so first. Post in the Teams channel and on the incident's Linear issue that you are using break-glass, why, and which environment.
  3. An organisation admin flips the boolean for the one repo's production environment: repo Settings, Environments, production, then allow administrators to bypass protection rules. Through the API it is the environment's can_admins_bypass field. Only an admin can do it, and the change appears in the organisation audit log.
  4. Make the change, then flip it back to false straight away. Do not leave the bypass open.
  5. Record it on the incident's Linear issue: who flipped it, when, which repo, the run that used it, and when it was closed.
  6. Post-facto review. The other production reviewer reviews the change and writes the review on the same issue. The break-glass is not complete until then.

Notes:

  • Self-approval is blocked, not tolerated. The person who cuts a release cannot approve its deployment (prevent_self_review: true). With two production reviewers, the other one approves.
  • Environments have no bypass_actors list, only the can_admins_bypass boolean. Do not plan around naming an actor on an environment.
  • Who in the organisation holds the admin role, and who is the second reviewer when both production reviewers are involved in the incident, is to decide.

6. Communication

Internal. Post updates in the Teams channel where the incident started. Say what you know, what you are doing and when the next update comes.

  • First post within the first ten minutes: severity, symptom, owner.
  • Then one update at every change, and at least every to decide minutes for S1 and S2.
  • Post when the incident is resolved, with the time and a pointer to the Linear issue.

Customers. There is no status page named in the repos or Terraform. What to tell customers, through which channel, and who sends it are all to decide:

  • Channel (status page, email, in-dashboard banner, support replies).
  • Owner of the message (a role, such as the person leading support).
  • Which severities trigger a customer message.
  • Wording rules: say what customers see, what is being done and when the next update is. Do not guess a cause.

Until decided, the incident owner tells support what is happening, and support answers the customers who write in.

7. Post-incident review

Write one for every S1 and S2, and for any break-glass. Write it within a few working days, while people still remember.

A review is a Linear issue on the owning team: Infrastructure & Security for infrastructure and deploy causes, Product & Engineering for product code. Link it to the incident's issue. Give it the label regression when a change broke something that worked, or maintenance when it was ageing, capacity or missing upkeep. Labels are single-select, so use one. Blameless: describe what happened and how the system let it, not who erred.

Template for the issue description:

## Summary
One sentence: what broke, for whom, for how long.

## Timeline
All times in UTC.
- hh:mm  First sign (alarm, card, customer report)
- hh:mm  Acknowledged by (role)
- hh:mm  Severity set to S?
- hh:mm  Cause identified
- hh:mm  Mitigation applied (rollback, fix, break-glass)
- hh:mm  Verified recovered (which checks)

## Impact
Who and what was affected, and how many. Any data lost or delayed.

## Cause
What changed or failed, and why it was not caught earlier.

## What worked
Alarms, runbooks, checks and decisions that helped.

## Actions
Each action is its own Linear issue with an owner:
- [ ] Prevent
- [ ] Detect
- [ ] Respond

If break-glass was used, the post-facto review from Break-glass goes on the incident's issue as well.