Incident response¶
What to do when production is broken: who hears about it, how bad it is, what to look at in the first ten minutes, how to roll back, when to use break-glass, who tells customers, and how to learn from it afterwards.
Facts on this page come from the Terraform in infra, the production deploy workflows and the other
runbooks, read on 12 November 2026. Where the team has not agreed a rule, the page says to decide. Where a
fact could not be established read-only, it says to confirm. Nothing here is an invented rota.
Required access¶
Reading this page needs nothing. Acting on it needs:
- AWS: read-only access to the production account (
759286286879):ecs:Describe*,ecs:List*,logs:FilterLogEvents,logs:StartLiveTailorlogs:GetLogEvents,cloudwatch:DescribeAlarms,elasticloadbalancing:Describe*,cloudfront:Get*,ssm:SendCommandandssm:GetCommandInvocationon the core instance. Regioneu-central-1, exceptatraxineu-west-1. - Rollback and break-glass:
ecs:UpdateServiceandecs:RegisterTaskDefinitionon the service, and for break-glass a GitHub organisation admin. Both are covered in Rollback and Break-glass. - Secrets / SSM: none for the checks on this page. Every request is a plain
GET. - VPN: the Client VPN, for
vault-apionly (internal hostname). See VPN Access. - GitHub: read access to the Actions runs of the affected repo, and the Teams channels listed below.
1. Who is paged and how¶
There is no pager tool and no on-call rota. Be plain about this. What exists today:
| What | Where it goes | Source |
|---|---|---|
| Production deploy result cards | The prod deploys Teams channel (TEAMS_WEBHOOK_DEPLOY_PROD) |
The deploy workflows |
| Stage deploy result cards | The stage deploys Teams channel (TEAMS_WEBHOOK_DEPLOY_STAGE) |
The deploy workflows |
| CI failure cards | The CI Teams channel (TEAMS_WEBHOOK_CI) |
Scheduled infra drift plan, and the scheduled-job failure notifier |
| Scheduled-job failures | The CI Teams channel, through a notifier Lambda | modules/core/scheduled-jobs (IS-200), declared on stage only so far |
CloudWatch alarms, production eu-central-1 |
The SNS topic named PagerDuty |
environments/prod/eu-central-1/monitoring.tf |
CloudWatch alarms, production eu-west-1 (atrax) |
The SNS topic named PagerDuty-Atrax |
environments/prod/eu-west-1/monitoring.tf |
Three things follow.
- Alarms do not post to Teams. They publish to two SNS topics that were created outside Terraform. Terraform references them by name only, so who or what is subscribed is to confirm in the SNS console. The names suggest a paging service. Do not assume a person's phone rings.
- Teams cards are not alerts. The deploy cards say a release finished or failed. The CI channel gets failures of scheduled work. None of them means a customer-facing check is red.
- A missing card is a signal. If a release was expected and no card arrived, open the Actions run. See A missing Teams card.
To decide:
- Who is on call, in what hours, and how the rota is published.
- Who is reached first when an alarm fires, and by what channel (phone, Teams, both).
- Whether alarms should post to a Teams channel, and which one.
- Who takes over if the first person does not answer, and after how long.
Until those are decided, whoever sees an alarm or a failed card owns the first response and posts in the Teams channel they saw it in (see Communication).
2. Severity levels¶
Pick the level from what customers and their visitors experience, not from how the problem feels. If unsure, pick the higher level and lower it later.
| Level | Meaning | Services that sit in it | Response |
|---|---|---|---|
| S1 | Customer-facing outage of the widget CDN, or consent writes are failing | cdn.cookiehub.eu (Bunny pull zone over the public cookiehub-cdn bucket); edge-api and its CloudFront distribution (edge.cookiehub.net, region-eu.cookiehub.net, consent-eu.cookiehub.net) |
Acknowledge: to decide. Update cadence: to decide. Target recovery: to decide |
| S2 | Dashboard or API degraded | core (dashboard on the EC2 box, dash.cookiehub.com); core-api and core-portal once live; admin-portal; atrax-api; vault-api; dsar-api once live |
Acknowledge: to decide. Update cadence: to decide. Target recovery: to decide |
| S3 | A single feature is broken, the rest works | atrax crawls and the cookiechecker scan; vault-ingest workers (analytics lag, not loss, while the queues hold); scheduled jobs; SES sending; docs and cookiechecker sites |
Acknowledge: to decide. Fix: next working day unless to decide says otherwise |
| S4 | Cosmetic | Copy, layout and styling anywhere | No incident. File a Linear issue |
Notes on placing a problem:
- Why
edge-apiis S1. It receives the consent events the widget sends. Edge ingestion is high volume, so waiting costs data. The widget keeps showing its banner whileedge-apiis down, but the consent log stops growing. - A widget that loads is not a widget that works. Customer domains serve bundles pinned to a
widget_version_id. A fresh release does not change them until a domain is upgraded and republished. A bad bundle on one domain is S3 for that domain, and S1 only if the bundle path itself is down. core-apiandcore-portalare not serving traffic today. The production Terraform holdscore-apiatdesired_count = 0and declares nocore-portalservice. Today thedashhostname is the legacy Laravel box. Re-rate when the flip happens.- Data loss and security issues are S1 or S2 whatever the user-visible effect. Who decides, and whether a security incident has its own path, is to decide.
3. The first ten minutes¶
The aim is to answer three questions: what changed, what is failing, and is it getting worse? Do not fix anything yet. Read first.
Every incident¶
- Say you are on it. Post one line in the Teams channel where the problem was reported: what you see and that you own the first response. Open a Linear issue on the owning team right away, even if empty. Its identifier is the incident's record for break-glass and the review.
- What changed last? Open the prod deploys Teams channel. Read the latest card for the affected repo: the
version, who published it, and the View deploy logs link. A release in the last hour is the first
suspect. If there is none, check whether
infrawas applied (ask the person who applies). - Check the alarms. List the ones in
ALARMstate:
aws cloudwatch describe-alarms --state-value ALARM --region eu-central-1 \
--query 'MetricAlarms[].{name:AlarmName,since:StateUpdatedTimestamp,reason:StateReason}'
aws cloudwatch describe-alarms --state-value ALARM --region eu-west-1 \
--query 'MetricAlarms[].{name:AlarmName,since:StateUpdatedTimestamp,reason:StateReason}'
- Run the service's health URL from the table below, or the check in the post-deploy checklist.
- Decide the severity and say it in the channel.
Which service is it¶
All names are from the production Terraform and the post-deploy checklist. Region is eu-central-1 unless noted.
| Service | Cluster | Service | Log group | Health URL |
|---|---|---|---|---|
edge-api |
prod-euc1-edge-ecs-cluster |
edge-api |
/ecs/prod-euc1-edge-edge-api |
https://edge.cookiehub.net/health and /ready |
core-api |
prod-euc1-core-ecs-cluster |
core-api |
/ecs/prod-euc1-core-core-api |
/client/health (hostname to confirm; service is at zero tasks) |
admin-portal |
prod-euc1-core-ecs-cluster |
admin-portal |
/ecs/prod-euc1-core-admin-portal |
https://cha.cookiehub.net/ redirects to sign-in |
atrax-api |
prod-euc1-core-ecs-cluster |
atrax-api |
/ecs/prod-euc1-atrax-atrax-api |
https://atrax-api.cookiehub.net/health |
vault-api |
prod-euc1-vault-ecs-cluster |
vault-api |
/ecs/prod-euc1-vault-vault-api |
https://vault-api.internal.cookiehub.net/health (VPN) |
atrax (eu-west-1) |
prod-euw1-atrax-ecs-cluster |
atrax-node |
/ecs/prod-euw1-atrax-atrax-node |
none |
vault-ingest runs seven services in prod-euc1-vault-ecs-cluster, with no HTTP endpoint. Their names and log
groups are in the post-deploy checklist. dsar-api is declared in the
core cluster at zero tasks. core-portal has no production service yet.
An ECS service¶
Replace the placeholders with the row above.
# 1. Is it running, and did a rollout settle?
aws ecs describe-services --cluster <cluster> --services <service> --region <region> \
--query 'services[0].{desired:desiredCount,running:runningCount,pending:pendingCount,td:taskDefinition,deployments:deployments[].{state:rolloutState,td:taskDefinition,created:createdAt}}'
# 2. Why did tasks stop or fail to start? Newest events first.
aws ecs describe-services --cluster <cluster> --services <service> --region <region> \
--query 'services[0].events[:10].[createdAt,message]' --output text
# 3. Application logs. Follow live, or read the last 30 minutes.
aws logs tail <log-group> --region <region> --since 30m --follow
aws logs tail <log-group> --region <region> --since 30m --filter-pattern '?ERROR ?error ?Error'
Read the result like this:
runningbelowdesiredwith repeated "unable to place" or "stopped" events: tasks are crashing or the cluster has no capacity. The core cluster is EC2-backed, so capacity is a real cause.- A second deployment that is
IN_PROGRESSorFAILED: a rollout is stuck. That points at the last release. - Running at the right count but the health URL fails: read the log group, then check the load balancer target health:
aws elbv2 describe-target-health --region eu-central-1 \
--target-group-arn <arn from `aws elbv2 describe-target-groups --names <name>`>
For the image the service is running, see Reading the version of an ECS service.
edge-api and consent writes (S1)¶
GET https://edge.cookiehub.net/healthand/ready. A503with"status":"degraded"names the dependency that is down:sqsorclickhouse.- Run the ECS commands above with
prod-euc1-edge-ecs-clusterandedge-api. Desired count is 2. - Look at the two alarms
edge-api - 5xx over 25/minandedge-api - Unhealthy hosts >= 1 (1 min). - CloudFront sits in front. Check the distribution is enabled and deployed:
aws cloudfront list-distributions \
--query "DistributionList.Items[?Comment=='edge.cookiehub.net — edge-api'].{id:Id,status:Status,enabled:Enabled}"
edge-apiruns against:latest. A new image pushed to the ECR repositoryedge-apimoves the running image at the next task start, whatever was released.
core on the EC2 box (S2)¶
The instance is cs150p, i-05301a95c43178747, in eu-central-1. It is not managed by Terraform and
Cloudflare fronts it, not a load balancer. Reach it over SSM SendCommand, as the deploy workflow does.
Do not open an interactive shell unless you must.
GET https://dash.cookiehub.com/loginshould answer200. If it answers a Cloudflare error, the problem is between Cloudflare and the box. If it answers5xxfrom the app, read the box.- Read the box. Each call needs two commands: send, then fetch the output.
CMD=$(aws ssm send-command --instance-ids i-05301a95c43178747 --region eu-central-1 \
--document-name AWS-RunShellScript \
--parameters 'commands=["uptime","free -m","df -h /","systemctl is-active apache2","tail -n 50 /home/cookiehub/core/storage/logs/laravel.log"]' \
--query 'Command.CommandId' --output text)
aws ssm get-command-invocation --instance-id i-05301a95c43178747 --region eu-central-1 \
--command-id "$CMD" --query 'StandardOutputContent' --output text
The Laravel log path is the framework default under the checkout the deploy updates
(/home/cookiehub/core). Whether production logs somewhere else is to confirm. Terraform declares no
CloudWatch log group for core.
3. Check the one Terraform alarm for the box, cs150p - CPU over 80%.
4. Read which tag is deployed, with the VERSION command in the
post-deploy checklist. If it shows the tag that was just released and the
login is failing, the release is the first suspect.
The CDN and the widget (S1)¶
Two different things carry the name CDN.
- Widget bundles on
cdn.cookiehub.eu. Production is a Bunny pull zone that pulls from the publiccookiehub-cdnS3 bucket. It is not CloudFront. Terraform declares no log group or alarm for this path. Check:
curl -sI https://cdn.cookiehub.eu/c2/54cd6cf3.js | head -5
curl -s https://cdn.cookiehub.eu/c2/54cd6cf3.js | grep -m1 "'version'"
Expect 200 and content-type: text/javascript. 54cd6cf3 is the sandbox domain used in the
post-deploy checklist. If it fails, check whether the bucket object exists
and whether the pull zone is serving. The Bunny dashboard login is to confirm as a requirement.
- edge.cookiehub.net on CloudFront. See the section above. This is the consent-write path.
A widget bundle that serves but is wrong is a different problem. See
the widget rollback: repoint the domain to an older widget_version_id
and republish.
Other services¶
vault-ingest, atrax and vault-api follow the ECS commands above. docs and cookiechecker are on
Vercel: check the newest production deployment in the Vercel project.
4. Rollback¶
Roll back when a release caused the incident and a fix forward is slower than going back. The per-service mechanics are in Rolling back. This page does not repeat them. In short:
| What | Rollback | Notes |
|---|---|---|
ECS service (core-api, edge-api, vault-api, vault-ingest, atrax, atrax-api, admin-portal) |
aws ecs update-service --cluster <cluster> --service <service> --task-definition <family>:<older revision> --region <region> |
Break-glass: no approval gate, only CloudTrail records it. A rollback is not yet proven end to end (IS-135): ECR repositories are mutable, so the image behind an old revision may have been overwritten. Verify after. |
| ECS service, through the pipeline | Re-run the earlier release's deploy run from the Actions UI (admin also accepts gh workflow run deploy.yml --ref <older tag>) |
Needs a reviewer approval. A re-run rebuilds the image. Works for 30 days after the original run. Slower than the command above. |
edge-api |
As above, but the running image tracks :latest, so a task-definition revision does not pin it |
Re-run the earlier release's deploy to push the old build as :latest. |
core |
Re-run the earlier release's deploy-prod.yml run on the previous tag, within 30 days, and approve it |
Restores PHP code and dashboard assets only. It does not revert migrations, compiled widget bundles or CDN objects. Old code meets a newer schema. |
Dashboard cutover (core-portal serving dash) |
The listener-rule change in Dashboard Cutover Rollback | One load-balancer change. Laravel stays deployed. Do not improvise beyond that page. |
| Widget | Repoint domains to the older widget_version_id and republish |
There is no deploy rollback by design. |
docs, cookiechecker |
Promote the previous deployment in Vercel, or re-run the earlier deploy |
Find the older revision:
After any rollback, run the service's section of the post-deploy checklist to prove the old version is serving, and keep the incident's Linear issue open.
A reviewer approval is the normal gate on a re-run. If waiting for one is worse than the risk, that is break-glass.
5. Break-glass¶
Break-glass is for an urgent production change when the normal approval cannot happen in time. The wording below is from Branching and Releases.
It should be possible, but not easy. There is no standing bypass. Every
productionenvironment is created withcan_admins_bypass: false, so using break-glass means an admin deliberately flipping that boolean — which the org audit log records.Operations Security Procedure 3.1.4 permits an urgent change to skip peer review only if it is peer-reviewed after the fact. So the bypass is not finished when the deploy lands: it is finished when the other approver has reviewed it and recorded that on the incident's Linear issue.
How to use it:
- Prefer the smaller step. A task-definition rollback needs AWS access and no environment bypass. Flip
can_admins_bypassonly if a gated workflow must run and no second reviewer can approve it. - Say so first. Post in the Teams channel and on the incident's Linear issue that you are using break-glass, why, and which environment.
- An organisation admin flips the boolean for the one repo's
productionenvironment: repo Settings, Environments,production, then allow administrators to bypass protection rules. Through the API it is the environment'scan_admins_bypassfield. Only an admin can do it, and the change appears in the organisation audit log. - Make the change, then flip it back to
falsestraight away. Do not leave the bypass open. - Record it on the incident's Linear issue: who flipped it, when, which repo, the run that used it, and when it was closed.
- Post-facto review. The other production reviewer reviews the change and writes the review on the same issue. The break-glass is not complete until then.
Notes:
- Self-approval is blocked, not tolerated. The person who cuts a release cannot approve its deployment
(
prevent_self_review: true). With two production reviewers, the other one approves. - Environments have no
bypass_actorslist, only thecan_admins_bypassboolean. Do not plan around naming an actor on an environment. - Who in the organisation holds the admin role, and who is the second reviewer when both production reviewers are involved in the incident, is to decide.
6. Communication¶
Internal. Post updates in the Teams channel where the incident started. Say what you know, what you are doing and when the next update comes.
- First post within the first ten minutes: severity, symptom, owner.
- Then one update at every change, and at least every to decide minutes for S1 and S2.
- Post when the incident is resolved, with the time and a pointer to the Linear issue.
Customers. There is no status page named in the repos or Terraform. What to tell customers, through which channel, and who sends it are all to decide:
- Channel (status page, email, in-dashboard banner, support replies).
- Owner of the message (a role, such as the person leading support).
- Which severities trigger a customer message.
- Wording rules: say what customers see, what is being done and when the next update is. Do not guess a cause.
Until decided, the incident owner tells support what is happening, and support answers the customers who write in.
7. Post-incident review¶
Write one for every S1 and S2, and for any break-glass. Write it within a few working days, while people still remember.
A review is a Linear issue on the owning team: Infrastructure & Security for infrastructure and deploy
causes, Product & Engineering for product code. Link it to the incident's issue. Give it the label
regression when a change broke something that worked, or maintenance when it was ageing, capacity or
missing upkeep. Labels are single-select, so use one. Blameless: describe what happened and how the system let
it, not who erred.
Template for the issue description:
## Summary
One sentence: what broke, for whom, for how long.
## Timeline
All times in UTC.
- hh:mm First sign (alarm, card, customer report)
- hh:mm Acknowledged by (role)
- hh:mm Severity set to S?
- hh:mm Cause identified
- hh:mm Mitigation applied (rollback, fix, break-glass)
- hh:mm Verified recovered (which checks)
## Impact
Who and what was affected, and how many. Any data lost or delayed.
## Cause
What changed or failed, and why it was not caught earlier.
## What worked
Alarms, runbooks, checks and decisions that helped.
## Actions
Each action is its own Linear issue with an owner:
- [ ] Prevent
- [ ] Detect
- [ ] Respond
If break-glass was used, the post-facto review from Break-glass goes on the incident's issue as well.
Related¶
- Post-deploy smoke checklist — what to check after each production release
- Branching and Releases — the release model, rolling back and break-glass
- Dashboard Cutover Rollback — reverting the
dashlistener rule - Deployments — per-service mechanics, names and status commands