Consent Domain Cutover — consent.cookiehub.net¶
Status (as of 2026-10-02):
- ✅ Part 1 — Deploy: executed 2026-04-20. edge-api v3.0.0 live at
edge.cookiehub.net(the first prod release was v3.0.0). Treat Part 1 as history and as a template for standing the stack up elsewhere. - 🟡 Part 2 — Legacy cutover: partly executed.
consent-euandregion-euwere cut over to the new distribution (infra commit3fff670, 2026-07-07). Still open:consent.cookiehub.netandds.cookiehub.net, which remain on the legacy CloudFront distribution. No target date yet. Customer-visible.
This runbook is the playbook for the open part of Part 2. Sections that describe the already-executed steps are kept for reference; the pre-flight checklist and the "Monday" execution plan below were written for the original April window and must be re-validated (dates, queue state, widget defaults) before use.
Who runs the Terraform steps
Every terraform apply in this runbook is run by Ástþór, from an Infrastructure & Security issue and a pull
request he reviews; nobody else applies, and never an assistant.
Terminology note: the identity values were renamed after this runbook was first written. Read edge_token as token, session_proof as token_proof and /v3/session/register as /v3/token.
Why¶
consent.cookiehub.net is the domain end-user browsers hit for consent event logging. It is baked into customer CSP allowlists and cannot be renamed. Today it is served by a CloudFront distribution with two Lambda@Edge functions (/region, /log) that push events into an SQS queue.
Moving this domain to edge-api unlocks:
- Server-issued session identity (
edge_token+ HMACsession_proof) - Straight-line path to ClickHouse via the existing vault SQS ingestion worker
- Retirement of Lambda@Edge (operational simplification)
- Foundation for future features that need a real API surface at the edge
Current state¶
What customers reach today¶
Four customer-facing hostnames, two different backends:
| URL pattern | Backend today | Behaviour |
|---|---|---|
POST consent.cookiehub.net/log |
CloudFront E2P43R3IK343PI → Lambda@Edge consent-log-entry:2 |
Wraps {payload, headers: {country, userAgent, ipAddress}} → ConsentLogQueue SQS |
* consent.cookiehub.net/* |
CloudFront → S3 cookiehub-static-root |
No active content (returns 403 on unknown paths) |
POST ds.cookiehub.net/* |
Same CloudFront E2P43R3IK343PI — alias |
Purpose unknown; kept as a safety alias on the new distribution |
POST consent-eu.cookiehub.net/ |
A record → Scaleway 51.158.227.132 → nginx + PHP index.php |
Same envelope → SQS. Dispatch by request shape, not URL (body → log; cookiehub-regions → region) |
POST region-eu.cookiehub.net/ |
Same Scaleway PHP | Same dispatch |
Not on this scope (confirmed): consent.cookiehub.net/region does not exist on the live distribution — the region Lambda@Edge lives on the separate cookiehub.net CloudFront (EK5O6OG2XQJTS). We don't touch that.
AWS footprint (prod, 759286286879)¶
| Resource | State |
|---|---|
CloudFront E2P43R3IK343PI for consent + ds |
Live, unmanaged by Terraform |
ACM cert (us-east-1) for consent + ds |
Live, unmanaged by Terraform, valid until 2026-08-18 |
Route53 A/ALIAS for consent, ds |
Live — ALIAS → d2f71kw5bfffrp.cloudfront.net |
Route53 A for consent-eu, region-eu |
Live — A → 51.158.227.132 (Scaleway) |
Route53 public zone cookiehub.net |
Live, ID Z039866524J05Y9YKGSI9 |
Lambda@Edge consent-log-entry:2, cors-region:1 |
Live in us-east-1; targets ConsentLogQueue |
SQS ConsentLogQueue |
Live, drained by vault ingest worker |
S3 cookiehub-static-root |
Live origin, no active content in scope |
| Scaleway PHP servers for EU hostnames | Live (not AWS) |
| Prod edge-api, ECS cluster, ECR, public ALB, Terraform-managed ACM certs | Do not exist yet — provisioned by infra PR #20 |
What we already have¶
- Stage edge-api is live at
edge-api.stage.cookiehub.netwith/v3/*+/v1/log+/v1/regionendpoints — see stage edge.tf - Vault SQS-ingest worker on prod already handles both v3 and legacy message formats — vault-ingest@4635498
- Widget 2.9 ready and stage-smoke verified — widget PR #178
Target state¶
flowchart LR
User[End user<br/>browser] --> CF[New CloudFront<br/>edge + consent + ds + consent-eu + region-eu]
CF --> ALB[Prod public ALB<br/>internet-facing]
ALB --> API[edge-api<br/>ECS service]
API --> SQS[SQS<br/>prod-euc1-edge-consent]
SQS --> Vault[Vault ingest → ClickHouse]
API --> CH[(ClickHouse<br/>read-only)]
edge.cookiehub.netis the canonical customer-facing hostname going forward (widget 2.9+ default)- Same CloudFront distribution serves the four legacy hostnames (consent / ds / consent-eu / region-eu) on the same ACM cert, so customer-pinned CSP allowlists keep working
- Default behavior forwards everything (including root
/) to edge-api — consent-eu / region-eu customers POST to/without path suffix - edge-api handles
/log,/region,/v1/*,/v3/*, and the/dispatcher - Lambda@Edge functions removed after the 48h grace period
- Old SQS
ConsentLogQueuedrained and retired - New SQS
prod-euc1-edge-consentreceives all consent events - Scaleway PHP servers decommissioned after EU customer traffic has converged
Rollout strategy — two independent operations¶
Split into:
- Deploy (today's PR) — stand up the new edge-api stack aliased only at
edge.cookiehub.net. Zero impact on customer-facing legacy traffic. The ACM cert and ALB listener rule already cover the legacy names so the stack is pre-wired for cutover, it just doesn't have them attached yet. - Cutover (later, separate PR) — attach the four legacy hostnames (
consent,ds,consent-eu,region-eu) to the new CloudFront distribution and flip their Route53 records. Customer-impacting. Run when ready.
flowchart TB
subgraph Deploy["Part 1 — Deploy (executed)"]
D1[terraform apply] --> D2[SSM values, first release, ECS healthy]
D2 --> D3[Smoke edge.cookiehub.net directly]
end
Deploy --> Later
subgraph Later[" "]
Ready[Stack live, serving edge.cookiehub.net only.<br/>Legacy hostnames unchanged.]
end
Later --> Cutover
subgraph Cutover["Part 2 — Cutover (separate PR, later)"]
C0[Phase 0 — Strip consent/ds aliases<br/>from legacy CloudFront] --> C1[Phase 1 — terraform apply cutover PR<br/>adds 4 aliases + Route53 overwrite]
C1 --> C2[Phase 2 — Smoke legacy hostnames]
C2 --> C3[Phase 3 — Monitor queue drain 24-48h]
C1 -.->|regression| CR[Re-add legacy aliases<br/>+ Route53 UPSERTs + revert PR]
C3 --> CC[Delete legacy CF, Lambda@Edge, queues, Scaleway]
end
Why split:
- Deploy risk is isolated — nothing customer-visible changes from the apply, so rollback is a simple
terraform destroyif we ever need one - Cutover can be scheduled separately (low-traffic window, with on-call ready, etc.) without the pressure of a day-of infra stand-up
- The stack can be exercised with real prod traffic via
edge.cookiehub.net(e.g. widget 2.9 release) before the cutover forces it on all legacy customers
Pre-flight checklist¶
Complete every item before the cutover window.
App changes¶
- [x] edge-api: expose
/logand/regionat root (in addition to/v1/logand/v1/region). The legacy CloudFront distribution routes/logdirectly to the origin — edge-api must answer the exact path customers hit. — edge-api@5739fab - [x] Unit test for the root-level routes, matching the
/v1/coverage — 5 new tests in tests/legacy.test.ts; OpenAPI spec now lists/log,/region,/v1/log,/v1/regionindependently - [x] Root dispatcher (
/) forconsent-eu.cookiehub.net+region-eu.cookiehub.net— pre-2.9 EU widgets POST to/(no path). Dispatches by request shape like the Scaleway PHPindex.php: body → consent log,cookiehub-regionsheader → region, else empty success. — edge-api PR #1 - [ ] Stage deploy + smoke:
curl -X POST https://edge-api.stage.cookiehub.net/log …returns the same response shape as the Lambda
Prod infrastructure (Terraform, cookiehub-com/infra)¶
Draft PR: infra#20 — branch feat/prod-edge-api. Mirrors stage/edge.tf and reuses existing modules (modules/edge/ecr, modules/edge/ecs, modules/edge/edge-api, modules/edge/consent-queue).
- [x]
module "edge_consent_queue"— createsprod-euc1-edge-consent(new queue, not the legacyConsentLogQueue) - [x]
module "edge_ecr" - [x]
module "edge_ecs"—t3.medium × 2, revisit after first day of prod traffic - [x]
module "edge_api"wired to new prod public ALB, public Route53 zone (cookiehub.net), prod ClickHouse security group, and the new SQS queue.create_dedicated_hostname = false+host_headersset to the four consent hostnames — prod does not provision a separateedge-api.cookiehub.netCloudFront/DNS. - [x] Module refactor supporting the above: new
host_headersandcreate_dedicated_hostnamevariables, both backwards-compatible with stage defaults - [x] New public ALB in prod with SG locked to CloudFront managed prefix list (prod previously had no public ALB); two new public subnets (public_b, public_c) added in AZ b/c for ALB HA
- [x] New us-east-1 ACM cert covering
consent.cookiehub.net+ds.cookiehub.net+consent-eu.cookiehub.net+region-eu.cookiehub.netfor the new CloudFront distribution - [x] New CloudFront distribution with single ALB origin and default-behavior forwarding (so consent-eu's root-path POSTs work). Aliases gated behind
var.consent_aliases_enabled(defaultfalse) — zero DNS change interraform planuntil the Monday flip. - [x] Wire the new queue into the prod vault ETL: added
edgeinstance toetl_consent_sqs_ingestinstancesmap in prod/eu-central-1/vault.tf - [ ] SSM params under
/edge/prod/edge-api/*—HMAC_SECRET(generated fresh, not copied from stage),CLICKHOUSE_URL,CLICKHOUSE_USERNAME,CLICKHOUSE_PASSWORD,CLICKHOUSE_DATABASE,SQS_QUEUE_URL. Values set manually post-apply (lifecycle ignore_changes). - [ ] GitHub OIDC deploy role (
gh-oidc-deploy-edge-api) in prod account — created by theedge_apimodule; verify on apply - [ ]
terraform planagainst prod state (needsprod-admin+ MFA; run during Friday dry-run) - [ ] Friday dry-run apply (everything except the
consent_aliases_enabled = trueflip)
edge-api deploy pipeline¶
- [x] Prod deploy workflow (edge-api/.github/workflows/deploy-prod.yml) already exists on main. Triggers on
release:published; builds image, pushes to prod ECR, force-deploys ECS. Targets match what Terraform creates (prod-euc1-edge-ecs-cluster, serviceedge-api, task familyprod-euc1-edge-edge-api). - [ ] Set GitHub secret
AWS_DEPLOY_ROLE_ARN_PRODto the ARN of thegh-oidc-deploy-edge-apiIAM role once Terraform has created it in prod - [ ] First prod release cut — verify ECS task healthy,
curl https://<ALB-dns>/healthreturns 200 via a Host header override to consent.cookiehub.net - [ ] Verify CORS from a real customer-origin
fetch()in a browser
Observability¶
- [ ] Grafana dashboard showing SQS queue depth on both
ConsentLogQueueandprod-euc1-edge-consent - [ ] Alert on either queue's ingestion rate dropping to 0 for > 5 minutes during cutover window
- [ ] Alert on edge-api ECS service unhealthy tasks > 0
- [ ] Tail edge-api CloudWatch logs during cutover
Communication¶
- [ ] Maintenance window announced internally (even though no customer-visible downtime is expected)
- [ ] Engineering on standby during cutover and for 2h after
- [ ] Incident response contact list current
Execution plan (written for the original 2026-04-20 window; re-date before use)¶
Dry-run the Terraform applies on Friday. Apply everything except the CloudFront aliases override and the Route53 flip on Friday. That means on Monday the only changes are adding the consent.cookiehub.net alias and updating the Route53 record — minimal surface during the live window.
Monday morning: final validation (08:00–09:30)¶
- Confirm all pre-flight items green
- Hit the new distribution directly via its
*.cloudfront.nethostname with aHost: consent.cookiehub.netoverride:NEW_DIST=d1234567890abc.cloudfront.net # from `terraform output` curl -sS -X POST "https://$NEW_DIST/log" \ -H "Host: consent.cookiehub.net" \ -H "Content-Type: application/json" \ -d '{"a":1,"r":1,"d":1,"u":"https://example.com/","t":"dryrun","s":"[]","n":"vault1","gsvs":{"a":false,"c":[],"v":[],"s":[]},"tcstring":""}' - Verify message lands in
prod-euc1-edge-consent(SQS console → ApproximateNumberOfMessagesReceived) - Verify vault ingest worker consumes it → row in ClickHouse
consent_events(or equivalent) - Verify
GET /regionreturns{r, c, s}with correct country from geo headers
Cutover window (10:00–10:15)¶
- Tail CloudWatch logs on edge-api ECS
- Open SQS console, keep both queue dashboards visible
- Route53 flip: update
consent.cookiehub.netto point at the new CloudFront distribution - Watch DNS propagation (
dig +trace consent.cookiehub.netfrom multiple vantage points) - Within 1–5 minutes, real customer traffic should start hitting the new distribution. Expect:
ConsentLogQueuemessage rate declining toward zeroprod-euc1-edge-consentmessage rate rising to previous baseline- edge-api ECS CPU/memory rising from idle
- No spike in 5xx rates on the ALB
Monitoring (10:15–Tuesday)¶
- Watch for 2h actively; check hourly for the next 24h
- If all healthy: proceed to cleanup after 48h
- If regression: immediately flip Route53 back to the old CloudFront. Both old and new infrastructure should be running unchanged — recovery is just DNS convergence time
Rollback¶
The Route53 flip is the only customer-facing change. To roll back:
- Update
consent.cookiehub.netRoute53 record back to the old CloudFront distribution - DNS TTL window (60s–5min) and customers re-converge on the old stack
- Old Lambda@Edge and
ConsentLogQueueare still operating (unchanged during cutover) - Events received by the new edge-api in between are already durably in SQS → ClickHouse; no data loss
- File incident notes, investigate, fix, retry
Do not delete old infrastructure until the grace period completes successfully.
Post-cutover cleanup (Wednesday or later)¶
After 48h of healthy operation with zero traffic on the legacy path:
- [ ] Delete old CloudFront distribution (or mark disabled for another week as insurance)
- [ ] Delete Lambda@Edge function versions (Lambda@Edge replicas auto-cleanup over 24h; this just removes the source)
- [ ] Drain and delete
ConsentLogQueue - [ ] Remove
ConsentLogQueuefrom the prod vaultetl_consent_sqs_ingestinstancesmap - [ ] Import the new prod edge resources into the infra repo if any were created manually (ideally zero)
- [ ] Update docs-internal/docs/services/edge.md to drop the Lambda@Edge section
- [ ] Close the Linear issue that tracks the remaining
consent/dscutover and the Lambda@Edge / Scaleway decommission
Open questions — resolved¶
- ✅ ACM cert — the existing cert covers only
consent+ds, notconsent-eu/region-eu. The infra PR provisions a new cert covering all four hostnames; we don't import the old one. - ✅ WAF — confirmed
WebACLId: (none)on the live distribution. Not applied manually either. Nothing to mirror. - ✅ Custom response headers — live distribution uses no response headers policy. The
cors-region:1viewer-response Lambda adds CORS headers today; edge-api already setsaccess-control-allow-origin: *itself so this is redundant under the new setup. - ⚠️ CloudFront access log destination — live distribution writes standard logging v1 to
cookiehub-netS3 bucket, prefixconsent-logs/. The new distribution mirrors this exactly so the vaultcloudfront-log-indexETL continues without change. - ⚠️
ds.cookiehub.net— purpose unknown. Preserved as an alias on the new distribution per explicit decision to keep until further notice. - ⚠️
/log2behavior — legacy distribution has a duplicate/log2path; not replicated on the new distribution per decision.
Open questions — still to resolve¶
- Prod ECS instance sizing — draft uses
t3.medium × 2based on stage'st3.small × 1. Revisit after first day of prod traffic (CloudWatch CPU / memory / throughput). - Widget
DEFAULT_EDGE_URL— widget 2.9 currently defaults tohttps://edge-api.cookiehub.net. Needs a flip tohttps://consent.cookiehub.net(or to stay as-is if we want customers to land on the API-shaped hostname). Small edit towidget/src/js/modules/edge-client/edge-client.ts:9.
Related (original)¶
- Widget PR #178 — widget 2.9 integration, held for release until this cutover completes
- docs-internal edge service overview
- docs-internal deployments runbook and Branching and Releases — how edge-api is released today
Command reference¶
Concrete, copy-pasteable commands for the sections above. All of them assume production AWS credentials are already in the shell (prod account 759286286879).
Known constants:
PROD_ACCOUNT=759286286879
PROD_REGION=eu-central-1
LEGACY_DIST_ID=E2P43R3IK343PI # consent.cookiehub.net + ds.cookiehub.net today
SCALEWAY_IP=51.158.227.132 # consent-eu + region-eu today
PUBLIC_ZONE_ID=Z039866524J05Y9YKGSI9 # cookiehub.net hosted zone
NEW_QUEUE_URL=https://sqs.eu-central-1.amazonaws.com/759286286879/prod-euc1-edge-consent
LEGACY_QUEUES="ConsentLogQueue qs313p"
The work is split into two independent operations, Part 1 (deploy the stack at edge.cookiehub.net, executed) and Part 2 (the legacy cutover, partly executed).
Part 1 — Deploy the stack at edge.cookiehub.net¶
✅ Executed 2026-04-20. (Part 2 below is only partly executed: consent-eu and region-eu are cut over; consent and ds are still open.) edge-api v3.0.0 live at edge.cookiehub.net. Commands below are kept as a reference and as the template for similar future deploys (e.g. recreating the stack or bringing up another environment). Some real-world adjustments from the actual run:
- ECS cluster is currently 1 ×
c7i.largewithdesired_count = 1— rolling deploys need more headroom; bump pending in a follow-up PR. - GitHub OIDC trust for
gh-oidc-deploy-edge-apiwas defaulting toref:refs/heads/main. Fix:github_oidc_ref = "ref:refs/tags/*"on the module. Manually applied viaaws iam update-assume-role-policy; Terraform reconcile PR: infra#21. - ClickHouse creds:
vault-api's SSM values (/vault/prod/vault-api/clickhouse_{url,username,password}) worked as-is. Reuse for edge-api rather than inventing new ones. - ClickHouse port from edge ECS: 8123 (HTTP), not 8443 (SG only opens 8123 for edge ECS → ClickHouse).
The current Terraform branch creates the new CloudFront distribution with only edge.cookiehub.net as its alias. The four legacy hostnames keep resolving to the legacy CloudFront / Scaleway IPs unchanged. No pre-apply step needed.
Phase 1 — Terraform apply¶
From infra/environments/prod/eu-central-1, on branch feat/prod-edge-api:
# Sideloaded in 4 batches so each step is reviewable. See the runbook above
# for the full -target list per batch. Or, if confident, one-shot apply:
terraform apply
Expected: ~56 resources added. CloudFront takes ~15 min to deploy globally.
After apply returns:
- New CloudFront exists with only
edge.cookiehub.netaliased; cert SANs cover the four legacy names (ready for cutover) - Route53 has
edge.cookiehub.net→ new CloudFront - ALB listener rule routes any of the five hostnames to edge-api (ready for cutover)
- ECS service exists but tasks are pending (no image in ECR yet)
- SSM params exist with placeholder values
Customer-facing traffic to consent / ds / consent-eu / region-eu is unchanged — still served by legacy CloudFront / Scaleway.
Phase 2 — Post-apply setup¶
2.1 Set real SSM parameter values¶
HMAC_SECRET=$(openssl rand -hex 32)
echo "HMAC_SECRET=$HMAC_SECRET" # save — needed if rotating later
aws ssm put-parameter --name /edge/prod/edge-api/hmac_secret \
--value "$HMAC_SECRET" --type SecureString --overwrite --region $PROD_REGION
# ClickHouse creds — copy from the existing core/prod SSM namespace
aws ssm put-parameter --name /edge/prod/edge-api/clickhouse_url \
--value "https://clickhouse.internal.cookiehub.net:8443" \
--type SecureString --overwrite --region $PROD_REGION
aws ssm put-parameter --name /edge/prod/edge-api/clickhouse_username \
--value "<fill in>" --type SecureString --overwrite --region $PROD_REGION
aws ssm put-parameter --name /edge/prod/edge-api/clickhouse_password \
--value "<fill in>" --type SecureString --overwrite --region $PROD_REGION
aws ssm put-parameter --name /edge/prod/edge-api/clickhouse_database \
--value "default" --type SecureString --overwrite --region $PROD_REGION
aws ssm put-parameter --name /edge/prod/edge-api/sqs_queue_url \
--value "$NEW_QUEUE_URL" --type SecureString --overwrite --region $PROD_REGION
2.2 Set the GitHub deploy secret¶
ROLE_ARN=$(aws iam get-role --role-name gh-oidc-deploy-edge-api \
--query 'Role.Arn' --output text --region $PROD_REGION)
echo "$ROLE_ARN"
# Expected: arn:aws:iam::759286286879:role/gh-oidc-deploy-edge-api
gh secret set AWS_DEPLOY_ROLE_ARN_PROD --body "$ROLE_ARN" \
--repo cookiehub-com/edge-api
2.3 Cut the first prod release¶
cd <your edge-api checkout>
git checkout main && git pull
gh release create v3.0.0 \
--title "edge-api v3.0.0" \
--notes "First prod release. Serves edge.cookiehub.net. Legacy hostnames land in a follow-up cutover PR."
Triggers deploy-prod.yml: build → push to prod ECR → force-redeploy ECS.
2.4 Wait for ECS service to be healthy¶
aws ecs wait services-stable \
--cluster prod-euc1-edge-ecs-cluster \
--services edge-api \
--region $PROD_REGION
aws ecs describe-services \
--cluster prod-euc1-edge-ecs-cluster \
--services edge-api \
--query 'services[0].{desired:desiredCount,running:runningCount,pending:pendingCount}' \
--region $PROD_REGION
# Expect desired=2, running=2, pending=0
Phase 3 — Smoke test edge.cookiehub.net¶
Direct hits. No Host-header tricks required — edge.cookiehub.net is fully live and pointing at the new stack.
# Health
curl -sS https://edge.cookiehub.net/health -w "\nHTTP %{http_code}\n"
curl -sS https://edge.cookiehub.net/ready -w "\nHTTP %{http_code}\n"
# /log (consent-style path)
curl -sS -X POST https://edge.cookiehub.net/log \
-H "Content-Type: application/json" \
-d '{"a":1,"r":1,"d":1,"u":"https://example.com/","t":"dryrun","s":"[]","n":"vault1","gsvs":{"a":false,"c":[],"v":[],"s":[]},"tcstring":""}' \
-w "\nHTTP %{http_code}\n"
# /region
curl -sS https://edge.cookiehub.net/region \
-H "cookiehub-regions: EU,C3" \
-w "\nHTTP %{http_code}\n"
# Root dispatcher (consent-eu / region-eu pattern)
curl -sS -X POST https://edge.cookiehub.net/ \
-H "Content-Type: application/json" \
-d '{"a":1,"r":1,"d":1,"u":"https://example.com/","t":"dryrun-root","s":"[]","n":"vault1","gsvs":{"a":false,"c":[],"v":[],"s":[]},"tcstring":""}' \
-w "\nHTTP %{http_code}\n"
All should return 200 (or 202 for the events endpoints).
Verify messages land in the new queue and ClickHouse¶
aws sqs get-queue-attributes --queue-url "$NEW_QUEUE_URL" \
--attribute-names ApproximateNumberOfMessages \
--region $PROD_REGION
# Expect > 0 immediately after the smoke curls
sleep 30
aws sqs get-queue-attributes --queue-url "$NEW_QUEUE_URL" \
--attribute-names ApproximateNumberOfMessages \
--region $PROD_REGION
# Expect back to 0 or close (new ingest drained it)
# ClickHouse (adjust to real session and table)
# clickhouse-client -q "SELECT count() FROM bronze.consent_events WHERE token LIKE 'dryrun%'"
If all green, the deploy is done. Legacy customer traffic is unchanged. Part 2 can happen any time later.
Part 2 — Legacy cutover (separate follow-up PR)¶
Customer-impacting. Schedule when ready. Requires a new PR that:
- Adds the four legacy hostnames to
aws_cloudfront_distribution.edge.aliases: - Adds Route53 alias records for each (with
allow_overwrite = truesince the four names already have records pointing at legacy CF / Scaleway):resource "aws_route53_record" "legacy_aliases" { for_each = toset([ "consent.cookiehub.net", "ds.cookiehub.net", "consent-eu.cookiehub.net", "region-eu.cookiehub.net", ]) zone_id = data.aws_route53_zone.public.zone_id name = each.value type = "A" allow_overwrite = true alias { name = aws_cloudfront_distribution.edge.domain_name zone_id = aws_cloudfront_distribution.edge.hosted_zone_id evaluate_target_health = false } }
Operational steps at apply time:
Phase 0 — Strip aliases from legacy CloudFront¶
Required before applying the cutover PR — CloudFront enforces alias uniqueness.
aws cloudfront get-distribution-config --id $LEGACY_DIST_ID \
--region $PROD_REGION > /tmp/legacy-dist.json
cp /tmp/legacy-dist.json /tmp/legacy-dist.backup.json
LEGACY_ETAG=$(jq -r .ETag /tmp/legacy-dist.json)
jq '.DistributionConfig.Aliases = {"Quantity": 0}' /tmp/legacy-dist.json \
| jq '.DistributionConfig' > /tmp/legacy-dist-no-aliases.json
aws cloudfront update-distribution \
--id $LEGACY_DIST_ID \
--if-match "$LEGACY_ETAG" \
--distribution-config file:///tmp/legacy-dist-no-aliases.json \
--region $PROD_REGION > /tmp/legacy-update.json
aws cloudfront wait distribution-deployed --id $LEGACY_DIST_ID --region $PROD_REGION
echo "Legacy aliases released"
From this moment, customer traffic to consent.cookiehub.net and ds.cookiehub.net is broken (legacy CF returns 403 — no alias match). Proceed immediately to Phase 1.
Phase 1 — Apply the cutover PR¶
Expected: ~5 changes — 4 new Route53 records (with overwrite), 1 CloudFront distribution update to add 4 aliases. CloudFront update takes ~15 min to deploy globally, but ALB traffic starts flowing for the four hostnames as soon as the CloudFront update is processed (first few minutes).
Phase 2 — Smoke the legacy hostnames¶
for HOST in consent.cookiehub.net ds.cookiehub.net consent-eu.cookiehub.net region-eu.cookiehub.net; do
echo "=== $HOST ==="
curl -sS -X POST "https://$HOST/log" \
-H "Content-Type: application/json" \
-d '{"a":1,"r":1,"d":1,"u":"https://example.com/","t":"cutover-smoke","s":"[]","n":"vault1","gsvs":{"a":false,"c":[],"v":[],"s":[]},"tcstring":""}' \
-w "HTTP %{http_code}\n"
curl -sS "https://$HOST/region" \
-H "cookiehub-regions: EU,C3" \
-w "HTTP %{http_code}\n"
curl -sS -X POST "https://$HOST/" \
-H "Content-Type: application/json" \
-d '{"a":1,"r":1,"d":1,"u":"https://example.com/","t":"cutover-smoke-root","s":"[]","n":"vault1","gsvs":{"a":false,"c":[],"v":[],"s":[]},"tcstring":""}' \
-w "HTTP %{http_code}\n"
echo
done
All should return 200 or 202.
Phase 3 — Monitor queue drain¶
watch -n 10 'for q in ConsentLogQueue qs313p prod-euc1-edge-consent; do
printf "%-30s " "$q"
aws sqs get-queue-attributes \
--queue-url https://sqs.eu-central-1.amazonaws.com/759286286879/$q \
--attribute-names ApproximateNumberOfMessages \
--region eu-central-1 \
--query "Attributes.ApproximateNumberOfMessages" --output text
done'
Expected over a few minutes:
ConsentLogQueue,qs313pdrop as customer traffic converges on new CF → edge-api → new SQSprod-euc1-edge-consentrises briefly, then drains via the new ECS ingest
5xx health on the new ALB:
aws cloudwatch get-metric-statistics \
--namespace AWS/ApplicationELB \
--metric-name HTTPCode_Target_5XX_Count \
--dimensions Name=LoadBalancer,Value=$(aws elbv2 describe-load-balancers \
--names prod-euc1-edge-public-alb \
--query 'LoadBalancers[0].LoadBalancerArn' --output text --region $PROD_REGION \
| awk -F'loadbalancer/' '{print $2}') \
--start-time $(date -u -v-10M '+%Y-%m-%dT%H:%M:%S') \
--end-time $(date -u '+%Y-%m-%dT%H:%M:%S') \
--period 60 --statistics Sum --region $PROD_REGION
Cutover rollback¶
If the cutover goes sideways, in this order:
# 1. Re-add aliases to the legacy CloudFront from the saved config
LEGACY_ETAG=$(aws cloudfront get-distribution-config --id $LEGACY_DIST_ID \
--region $PROD_REGION --query 'ETag' --output text)
jq '.DistributionConfig' /tmp/legacy-dist.backup.json > /tmp/legacy-dist-restore.json
aws cloudfront update-distribution \
--id $LEGACY_DIST_ID \
--if-match "$LEGACY_ETAG" \
--distribution-config file:///tmp/legacy-dist-restore.json \
--region $PROD_REGION
aws cloudfront wait distribution-deployed --id $LEGACY_DIST_ID --region $PROD_REGION
# 2. Point Route53 back at the legacy CloudFront / Scaleway IPs
cat > /tmp/rollback-route53.json <<EOF
{
"Changes": [
{ "Action": "UPSERT", "ResourceRecordSet": { "Name": "consent.cookiehub.net", "Type": "A", "AliasTarget": { "HostedZoneId": "Z2FDTNDATAQYW2", "DNSName": "d2f71kw5bfffrp.cloudfront.net", "EvaluateTargetHealth": false } } },
{ "Action": "UPSERT", "ResourceRecordSet": { "Name": "ds.cookiehub.net", "Type": "A", "AliasTarget": { "HostedZoneId": "Z2FDTNDATAQYW2", "DNSName": "d2f71kw5bfffrp.cloudfront.net", "EvaluateTargetHealth": false } } },
{ "Action": "UPSERT", "ResourceRecordSet": { "Name": "consent-eu.cookiehub.net", "Type": "A", "TTL": 300, "ResourceRecords": [{"Value": "$SCALEWAY_IP"}] } },
{ "Action": "UPSERT", "ResourceRecordSet": { "Name": "region-eu.cookiehub.net", "Type": "A", "TTL": 300, "ResourceRecords": [{"Value": "$SCALEWAY_IP"}] } }
]
}
EOF
aws route53 change-resource-record-sets \
--hosted-zone-id $PUBLIC_ZONE_ID \
--change-batch file:///tmp/rollback-route53.json
Then revert the cutover PR and terraform apply. Customer traffic re-converges on legacy within a DNS TTL. Events the new edge-api received between apply and rollback are already in prod-euc1-edge-consent and will be drained by the new ingest — no data loss.
Post-cutover cleanup (48h+ after successful Part 2)¶
# Confirm legacy queues are idle
for q in ConsentLogQueue qs313p; do
aws sqs get-queue-attributes \
--queue-url https://sqs.eu-central-1.amazonaws.com/$PROD_ACCOUNT/$q \
--attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible \
--region $PROD_REGION
done
# Both should show 0 / 0 for at least several hours
# Retire legacy CloudFront (disable first, delete later)
aws cloudfront get-distribution-config --id $LEGACY_DIST_ID --region $PROD_REGION > /tmp/legacy-dist.json
jq '.DistributionConfig.Enabled = false' /tmp/legacy-dist.json | jq '.DistributionConfig' > /tmp/legacy-disabled.json
LEGACY_ETAG=$(jq -r .ETag /tmp/legacy-dist.json)
aws cloudfront update-distribution \
--id $LEGACY_DIST_ID \
--if-match "$LEGACY_ETAG" \
--distribution-config file:///tmp/legacy-disabled.json \
--region $PROD_REGION
# Wait 24h more, then: aws cloudfront delete-distribution --id $LEGACY_DIST_ID
# Drain and delete legacy queues
aws sqs delete-queue --queue-url https://sqs.eu-central-1.amazonaws.com/$PROD_ACCOUNT/ConsentLogQueue --region $PROD_REGION
aws sqs delete-queue --queue-url https://sqs.eu-central-1.amazonaws.com/$PROD_ACCOUNT/qs313p --region $PROD_REGION
# Remove legacy queue entries from the vault ETL instances map
# (edit environments/prod/eu-central-1/vault.tf, then terraform apply)
Lambda@Edge functions auto-clean up their replicas over ~24h once the distribution referencing them is deleted.