Skip to content

Dashboard cutover: rollback to Laravel

If the React dashboard (core-portal) misbehaves after the flip, put the legacy Laravel dashboard back in front of customers with one load-balancer change. This page says when to do it, how, what it leaves behind, and how to rehearse it on stage. It pairs with Dashboard cutover: legacy allowlist on the dash listener, which describes the listener, its rules and the flip itself; read its "How the routing works" section first. The plan behind both is the React Dashboard plan, §6.4 (the flip) and §6.6 (rollback), a Linear document.

Who this is for: whoever is on call after the flip, possibly at night. The rollback is deliberately small. Do not improvise beyond it.

Required access

  • AWS: in the target account (stage 258618559895, prod 759286286879, region eu-central-1): elasticloadbalancing:DescribeListeners, DescribeRules, DescribeTargetGroups, DescribeTargetHealth and ModifyRule. The emergency path needs only these. The Terraform path needs the role that applies the dash root; Ástþór holds it.
  • Terraform: the infra repository (environments/<env>/eu-central-1/dash.tf). Chats and agents never run terraform apply; a person does.
  • Secrets / SSM: none. VPN: none for the rollback; the React dashboard's private host may need the VPN (see step 4 below).
  • Prod only: a Cloudflare login with DNS access, in case the hostname is reached through Cloudflare (see "Prod differs" below).

Decision: when to roll back

Roll back when customers cannot do something essential and there is no quick fix in the React dashboard:

  • Login is broken for many users: password, TOTP, magic link, or SSO (OIDC or SAML). A broken SAML ACS locks out every SSO customer.
  • A bug writes wrong data: money paths (upgrade, coupon, proration, invoice payment), settings that save the wrong values, domains or accounts changed unexpectedly.
  • Widget publishing is broken, so customers cannot get a consent-banner change live.
  • Sustained 5xx from the React target group (more than a few minutes, not one noisy deploy), or the React service cannot stay healthy.

Do not roll back for:

  • A page or action that is broken in the React app while everything else works. Fix forward, or, if the path really belongs to Laravel, route just that path back with the hot-fix rule in the allowlist runbook (Hot-fix: add a rule).
  • A missed legacy URL (an external caller getting 404). That is the same hot-fix rule.
  • Cosmetic or copy issues.

When unsure about data-affecting bugs, roll back first and investigate afterwards: the rollback loses nothing (see "What is preserved") and repeating the flip takes the same one change.

The rollback

One change: the dashboard catch-all on the public HTTPS listener goes back to forwarding to Laravel's target group. The allowlist rules do not change; they already point at Laravel. The change takes effect within seconds.

On stage the catch-all is the rule aws_lb_listener_rule.dash_https (priority 1000, host dash.stage.cookiehub.net) in environments/stage/eu-central-1/dash.tf. Laravel's target group there is aws_lb_target_group.dash (named <name_prefix>-dash-tg). The prod root's resource names are in environments/prod/eu-central-1/dash.tf; read them there rather than assuming they match stage.

Look up the values first

Never hard-code ARNs, and never copy them from a previous incident.

export AWS_REGION=eu-central-1            # and the profile/role of the target account

# The public ALB's HTTPS listener
LB_ARN=$(aws elbv2 describe-load-balancers --query "LoadBalancers[?contains(LoadBalancerName,'public')].LoadBalancerArn" --output text)
LISTENER_ARN=$(aws elbv2 describe-listeners --load-balancer-arn "$LB_ARN" \
  --query "Listeners[?Port==\`443\`].ListenerArn" --output text)

# The catch-all rule: priority 1000, host = the dashboard hostname
aws elbv2 describe-rules --listener-arn "$LISTENER_ARN" \
  --query "Rules[?Priority=='1000'].[RuleArn,Conditions[0].HostHeaderConfig.Values[0],Actions[0].ForwardConfig.TargetGroups[0].TargetGroupArn]" --output text

# Laravel's target group (stage name ends in -dash-tg; confirm the prod name in dash.tf) and its health
aws elbv2 describe-target-groups --query "TargetGroups[?contains(TargetGroupName,'dash-tg')].[TargetGroupName,TargetGroupArn]" --output text
aws elbv2 describe-target-health --target-group-arn <laravel-target-group-arn>

If the load balancer name filter returns more than one ALB, or the catch-all is not at priority 1000, stop and read the listener's rules (describe-rules) before changing anything. Check that the Laravel target is healthy. If it is not, rolling back would send customers to an error page: fix Laravel first (see Deployments).

(a) Terraform (the normal path)

  1. In infra, edit the catch-all rule's target_group_arn from the React dashboard's target group (the core-portal module's target_group_arn output) back to Laravel's (aws_lb_target_group.dash.arn on stage). Revert the flip commit if that is the whole change.
  2. terraform plan: it must show exactly one change, on the catch-all rule.
  3. Apply it (whoever holds the role), then open the pull request.

In an incident, apply first and review after: Operations Security Procedure 3.1.4 allows an urgent change to skip peer review only if it is peer-reviewed afterwards. The pull request must exist the same day, and the other approver's review is recorded on the incident's Linear issue (see Branching and Releases, Break-glass).

(b) Console or CLI (the emergency path)

Use this when nobody with the Terraform role is available. It changes only the live rule and creates drift, which step (a) must then codify.

RULE_ARN=<catch-all rule arn from the lookup above>
LARAVEL_TG_ARN=<laravel target group arn from the lookup above>

aws elbv2 modify-rule --rule-arn "$RULE_ARN" \
  --actions Type=forward,TargetGroupArn="$LARAVEL_TG_ARN"

modify-listener is not the tool for this: it changes the listener's default action (a fixed 404), which is not the dashboard's catch-all. Do not touch it.

The same session, put the change into Terraform (a), so the next apply does not undo the rollback. Record it on the incident issue.

Check afterwards

  1. curl -sI https://dash.cookiehub.com/login answers from Laravel (stage: https://dash.stage.cookiehub.net/login). Then log in in a browser: it lands on the legacy dashboard (you are asked to log in again).
  2. From the legacy dashboard, publish a widget on a test domain and confirm the change is live.
  3. One allowlisted URL still answers from Laravel, for example GET /scanner-ip/json.
  4. The React dashboard is still reachable on its own host for debugging: on stage https://dash-next.stage.cookiehub.net. For prod, find the portal's hostname in environments/prod/eu-central-1 (the core-portal module's hostname); it may be private.
  5. The Laravel target group shows healthy targets and its 5xx count is flat.

Prod differs from stage

Before the prod flip, dash.cookiehub.com goes through Cloudflare straight to the Laravel server cs150p, outside the ALB (see Production differs from stage). If the flip moved that hostname onto the ALB, the ALB catch-all is the lever above. If the ALB itself is the problem, the second lever is Cloudflare: point the hostname's record back at cs150p. Record which one the flip used on the cutover issue so this page is not guessed at night.

What is preserved and what is lost

Nothing is deleted by a rollback. The only change is which application answers.

  • Sessions. Sessions are owned by core-api. Rolling back at most asks users to log in again once.
  • Data. Laravel and core-api read and write the same MySQL database. Whatever the React dashboard saved through core-api (settings, drafts, domain and account changes) is in the tables Laravel reads, so the legacy dashboard shows it. Check one write made during the incident window after rolling back; if a field looks wrong, that is a bug to file, not a reason to restore from backup.
  • Orphaned but harmless. Tables that only core-api uses stay in the database and Laravel ignores them: for example user_email_reverts and the OAuth and MCP grant tables. Do not drop or clean them during the rollback window.
  • Not covered by the rollback. Anything the React dashboard did that Laravel has no screen for (the new URLs with no legacy equivalent: the AI assistants page, the OAuth consent screen). Those pages disappear until the flip is repeated; links to them show Laravel's not-found page.

Preconditions that keep the rollback cheap

From the plan, §6.6. All must hold for the whole rollback window (2 to 4 weeks after the flip):

  • Laravel stays deployed and healthy: the server runs, the target group is healthy, deploys of core continue by v* tag on main, and its allowlist paths keep working.
  • No destructive core cleanup starts until the window closes: no seven-day deletions, no table drops, no freeze-then-delete of the portal code.
  • The window's end is a dated decision recorded on the cutover issue. Until that decision exists, the window is open.
  • The listener rules stay as the allowlist runbook describes them: the allowlist at 200 to 299, the catch-all at 1000. Do not renumber them during the window.

Rolling forward again

After the fix is released and verified on stage, repeat the flip: the same single edit as the allowlist runbook's flip change, the catch-all's target_group_arn to the React dashboard's target group. Re-run the sweep and the IdP round trips first if the incident touched login. Users get one more forced re-login; announce it. Note the date on the cutover issue; the rollback window restarts from the last flip.

Stage rehearsal

The plan requires the rollback to be executed once on stage before the prod flip (plan §6.1, rule 6). This runbook is not proven, and CH-358 is not done, until the two blank fields below are filled in from a real run.

Preconditions: the allowlist rules (CH-357) are applied on stage, and the stage flip (CH-359) has been done once, so dash.stage.cookiehub.net is currently served by the React dashboard.

Rehearsed on: ________ (date) by: ________ Total time: ________

# Step Command or action Expected result Time taken
1 Confirm the stage flip is in place curl -s https://dash.stage.cookiehub.net/login shows the React dashboard; log in React dashboard answers
2 Look up the values The lookup block above, stage account Listener, catch-all rule and Laravel target group found; Laravel target healthy
3 Roll back by CLI aws elbv2 modify-rule from (b) Command succeeds; rule now forwards to Laravel's target group
4 Check login Browser login on dash.stage.cookiehub.net Legacy dashboard; one re-login
5 Check publish and an allowlist URL Publish a widget on a stage test domain; GET /scanner-ip/json Change live; IP list returned
6 Check the React host Open https://dash-next.stage.cookiehub.net Still reachable, can log in
7 Check preserved data Save a setting in the React dashboard before step 3; read it in legacy after Same value
8 Codify in Terraform Edit dash.tf, terraform plan Plan shows no change beyond the one rule (the live state already matches)
9 Roll forward The flip change, by Terraform React dashboard answers again

Fill in "time taken" per step from the run, not from estimates. If any step fails, fix this page in the same pull request that records the run.

Communication

  • Tell the team first: the prod deploys Teams channel, and anyone cutting a release, before and after the change. (There is no separate incident runbook on this site yet; this page names the channels the deploy workflows already post to.)
  • Customers: if announced users noticed, the line is: "We returned to the previous dashboard while we fix an issue. You may be asked to sign in once more. Nothing you saved was lost."
  • Linear: open an incident issue before or right after the change (Infrastructure & Security for infra, Product & Engineering for the dashboard bug). Record the time, the command used, who applied it, and the post-facto review by the other approver (Operations Security Procedure 3.1.4). Link it from the cutover issue.
  • Afterwards: note on the cutover issue that the rollback window restarts when the flip is repeated.