ClickHouse Upgrade¶

Apply OS security updates and, optionally, upgrade ClickHouse on the production ClickHouse host. This is a planned maintenance operation on a single, unreplicated database instance.

Maintenance window required

Run this off hours. ClickHouse is unavailable from the moment ingest is stopped until the service is confirmed healthy. vault-api will return no data for the duration — this is expected, do not escalate.

The procedure is deliberately split into two halves that can be run independently:

OS half — patch, reboot, verify the mount, bring ClickHouse back on its existing version. This is a complete, legitimate stopping point.
ClickHouse half — upgrade the ClickHouse packages themselves. Riskier, version-pinned, and skippable.

Running only the OS half and closing the window is a normal outcome, not a failure.

flowchart TD
    A[Stop ingest] --> B[Stop + disable clickhouse-server]
    B --> C[Snapshot instance<br/>root + data]
    C --> D[Pre-reboot checks]
    D --> E[OS upgrade]
    E --> F[Reboot]
    F --> G{Mount gate<br/>device + size}
    G -->|wrong| H[HARD STOP<br/>fix the mount<br/>do not start the server]
    G -->|correct| I[Start ClickHouse<br/>on current version]
    I --> J{Healthy?}
    J -->|no| K[Rollback — see table]
    J -->|yes| L{Continue to<br/>ClickHouse upgrade?}
    L -->|no| M[Re-enable service<br/>restore ingest<br/>window closed]
    L -->|yes| N[Stop server<br/>unlock, upgrade to pinned target]
    N --> O[Start + verify + re-lock]
    O --> M

Required access¶

Requirement	Detail
AWS profile	`cookiehub-prod` (account `759286286879`), region `eu-central-1`
Shell on the host	SSM Session Manager only — the instance has no public IP and no SSH path
EC2 permissions	`ec2:CreateSnapshots`, `ec2:DescribeSnapshots`, `ec2:CreateReplaceRootVolumeTask`
ECS permissions	`ecs:UpdateService`, `ecs:DescribeServices` on `prod-euc1-vault-ecs-cluster`
ClickHouse credential	SSM Parameter Store, SecureString `/vault/prod/clickhouse/admin_password` — needs `ssm:GetParameter` + `kms:Decrypt`
CloudWatch	Read access to `/ecs/prod-euc1-vault-*` log groups

SSM is the only way in

There is no SSH, no bastion and no public IP on this host. If amazon-ssm-agent does not come back after the reboot you have no shell on an instance whose database is deliberately stopped. The pre-reboot checks below verify the agent is enabled. There is currently no documented break-glass path — this is an accepted risk, and it is tracked in Follow-ups.

Run parameters¶

Fill these in before the window opens. Nothing in this runbook takes a version from whatever the repository happens to be serving on the night.

Parameter	Value for this run	Notes
Instance ID		`terraform output` in `environments/prod/eu-central-1`, or EC2 console
Current OS release		`cat /etc/dnf/vars/releasever`
Target OS release	`latest`	Record the release `dnf` actually resolves — see OS upgrade
Current ClickHouse		`SELECT version()`
Target ClickHouse		Explicit `x.y.z.n`, chosen after changelog review. Leave blank to skip the ClickHouse half
Pre-window data used		`df -h /var/lib/clickhouse` — needed by the mount gate
Pre-window active parts		`SELECT count() FROM system.parts WHERE active`
Abort deadline		Wall-clock time. Decided now, not during the window
Root snapshot ID		Filled in at step 3
Data snapshot ID		Filled in at step 3

Set the abort deadline before you start

The failure mode on this host has never been a bad upgrade on its own. It is a bad upgrade followed by two hours of optimistic debugging with the window gone. Write a wall-clock time down. When you reach it, roll back.

Why this host drifts¶

Two separate mechanisms hold this host back, and both are deliberate. Understand them before you touch anything, because between them they let the instance sit 6.5 months behind on OS patches with dnf honestly reporting "nothing to do".

AL2023 repository pinning. Amazon Linux 2023 pins each instance to a frozen, versioned repository snapshot recorded in /etc/dnf/vars/releasever. A bare sudo dnf upgrade resolves against that frozen snapshot, finds nothing newer in it, and exits successfully while fixes accumulate upstream. Moving forward is an explicit act — it only happens when you pass --releasever.

A deliberate versionlock on ClickHouse. The three ClickHouse packages are locked at 26.1.2.11-1:

sudo dnf versionlock list

The lock is intentional and must not be removed casually. Releasing it is step one of an intentional ClickHouse upgrade and re-locking at the new version is the last step. It is not something to clear because dnf complained.

Before the window¶

1. Rehearse¶

Run the whole procedure somewhere else first. A rehearsal is only worth something if the rehearsal host matches production in the ways that actually break. Compare explicitly:

Must match	Why
Starting OS release	A different starting `releasever` crosses a different package set
ClickHouse version	Backward-incompatible changes are per-minor
Volume layout	Separate data volume, same mountpoint, same `fstab` style
Mount configuration	Device path vs UUID is the single most dangerous difference

Stage does not match prod's storage

Per environments/{stage,prod}/eu-central-1/vault.tf, stage runs a t3.medium with a 50 GB data volume; prod runs an m7i.xlarge with 1000 GB. Stage is a fine rehearsal for the OS transaction and the reboot. It proves nothing about how the real data volume behaves or how long the first post-upgrade start takes.

For the ClickHouse version upgrade specifically, rehearse on a throwaway instance built from a restore of the production data-volume snapshot. That is the only way to learn what the first start on the new version actually does to real data.

2. Choose the ClickHouse target version¶

Skip this section entirely if you are only doing the OS half.

Record the current version (SELECT version()) and the intended target. The target is an explicit version string, never "whatever stable holds tonight".
Read the backward-incompatible changes section of the changelog for every minor release crossed, not just the target's.
ClickHouse supports roughly a one-year compatibility window. If the jump is wider than that, go via an intermediate version rather than in one hop.

The gap here is large

As of the 2026-09-01 window the host runs 26.1.2.11 and the stable channel holds 26.8.1.2041 — eight months and multiple minor releases, each with its own backward-incompatible changes. Do not take that in one unrehearsed hop. sudo dnf upgrade -y clickhouse-server clickhouse-client clickhouse-common-static with no version pin is the single riskiest command that can be run against this host.

Rollback is not symmetric

Once the new version has started and written to the data directory, dnf downgrade may not be available and the on-disk format may no longer be readable by the old binary. Past that point the pre-upgrade snapshot is the only way back.

3. Capture the baseline¶

From your workstation or an SSM session, with ClickHouse still serving normally:

# Row counts per table — the thing that actually proves the data came back
clickhouse-client --password --query "SELECT database, table, sum(rows) AS rows
  FROM system.parts WHERE active GROUP BY 1,2 ORDER BY rows DESC" > ~/ch-rowcounts-pre.tsv

# Active part count, version, and disk usage
clickhouse-client --password --query "SELECT count() FROM system.parts WHERE active"
clickhouse-client --password --query "SELECT version()"
df -h /var/lib/clickhouse
cat /etc/dnf/vars/releasever

Write the active-part count, the used-disk figure and the versions into the run parameters table. The mount gate and the verification step both need them.

4. Dry run the OS transaction¶

sudo dnf upgrade --releasever=latest --exclude='clickhouse*' --assumeno 2>&1 | tee /tmp/dryrun.txt
sudo dnf history list | head -5   # record the current transaction ID

Read the output for two things:

Package count — it sets the expected duration. On 2026-09-01 the dry run resolved 292 packages / 357 MB.
Anything removed or obsoleted — this is where surprises live, and removals are exactly what dnf history undo handles worst. On 2026-09-01 there were zero removals.

Record the dnf history transaction ID. It is the rollback handle for the pre-reboot failure case.

The window¶

1. Stop ingest¶

Two vault-ingest tasks write directly to ClickHouse and must be stopped before anything else. Set desired count to 0 for both services on prod-euc1-vault-ecs-cluster:

clickhouse-session_ingest
consent-clickhouse-ingest

for svc in clickhouse-session_ingest consent-clickhouse-ingest; do
  aws ecs update-service --cluster prod-euc1-vault-ecs-cluster --service "$svc" \
    --desired-count 0 --region eu-central-1 --query 'service.serviceName' --output text
done

Wait for running tasks to drain to zero before continuing:

aws ecs describe-services --cluster prod-euc1-vault-ecs-cluster \
  --services clickhouse-session_ingest consent-clickhouse-ingest \
  --region eu-central-1 \
  --query 'services[].{name:serviceName,desired:desiredCount,running:runningCount}'

All other vault-ingest tasks and vault-api can remain running. vault-api returning no data is expected.

2. Connect, then stop and disable ClickHouse¶

aws ssm start-session --target <instance-id> --region eu-central-1

sudo systemctl stop clickhouse-server
sudo systemctl disable clickhouse-server
systemctl is-enabled clickhouse-server   # expect: disabled

Why disable it before rebooting

Disabling the unit makes it structurally impossible for ClickHouse to auto-start into an unmounted data directory after the reboot (see the mount gate). You start it by hand, after you have proven the volume is there. This is paired with an explicit re-enable at step 11 — do not skip that step.

Stopping the server here is also what makes the snapshot worth taking.

3. Snapshot the whole instance¶

With the writers stopped and the server down, the snapshot is application-consistent rather than crash-consistent. That distinction is precisely what makes a ClickHouse version upgrade reversible.

aws ec2 create-snapshots \
  --instance-specification InstanceId=<instance-id>,ExcludeBootVolume=false \
  --description "pre-upgrade $(date +%F)" \
  --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Purpose,Value=pre-upgrade}]' \
  --region eu-central-1

ExcludeBootVolume=false is not optional. Without a root snapshot the OS upgrade has no rollback path at all. Record both snapshot IDs in the run parameters table — the root snapshot and the data snapshot do different jobs in a rollback.

Do not wait for the snapshot to complete

An EBS snapshot fixes its point in time at initiation. The progress percentage is background copying to S3; the restore point is already captured while the snapshot reads pending, and the volume is free to be changed. On a 436 GB volume, waiting for 100% burns 15–30 minutes of window for no benefit.

Confirm a snapshot ID exists per volume, record them, continue:

aws ec2 describe-snapshots --snapshot-ids <snap-1> <snap-2> --region eu-central-1 \
  --query 'Snapshots[].{id:SnapshotId,vol:VolumeId,state:State,size:VolumeSize}'

4. Pre-reboot checks¶

After a long uptime the reboot is the risk, not the patch. State that was applied live but never persisted only surfaces at boot — and this host reached 28 weeks of uptime before the 2026-09-01 window, so none of it had ever been tested.

grep clickhouse /etc/fstab                  # must be UUID=..., NOT a device path
sudo findmnt --verify
sudo grubby --default-kernel                # must be the NEW kernel
systemctl is-enabled amazon-ssm-agent       # or you have no way back in
systemctl is-enabled clickhouse-server      # expect: disabled, from step 2
sudo find /etc \( -name '*.rpmnew' -o -name '*.rpmsave' \) -print
systemctl --failed

Check	Pass	If it fails
`fstab` entry	`UUID=...`	Fix it before rebooting — see the warning below
`findmnt --verify`	No errors	Fix the entry it names
`grubby --default-kernel`	The kernel just installed	Do not reboot; investigate the bootloader
`amazon-ssm-agent`	`enabled`	Do not reboot. Enable it — this is your only access path
`.rpmsave` files	None	Your config was replaced; reconcile before rebooting
`systemctl --failed`	Empty	Understand each unit before adding a reboot to the mix

.rpmnew means your configuration file was kept and the new default was parked alongside it — safe. .rpmsave means the reverse: your file was moved aside and the package default is now live. That needs attention before rebooting.

fstab must reference the data volume by UUID

Until 2026-09-01 /etc/fstab referenced the data volume as /dev/nvme1n1 with nofail. NVMe device names are not stable on Nitro. On a rename the instance boots cleanly, the mount silently does not happen, /var/lib/clickhouse resolves to an empty directory on the root volume, and a ClickHouse start initialises a fresh, empty store over the top of it.

The entry was corrected by hand to UUID=d937a310-7eae-4271-ab59-3a3a325dbbf7. On the live host that correction is not durable — the Terraform userdata that built it writes the entry by device path, so an instance rebuild would undo it. Re-check this line every time. See Bootstrap script for the state of the fix.

5. Upgrade the OS¶

Do not use a hardcoded --releasever. A pinned value silently under-patches and rots between windows — the runbook carried 2023.11.20260514 (a May 2026 release) into a September window whose actual target was 2023.12.20260831.

dnf check-update on its own is misleading here

Run without --releasever it resolves against the frozen pinned repo and reports almost nothing. That reads as reassuring immediately before a ~290-package upgrade. If you want the check, run it against the real target:

sudo dnf check-update --releasever=latest

Run the transaction detached. It takes 10–15 minutes, a dnf transaction cannot be safely interrupted, and on SSM Session Manager a dropped session sends SIGHUP to a foreground dnf:

sudo nohup dnf upgrade -y --releasever=latest --exclude='clickhouse*' > ~/upgrade.log 2>&1 &
tail -f ~/upgrade.log

--exclude='clickhouse*' covers clickhouse-server, clickhouse-client and clickhouse-common-static. Keep it even though versionlock should already hold them — the two mechanisms are cheap and independent.

When it finishes, record the release that was actually applied:

cat /etc/dnf/vars/releasever    # write this into the run log
sudo dnf history list | head -3

6. Reboot¶

sudo reboot

Reconnect via SSM once the instance is back.

7. Mount gate (hard stop)¶

Do not start ClickHouse until this gate passes

This is the step that separates a five-minute problem from a restore. If the data volume did not come back and you start the server, ClickHouse initialises an empty store in /var/lib/clickhouse on the root volume, and the only way back is the snapshot.

uname -r
findmnt -T /var/lib/clickhouse
df -h /var/lib/clickhouse
systemctl --failed

The gate is not "is something mounted". It is device and size:

Check	Expected
`uname -r`	The new kernel
Source	`/dev/nvme1n1` (a real block device, not the root filesystem)
Total size	~1000 G
Used	Approximately the pre-window figure from the run parameters — 436 G at the 2026-09-01 window
Failed units	None

Size is the tell. An empty directory on the root volume will show a mounted-looking path backed by a ~64 G root filesystem with almost nothing used. That is the failure, and it looks superficially fine.

If the mount is wrong: stop. Do not start ClickHouse. Fix /etc/fstab to reference the volume by UUID, mount -a, and re-run this gate. Only then continue.

8. Checkpoint — start ClickHouse on its current version¶

Bring the database back on the version it was already running, before changing anything else. This is what makes the two halves separable: if something misbehaves after this point, you know the ClickHouse upgrade caused it, and you can unwind that alone rather than unwinding an OS jump and a database upgrade together.

sudo systemctl start clickhouse-server

The unit is Type=notify, so systemctl start blocks until the server is actually ready to serve. When the command returns, it is serving.

curl -sf localhost:8123/ping           # no auth needed — the right liveness check
sudo systemctl status clickhouse-server --no-pager

Then run the verification below. If it passes, you have a patched OS and an unchanged database.

This is a valid place to stop

If the window is running long, or anything above was harder than expected, skip to step 11 and close out here. A patched OS with an untouched database is a good night's work. This is exactly what happened on 2026-09-01.

9. Upgrade ClickHouse (optional)¶

Only proceed if you have a target version from Before the window, you have rehearsed it, and you are comfortably inside the abort deadline.

sudo systemctl stop clickhouse-server

# 1. Release the deliberate lock
sudo dnf versionlock delete 'clickhouse*'

# 2. Upgrade to the EXPLICIT target — never an unpinned upgrade
sudo nohup dnf upgrade -y \
  clickhouse-server-<TARGET> \
  clickhouse-client-<TARGET> \
  clickhouse-common-static-<TARGET> > ~/ch-upgrade.log 2>&1 &
tail -f ~/ch-upgrade.log

10. Start ClickHouse and re-lock¶

sudo systemctl start clickhouse-server
curl -sf localhost:8123/ping
clickhouse-client --password --query "SELECT version()"    # confirm it is the target

Run verification. Once healthy, re-lock at the new version:

sudo dnf versionlock add \
  clickhouse-server-<TARGET> clickhouse-client-<TARGET> clickhouse-common-static-<TARGET>
sudo dnf versionlock list

Check authentication survived the upgrade

The default user's password is currently set by a sed into /etc/clickhouse-server/users.xml, which is an RPM-managed file. A package upgrade can replace it and take authentication with it. If clickhouse-client --password starts failing right after the upgrade, this is the first thing to check. Moving the credential to a users.d/ drop-in is tracked in Follow-ups.

11. Re-enable the service¶

Do not close the window without this

A running-but-disabled service looks completely healthy tonight. Then the next unplanned reboot, weeks later, brings the instance back with no database and nobody understands why.

sudo systemctl enable clickhouse-server
systemctl is-enabled clickhouse-server   # must read: enabled

You disabled it yourself at step 2. Separately, the ClickHouse RPM re-applies its vendor preset — which is disabled — on package upgrade, so step 9 will also have left the unit disabled.

Accuracy note on the vendor preset

On 2026-09-01 the unit was found enabled before the window, on a host that had been through OS upgrades but not a ClickHouse package upgrade. The evidence points at the ClickHouse package upgrade re-applying the disabled preset, not the OS upgrade. That mechanism has not yet been confirmed directly. Verify is-enabled at the end of every run regardless of which half you ran.

12. Restore ingest¶

Only once ClickHouse is verified healthy and is-enabled reads enabled:

aws ecs update-service --cluster prod-euc1-vault-ecs-cluster \
  --service clickhouse-session_ingest --desired-count <previous> --region eu-central-1
aws ecs update-service --cluster prod-euc1-vault-ecs-cluster \
  --service consent-clickhouse-ingest --desired-count <previous> --region eu-central-1

Confirm tasks reach running state and check CloudWatch logs for ingest errors.

Verification¶

SELECT version() and SELECT 1 prove only that a process is listening. They do not prove the data came back. Run all of the following, at both the checkpoint and after any ClickHouse upgrade.

Every client command needs --password

The default user has a password, set at bootstrap from SSM Parameter Store (/vault/prod/clickhouse/admin_password). clickhouse-client -q "..." as written in older versions of this runbook fails with Code: 516 ... Authentication failed, which at 23:00 reads exactly like a broken upgrade. Use clickhouse-client --password --query "..." and enter the password when prompted, or fetch it with:

aws ssm get-parameter --name /vault/prod/clickhouse/admin_password \
  --with-decryption --region eu-central-1 --query Parameter.Value --output text

1. Liveness and version

curl -sf localhost:8123/ping
clickhouse-client --password --query "SELECT version()"

2. Row counts against the baseline

clickhouse-client --password --query "SELECT database, table, sum(rows) AS rows
  FROM system.parts WHERE active GROUP BY 1,2 ORDER BY rows DESC" > ~/ch-rowcounts-post.tsv

diff <(grep '^default' ~/ch-rowcounts-pre.tsv) <(grep '^default' ~/ch-rowcounts-post.tsv)

Compare only default.*. The system.* log tables write continuously; a difference there is the server working correctly, not a fault. Then read SummingMergeTree row counts before concluding anything from a drop.

3. Active part count

clickhouse-client --password --query "SELECT count() FROM system.parts WHERE active"

4. Errors since the restart

clickhouse-client --password --query "SELECT name, value, last_error_time, last_error_message
  FROM system.errors WHERE last_error_time > now() - INTERVAL 30 MINUTE ORDER BY value DESC"

5. One real application query

Run something the application actually runs — a recent-window aggregate over default.consent_category_daily or default.sessions_dim_daily. Do this before restoring ingest: a quiet database is a far easier place to find a problem than a loaded one.

If any of this fails, go to Rollback rather than debugging past the abort deadline. Logs:

sudo journalctl -u clickhouse-server -n 50 --no-pager
sudo tail -n 50 /var/log/clickhouse-server/clickhouse-server.err.log

Rollback¶

Indexed by where it went wrong. Pick the row, do not improvise.

Failure point	Action
OS transaction misbehaves, pre-reboot	`sudo dnf history undo <id>` — the ID recorded during the dry run
Instance does not boot	`aws ec2 create-replace-root-volume-task --instance-id <id> --snapshot-id <root snap>` — same instance, same IPs, the data volume stays attached
Data volume did not remount	Fix `/etc/fstab` by UUID and `mount -a`. Do not start the server first
ClickHouse bad, before it serves writes	`sudo dnf downgrade clickhouse-server-<OLD> clickhouse-client-<OLD> clickhouse-common-static-<OLD>`
ClickHouse bad, already writing	Restore the data volume from the pre-upgrade snapshot. `dnf downgrade` is no longer trustworthy

Ingest stays at desired count 0 throughout any rollback. Bring it back only after verification passes.

Reference — what "normal" looks like¶

SummingMergeTree row counts legitimately fall¶

Read this before rolling back on a row-count drop

Row counts on SummingMergeTree tables legitimately fall after a restart. Merges resume on startup and collapse rows that share a sorting key. On a plain MergeTree the same drop would be data loss. Do not roll back a healthy database because of this.

Measured across the 2026-09-01 restart:

Table	Change
`default.consent_category_daily`	−241 rows
`default.sessions_dim_daily`	−4,762 rows
Active parts (whole server)	2,982 → 2,967

The signature is row count and part count moving together. Confirm it rather than assuming it:

# Is the table actually a summing engine?
clickhouse-client --password --query "SELECT database, name, engine
  FROM system.tables WHERE database = 'default' ORDER BY name"

# Did merges actually run?
clickhouse-client --password --query "SELECT event_time, database, table, part_name, rows
  FROM system.part_log WHERE event_type = 'MergeParts'
  AND event_time > now() - INTERVAL 1 HOUR ORDER BY event_time DESC LIMIT 20"

A row-count drop on a SummingMergeTree table, with the active part count falling at the same time and matching MergeParts entries in system.part_log, is the merge signature and is expected. A drop on a plain MergeTree table, or a drop with no corresponding merge activity, is not — that is a rollback.

Startup time¶

Startup is seconds, not minutes. Older revisions of this runbook claimed 3–5 minutes and attributed it to "the data volume mounting and tables loading". Both parts were wrong: the volume mounts at boot, not at service start, and the measured start on 2026-09-01 with ~2,970 active parts was seconds.

Startup time scales with active part and table count, so measure it rather than trusting a number in a document:

sudo grep -E "Ready for connections|Application: Ready" \
  /var/log/clickhouse-server/clickhouse-server.log | tail -3

Because the unit is Type=notify, systemctl start does not return until the server is ready. If the command has returned, the server is up — a slow start command is the server genuinely still loading, not systemd lying to you.

Parts appearing with ingest stopped¶

Several *_filtered tables produced new parts immediately after the restart, with both ingest services at desired count 0. This looks like refreshable materialized views firing on startup. It has not been confirmed, and it means "quiesced" may not mean "nothing is writing" on this host. Take it into account when reading a row-count diff, and see Follow-ups.

Bootstrap script¶

modules/vault/clickhouse/clickhouse_bootstrap.sh in the infra/ repo is the userdata that builds this host. Fixes for the problems the 2026-09-01 window exposed are prepared on the branch fix/clickhouse-bootstrap-fstab-uuid:

Change	Effect on this runbook
`fstab` written as `UUID=` read from `blkid` after `mkfs`	The device-path trap does not get recreated on a rebuild
Idempotency check matches the mountpoint, and rewrites rather than appends	A re-run no longer appends a second, conflicting entry
`mount -a` followed by an assertion that the mountpoint is backed by that UUID	Bootstrap fails loudly instead of continuing onto the root volume
`RequiresMountsFor=/var/lib/clickhouse` drop-in for the unit	systemd refuses to start the server without the mount
Password moved to a `users.d/` drop-in as `password_sha256_hex`, `chmod 600`	A ClickHouse package upgrade can no longer take authentication with it
`gpgcheck=1`	Packages are signature-verified
Bounded device wait, `enable` no longer swallows failure	An unattached volume or a failed `enable` fails the build instead of going quiet

The fix does not reach the running host on its own

The aws_instance resource carries ignore_changes = [user_data] and prevent_destroy = true, so merging that branch changes nothing on the live instance — it only affects a rebuild. The hand-fixed fstab on the running host stays hand-fixed, and the pre-reboot check stays mandatory.

The users.d/ password drop-in and RequiresMountsFor have not been exercised on a real instance yet. Validate them on stage before relying on them.

Follow-ups (not part of this procedure)¶

Open items from the 2026-09-01 window. None of these are steps in the runbook; they need their own work.

Item	Detail
Bootstrap script fixes	`fstab` by device path, the broken idempotency check, the `users.xml` password write, `gpgcheck=0`, an unbounded device-wait loop, and a missing mount dependency. Prepared on `infra` branch `fix/clickhouse-bootstrap-fstab-uuid` — see Bootstrap script. Needs stage validation and a rebuild to take effect
System log table growth	`system.trace_log` 6.76 billion rows, `system.asynchronous_metric_log` 4.06 billion, `system.text_log` 1.28 billion. Almost certainly a large share of the 436 GB in use. Needs TTLs — this is a capacity finding, not housekeeping
`aws-neuronx` DKMS module	The Inferentia/Trainium driver is installed on this database host and rebuilds on every kernel upgrade, adding a failure surface to every OS window. Establish whether it is needed at all, and remove it if not
*`_filtered` write activity**	Confirm whether refreshable materialized views fire on startup. It changes what "ingest stopped" actually guarantees
No break-glass access	SSM is the only path to this host. Either document an alternative or formally accept the risk