Secrets

Overview

Secrets are stored in AWS Systems Manager Parameter Store as SecureString parameters, encrypted with KMS. ECS tasks read secrets at startup via the secrets block in their container definitions.

Terraform creates the parameters with placeholder values. Real values are set manually after the initial apply.

Naming Convention

SSM parameters follow the pattern:

/<service>/<environment>/<component>/<secret_name>

Examples:

/atrax/stage/atrax-api/database_url
/vault/prod/vault-api/clickhouse_password
/core/stage/core-api/auth_secret

Current Parameters

Core API

Path	Description	Status
/core/stage/core-api/auth_secret	JWT/HMAC signing secret	Set
/core/stage/core-api/database_url	MariaDB connection URL	Not yet created

Atrax (stage only)

Path	Description
/atrax/stage/atrax-api/database_url	PostgreSQL connection URL
/atrax/stage/atrax-api/controller_auth	Bearer tokens for auth
/atrax/stage/atrax-api/s3_access_key_id	S3 access key
/atrax/stage/atrax-api/s3_secret_access_key	S3 secret key
/atrax/stage/atrax-node/controller_url	Atrax API controller URL (plain String)
/atrax/stage/atrax-node/controller_auth	Bearer token for controller
/atrax/stage/atrax-node/bugsnag_api_key	Error tracking key

Vault

Path	Description
/vault/{env}/postgres/password	PostgreSQL master password
/vault/{env}/postgres/db_url	PostgreSQL connection URL
/vault/{env}/mariadb/password	MariaDB master password (stage only)
/vault/{env}/mariadb/db_url	MariaDB connection URL (stage only)
/{group}/{env}/vault-api/clickhouse_url	ClickHouse HTTP URL
/{group}/{env}/vault-api/clickhouse_username	ClickHouse username
/{group}/{env}/vault-api/clickhouse_password	ClickHouse password
/{group}/{env}/vault-api/vault_api_key	Static API key for auth
/{group}/{env}/etl/vault_ingest_config	JSON config for ETL pipeline
/{group}/{env}/clickhouse/admin_password	ClickHouse admin password

How Secrets Work in Terraform

Secrets are defined with a placeholder value and ignore_changes on the value:

resource "aws_ssm_parameter" "database_url" {
  name        = "/atrax/${var.environment}/atrax-api/database_url"
  description = "PostgreSQL connection URL for atrax-api"
  type        = "SecureString"
  value       = "placeholder"

  lifecycle {
    ignore_changes = [value]
  }

  tags = merge(var.base_tags, { Service = "atrax-api" })
}

This means:

• terraform apply creates the parameter once with "placeholder"
• After apply, you set the real value manually via CLI
• Subsequent terraform apply runs leave the value untouched

How ECS Tasks Consume Secrets

In the task definition, plain config goes in environment and secrets go in secrets:

container_definitions = jsonencode([{
  name  = "atrax-api"
  image = "${ecr_repo_url}:latest"

  environment = [
    { name = "PORT", value = "3000" },
  ]

  secrets = [
    { name = "DATABASE_URL", valueFrom = aws_ssm_parameter.database_url.arn },
    { name = "AUTH_SECRET",  valueFrom = aws_ssm_parameter.auth_secret.arn },
  ]
}])

The ECS agent fetches secrets from SSM at task startup and injects them as environment variables. This requires the execution role to have ssm:GetParameter and kms:Decrypt permissions:

resource "aws_iam_policy" "read_ssm" {
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["ssm:GetParameter", "ssm:GetParameters"]
        Resource = [aws_ssm_parameter.database_url.arn]
      },
      {
        Effect   = "Allow"
        Action   = ["kms:Decrypt", "kms:DescribeKey"]
        Resource = "*"
      }
    ]
  })
}

KMS Encryption

• SSM parameters use the AWS-managed aws/ssm KMS key automatically
• RDS instances use dedicated customer-managed KMS keys with:
• Automatic annual key rotation (enable_key_rotation = true)
• 30-day deletion window
• prevent_destroy lifecycle protection

Setting and Rotating Secrets

Set a new secret

aws ssm put-parameter \
  --name "/core/stage/core-api/database_url" \
  --value "mysql://user:pass@host:3306/dbname" \
  --type SecureString \
  --region eu-central-1

Update an existing secret

aws ssm put-parameter \
  --name "/core/stage/core-api/database_url" \
  --value "mysql://user:newpass@host:3306/dbname" \
  --type SecureString \
  --overwrite \
  --region eu-central-1

Restart service to pick up new value

ECS tasks only read secrets at startup. After changing a parameter, force a new deployment:

aws ecs update-service \
  --cluster stage-euc1-core-ecs-cluster \
  --service core-api \
  --force-new-deployment \
  --region eu-central-1

Read a secret (for debugging)

aws ssm get-parameter \
  --name "/core/stage/core-api/auth_secret" \
  --with-decryption \
  --region eu-central-1 \
  --query "Parameter.Value" \
  --output text

Database passwords

RDS passwords have ignore_changes = [password] in Terraform. If you rotate a database password, you must update both the RDS instance password and the corresponding SSM parameter, then restart dependent services.