DSAR Storage and Encryption¶
How Core stores and protects the files and personal data of Data Subject Access Requests (DSARs): a dedicated S3 "vault" bucket for temporary disclosure files, and application-level AES-256-GCM encryption on top of it. The first part sets up the bucket and its application configuration; the second covers the encryption key.
The encryption key is not settled
Whether the DSAR_ENCRYPTION_KEY in use is the one that was once committed to the Core repository, and whether it
must be rotated, is being resolved in IS-132. Until that is closed, do not copy a key from anywhere: generate a
fresh one with the helper below, and do not rotate a live key without reading the rotation warning. Keys never go in
this page or in git.
Part 1: the S3 vault¶
This guide explains how to set up the S3 "Vault" for the DSAR Processor Model migration. The vault provides secure, temporary storage for sensitive data disclosure files.
Overview¶
The DSAR system now uses a dedicated AWS S3 bucket as a "vault" for temporary storage of sensitive data. Files are: - Never stored on disk (streamed directly to S3) - Automatically deleted after 14 days (via S3 Lifecycle policies) - Accessed via presigned URLs (15-minute validity) - Tracked in the database with expiration dates
This implements the "Processor" model where the system holds data temporarily only on the Controller's (account owner's) instructions.
AWS S3 Setup¶
1. Create a Dedicated S3 Bucket¶
Bucket naming convention: {org}-dsar-vault-{environment}
- Example: cookiehub-dsar-vault-prod
2. Enable Versioning (Optional)¶
For audit trail protection, enable versioning:
aws s3api put-bucket-versioning \
--bucket your-org-dsar-vault-prod \
--versioning-configuration Status=Enabled
3. Set Bucket Encryption¶
Enable default encryption with AES-256:
aws s3api put-bucket-encryption \
--bucket your-org-dsar-vault-prod \
--server-side-encryption-configuration '
{
"Rules": [
{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "AES256"
}
}
]
}'
4. Block Public Access¶
Ensure the bucket is completely private:
aws s3api put-public-access-block \
--bucket your-org-dsar-vault-prod \
--public-access-block-configuration \
"BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
5. Set Lifecycle Policy (14-Day Auto-Purge)¶
Create a lifecycle policy to automatically delete files after 14 days:
aws s3api put-bucket-lifecycle-configuration \
--bucket your-org-dsar-vault-prod \
--lifecycle-configuration '{
"Rules": [
{
"Id": "DeleteDSARDisclosuresAfter14Days",
"Status": "Enabled",
"Prefix": "disclosures/",
"Expiration": {
"Days": 14
},
"NoncurrentVersionExpiration": {
"NoncurrentDays": 14
}
},
{
"Id": "CleanupIncompleteMultipartUploads",
"Status": "Enabled",
"Prefix": "disclosures/",
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 7
}
}
]
}'
6. Create IAM Policy¶
Create an IAM policy for the application to access the vault:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DSARVaultAccess",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:HeadObject"
],
"Resource": "arn:aws:s3:::your-org-dsar-vault-prod/disclosures/*"
},
{
"Sid": "DSARVaultListBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::your-org-dsar-vault-prod",
"Condition": {
"StringLike": {
"s3:prefix": "disclosures/*"
}
}
}
]
}
7. Create IAM User or Role¶
Create a dedicated IAM user or assume role for the application:
aws iam create-user --user-name dsar-vault-processor
aws iam put-user-policy \
--user-name dsar-vault-processor \
--policy-name dsar-vault-policy \
--policy-document file://policy.json
aws iam create-access-key --user-name dsar-vault-processor
Note the AccessKeyId and SecretAccessKey from the output.
Application Configuration¶
1. Set Environment Variables¶
Add the following to your .env file:
# DSAR S3 Vault (Processor Model)
DSAR_VAULT_ACCESS_KEY_ID=your-iam-user-access-key
DSAR_VAULT_SECRET_ACCESS_KEY=your-iam-user-secret-key
DSAR_VAULT_REGION=us-east-1
DSAR_VAULT_BUCKET=your-org-dsar-vault-prod
DSAR_VAULT_URL=https://your-org-dsar-vault-prod.s3.amazonaws.com
DSAR_VAULT_ENDPOINT=https://s3.amazonaws.com
DSAR_VAULT_USE_PATH_STYLE_ENDPOINT=false
2. Run Database Migrations¶
This will:
- Add expires_at column to dsar_disclosures table
- Create dsar_instruction_logs table for compliance tracking
3. Test the Configuration¶
php artisan tinker
# Test S3 connection
Storage::disk('s3-dsar-vault')->put('test.txt', 'test content');
Storage::disk('s3-dsar-vault')->get('test.txt'); // Should return 'test content'
Storage::disk('s3-dsar-vault')->delete('test.txt');
How It Works¶
File Upload Flow (Processor Model)¶
- Admin completes DSAR request with attached files
- System creates ZIP in memory (never on disk)
- ZIP is streamed to S3 vault immediately
- Disclosure record created with S3 path, hash, and 14-day expiration
- Instruction log entry created documenting the Controller's instruction
- Email sent with presigned URL (15-minute validity) and password
File Download Flow (Presigned URLs)¶
- User clicks download link from email
- Application verifies request is still valid (not expired, not rejected)
- Application generates presigned URL (15-minute validity)
- User is redirected directly to S3
- Browser downloads directly from S3
- S3 Lifecycle policy automatically deletes after 14 days
Processor Model Compliance¶
| Aspect | Implementation |
|---|---|
| Data Minimization | Files only in S3 vault, temp storage on local disk |
| Temporary Storage | 14-day auto-delete via S3 lifecycle policy |
| Instruction Logging | dsar_instruction_logs table tracks all actions |
| Access Control | Presigned URLs prevent unauthorized access |
| Audit Trail | Complete action logging with IP, user agent |
| Server Isolation | Server acts as gatekeeper, not delivery truck |
Monitoring and Maintenance¶
Monitor S3 Storage Usage¶
Verify Lifecycle Policies¶
Check for Failed Uploads¶
Monitor CloudWatch logs for DsarVaultService errors:
aws logs filter-log-events \
--log-group-name /aws/laravel/dsar \
--filter-pattern "DsarVaultService"
Troubleshooting¶
Presigned URL Generation Fails¶
Problem: Failed to generate presigned URL
Solution:
1. Verify IAM credentials are correct
2. Check bucket name in .env
3. Ensure IAM policy includes s3:GetObject
4. Check CloudWatch logs for detailed error
Files Not Automatically Deleted¶
Problem: Files older than 14 days still exist in S3
Solution: 1. Verify lifecycle policy is enabled:
2. Note: S3 may take up to 24 hours to delete files 3. You can manually clean up older files:aws s3 rm s3://your-bucket/disclosures/ --recursive --exclude "*" \
--include "disclosures/*" \
--older-than 14d
Access Denied Errors¶
Problem: Access Denied when uploading to S3
Solution:
1. Verify IAM user/role has correct policy
2. Check that the policy resource is correct: arn:aws:s3:::bucket-name/disclosures/*
3. Verify AWS credentials in .env
4. Test manually:
aws s3 put-object \
--bucket your-org-dsar-vault-prod \
--key disclosures/test.txt \
--body /path/to/test.txt
Migration from Local Storage¶
For existing requests in local storage, there are two options:
- Keep existing files local - Only new requests use S3 vault
- Migrate existing data - Create an artisan command to copy to S3
To migrate existing disclosures (optional):
Security Considerations¶
✅ What we've implemented: - Presigned URLs with 15-minute validity (prevents sharing) - Password-protected ZIPs - Files deleted after 14 days - Instruction logging for accountability - No PII in file paths (UUIDs only) - Encrypted transport (HTTPS)
⚠️ Additional measures to consider: - Enable S3 access logging - Enable CloudTrail for API audit - Use VPC endpoints to prevent internet exposure - Implement IP whitelisting for API access - Encrypt at rest with customer-managed KMS keys - Use MFA delete protection
Legal Compliance¶
This implementation supports GDPR Article 15 (Right of Access) by:
- Data Minimization - Temporary storage only
- Purpose Limitation - Only for fulfilling the request
- Accountability - Complete instruction and action logs
- Lawfulness - Presigned URLs prevent unauthorized access
- Integrity & Confidentiality - Encrypted transport and storage
See dsar_instruction_logs table for proof of:
- Who authorized the disclosure
- When the instruction was given
- What action was taken
- Legal basis for processing
Part 2: encryption¶
This guide explains how to set up encryption for DSAR (Data Subject Access Request) files and personal data.
Overview¶
The DSAR system uses AES-256-GCM symmetric encryption to protect sensitive data. This provides both confidentiality and authenticity (authenticated encryption). A single shared key is used for all encryption and decryption operations.
Key Components¶
- Encryption Key: A 256-bit (32-byte) key stored in
.envasDSAR_ENCRYPTION_KEYin Laravel'sbase64:format - DsarEncryptionService: Handles encrypt/decrypt for file contents and personal data
- Blind-Index Search: HMAC-SHA256 trigram tokens allow searching encrypted PII without decryption
What Gets Encrypted¶
- Personal data in
dsar_requeststable (name, email, requester_name, requester_email, details) — via Laravel'sencryptedcast - Uploaded files — encrypted with AES-256-GCM before S3 storage
- Disclosure ZIPs — encrypted before S3 storage
- S3 vault credentials in
dsar_settings— via Laravel'sencryptedcast - S3 server-side encryption — AES-256 at rest in S3
Setup Instructions¶
1. Generate an Encryption Key¶
Use the static helper method to generate a key:
php artisan tinker
>>> App\Services\DsarEncryptionService::generateKey()
=> "base64:<generated by the tinker helper>"
2. Add to .env File¶
Important:
- The key must include the base64: prefix
- Do not wrap in quotes
- Do not commit the .env file to version control
3. Verify Setup¶
The key is loaded via config('services.dsar.encryption_key') (defined in config/services.php). To verify:
php artisan tinker
>>> $svc = App\Services\DsarEncryptionService::forAccount();
>>> $encrypted = $svc->encrypt('test');
>>> $svc->decrypt($encrypted);
=> "test"
Encryption Format¶
Encrypted data is stored as a base64-encoded JSON package:
{
"v": 1,
"iv": "<base64 encoded 96-bit IV>",
"data": "<base64 encoded ciphertext>",
"tag": "<base64 encoded GCM auth tag>"
}
Each encryption operation generates a unique random IV, so encrypting the same data twice produces different ciphertext.
Where Encryption Is Used¶
- File Upload (
FileController::uploadStagedFile) -
Encrypts file content before storing in S3 vault
-
ZIP Creation (
ItemController::complete) - Decrypts staged files, bundles into ZIP
-
Encrypts final ZIP before S3 storage
-
User Download (
UserController::download) -
Decrypts ZIP before streaming to user
-
Admin Preview (
FileController::downloadStagedFile) - Decrypts file before returning to admin
Key Rotation¶
To rotate the encryption key:
1. Generate a new key using DsarEncryptionService::generateKey()
2. Update DSAR_ENCRYPTION_KEY in .env
3. Warning: Files encrypted with the old key will no longer be decryptable. Ensure all active disclosures have expired before rotating.
Troubleshooting¶
"Encryption key must be 32 bytes" Error¶
The key is not the correct length. Regenerate using DsarEncryptionService::generateKey() and ensure the base64: prefix is present.
"DSAR_ENCRYPTION_KEY not configured" Error¶
The key is missing from .env or config/services.php is not loading it. Run php artisan config:clear and verify the key is set.
"Failed to decrypt data or authentication failed" Error¶
The data was encrypted with a different key, or the encrypted package has been corrupted. Check that the current DSAR_ENCRYPTION_KEY matches the one used when the data was encrypted.
Security Notes¶
- The encryption key must be kept secure and never committed to version control
- AES-256-GCM provides authenticated encryption — tampered ciphertext is detected and rejected
- Use a secrets manager (e.g., AWS Secrets Manager) in production
- Back up the encryption key securely — losing it means losing access to all encrypted DSAR data
- The same key is used for all accounts; per-account keys can be implemented by extending
DsarEncryptionService::forAccount()