You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem?
The current encrypt_credentials implementation sends entire credential JSON to AWS KMS, which has a 4096-byte limit. This poses issues for schema-agnostic credentials, especially as larger payloads are planned.
Describe the solution you'd like
Implement envelope encryption for KMS credentials:
Write: Use GenerateDataKey to obtain a 32-byte data key; encrypt credentials locally with AES-256-GCM and store as kms.v2:<wrapped_dek>:<nonce>:<ciphertext>.
Read: Decrypt the wrapped DEK using KMS, then decrypt the payload locally.
Ensure:
Update backend/app/core/security.py for the kms.v2: encryption/decryption without changing function signatures.
Create a new Alembic migration to mirror 073_reencrypt_credentials.py.
Add tests in backend/app/tests/core/test_security.py and .../services/credentials/test_reencrypt.py.
Original issue
Describe the current behavior
encrypt_credentials in backend/app/core/security.py sends the entire credential JSON as plaintext to AWS KMS Encrypt. That API caps plaintext at 4096 bytes, so the size of a credential is bounded by the KMS request limit.
This is fine for the API-key-shaped credentials we store today (~50 bytes), but the credential column is deliberately schema-agnostic — a free-form JSON blob whose interpretation is driven by the provider column — and larger payloads are already on the roadmap:
A schema-agnostic column with a 4 KB ceiling is a contradiction that surfaces in production on whichever tenant happens to have the largest credential.
Describe the enhancement you'd like
Switch KMS credential encryption to envelope encryption, stored under a new kms.v2: ciphertext prefix.
Write:GenerateDataKey returns a 32-byte data key (DEK) both in plaintext and KMS-wrapped, in a single call. Encrypt the credential locally with AES-256-GCM, discard the plaintext DEK, store kms.v2:<wrapped_dek>:<nonce>:<ciphertext> (base64 segments).
Read: KMS-decrypt the ~200-byte wrapped DEK, then decrypt the payload locally.
The 4096-byte limit is a cap on the KMS API request body, not on how much data a KMS key can protect. A DEK is always small regardless of payload size, which is what removes the ceiling. This is the same pattern S3 SSE-KMS, EBS, and RDS encryption use internally.
Scope:
backend/app/core/security.py — add the kms.v2: branch to encrypt_credentials / decrypt_credentials. Function signatures do not change, so none of the 6 call sites are touched.
New one-shot Alembic migration mirroring 073_reencrypt_credentials.py, calling the existing execute_credential_reencrypt() — that service needs zero changes.
Tests in backend/app/tests/core/test_security.py and .../services/credentials/test_reencrypt.py.
Why is this enhancement needed?
Unblocks GCP/Vertex credential storage. Storing a service-account JSON per org/project is a hard requirement of the google-gcp routing work; today it fits only barely, and any second key or added field breaks it.
No cost or latency regression. Still exactly one KMS API call per read and per write. (GenerateDataKey returns both DEK copies in one round trip.)
Backwards compatible by construction.decrypt_credentials already routes by ciphertext prefix, so legacy Fernet rows and existing kms.v1: rows keep decrypting indefinitely. Nothing needs a coordinated cutover.
No new dependencies.boto3 and cryptography (which provides AESGCM) are already in backend/pyproject.toml.
Enables DEK caching later if credential-read volume ever justifies it — impossible with direct KMS encryption, which forces a network call on every read.
Additional context
Deployment state (verified against release tags):
Prod is on v1.3.1, alembic head 071.security.py at that tag contains no KMS code — prod credentials are still pure Fernet.
Staging (v1.4.0-main.8) is on head 075 and has already run 073_reencrypt_credentials, so it holds kms.v1: rows.
Therefore, if this lands before the 1.4.0 prod release, prod goes Fernet → kms.v2 in one hop (073 simply calls encrypt_credentials, which will emit v2 by then) and the new backfill migration is a near no-op there. The backfill is still required for staging's existing kms.v1: rows, and is idempotent everywhere.
Related: KMS: Deployment and backup strategy #1096 (KMS: Deployment and backup strategy) — the credential-table backup and recovery test described there should account for this format change before the prod KMS rollout.
Deliberately out of scope:
AAD / KMS encryption-context binding (binding ciphertexts to org_id/project_id so a blob lifted from one tenant's row can't decrypt against another's). Worth doing, but it changes function signatures across all call sites; it can be layered on later as a kms.v3 format.
DEK caching — optimization, not needed at current read volume.
Is your feature request related to a problem?
The current
encrypt_credentialsimplementation sends entire credential JSON to AWS KMS, which has a 4096-byte limit. This poses issues for schema-agnostic credentials, especially as larger payloads are planned.Describe the solution you'd like
Implement envelope encryption for KMS credentials:
GenerateDataKeyto obtain a 32-byte data key; encrypt credentials locally with AES-256-GCM and store askms.v2:<wrapped_dek>:<nonce>:<ciphertext>.Ensure:
backend/app/core/security.pyfor thekms.v2:encryption/decryption without changing function signatures.073_reencrypt_credentials.py.backend/app/tests/core/test_security.pyand.../services/credentials/test_reencrypt.py.Original issue
Describe the current behavior
encrypt_credentialsinbackend/app/core/security.pysends the entire credential JSON as plaintext to AWS KMSEncrypt. That API caps plaintext at 4096 bytes, so the size of a credential is bounded by the KMS request limit.This is fine for the API-key-shaped credentials we store today (~50 bytes), but the
credentialcolumn is deliberately schema-agnostic — a free-form JSON blob whose interpretation is driven by theprovidercolumn — and larger payloads are already on the roadmap:EncryptValidationExceptionA schema-agnostic column with a 4 KB ceiling is a contradiction that surfaces in production on whichever tenant happens to have the largest credential.
Describe the enhancement you'd like
Switch KMS credential encryption to envelope encryption, stored under a new
kms.v2:ciphertext prefix.GenerateDataKeyreturns a 32-byte data key (DEK) both in plaintext and KMS-wrapped, in a single call. Encrypt the credential locally with AES-256-GCM, discard the plaintext DEK, storekms.v2:<wrapped_dek>:<nonce>:<ciphertext>(base64 segments).The 4096-byte limit is a cap on the KMS API request body, not on how much data a KMS key can protect. A DEK is always small regardless of payload size, which is what removes the ceiling. This is the same pattern S3 SSE-KMS, EBS, and RDS encryption use internally.
Scope:
backend/app/core/security.py— add thekms.v2:branch toencrypt_credentials/decrypt_credentials. Function signatures do not change, so none of the 6 call sites are touched.073_reencrypt_credentials.py, calling the existingexecute_credential_reencrypt()— that service needs zero changes.backend/app/tests/core/test_security.pyand.../services/credentials/test_reencrypt.py.Why is this enhancement needed?
GenerateDataKeyreturns both DEK copies in one round trip.)decrypt_credentialsalready routes by ciphertext prefix, so legacy Fernet rows and existingkms.v1:rows keep decrypting indefinitely. Nothing needs a coordinated cutover.boto3andcryptography(which providesAESGCM) are already inbackend/pyproject.toml.Additional context
Deployment state (verified against release tags):
security.pyat that tag contains no KMS code — prod credentials are still pure Fernet.073_reencrypt_credentials, so it holdskms.v1:rows.kms.v2in one hop (073 simply callsencrypt_credentials, which will emit v2 by then) and the new backfill migration is a near no-op there. The backfill is still required for staging's existingkms.v1:rows, and is idempotent everywhere.Deliberately out of scope:
org_id/project_idso a blob lifted from one tenant's row can't decrypt against another's). Worth doing, but it changes function signatures across all call sites; it can be layered on later as akms.v3format.