Implement Vault Event Notification System for Kvv2 Hot Reload

[drewmullen]

Users would like to be able to reload Kvv2 but are currently not capable because the secret engine does not implement a ttl. Without a TTL the kvv2 engine cannot utilize the LeaseListener.

I propose we leverage Vaults Event Notification System (enterprise only) to listen for changes to kvv2 secrets and enable the client to hot reload. The mechanism works via websockets which can be implemented via via springs built in websockets functionality. The specific events would be kv-v2/data-{patch,write}.

Project Specific Questions:

• What project requirements are there for implementing a Enterprise only feature (any flags I need to set, etc?)

Open Questions:

• What changes need, if any, to bubble up to spring config vault (i think none)?
• Design for implementing filtering where applications may need to subscribe to many secrets (details link)
• Is it possible for users who use spring config vault's kvv2 implementation to benefit from this function (i dont think so but worth exploring)?

[mp911de]

We currently cannot test enterprise features and that is a showstopper for support of any enterprise feature really. Overall, a consistent rotation and update propagation mechanism within Vault would have been superb, especially if it would simplify the integration side so that e.g. Spring Vault would not be required to implement its own lease renewal/rotation.

Taking a step back, Vault has a concerning sprawl of mechanisms to support rotation/updates of secrets:

• KV version 1 started with TTL support
• KV version 2 never supported TTLs
• Database engines use primarily leases
• PKI Certificates could use leases but it's discouraged because of load concerns, so really checking not-after of x.509 certificates seems the only viable option

Now there's an event stream. Concerning reliability, there are several pub/sub systems that are designed to emit interesting events with the limitation if you don't listen in the right moment, updates are lost (e.g. Redis Pub/Sub). WebSocket connections may experience interruptions. In such case, there's no way to provide a last seen event Id to the subscription meaning that events emitted between the last seen event and after recovery are lost for the application.

We deliberately introduced SecretLeaseContainer as primary event bus for secret creation and rotation to keep an event-driven abstraction. A Vault built-in event bus throws another concept at us requiring careful design consideration how such a component would actually fit into property sources and how we could notify components that are interested on updates to secrets.

Propagating updates of secrets to consumers (such as connection pools or even application components that require secrets) is a sensible concern that we're interested in, however, there's quite some sprawl of approaches (e.g. HikariCP allows direct reconfiguration while e.g. MongoDB would require a newly configured client instance). That leaves us with merely providing infrastructure to publish changes in secrets so that downstream components such as Spring Cloud Vault can deliver these changes to actual database and integration consumers.

As this is an enterprise-only feature, we don't have means of testing it. We perceive it also as huge skew: OSS maintainers are asked through enterprise-only feature motivation to invest time so that an exclusive audience can consume some feature. At the same time, exclusive audiences are easier to convince of discontinuing features because of $randomCompanyReasons leading to frustration on the OSS side.

While it may not be obvious, removal of inmem in Vault Enterprise is the most recent instance of discontinuing features because of $reasons.