Substrate Infrastructure Upgrade mechanism

[HavenXia]

We need to figure out a best upgrade mechanism for upgrading the substrate system.

I have in-place upgrade, in-cluster green/blue and across cluster green/blue - 3 mode to evaluate. We need to take cost and performance into consideration.

[HavenXia]

Some general ideas

Option1: in-place rolling upgrade

Upgrade the existing install component by component: apply new images, let each Deployment roll, during the upgrade both old and new components are accepting traffic.

        clients
           │   (endpoint never moves)
           ▼
┌─ cluster ───────────────────────────┐
│  one stack, upgraded in place:      │
│  v1 pods and v2 pods run mixed      │
│  while each Deployment rolls        │
│              │                      │
│              ▼                      │
│      Valkey: both versions          │
│      write it at the same time      │
└─────────────────────────────────────┘

Option2: Blue/green inside the same cluster

Install a second, renamed copy of the stack ("green") in the same cluster, plus a new worker node pool. Both stacks can reach the same Valkey. To upgrade: suspend all actors, switch the endpoint to green, resume the actors on green, then delete the blue stack.

        clients
           │   (endpoint flips: blue → green)
           ▼
┌─ cluster ─────────────────────────────────┐
│  blue stack (v1)      green stack (v2)    │
│  old node pool        new node pool       │
│        │                    │             │
│        └─────────┬──────────┘             │
│                  │                        │
│                  ▼                        │
│         Valkey (shared, stays put)        │
└───────────────────────────────────────────┘

Option3: Blue/green with a new cluster

Create a brand-new cluster running the new version, get the data over, point clients and DNS at it, delete the old cluster.

        clients
           │   (DNS / address repoints to the new cluster)
           ▼
┌─ old cluster ─────┐     ┌─ new cluster ─────┐
│  stack v1         │     │  stack v2         │
│  workers          │     │  workers          │
└────────┬──────────┘     └────────┬──────────┘
         │                         │
         └────────────┬────────────┘
                      ▼
        store reachable from both clusters:
        external Valkey - (does not exist today),
        or an export / import mechanism we build?