Цю сторінку ще не перекладено.

Upgrade and roll back

This page is for operators of a self-hosted instance. It shows how to move the instance to a newer commit of the repository in the right order, what has to restart together, and how to undo a bad upgrade. Installing is Self-hosting.

An upgrade is the install's Helm sequence again, from a newer commit, with new image tags. The schema migration runs before the new imd-svc starts, and a new gateway release restarts every gateway replica together.

Before you start

  • Know what you're shipping. A deploy is a commit: the charts, the generated configs and the image tags all come from it. Read what changed on Changes and status.
  • Check the generated files. The committed gateway config and realms must be what the generators write from that commit. A difference means someone edited a generated file, or didn't regenerate:
Shell
# the committed gateway config and realms are what the generators write
python3 deploy/mcpg/gen-config.py <profile> && python3 deploy/mcpg/check-config.py <profile>
python3 deploy/keycloak/gen-realms.py <profile>
git diff --exit-code deploy/mcpg deploy/keycloak/realms
  • Check the backups. Schema migrations run once and only forward. If one goes wrong, the way back is a restore. Make sure the last base backup completed and WAL is archiving (kubectl -n instruction-md get backup).
  • A new gateway release passes its own check first: see below.

Check a new gateway release

This page was built against MCPG 0.9.1 (ghcr.io/mcpg-dev/mcpg:0.9.1). Before image.tag in deploy/k8s/imd-gateway/values.yaml moves to another release, prove it runs your whole server:

Shell
# is the release published? 200 means it can be pulled anonymously
tok=$(curl -s "https://ghcr.io/token?scope=repository:mcpg-dev/mcpg:pull&service=ghcr.io" \
      | python3 -c 'import json,sys;print(json.load(sys.stdin)["token"])')
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $tok" \
  -H "Accept: application/vnd.oci.image.index.v1+json" \
  "https://ghcr.io/v2/mcpg-dev/mcpg/manifests/<tag>"

# does it run the whole server? the production charts on one machine, with that release
IMD_GATEWAY_IMAGE=ghcr.io/mcpg-dev/mcpg:<tag> deploy/k8s/stack/up.sh

The public image is ghcr.io/mcpg-dev/mcpg. A 200 means it can be pulled; it says nothing more. The second command runs the production charts on one machine with that release; then run your own checks against http://localhost:18080.

When a release comes with new builds of the plugins, move the versions in deploy/mcpg/pins.json in the same change as image.tag, then regenerate the configs. The plugin refs are pinned to a release version, never floating, and check-config.py refuses anything else. Never move the tag alone: a gateway release loads only the plugin builds made for its layout, and may refuse a config key the other release doesn't know.

Upgrade

Charts upgrade in install order; a new gateway release restarts every replica together.
  1. Make the secrets

    Run deploy/k8s/bootstrap-secrets.sh instruction-md again. It keeps every value it finds and writes the derived Secrets anew, so a new key a release needs is made.

  2. The edge, when it changed

    helm upgrade --install imd-edge … with the same values as at install. Never with gateway.create=false: see The edge.

  3. Postgres, NATS and Keycloak

    In that order, each waited for, as at install. Keycloak's job then applies the realm content: it adds what is new and leaves what exists. When the realm files changed a client, upgrade Keycloak with apply.overwrite=true, which recreates the changed client:

    Shell
    # the realm files changed: recreate a changed client (the job otherwise adds what is new and leaves what exists)
    helm upgrade imd-keycloak deploy/k8s/imd-keycloak -n instruction-md --reuse-values --set apply.overwrite=true --wait --timeout 20m
  4. The site

    Upgrade imd-web with the new site image before imd-svc: the pages imd-svc serves load a script from the site, so the site goes first.

  5. The embedding model and imd-svc

    Upgrade imd-embeddings, then imd-svc with the new image tag. Pass a hash of the Secret imd-svc as secretRevision, as in Optional services, so a key added to the Secret rolls the pods even on the same image. The migration job imd-svc-migrate runs first, as the owner role, and its log stays until the next upgrade. Each migration runs in its own transaction, and the run stops at the first error: the failed migration leaves no trace, the ones before it in the same run stay applied, and the old pods keep running.

  6. The gateway

    A new gateway release moves with a full restart: every replica stops, then the upgrade starts them together. The same release with a new config rolls one replica at a time.

    Shell
    NS=instruction-md
    want="$(sed -n 's/^  repository: //p' deploy/k8s/imd-gateway/values.yaml):$(sed -n 's/^  tag: //p' deploy/k8s/imd-gateway/values.yaml)"
    have=$(kubectl -n $NS get statefulset imd-gateway \
      -o 'jsonpath={.spec.template.spec.containers[?(@.name=="gateway")].image}' 2>/dev/null || true)
    if [ -n "$have" ] && [ "$have" != "$want" ]; then
      echo "gateway: $have → $want, a full restart"
      kubectl -n $NS scale statefulset imd-gateway --replicas=0
      kubectl -n $NS wait --for=delete pod -l app.kubernetes.io/name=imd-gateway --timeout=5m || true
    fi
    helm upgrade --install imd-gateway deploy/k8s/imd-gateway -n $NS -f <gateway-values.yaml> \
      --set-file config=deploy/mcpg/<profile>.yaml --set route.enabled=true --wait --timeout 10m

Why the gateway restarts whole

Two different gateway releases must never serve side by side: their wire and their plugin sets differ. A rolling update of a new release would run one replica of each for a while, so the old ones stop first. /mcp is unavailable from the moment the last old replica stops until a new one is ready. Clients reopen their streams when the new replicas come up, and every request stands alone on MCP 2026-07-28, so nothing else has to be redone.

Check it worked

Shell
kubectl -n instruction-md logs job/imd-svc-migrate --tail=40      # what the migration applied
kubectl -n instruction-md logs job/imd-keycloak-apply --tail=60   # each realm applied
kubectl -n instruction-md logs statefulset/imd-gateway | grep -m1 'application bootstrap complete'
kubectl -n instruction-md get pods                                # every pod Running, none restarting
curl -s -o /dev/null -w '%{http_code}\n' https://<your host>/mcp   # 401: the door asks for a sign-in

Then sign in to the editor, open an instruction, and have an agent read one over MCP.

Roll back

Two different things are called rollback, and confusing them makes an incident worse:

  • A deployment rollback puts the previous software or config back. The rest of this section.
  • A content rollback moves a ref to an earlier version, so followers get the earlier text: instruction refs <id> set latest <version>. That is a product action, audited, and needs no deploy.

Roll back first and diagnose afterwards, from the logs and the failed release. Roll back only what the bad change touched, newest part first: the gateway, then imd-svc, then the site.

Shell
helm -n instruction-md history imd-svc
helm -n instruction-md rollback imd-svc <REV> --wait --timeout 10m
helm -n instruction-md history imd-web
helm -n instruction-md rollback imd-web <REV> --wait --timeout 10m

A different gateway release goes back with the same full restart:

Shell
# a different gateway release: every replica stops first, then the rollback starts them together
kubectl -n instruction-md scale statefulset imd-gateway --replicas=0
kubectl -n instruction-md wait --for=delete pod -l app.kubernetes.io/name=imd-gateway --timeout=5m
helm -n instruction-md history imd-gateway
helm -n instruction-md rollback imd-gateway <REV> --wait --timeout 10m

A Helm rollback of imd-svc doesn't run the migration. The migration is a pre-install and pre-upgrade hook, so the schema and the SQL functions stay where the newer run left them, and the older image runs against them. When the bad change touched the SQL functions or the gateway's config, revert the commit and upgrade forward instead: every migration applies the functions anew from its own commit, and the gateway rolls with the matching config.

Change Rolls back cleanly
imd-svc's image, a chart's values Yes, with helm rollback. imd-svc's migration doesn't run again
The gateway's image and config Yes, with helm rollback imd-gateway, and a full restart for another release
The site (imd-web) Yes, with helm rollback imd-web. Roll imd-svc back first if the site's page script changed with it
SQL functions Yes, by a revert and an upgrade: every migration applies them anew
Schema migrations No. They run once, forward only. Going back means a restore
Settings the migration writes into the database Not by helm rollback, which runs no migration. Set the chart's value and upgrade
Keycloak realm content No. Revert the realm files and upgrade Keycloak with apply.overwrite=true, or change it in the admin console. Never delete and re-import a live realm: that destroys organizations and identity-provider links
Secrets Set the previous value again in your secret store, run bootstrap-secrets.sh, and restart what reads it
A revoked key or version No, by design. Publish a new version and move the refs
The database's contents No, and must not: they are the data

The edge

  • imd-edge goes forward only. Never helm rollback imd-edge to a revision from before the Gateway was part of the release, and never set gateway.create=false on it. A Gateway that leaves the manifest is deleted, and with it Envoy's Service and the data plane: every address of your instance at once. The live Gateway should carry helm.sh/resource-policy: keep as a guard. Check it is there before any change to the release:
Shell
kubectl -n envoy-gateway-system get gateway default -o jsonpath='{.metadata.annotations}'
# must include "helm.sh/resource-policy":"keep"
  • To undo a change to the edge, revert the commit with gateway.create still true, and upgrade.
  • A policy that Envoy Gateway can't apply fails closed: the routes it guards answer 500, never unguarded. After 500s on one route right after an upgrade, read the policies' status and revert that change first:
Shell
kubectl -n instruction-md get backendtrafficpolicy -o yaml
kubectl -n envoy-gateway-system get clienttrafficpolicy imd-edge -o yaml
  • The hop counts move together. The edge's trustedHops, imd-svc's config.forwardedHops and the site's clientHops change in one commit: the last two are always the first plus one.

Restore Postgres

A restore is the way back from a bad schema migration or lost data. It takes the database to a point in time; everything written after that point is gone, including the audit.

Shell
# 1. stop the writers
kubectl -n instruction-md scale statefulset imd-gateway --replicas=0
kubectl -n instruction-md scale deploy imd-svc --replicas=0
# 2. bring up a CloudNativePG cluster from the bucket, with bootstrap.recovery and the recoveryTarget you chose
# 3. point the database URLs at it, or rename the clusters so the URLs stand
# 4. make it instruction.md's again: the migration with --after-restore
#    (it reloads the functions, confirms the recorded migrations and raises the events' epoch)
# 5. check it, then start the writers (the charts' replicaCount: 2 each)
kubectl -n instruction-md scale deploy imd-svc --replicas=2
kubectl -n instruction-md scale statefulset imd-gateway --replicas=2
  1. Stop the writers: scale the gateway and imd-svc to zero. The editor says the workspace didn't open and keeps what is on the device.
  2. Restore: bring up a CloudNativePG cluster from the bucket with bootstrap.recovery and the recoveryTarget you chose. Keep the failed cluster until the restore is checked.
  3. Point at it: change the database URLs, or rename the clusters so the URLs stand.
  4. Make it instruction.md's again: run the imd-svc image as imd-svc migrate --after-restore, with the migration job's environment and Secret. It reloads the functions, confirms the recorded migrations against the release, and raises the events' epoch. After a point-in-time restore, transaction ids and sequences go back to the restore point; without a new epoch, the gateway would pass over new events until the counters caught up.
  5. Check it: the instance answers, and the newest versions before the restore point are there.
  6. Start the writers: scale imd-svc and the gateway back up.

Events the gateway delivered for writes that are now lost can't be taken back. Clients read again on their next change: the editor re-reads its workspace after a gap in the activity stream.

If it doesn't work

What you see Why What to do
The imd-svc upgrade fails and imd-svc-migrate's log shows an error A migration failed. It ran in its own transaction, so nothing of it was applied; earlier migrations of the same run were Fix forward: a migration is never edited once applied, and the migrate command refuses a changed checksum
another operation (install/upgrade/rollback) is in progress An earlier run stopped half-way and left the release pending helm rollback the release to its last good revision, or helm uninstall an install that never finished
The gateway doesn't start after a release change Its plugin versions or config don't match the release Move image.tag, pins.json and the regenerated config together, or roll back with the full restart
imd-svc refuses the gateway as unauthenticated The gateway's key pair changed and only one side restarted Restart imd-svc and the gateway together
Changed realm settings don't take The apply job leaves what exists Upgrade Keycloak with apply.overwrite=true
Still stuck Ask us

Questions

Do I need downtime for an upgrade?

Not for most. imd-svc and the site roll one pod at a time, and a gateway config change rolls one replica at a time. A new gateway release is a full restart, so /mcp is down until a new replica is ready. See Why the gateway restarts whole.

Can I skip commits when I upgrade?

The migrations apply in order from whatever the database records, so one upgrade applies every migration it hasn't seen. Generated configs and pinned plugin versions are always the commit's own. Check the generated files before you deploy.

How do I undo a schema migration?

You can't roll it back: migrations run once, forward only. A wrong one is fixed by the next migration, or, if data was lost, by a restore.

Does helm rollback undo everything a release changed?

No. It runs no migration, so the schema, the SQL functions and the settings the migration writes stay as the newer run left them. Revert and upgrade forward instead. See Roll back.

Last checked against the product:

Was this page helpful? Report a problem