Kubernetes install

For a single host see Docker Compose. For local dev see Manual.

The committed manifests are templates. secrets.yaml ships with CHANGE_ME placeholders and ingress.yaml references novamem.example.com. Do not kubectl apply -k deploy/k8s/ as-is — follow Configure first.

Layout

deploy/k8s/
├── kustomization.yaml   # bundles the rest into namespace `novamem`
├── namespace.yaml
├── secrets.yaml         # TEMPLATE — placeholders only, see "Secrets"
├── postgres.yaml        # StatefulSet · Service · PVC
├── qdrant.yaml          # StatefulSet · Service · PVC
├── novamem.yaml         # ConfigMap · Deployment · ClusterIP Service
└── ingress.yaml         # cert-manager Ingress with TLS

Image

The manifest pulls a public multi-arch image (linux/amd64 + linux/arm64) from GitHub Container Registry:

ghcr.io/azrtydxb/novamem:main         # mutable, rebuilt on every main push by CI
ghcr.io/azrtydxb/novamem:sha-<short>  # immutable per commit, also multi-arch

CI builds, scans (Trivy HIGH/CRITICAL with --ignore-unfixed), and pushes both tags from the CI workflow. The :main tag updates on every push to the main branch; pin image: to a :sha-… tag in novamem.yaml if you want immutability.

To roll forward after a CI publish:

kubectl -n novamem rollout restart deploy/novamem

To build locally and bypass the registry (e.g. in an air-gapped lab) — set image: novamem:dev and imagePullPolicy: Never in novamem.yaml, then:

docker build -f go/Dockerfile -t novamem:dev .
docker save novamem:dev | sudo k3s ctr images import -

Configure

Secrets

deploy/k8s/secrets.yaml is a template. The values are intentionally CHANGE_ME — anything you paste in there ends up in git history. Pick one of the patterns below.

Pattern A — kubectl create secret (simplest). Skip the templated secrets.yaml entirely: remove it from kustomization.yaml (or apply the rest with --prune-style discipline) and create the Secret out-of-band:

kubectl create namespace novamem

POSTGRES_PASSWORD="$(openssl rand -base64 24 | tr -d '/+=' )"

kubectl create secret generic novamem-secrets -n novamem \
  --from-literal=NOVAMEM_COOKIE_SECRET="$(openssl rand -hex 32)" \
  --from-literal=NOVAMEM_BOOTSTRAP_ADMIN_EMAIL="admin@example.com" \
  --from-literal=NOVAMEM_BOOTSTRAP_ADMIN_PASSWORD="$(openssl rand -base64 24)" \
  --from-literal=POSTGRES_PASSWORD="${POSTGRES_PASSWORD}" \
  --from-literal=NOVAMEM_WARM_URL="postgres://novamem:${POSTGRES_PASSWORD}@postgres.novamem.svc.cluster.local:5432/novamem"

Pattern B — Sealed Secrets. Install bitnami-labs/sealed-secrets and kubeseal your filled-in secrets.yaml into a SealedSecret CR. The encrypted CR is safe to commit; the controller decrypts it in-cluster.

Pattern C — external-secrets-operator. Install external-secrets and define an ExternalSecret that pulls from Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. Drop secrets.yaml from the kustomization in favor of the ExternalSecret.

The keys consumed by the Deployment (envFrom: secretRef) are:

Key	Notes
NOVAMEM_COOKIE_SECRET	Session signing. openssl rand -hex 32. Rotate to invalidate all sessions.
NOVAMEM_BOOTSTRAP_ADMIN_EMAIL	First admin (only consulted when no admin user exists).
NOVAMEM_BOOTSTRAP_ADMIN_PASSWORD	First admin password. Auto-scrubbed from process.env after the seed runs.
POSTGRES_PASSWORD	Mounted into the Postgres StatefulSet as well.
NOVAMEM_WARM_URL	Full Postgres DSN — embeds the password, so it lives in the Secret, not the ConfigMap.

App config

deploy/k8s/novamem.yaml (ConfigMap):

• NOVAMEM_BASE_URL — set to your TLS-terminated origin (the Ingress hostname). Better Auth's trusted-origin check rejects mismatches.
• NOVAMEM_INSECURE_COOKIES — leave at "0" in production. Only flip to "1" for a local HTTP-only smoke test.

Ingress + TLS

deploy/k8s/ingress.yaml is a working example for cert-manager + nginx-ingress on Let's Encrypt. To use it:

1. Install an Ingress controller. nginx-ingress works out of the box; Traefik / HAProxy / cloud-native (GKE / EKS / AKS) controllers all work — adjust ingressClassName and the ssl-redirect annotation accordingly.
2. Install cert-manager and create a ClusterIssuer. The example references letsencrypt-prod:

   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata: { name: letsencrypt-prod }
   spec:
     acme:
       server: https://acme-v02.api.letsencrypt.org/directory
       email: ops@example.com
       privateKeySecretRef: { name: letsencrypt-prod }
       solvers:
         - http01: { ingress: { class: nginx } }

3. Replace novamem.example.com in ingress.yaml with your DNS name and point that name at your Ingress controller's external IP.
4. Set NOVAMEM_BASE_URL in novamem.yaml to https://<your-domain>.

Alternatives. If you don't want cert-manager, terminate TLS at a cloud load balancer (GCLB managed certs, AWS ALB + ACM, Azure Application Gateway) and point a Service: type=LoadBalancer at it — but only if the LB does the TLS, not the app. The app speaks plain HTTP and never gets a certificate of its own.

Apply

After you've created the Secret out-of-band (Pattern A) or sealed/synced it (B/C), and after you've updated NOVAMEM_BASE_URL and the Ingress hostname:

kubectl apply -k deploy/k8s/

kubectl -n novamem rollout status statefulset/postgres
kubectl -n novamem rollout status statefulset/qdrant
kubectl -n novamem rollout status deploy/novamem

Reach the API

The Service is ClusterIP — reach it through the Ingress on https://<your-domain>. Sign in at https://<your-domain>/admin with the bootstrap admin email + password.

For a quick in-cluster sanity check without DNS, port-forward:

kubectl -n novamem port-forward svc/novamem 7778:7778
# then http://localhost:7778/health   (note: cookies won't work on http,
# this is for /health probes only)

Persistence

Each StatefulSet uses a volumeClaimTemplates against the cluster's default StorageClass. On k3s that's local-path — node-local, not migratable. For HA storage swap in your CSI provider before first apply.

To back up:

kubectl -n novamem exec sts/postgres -- pg_dump -U novamem -d novamem -Fc > novamem-warm.dump
# Qdrant: kubectl exec into the pod and POST /collections/<name>/snapshots

Scaling out

novamem.yaml ships replicas: 1. Raising it is safe for the API and for the streamable MCP endpoint (POST /mcp), on two conditions:

• NOVAMEM_COOKIE_SECRET must be set and identical on every pod. MCP session ids are signed with a key derived from it, which is how a pod that never saw a client's initialize can still verify and serve that client's session. Without the secret (NOVAMEM_AUTH_MODE=none), ids are unsigned and confined to the pod that minted them — clients then fail with 404 {"error":"unknown sessionId"} on roughly 1 - 1/N of their calls, which looks like a client bug but is a scaling one.
• strategy: Recreate and the ReadWriteOnce data volume assume one pod at a time. Switch the strategy and provision RWX (or node-pin) before scaling the pods that mount them.

No MCP transport needs load-balancer affinity any more. The legacy HTTP+SSE pair did — a message POST had to reach the goroutine holding that session's open stream, which no amount of signing can make portable — and it was kept alive on multi-replica deployments by a dedicated Service plus an Ingress annotated nginx.ingress.kubernetes.io/upstream-hash-by: "$http_authorization".

That transport is removed (ADR 0007), so those objects can go with it. If you are running them, delete the extra Service and Ingress together: two Ingresses pointing at one Service collapse onto a single nginx upstream and the annotation is silently ignored, so a half-deletion looks like it still works.

Updates

:main is mutable, so CI publishing a new image + a rollout restart is enough:

kubectl -n novamem rollout restart deploy/novamem
kubectl -n novamem rollout status  deploy/novamem

To pin to a specific commit instead of tracking :main:

kubectl -n novamem set image deploy/novamem novamem=ghcr.io/azrtydxb/novamem:sha-<short>

Schema migrations are forward-only — back up Postgres before rolling.

Troubleshooting

kubectl -n novamem get pods
kubectl -n novamem logs deploy/novamem
kubectl -n novamem describe pod -l app=novamem

• Pod stuck in CrashLoopBackOff → almost always missing NOVAMEM_COOKIE_SECRET or unreachable Postgres. Check the Secret was created (kubectl -n novamem get secret novamem-secrets) and that NOVAMEM_WARM_URL is set inside it.
• 403 Invalid origin on sign-in → NOVAMEM_BASE_URL doesn't match the browser's URL. It must be the exact https:// Ingress hostname.
• Cookies missing on sign-in over HTTPS → confirm NOVAMEM_INSECURE_COOKIES=0 and that the Ingress is terminating TLS (not passing through).
• cert-manager Certificate stuck Pending → check the Order / Challenge resources. Most often DNS for the host doesn't yet resolve to the Ingress IP.
• Slow first search → local embedder is downloading the model. Subsequent calls are fast; the model lives in the pod's ephemeral volume so it re-downloads on every restart unless you mount a PVC for it.

Running with pgvector instead of Qdrant

Set NOVAMEM_COLD_PROVIDER: "pgvector" and remove NOVAMEM_COLD_URL from the ConfigMap — left pointing at Qdrant it is silently used as a Postgres connection string and every pod fails readiness (env vars from envFrom cannot be overridden by kubectl set env; edit the ConfigMap itself). Unset, the cold store shares the warm database.

The Postgres pod needs: a pgvector-enabled image (pgvector/pgvector:pg16 is drop-in for postgres:16), and a memory-backed /dev/shm (see the example in postgres.yaml) sized well below the container memory limit. For migrating an existing Qdrant deployment, go run ./cmd/sync-qdrant-to-pgvector copies vectors without re-embedding.

Postgres sizing for pgvector. Measured with pgbench (persistent connections) on a ~380k-vector corpus with per-partition HNSW indexes: one k=200 query runs ~8–10 ms, and 8 concurrent queries run ~20 ms each — HNSW search is cheap, and a modest CPU limit absorbs moderate concurrent load. What does matter is shared_buffers: raise it from its 128 MB default so the hot index pages stay resident instead of round-tripping through the OS cache — but size it against the container memory limit, not in isolation. Keep shared_buffers at roughly 25% of the limit: shared_buffers=1GB requires raising the default manifest's limits.memory: 1Gi to ~4Gi first (e.g. args: ["-c", "shared_buffers=1GB"] with limits.memory: 4Gi); copy-pasting the 1GB setting against a 1Gi limit OOM-kills the pod under load.

If you suspect CPU throttling (search latency that degrades with client concurrency while the node looks idle), confirm it from the cgroup counters — v2: throttled_usec in /sys/fs/cgroup/cpu.stat; v1: throttled_time in /sys/fs/cgroup/cpu/cpu.stat — but measure query latency from a remote client with persistent connections (pgbench with a custom script). A forked-psql loop inside the pod spends more CPU on client startup and vector-literal parsing than on the queries, throttles the container's own limit, and implicates the database falsely; an earlier revision of this section published numbers produced exactly that way.