Install Xberg Pro with Helm
This guide installs the default singleton deployment: one Pro StatefulSet at one replica, backed by your own pgvector-enabled PostgreSQL database. For fixed replicas and red/green deployments, see Run Pro with multiple replicas.
The chart and the Pro image are public, so you need no access to any private repository or registry. The install has seven steps, and each ends with a check. Do not continue until the check passes.
- Download the chart
- Prepare the database
- Create the Secrets
- Configure the values
- Install the chart
- Verify the deployment
- Create a project and an API key
To upgrade an existing installation, use Upgrades and migrations.
Prerequisites
Section titled “Prerequisites”- Helm 4,
kubectl,curlandjqon the machine that runs the install. - Kubernetes 1.29 or newer within Helm’s support matrix, with
amd64orarm64nodes. - PostgreSQL 15 to 18 with the pgvector (
vector) extension, version 0.8.0 or newer. The install machine must reach it. psqland the GNUtimeoutcommand on the install machine, for the database bootstrap.- A Pro license token and a commercial license to run the Pro image. See Licensing. Contact contact@xberg.io if you do not have them.
- Cluster nodes that can pull
ghcr.io/xberg-io/xberg-proanonymously, or an authorized mirror. - An ingress controller if you expose Pro externally. Use at most one of
ingress,gke.ingressoristio.
Download the chart
Section titled “Download the chart”Use an empty working directory. The public chart needs no registry login. Public availability does not grant an open-source license: the archive’s LICENSE and your commercial agreement apply.
Set VERSION to the chart release you were given, or to the latest published release. The example uses 0.7.4:
export VERSION=0.7.4export PRO_CHART="oci://us-central1-docker.pkg.dev/xberg-public-assets/helm-charts/pro"helm pull "$PRO_CHART" --version "$VERSION" --untarexport PRO_CHART_DIR="$PWD/pro"cp "$PRO_CHART_DIR/charts/_distribution/examples/values.yaml" pro-values.yamlexport PRO_VALUES_FILE="$PWD/pro-values.yaml"If you received a chart archive directly, extract it with tar -xzf pro-$VERSION.tgz and set PRO_CHART_DIR to the extracted pro directory.
The extracted chart holds three things you use during the install:
| Path | Content |
|---|---|
$PRO_CHART_DIR/charts/_distribution/files/ |
The database bootstrap scripts and the complete SQL |
$PRO_CHART_DIR/charts/_distribution/examples/ |
The example values file |
$PRO_CHART_DIR/values.yaml |
Every value the chart accepts, with comments |
Keep the extracted chart for bootstrap, configuration, installation and upgrades. Use the bootstrap files from the same chart version you install.
The released chart pins the Pro image to an exact digest, and the example values inherit it. Every chart-managed container, including database Jobs and helm test, uses that image. Nodes pull it anonymously, so imagePullSecrets stays empty.
If you use a private mirror:
- Copy the image and set both
image.repositoryandimage.digestfrom the copied artifact. - Mirror the migration artifact image (
migrations.artifactImage) too, when nodes cannot reach the public registry. - Configure node identity, a kubelet credential provider or a registry Secret for the mirror.
Do not invent a version tag.
Check. The chart renders its metadata and the files exist:
helm show chart "$PRO_CHART_DIR" | grep -E '^(name|version):'test -f "$PRO_CHART_DIR/charts/_distribution/files/pro-db-bootstrap.sh" && echo OKtest -f "$PRO_VALUES_FILE" && echo OKThe chart reports name: pro and the version you set. Both test commands print OK.
Database bootstrap
Section titled “Database bootstrap”Use a dedicated PostgreSQL server or cluster for each independently administered Xberg deployment. Separate databases or custom login names do not isolate the fixed Xberg capability roles. Do not install independent Pro and Enterprise releases on the same PostgreSQL server, because migrations and rollbacks change shared role memberships. A separate PostgreSQL instance is enough; a separate Kubernetes cluster is not required.
Create an empty target database and run the bootstrap once as its PostgreSQL administrator. The bootstrap creates the capability roles, the eight runtime logins and the schema.
PRO_DB_BOOTSTRAP_MODE=standard works on every provider and needs no superuser authority. For a restricted administrator it selects the managed-PostgreSQL path automatically. AWS RDS and Google Cloud SQL use this same command. The restricted path needs:
- PostgreSQL 16 or newer.
- An administrator login with direct
CREATEROLE,CREATEDBandCREATEon the target database, and withoutBYPASSRLSorREPLICATION. - The ability to create the
uuid-ossp,pgcrypto,btree_ginandvectorextensions, or a provider that preinstalls them with routines the administrator can transfer.
Check extension availability for your engine version. The bootstrap fails before it creates Xberg roles when it cannot prepare the extensions.
For AWS RDS, set PGHOST to the RDS endpoint and POSTGRES_USER to the instance master user. For RDS runtime connections, combine AWS’s RDS CA bundle with the Pro image’s public roots and mount it with SSL_CERT_FILE. For Cloud SQL, set them to the Auth Proxy endpoint and the database administrator; see Cloud SQL.
Set the connection details and the eight role passwords, then run the bootstrap. Generate each password as a distinct random value and store it in your secret manager:
export PGHOST=postgres.example.comexport PGPORT=5432export POSTGRES_DB=xbergexport POSTGRES_USER=postgres_adminexport POSTGRES_PASSWORD=REPLACE_WITH_ADMINISTRATOR_PASSWORDexport PGPASSWORD="$POSTGRES_PASSWORD"export POSTGRES_APPLICATION_PASSWORD=REPLACE_WITH_APPLICATION_PASSWORDexport POSTGRES_MAINTENANCE_PASSWORD=REPLACE_WITH_MAINTENANCE_PASSWORDexport POSTGRES_SCHEMA_OWNER_PASSWORD=REPLACE_WITH_SCHEMA_OWNER_PASSWORDexport POSTGRES_MIGRATION_PASSWORD=REPLACE_WITH_SCHEMA_MIGRATOR_PASSWORDexport POSTGRES_ROLE_ADMIN_PASSWORD=REPLACE_WITH_ROLE_ADMIN_PASSWORDexport POSTGRES_INDEX_WORKER_PASSWORD=REPLACE_WITH_VECTOR_INDEX_WORKER_PASSWORDexport POSTGRES_VECTOR_BROKER_PASSWORD=REPLACE_WITH_VECTOR_BROKER_PASSWORDexport POSTGRES_PHYSICAL_MIGRATION_PASSWORD=REPLACE_WITH_PHYSICAL_MIGRATION_PASSWORDexport PRO_DB_BOOTSTRAP_MODE=standardexport PRO_DB_BOOTSTRAP_PHASE=allexport PRO_DB_INITIALIZE_SCHEMA=trueexport PRO_DB_DEFER_CLAIMS_ENABLE=trueexport MIGRATION_PLAN_DATABASE=xberg_schema_planexport PLANNING_DATABASE_BOOTSTRAP=trueexport PLANNING_EXTENSIONS_FILE="$PRO_CHART_DIR/charts/_distribution/files/extensions-prepare.sql"export DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${PGHOST}:${PGPORT}/${POSTGRES_DB}"export MIGRATION_DATABASE_URL="postgresql://xberg_schema_migrator:${POSTGRES_MIGRATION_PASSWORD}@${PGHOST}:${PGPORT}/${POSTGRES_DB}"export SCHEMA_DIRECTORY="$PRO_CHART_DIR/charts/_distribution/files"sh "$PRO_CHART_DIR/charts/_distribution/files/pro-db-bootstrap.sh"sh "$PRO_CHART_DIR/charts/_distribution/files/prepare-schema-planning-database.sh"The bootstrap also creates the separate planning database named in MIGRATION_PLAN_DATABASE, which Helm uses to compute schema changes. The last command validates the result, and on self-managed PostgreSQL it creates the database.
Run this procedure on an empty database only. If it was interrupted, rerun the same command with the same administrator and role credentials. It validates the existing roles and schema, then resumes without recreating tables or data, and it rejects unknown or incomplete schemas before any repair. For an established installation, use Upgrades and migrations instead.
Never give administrator, schema-owner or physical-migration credentials to runtime pods. Treat the maintenance and administrator credentials as administrative credentials.
Check. Both scripts finish without an error message, and the application login connects:
PGPASSWORD="$POSTGRES_APPLICATION_PASSWORD" \ psql -h "$PGHOST" -p "$PGPORT" -U xberg_app -d "$POSTGRES_DB" -c 'SELECT current_user'psql prints xberg_app. If a script fails, read its last message. It names the missing capability or extension.
Secrets
Section titled “Secrets”Create the namespace and the seven Secrets that the example values reference. Percent-encode passwords inside database URLs, and replace every placeholder with the value it names. Run the snippet in Bash or Zsh:
kubectl create namespace xberg-prodatabase_suffix='@postgres.example.com:5432/xberg'application_url="postgresql://xberg_app:REPLACE_APPLICATION_PASSWORD${database_suffix}"maintenance_url="postgresql://xberg_maintenance_login:REPLACE_MAINTENANCE_PASSWORD${database_suffix}"migration_url="postgresql://xberg_schema_migrator:REPLACE_SCHEMA_MIGRATOR_PASSWORD${database_suffix}"role_admin_url="postgresql://xberg_role_admin:REPLACE_WITH_ROLE_ADMIN_PASSWORD${database_suffix}"worker_url="postgresql://xberg_index_worker_login:REPLACE_VECTOR_INDEX_WORKER_PASSWORD${database_suffix}"broker_url="postgresql://xberg_vector_index_broker_login:REPLACE_VECTOR_BROKER_PASSWORD${database_suffix}"physical_url="postgresql://xberg_physical_migration_login:REPLACE_PHYSICAL_MIGRATION_PASSWORD${database_suffix}"license_file='REPLACE_WITH_LICENSE_FILE_PATH'kubectl -n xberg-pro create secret generic xberg-pro-secrets \ --from-literal=DATABASE_URL="$application_url" \ --from-literal=MAINTENANCE_DATABASE_URL="$maintenance_url" \ --from-literal=JWT_SECRET='REPLACE_WITH_RANDOM_SESSION_SECRET' \ --from-literal=XBERG_ADMIN_KEY='REPLACE_WITH_RANDOM_ADMIN_API_KEY' \ --from-file=XBERG_LICENSE_KEY=<(tr -d '\r\n' < "$license_file")kubectl -n xberg-pro create secret generic xberg-pro-schema-migrator \ --from-literal=DATABASE_URL="$migration_url"kubectl -n xberg-pro create secret generic xberg-pro-role-admin \ --from-literal=ROLE_ADMIN_DATABASE_URL="$role_admin_url"kubectl -n xberg-pro create secret generic xberg-pro-vector-index-worker \ --from-literal=DATABASE_URL="$worker_url"kubectl -n xberg-pro create secret generic xberg-pro-vector-index-broker \ --from-literal=DATABASE_URL="$broker_url"kubectl -n xberg-pro create secret generic xberg-pro-vector-index-physical-migration \ --from-literal=DATABASE_URL="$physical_url"kubectl -n xberg-pro create secret tls pro-tls \ --cert=REPLACE_WITH_CERTIFICATE_PATH --key=REPLACE_WITH_PRIVATE_KEY_PATHThe license-file substitution removes CR and LF bytes that text files may end with. Pro requires the exact token without a trailing newline, and the token never appears in a command argument. Remember XBERG_ADMIN_KEY: you use it in the last step.
The TLS Secret is unnecessary when ingress.enabled=false or your ingress controller issues the certificate.
For any source connector, add both INTEGRATION_KEK and INTEGRATION_STATE_SECRET to xberg-pro-secrets. Each must be the base64 encoding of 32 random bytes. Without them the integration hub answers 503, and Pro refuses a configuration that contains only one of them.
Check. All seven Secrets exist:
kubectl -n xberg-pro get secretsThe list shows xberg-pro-secrets, xberg-pro-schema-migrator, xberg-pro-role-admin, xberg-pro-vector-index-worker, xberg-pro-vector-index-broker, xberg-pro-vector-index-physical-migration and pro-tls.
Configuration
Section titled “Configuration”Edit $PRO_VALUES_FILE, which you copied from the chart in step 1; the example below matches it. Replace the hostnames, OIDC settings, administrator addresses, TLS Secret name and PostgreSQL CIDR. The example names every Secret the chart reads, so keep its Secret keys complete.
imagePullSecrets: []
secrets: name: xberg-pro-secrets
maintenance: databaseUrlSecret: name: xberg-pro-secrets key: MAINTENANCE_DATABASE_URL
vectorIndexWorker: databaseUrlSecret: name: xberg-pro-vector-index-worker key: DATABASE_URL brokerDatabaseUrlSecret: name: xberg-pro-vector-index-broker key: DATABASE_URL physicalMigrationDatabaseUrlSecret: name: xberg-pro-vector-index-physical-migration key: DATABASE_URL
migrations: planningDatabaseBootstrap: false databaseUrlSecret: name: xberg-pro-schema-migrator key: DATABASE_URL roleAdminDatabaseUrlSecret: name: xberg-pro-role-admin key: ROLE_ADMIN_DATABASE_URL
configMap: FRONTEND_BASE_URL: https://pro.example.com PRO_PUBLIC_API_URL: https://pro.example.com OAUTH_CALLBACK_BASE_URL: https://pro.example.com OIDC_ISSUER_URL: https://issuer.example.com OIDC_AUDIENCE: xberg-pro OIDC_JWKS_URI: https://issuer.example.com/.well-known/jwks.json NEXT_PUBLIC_OIDC_ISSUER_URL: https://issuer.example.com NEXT_PUBLIC_OIDC_CLIENT_ID: xberg-dashboard NEXT_PUBLIC_OIDC_REDIRECT_URI: https://pro.example.com/auth/callback PRO_ALLOWED_EMAIL_DOMAINS: example.com PRO_ADMIN_EMAILS: admin@example.com
ingress: enabled: true className: nginx hosts: - pro.example.com tls: - secretName: pro-tls hosts: - pro.example.com
networkPolicy: enabled: true ingressControllerNamespace: ingress-nginx postgresCidrs: - 10.20.0.0/24
persistence: claimTemplateMode: freshpersistence.claimTemplateMode: fresh fits a new release. GitOps reconciliation must use existing and the installed request; see Persistent volume.
OIDC needs two sets of keys:
- Pro verifies tokens with
OIDC_ISSUER_URL,OIDC_AUDIENCEandOIDC_JWKS_URI. - The web console starts the sign-in in the browser with
NEXT_PUBLIC_OIDC_ISSUER_URL,NEXT_PUBLIC_OIDC_CLIENT_IDandNEXT_PUBLIC_OIDC_REDIRECT_URI.
With only the first set, Pro accepts tokens but the sign-in page refuses to start.
Network policies
Section titled “Network policies”With networkPolicy.enabled: true, the chart restricts pod traffic:
- For PostgreSQL in the same cluster, set
networkPolicy.postgresNamespaceto its namespace and removenetworkPolicy.postgresCidrs. - For PostgreSQL outside the cluster, set
networkPolicy.postgresCidrsto its reachable CIDRs. - Add other destinations, such as a private object store, under
networkPolicy.extraEgress. - Crawling and webhook delivery reach the public web on the ports in
networkPolicy.publicEgressPorts, which holds 80 and 443. A crawl seed or webhook receiver on any other port needs its ownextraEgressrule, or the request times out with no message that names the policy. - Set
configMap.OTEL_EXPORTER_OTLP_ENDPOINTonly for an in-cluster collector, then setnetworkPolicy.otlpNamespaceandnetworkPolicy.otlpPortto match it. Admit an external collector throughextraEgress.
The migration hooks call the Kubernetes API, and the chart policy leaves their pods out so they can reach it. A policy of your own that selects every pod in the namespace selects the hooks again. It must then admit the exact Kubernetes API destination CIDRs and TCP ports your cluster enforces after Service address translation, or the hooks fail on a deadline. Do not open all private ranges.
Authentication
Section titled “Authentication”To turn authentication off for an evaluation, set configMap.PRO_AUTH_MODE: "disabled". The UI and API then use a persisted local workspace without OAuth, login or an API key.
- Omit
JWT_SECRETandXBERG_ADMIN_KEYfrom the runtime Secret and clear the OIDC values. - The database Secrets stay required.
- Anyone who can reach the service can use this workspace, so restrict network access.
The default is required.
Storage
Section titled “Storage”The default filesystem storage uses the /data volume. For object storage, set configMap.STORAGE_PROVIDER and configMap.STORAGE_BUCKET:
- GCS supports workload identity.
- S3 uses the default AWS credential chain, including EKS Pod Identity and IRSA; see AWS storage. For explicit credentials, set
S3_ACCESS_KEY_IDandS3_SECRET_ACCESS_KEYtogether inxberg-pro-secrets, plusS3_SESSION_TOKENfor temporary credentials. - For Azure, add
AZURE_STORAGE_ACCOUNTandAZURE_STORAGE_ACCESS_KEYto that Secret.
Keep configMap.PRO_PUBLIC_API_URL set to the externally reachable HTTP(S) origin for every storage provider. Upload capabilities are one-use PUT URLs on Pro, and an unset origin makes presign return 503.
Route /v1/uploads/local directly to Pro and allow request bodies up to PRO_MAX_UPLOAD_BYTES at your ingress. The included standard, GKE and Istio routes do this. A customer-managed proxy needs the same rule.
Pro keeps minimal cleanup records for uploads whose writers cannot be confirmed finished. Monitor pro_upload_retained_unknown_writers and pro_upload_oldest_unknown_writer_age_seconds on /metrics. Both are gauges without project labels.
The count can stay above zero after an upgrade. Investigate sustained growth, or an age beyond what you expect, for writer disconnects and cleanup failures. Do not delete these records or treat URL expiry or lifecycle rules as proof that a writer finished.
Persistent volume
Section titled “Persistent volume”The default 100 GiB claim reserves about 20 GiB for model and result caches, including the 10 GiB configMap.CACHE_MAX_BYTES ceiling. Accepted documents and results are retained with the corpus, and upload cleanup only removes expired staging data. Set persistence.size above 100 GiB when the corpus needs more than the remaining 80 GiB or so.
persistence.size applies to fresh claims only. Kubernetes does not resize an existing claim when you change this value, and the StatefulSet claim template is immutable. Set persistence.claimTemplateMode: fresh only while creating a release. A live helm upgrade preserves the installed claim template automatically.
Argo CD, GitOps and other renderers that cannot read the cluster cannot see the installed template. Use fresh for the initial sync. Before the next sync, capture the installed template as a values file and include that file in every later render:
export RELEASE=xberg-proexport NAMESPACE=xberg-prokubectl -n "$NAMESPACE" get statefulset \ -l "app.kubernetes.io/name=pro,app.kubernetes.io/instance=$RELEASE" -o json | jq -e ' [.items[].spec.volumeClaimTemplates[] | select(.metadata.name == "data")] | if length != 1 then error("exactly one installed data template is required") else {persistence: {claimTemplateMode: "existing", existingClaimTemplate: { metadata: .[0].metadata, spec: .[0].spec }}} end' > pro-existing-claim.jsonKeep this captured template unchanged, even after you expand the claim. The optional persistence.existingClaimTemplateSize must match the captured request. Do not delete the StatefulSet or the claim to bypass an immutable-template error.
To grow the volume, expand the claim directly:
STATEFULSET="$(kubectl -n "$NAMESPACE" get statefulset \ -l "app.kubernetes.io/name=pro,app.kubernetes.io/instance=$RELEASE" \ -o jsonpath='{.items[0].metadata.name}')"test -n "$STATEFULSET"DATA_PVC="data-${STATEFULSET}-0"STORAGE_CLASS="$(kubectl -n "$NAMESPACE" get pvc "$DATA_PVC" -o jsonpath='{.spec.storageClassName}')"test "$(kubectl get storageclass "$STORAGE_CLASS" -o jsonpath='{.allowVolumeExpansion}')" = truekubectl -n "$NAMESPACE" patch pvc "$DATA_PVC" --type merge \ -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'kubectl -n "$NAMESPACE" wait pvc "$DATA_PVC" \ --for=jsonpath='{.status.capacity.storage}'=200Gi --timeout=10mReplace 200Gi with the new size. When allowVolumeExpansion is not true, follow your storage provider’s data migration procedure to a larger claim. Do not delete the existing claim.
Memory
Section titled “Memory”The chart requests 16 GiB of memory and limits it to 24 GiB, which fits PRO_MAX_CONCURRENCY of 4 with the RAG models loaded. Size resources.limits.memory from the models you use and the number of concurrent documents.
Large multi-page documents can use around 1 GiB of working memory each, on top of the models. PRO_ENGINE_MAX_THREADS sets the shared extraction thread budget, and peak memory grows with it on multi-page scans, so raise the memory limit when you raise it. See Pro configuration.
Pro keeps two embedding engines and two reranker engines resident by default. Raise configMap.XBERG_EMBEDDING_ENGINE_CACHE_MAX_RESIDENT and configMap.XBERG_RERANKER_ENGINE_CACHE_MAX_RESIDENT to retain more. Eviction is least recently used, and an active request can keep an evicted engine alive until it finishes, so the count is not a hard memory ceiling.
Set each bound at least as high as the number of collection presets used at the same time, or alternating requests rebuild engines after eviction. Recurring RAG warmup is off by default, because warming more presets than the bound would rebuild and evict them every 15 minutes.
Migration safety flags
Section titled “Migration safety flags”Keep migrations.allowDestructive, migrations.allowLocking and migrations.allowDataRepair set to false unless a reviewed, coordinated schema change or data repair needs them. The extracted $PRO_CHART_DIR/values.yaml lists the complete value surface, and Pro configuration lists every environment variable.
Check. The values render without errors:
helm lint --strict "$PRO_CHART_DIR" -f "$PRO_VALUES_FILE"helm template xberg-pro "$PRO_CHART_DIR" -n xberg-pro -f "$PRO_VALUES_FILE" >/dev/null && echo OKhelm lint reports 0 chart(s) failed, and the second command prints OK. An error that names a missing Secret or value tells you which entry to fix.
Install
Section titled “Install”Install the chart. The command waits for the migration hooks and the StatefulSet to finish, and allows 20 minutes:
helm install xberg-pro "$PRO_CHART_DIR" \ -n xberg-pro -f "$PRO_VALUES_FILE" \ --wait --timeout 20mCheck. Helm reports the release as deployed:
helm status xberg-pro -n xberg-pro | grep -E '^STATUS'The output is STATUS: deployed. If the install stops with Job Failed for a migration hook, see Troubleshooting.
Verification
Section titled “Verification”Wait for the rollout, run the chart tests and forward the service ports:
kubectl -n xberg-pro rollout status statefulset/xberg-pro --timeout=10mkubectl -n xberg-pro get pods,services,ingresseshelm test xberg-pro -n xberg-pro --logskubectl -n xberg-pro port-forward service/xberg-pro 8080:8080 3000:3000In a second terminal, while the port-forward runs:
curl --fail http://127.0.0.1:8080/healthzcurl --fail http://127.0.0.1:8080/readyz/healthz proves the process is alive. /readyz also verifies the database identities and the object store. helm test ends with Phase: Succeeded.
Open http://127.0.0.1:3000 to check that the web console sign-in page loads. OIDC sign-in needs the configured public origin and the registered redirect URI, so use the public URL for an authenticated test.
Create a project and an API key
Section titled “Create a project and an API key”Pro keeps work in projects, and API keys belong to a project. With the port-forward still running, use the XBERG_ADMIN_KEY you put in xberg-pro-secrets:
export XBERG_URL=http://127.0.0.1:8080export XBERG_ADMIN_KEY=REPLACE_WITH_THE_ADMIN_KEY_FROM_THE_SECRET
export PROJECT_ID=$(curl -sX POST "$XBERG_URL/v1/projects" \ -H "Authorization: Bearer $XBERG_ADMIN_KEY" \ -H "Content-Type: application/json" \ -d '{"name": "Documents", "slug": "documents"}' | jq -r .id)
export XBERG_API_KEY=$(curl -sX POST "$XBERG_URL/v1/projects/$PROJECT_ID/api-keys" \ -H "Authorization: Bearer $XBERG_ADMIN_KEY" \ -H "Content-Type: application/json" \ -d '{"name": "ingest-service"}' | jq -r .key)The key field is the full plaintext key and appears only once. Store it in your secret manager. See Projects and API keys.
Check. The key authenticates:
curl --fail -H "Authorization: Bearer $XBERG_API_KEY" "$XBERG_URL/v1/usage"The response is a JSON usage report. Continue with the quickstart.
Troubleshooting
Section titled “Troubleshooting”When helm install or helm upgrade stops with Job Failed for a migration hook, read the hook’s pod first:
kubectl -n xberg-pro describe pod -l job-name=xberg-pro-topology-guardkubectl -n xberg-pro get events --sort-by=.lastTimestamp | grep -i -E 'pull|deadline'An image pull failure is the first cause to check. The pod shows ImagePullBackOff with the image name, and the events carry the registry response, for example 401 Unauthorized. Confirm the image reference and anonymous pull access for the public defaults. For a private mirror, configure its credentials in imagePullSecrets and run the install again.
The first hook, the topology guard, pulls the same Pro image as the StatefulSet, so a pull failure stops it within two minutes and leaves the failed pod for inspection. For another hook, replace the job-name label value with the Job name from the Helm message.
If the migration hook fails
Section titled “If the migration hook fails”A migration hook that fails for a reason other than an image pull leaves Pro stopped. This is intended: the chart keeps the workload down and records its replica count for a corrected retry.
Symptoms:
helm installorhelm upgradeends withJob Failedfor amaintenance,roleorschemahook, and the pod is not inImagePullBackOff.- A retry after you changed the chart or values is refused by the maintenance hook.
- Pro pods stay at zero replicas after the command returns.
Collect this before you contact support:
helm history xberg-pro -n xberg-prohelm get hooks xberg-pro -n xberg-pro > hooks.yamlkubectl -n xberg-pro get jobs,podskubectl -n xberg-pro logs job/REPLACE_WITH_FAILED_JOB_NAME > hook.logkubectl -n xberg-pro get events --sort-by=.lastTimestamp > events.logAlso note the chart version, the target chart version for an upgrade, and your values file with secrets removed.
Then:
- Stop every reconciler of the release. Suspend Argo CD automated sync, and do not run
helm upgrade. - Do not change the chart or the values, and do not run SQL against the database to clear the failure.
- Contact support with the collected files. Support validates the database phase and gives a recovery plan for your version.
Migration modes
Section titled “Migration modes”migrations.mode defaults to maintenance. It quiesces the workload and runs the role-contract and restoration hooks around each schema change. Pro is unavailable during that interval.
online renders one restricted schema runner instead and does not stop workloads. It accepts only a converged no-op today, and a nonempty plan fails before it changes the database. Use maintenance for every upgrade.
When an installation already has a fleet control record, maintenance needs two values:
migrations.maintenanceOperationId, a UUID that identifies the operation.migrations.maintenanceExpectedEpoch, which must match the observed control epoch.
A failed operation keeps its fence until it is recovered, and a different operation cannot take over implicitly. Support handles that recovery; see If the migration hook fails. Migration hooks require the READ COMMITTED transaction default.
An installation from before chart 0.7 has no initialized schema bridge, so its first upgrade needs a one-time bootstrap. Contact support before that upgrade. Other version-specific preparation, including SQL supplements for older bootstraps, is in Upgrades and migrations.
Run Pro with multiple replicas
Section titled “Run Pro with multiple replicas”Use deployment.mode=replicated for extraction-only Pro with 2 to 32 replicas per generation, on distinct nodes. A replicated generation needs:
- An external PostgreSQL service and a shared AWS S3, Google Cloud Storage or Azure Blob namespace.
- Required API authentication,
dataPolicy=extraction_only, and Pro RAG and Auto-Tune disabled. persistence.enabled=falseandmigrations.enabled=falseon the generation releases. Plan database maintenance separately, at installation scope.- A customer-managed layer-7 upstream whose connection draining you verified, before you set
deployment.traffic.connectionDrainingConfirmed=true.
To deploy:
- Register the installation, qualify cross-Pod storage, admit the generation and verify serving readiness with the deployment operator before you route traffic.
- On native Kubernetes, set
deployment.adapter=nativeand install each immutable generation as its own Helm release, with the registered installation and generation IDs, the storage namespace identity and the configuration and policy digests. - Pin the platform image manifest and match its architecture in
nodeSelector. - With a separately installed Argo Rollouts controller, set
deployment.adapter=argo-rolloutsanddeployment.argo.publishOnly=trueinstead. The chart publishes the desired generation, and the installation operator creates the Rollout and manages the public Service.
An existing filesystem-backed singleton stays a singleton on upgrade. Plan a separate move to shared storage and a verified Service handoff before you route traffic to a replicated generation. Replicated deployments exclude PVC-backed models and arbitrary sidecar or service-account token injection. EKS Pod Identity and IRSA credentials work through the AWS storage profile.
Clean up abandoned S3 inputs
Section titled “Clean up abandoned S3 inputs”On a replicated generation that uses the installation’s registered, versioned AWS S3 namespace, Pro can delete extraction inputs that were uploaded but never attached to a job. The cleanup is off. To enable it:
- Upgrade the schema and every application and maintenance binary that can start against the installation, rollback candidates included. Older binaries reject the expanded maintenance privileges.
- With the cleanup still disabled, set
migrations.inputS3CleanupGrants: grantfor one maintenance release. For a database you manage yourself, run the maintenance grant phase withPRO_INPUT_S3_CLEANUP_GRANTS=grant. Then return the chart value toverify. - Give the runtime identity
s3:DeleteObjectVersionon the installation’sdocuments/prefix under the storage root. - Set
configMap.PRO_INPUT_S3_CLEANUP_ENABLED: "true"on the admitted generation. Pro refuses to start with the cleanup enabled when the maintenance login lacks the grants.
Before you roll back to an older binary, disable the cleanup and let it drain. Then run the maintenance grant phase with migrations.inputS3CleanupGrants: revoke (or PRO_INPUT_S3_CLEANUP_GRANTS=revoke) and confirm that the grants are gone.
A failed or interrupted delete keeps the staging credit held for a later retry. Each replica reports its own events on the metrics endpoint:
| Metric | Meaning |
|---|---|
pro_abandoned_s3_input_cleanup_candidate_events_total |
Candidate events by outcome: started, acknowledged, not_authorized, unconfirmed, failed. |
pro_abandoned_s3_input_cleanup_sweep_events_total |
Discovery sweeps: completed, not_authorized, failed. |
pro_abandoned_s3_input_cleanup_last_completed_sweep_timestamp_seconds |
When an authorized replica last finished a scan. Absent until the first sweep completes. |
A completed sweep does not mean every delete finished, so read the sweep timestamp next to the candidate outcomes. A preview or fenced replica can report not_authorized while another replica keeps processing. For the database-side census and reconciliation procedures, contact support.
Private MCP listener
Section titled “Private MCP listener”The chart can start a private MCP listener on loopback for local clients. It is off by default. See Private MCP listener.