Back to overview
Downtime

Internal cluster failure

Oct 30, 2025 at 2:00pm UTC
Affected services
Dashboard
Authentication Service
Users Service

Resolved
Nov 2, 2025 at 11:13am UTC

Complete the fixes, backfilled all migrations

We are back, sorry for the inconvenience caused

Updated
Nov 1, 2025 at 5:24pm UTC

Due to changes in a build step, the decision was made to bring a beta branch into production. This is soon to be up and running, some database migrations and data backfils are needed to run. Projected 1 hour to complete

Updated
Oct 30, 2025 at 2:57pm UTC

Accounts API is back up and running. Dashboard should be back within the next 10-15 minutes

Created
Oct 30, 2025 at 2:00pm UTC

All services down due to unresolved updates within our cluster linked to bitnami. Helm charts were updated to handle the change to bitnamilegacy but were not implemented. During an K8s cluster update, when argocd span back up it propagated these changes, unbeknownst to us at the time the versions were not compaible with some dependencies.

These have all been reciftied, and the images are being recreated to be re-deployed.