Back to overview
Downtime

Internal cluster failure

Oct 30 at 02:00pm GMT
Affected services
Dashboard
Authentication Service
Users Service

Resolved
Nov 02 at 11:13am GMT

Complete the fixes, backfilled all migrations

We are back, sorry for the inconvenience caused

Updated
Nov 01 at 05:24pm GMT

Due to changes in a build step, the decision was made to bring a beta branch into production. This is soon to be up and running, some database migrations and data backfils are needed to run. Projected 1 hour to complete

Updated
Oct 30 at 02:57pm GMT

Accounts API is back up and running. Dashboard should be back within the next 10-15 minutes

Created
Oct 30 at 02:00pm GMT

All services down due to unresolved updates within our cluster linked to bitnami. Helm charts were updated to handle the change to bitnamilegacy but were not implemented. During an K8s cluster update, when argocd span back up it propagated these changes, unbeknownst to us at the time the versions were not compaible with some dependencies.

These have all been reciftied, and the images are being recreated to be re-deployed.