Internal cluster failure
Resolved
Nov 02 at 11:13am GMT
Complete the fixes, backfilled all migrations
We are back, sorry for the inconvenience caused
Affected services
Updated
Nov 01 at 05:24pm GMT
Due to changes in a build step, the decision was made to bring a beta branch into production. This is soon to be up and running, some database migrations and data backfils are needed to run. Projected 1 hour to complete
Affected services
Updated
Oct 30 at 02:57pm GMT
Accounts API is back up and running. Dashboard should be back within the next 10-15 minutes
Affected services
Created
Oct 30 at 02:00pm GMT
All services down due to unresolved updates within our cluster linked to bitnami. Helm charts were updated to handle the change to bitnamilegacy but were not implemented. During an K8s cluster update, when argocd span back up it propagated these changes, unbeknownst to us at the time the versions were not compaible with some dependencies.
These have all been reciftied, and the images are being recreated to be re-deployed.
Affected services