Starling Connect Partial Outage

What happened?

Between 00:38 and 20:50 UTC on 2024-11-22, customers experienced connection errors related to the Starling Connect service.

What went wrong and why?

At 16:06 UTC on 2024-11-21, a change was introduced to Connect Supervisor restricting certain content to JSON format. The Connector application used by Safeguard Privileged Passwords (SPP) was not configured to reflect this limitation. Consequently, the Connector application accepted content in other formats such as TXT and plain text. When we introduced the JSON format restriction, connections to SPP in non-JSON format began to fail.

This impacted some, but not all, of our customers.

How did we respond?

This incident was detected at 00:38 UTC on 2024-11-22.

After receiving the alert, we started to investigate the incident by analyzing support bundles of various customer instances.

At 17:14 UTC, a decision was made to roll back Connect Supervisor to its prior version. After successfully testing the rollback in a non-production environment, the rollback change was applied to the production environment at 20:43 UTC.

The Statuspage incident was updated to reflect that a fix was applied at 20:44 UTC, and we confirmed that the fix had resolved the issue at 21:05 UTC on 2024-11-22.

How are we making incidents like this less likely or less impactful?

We have updated our test plan to ensure SPP teams are included.