Starling Connect Partial Outage

Incident Report for One Identity Starling

Postmortem

What happened?

Between 00:38 and 20:50 UTC on 2024-11-22, customers experienced connection errors related to the Starling Connect service.

What went wrong and why?

At 16:06 UTC on 2024-11-21, a change was introduced to Connect Supervisor restricting certain content to JSON format. The Connector application used by Safeguard Privileged Passwords (SPP) was not configured to reflect this limitation. Consequently, the Connector application accepted content in other formats such as TXT and plain text. When we introduced the JSON format restriction, connections to SPP in non-JSON format began to fail.

This impacted some, but not all, of our customers.

How did we respond?

This incident was detected at 00:38 UTC on 2024-11-22.

After receiving the alert, we started to investigate the incident by analyzing support bundles of various customer instances.

At 17:14 UTC, a decision was made to roll back Connect Supervisor to its prior version. After successfully testing the rollback in a non-production environment, the rollback change was applied to the production environment at 20:43 UTC.

The Statuspage incident was updated to reflect that a fix was applied at 20:44 UTC, and we confirmed that the fix had resolved the issue at 21:05 UTC on 2024-11-22.

How are we making incidents like this less likely or less impactful?

We have updated our test plan to ensure SPP teams are included.

Posted Jan 08, 2025 - 04:21 PST

Resolved

The issue has been verified to be resolved, all related services have returned to normal functional status.
Posted Nov 22, 2024 - 13:03 PST

Monitoring

A fix has been applied and we are currently monitoring.
Posted Nov 22, 2024 - 12:44 PST

Identified

Connector customers with Safeguard Privileged Password may experience issues performing certain tasks against the Registered Connector Asset [Test system, Check Password, Change Password, etc...]. The issue has been identified and we are working on a solution.

Then next update will occur at 1PM PST
Posted Nov 22, 2024 - 11:05 PST
This incident affected: One Identity Starling EMEA (Connect), One Identity Starling NA (Connect), and One Identity Starling (Connect).