Summary
From 17:15 UTC on 27 October to 08:42 UTC on 28 October, some customers in our US and EU regions could not open the Settings page, and some could not log in with SSO. Both problems are now fixed and have not come back.
Impact
Affected users saw the Settings page fail to load, and some SSO logins failed. Calls and voice agents kept running as normal.
Data impact
A small number of workspaces may have lost some workspace-level settings, such as the workspace name. No customer data was lost. We will contact each affected workspace directly to help restore these settings.
Root cause
An issue with one of our third-party providers caused some workspace settings, including the workspace’s region, to be cleared. Without these settings, the Settings page and SSO login stopped working for affected workspaces. Our first fix restored the settings, but they were cleared again until we found and fixed the underlying cause.
Timeline (UTC)
27 Oct, 17:15: We started investigating reports of Settings and SSO login failures.
27 Oct, 18:33: We restored settings for the affected workspaces.
27 Oct, 21:11: Some settings were being cleared again. We found the cause and began restoring all affected workspaces.
28 Oct, 08:42: We finished restoring every affected workspace and added safeguards. Our checks found no affected workspaces after this.
What we’re doing to prevent this
We’ve added safeguards that stop workspace settings from being cleared if a third-party provider fails.
To migrate off this provider to prevent this issue from happening again in the future
We’re sorry for the disruption. If you still see problems with Settings or SSO login, please contact support at support@synthflow.ai, or reach out to your account representative.
Resolved
Summary
From 17:15 UTC on 27 October to 08:42 UTC on 28 October, some customers in our US and EU regions could not open the Settings page, and some could not log in with SSO. Both problems are now fixed and have not come back.
Impact
Affected users saw the Settings page fail to load, and some SSO logins failed. Calls and voice agents kept running as normal.
Data impact
A small number of workspaces may have lost some workspace-level settings, such as the workspace name. No customer data was lost. We will contact each affected workspace directly to help restore these settings.
Root cause
An issue with one of our third-party providers caused some workspace settings, including the workspace’s region, to be cleared. Without these settings, the Settings page and SSO login stopped working for affected workspaces. Our first fix restored the settings, but they were cleared again until we found and fixed the underlying cause.
Timeline (UTC)
27 Oct, 17:15: We started investigating reports of Settings and SSO login failures.
27 Oct, 18:33: We restored settings for the affected workspaces.
27 Oct, 21:11: Some settings were being cleared again. We found the cause and began restoring all affected workspaces.
28 Oct, 08:42: We finished restoring every affected workspace and added safeguards. Our checks found no affected workspaces after this.
What we’re doing to prevent this
We’ve added safeguards that stop workspace settings from being cleared if a third-party provider fails.
To migrate off this provider to prevent this issue from happening again in the future
We’re sorry for the disruption. If you still see problems with Settings or SSO login, please contact support at support@synthflow.ai, or reach out to your account representative.
