Summary
From 13:02 to 14:48 GMT on 17 November, some inbound calls on our US region failed to start. This affected voice agents that have a call transfer action set up. We rolled back the release that caused it, and calls have connected normally since then.
Impact
For about one hour and 48 minutes, some inbound calls to voice agents with a call transfer action did not start. This happened whether or not the “transfer availability” setting was on. Agents without a transfer action were not affected. Customers on our EU region or Global region were not affected either.
Data impact
No customer data was lost or changed. Calls that failed during this window never started, so there are no recordings or transcripts for them.
Root cause
We released an update to the service that handles live calls, and it was built on a new Docker base image. That image was missing time zone information. Agents with a transfer action check the current time to decide whether a transfer is available. Without time zone information that check failed, and the call could not start. Our automated pre-release tests did not cover this case, so the problem reached production.
Timeline (GMT)
17 Nov, 13:02: We released the update to the US region.
17 Nov, 13:32: We received reports of inbound calls failing to start.
17 Nov, 14:30: We traced the failures to the new release and the missing time zone information.
17 Nov, 14:35: We started rolling back the release.
17 Nov, 14:48: The rollback finished and inbound calls were connecting normally again.
What we’re doing to prevent this
We’re fixing the missing time zone information before we release this update again.
We’re adding call transfer actions to the automated tests that every release must pass.
We’re looking into why our existing pre-release checks and alerts didn’t catch the failure.
We’re sorry for the disruption.
If you have any questions or are still seeing issues, please contact Synthflow Support by submitting a ticket at https://helpcenter.synthflow.ai.
Resolved
Summary
From 13:02 to 14:48 GMT on 17 November, some inbound calls on our US region failed to start. This affected voice agents that have a call transfer action set up. We rolled back the release that caused it, and calls have connected normally since then.
Impact
For about one hour and 48 minutes, some inbound calls to voice agents with a call transfer action did not start. This happened whether or not the “transfer availability” setting was on. Agents without a transfer action were not affected. Customers on our EU region or Global region were not affected either.
Data impact
No customer data was lost or changed. Calls that failed during this window never started, so there are no recordings or transcripts for them.
Root cause
We released an update to the service that handles live calls, and it was built on a new Docker base image. That image was missing time zone information. Agents with a transfer action check the current time to decide whether a transfer is available. Without time zone information that check failed, and the call could not start. Our automated pre-release tests did not cover this case, so the problem reached production.
Timeline (GMT)
17 Nov, 13:02: We released the update to the US region.
17 Nov, 13:32: We received reports of inbound calls failing to start.
17 Nov, 14:30: We traced the failures to the new release and the missing time zone information.
17 Nov, 14:35: We started rolling back the release.
17 Nov, 14:48: The rollback finished and inbound calls were connecting normally again.
What we’re doing to prevent this
We’re fixing the missing time zone information before we release this update again.
We’re adding call transfer actions to the automated tests that every release must pass.
We’re looking into why our existing pre-release checks and alerts didn’t catch the failure.
We’re sorry for the disruption.
If you have any questions or are still seeing issues, please contact Synthflow Support by submitting a ticket at https://helpcenter.synthflow.ai.