Voice Agents Not Responding During Phone Calls
Summary
Impact: Customers in all regions – Global, US, and EU – experienced unresponsive voice agents. The most common symptom was a silent agent. In some cases, silent agent failures were intermittent.
Cause: Our primary inference provider blocked access to models via automated detection of unusual behavior, per the provider. At the time of this writing, the provider has not provided an explanation of the reason for blocking access. When access is blocked, a 403 is returned for any inference call to that provider, leading to our agents failing.
Steps to resolve: We removed usage of the primary provider and rerouted inference traffic to another provider, most notably, OpenAI.
Service Disruption Impact & Timeline
Between approximately 16:48 UTC and 19:39 UTC on Thursday, May 14, 2026, Synthflow experienced a disruption to voice agent functionality across all three production clusters: Global, US, and EU. The trigger was an abrupt block applied to almost all our GPT models by our primary inference provider’s automated abuse-detection system, which caused requests to our inference provider to fail with HTTP 403 Forbidden errors simultaneously across every region. Because the block was applied across models, our region-to-region failover provided little protection, and all customer traffic dependent on the affected models began failing the moment enforcement took effect. The first internal failure reports arrived at 16:48 UTC.
For customers, the disruption surfaced primarily as:
• Answered calls with silence when a model response was expected (i.e. an agent should respond with voice);
• Calls that terminated unexpectedly mid-conversation;
• A sharp increase in failed or unanswered calls in customer dashboards.
Initial mitigation, which moved most inference traffic to OpenAI, restored service for the majority of agents within approximately 50 minutes of the incident being declared. A subset of customers continued to experience intermittent failures for an additional \~90 minutes due to a remainder of LLM-dependent traffic still routing to the pre-existing inference provider. A hotfix was applied to reroute the remainder of the traffic.
The incident was declared resolved at 19:39 UTC once all inference traffic had been migrated to the secondary provider, error rates had returned to baseline across all clusters, and customer-initiated test calls were completed end-to-end successfully.
Root Cause
Synthflow did not successfully handle a failure scenario where an inference provider begins erroring across the majority of API requests. During this incident, our primary inference provider blocked almost all inference traffic due to unusual activity. Synthflow is designed with redundancy built into its inference layer for scenarios where our inference provider is unavailable. Synthflow’s architecture includes an inference router to handle such failure scenarios. In certain situations, the inference router did not apply failover mechanisms in real-time.
Background on Synthflow’s Inference Routing
LLM API calls required by our voice agents are funneled through our inference router rather than each of our components communicating to inference providers directly. The router maintains a continuously updated pool of model deployments: dozens of endpoints spanning multiple inference providers, multiple geographic regions, and multiple model variants of each generation. For every deployment, it keeps a rolling window of recent latency and error statistics and uses those signals to choose the endpoint expected to give the fastest healthy response for the model the caller has asked for. Deployments that start failing or slowing down are automatically demoted. Deployments that recover are reintroduced. As designed, this architecture provides us with a way to automatically handle poorly performing, or offline inference providers.
The same mechanism is what gives us cross-provider failover: because the pool contains deployments from more than one inference provider, a localized failure on one of them naturally drains away as its endpoints lose health, and traffic concentrates on whichever provider is currently responding well for the model in question. Under normal failure modes – partial outages, regional brownouts, individual model deployments becoming blocked – this typically resolves itself fast enough that the average customer never sees a failed call.
Why failover was slower than expected this time
This incident exposed some parts of our system that did not route through our router, showing points of failure previously not observed. We initially believed failovers were not happening and took action to remove the failing provider manually. We made a configuration change to remove the unhealthy (primary) provider from the overall pool of available providers. However, after taking this action and monitoring, we observed a number of failing API requests continuing to route to that provider.
After further review, we observed some older parts of the codebase still make direct connections for inference. That meant the router's shift to the healthy provider only resolved part of the impact. Two areas of our codebase kept hitting the unhealthy provider directly. Moving them to the secondary provider required a code change and hotfix release to remove direct dependencies on the failing provider.
Next Steps
We understand how an outage such as this can affect you and your business. It’s important we’ve not only resolved this incident, but that we are also taking crucial steps to improve our service to avoid a scenario such as this again. Here are the primary actions we’re taking throughout the coming week:
• Continue working with our primary inference provider to understand why our model deployments were blocked; we will share an amended update here when we have additional information.
• Provide per-customer inference deployments to our enterprise customers in order to reduce the impact of a regional deployment outage.
• Improve our inference router to handle wide-scale issues with a given provider as described in the failure scenario above.
• Finalize the removal of any direct inference communication that does not currently go through our router.
Resolved
Voice Agents Not Responding During Phone Calls
Summary
Impact: Customers in all regions – Global, US, and EU – experienced unresponsive voice agents. The most common symptom was a silent agent. In some cases, silent agent failures were intermittent.
Cause: Our primary inference provider blocked access to models via automated detection of unusual behavior, per the provider. At the time of this writing, the provider has not provided an explanation of the reason for blocking access. When access is blocked, a 403 is returned for any inference call to that provider, leading to our agents failing.
Steps to resolve: We removed usage of the primary provider and rerouted inference traffic to another provider, most notably, OpenAI.
Service Disruption Impact & Timeline
Between approximately 16:48 UTC and 19:39 UTC on Thursday, May 14, 2026, Synthflow experienced a disruption to voice agent functionality across all three production clusters: Global, US, and EU. The trigger was an abrupt block applied to almost all our GPT models by our primary inference provider’s automated abuse-detection system, which caused requests to our inference provider to fail with HTTP 403 Forbidden errors simultaneously across every region. Because the block was applied across models, our region-to-region failover provided little protection, and all customer traffic dependent on the affected models began failing the moment enforcement took effect. The first internal failure reports arrived at 16:48 UTC.
For customers, the disruption surfaced primarily as:
• Answered calls with silence when a model response was expected (i.e. an agent should respond with voice);
• Calls that terminated unexpectedly mid-conversation;
• A sharp increase in failed or unanswered calls in customer dashboards.
Initial mitigation, which moved most inference traffic to OpenAI, restored service for the majority of agents within approximately 50 minutes of the incident being declared. A subset of customers continued to experience intermittent failures for an additional \~90 minutes due to a remainder of LLM-dependent traffic still routing to the pre-existing inference provider. A hotfix was applied to reroute the remainder of the traffic.
The incident was declared resolved at 19:39 UTC once all inference traffic had been migrated to the secondary provider, error rates had returned to baseline across all clusters, and customer-initiated test calls were completed end-to-end successfully.
Root Cause
Synthflow did not successfully handle a failure scenario where an inference provider begins erroring across the majority of API requests. During this incident, our primary inference provider blocked almost all inference traffic due to unusual activity. Synthflow is designed with redundancy built into its inference layer for scenarios where our inference provider is unavailable. Synthflow’s architecture includes an inference router to handle such failure scenarios. In certain situations, the inference router did not apply failover mechanisms in real-time.
Background on Synthflow’s Inference Routing
LLM API calls required by our voice agents are funneled through our inference router rather than each of our components communicating to inference providers directly. The router maintains a continuously updated pool of model deployments: dozens of endpoints spanning multiple inference providers, multiple geographic regions, and multiple model variants of each generation. For every deployment, it keeps a rolling window of recent latency and error statistics and uses those signals to choose the endpoint expected to give the fastest healthy response for the model the caller has asked for. Deployments that start failing or slowing down are automatically demoted. Deployments that recover are reintroduced. As designed, this architecture provides us with a way to automatically handle poorly performing, or offline inference providers.
The same mechanism is what gives us cross-provider failover: because the pool contains deployments from more than one inference provider, a localized failure on one of them naturally drains away as its endpoints lose health, and traffic concentrates on whichever provider is currently responding well for the model in question. Under normal failure modes – partial outages, regional brownouts, individual model deployments becoming blocked – this typically resolves itself fast enough that the average customer never sees a failed call.
Why failover was slower than expected this time
This incident exposed some parts of our system that did not route through our router, showing points of failure previously not observed. We initially believed failovers were not happening and took action to remove the failing provider manually. We made a configuration change to remove the unhealthy (primary) provider from the overall pool of available providers. However, after taking this action and monitoring, we observed a number of failing API requests continuing to route to that provider.
After further review, we observed some older parts of the codebase still make direct connections for inference. That meant the router's shift to the healthy provider only resolved part of the impact. Two areas of our codebase kept hitting the unhealthy provider directly. Moving them to the secondary provider required a code change and hotfix release to remove direct dependencies on the failing provider.
Next Steps
We understand how an outage such as this can affect you and your business. It’s important we’ve not only resolved this incident, but that we are also taking crucial steps to improve our service to avoid a scenario such as this again. Here are the primary actions we’re taking throughout the coming week:
• Continue working with our primary inference provider to understand why our model deployments were blocked; we will share an amended update here when we have additional information.
• Provide per-customer inference deployments to our enterprise customers in order to reduce the impact of a regional deployment outage.
• Improve our inference router to handle wide-scale issues with a given provider as described in the failure scenario above.
• Finalize the removal of any direct inference communication that does not currently go through our router.
Resolved
We have resolved the issue with our provider and calls have resumed working. We will share more details separately.
Monitoring
We have identified the issue was due to a provider failing, we have migrated traffic to another provider. Calls should be working again. We are monitoring the situation.
Identified
We are investigating a spike in calls failing. We will report here as we find more.
