June 16, 2026 Synthflow RCA
**Increased Rate Limiting (HTTP 429\) in the Synthflow API**
**Summary**
On June 16, 2026, we experienced a significant increase in overall traffic, which triggered Synthflow’s service-wide rate limiting rules. Users of the Synthflow API would have experienced HTTP 429 Too Many Requests errors. Our service-wide rate limits were set too low for historical purposes. We increased our service-wide rate limits and we are planning further improvements to monitoring and rate limiting.
**Customer impact**
From 18:39-20:48 UTC on June 16, 2026, customers experienced intermittent rate limiting when making API requests or using the user interface. Phone calls were unaffected.
**Root cause**
Synthflow’s rate limits have been in place for more than 6 months as-is. When a request is sent to Synthflow, the request routes through Cloudflare, then GCP Cloud Armor, then various internal Synthflow components. Within this pipeline, Cloudflare IPs were treated as the originating traffic IP. As a result, rate limits were applied across the service rather than per IP. Similarly, we relaxed the time windows within our policies to match traffic patterns.
The combination of limiting by Cloudflare IPs instead of originating IPs and an increase in traffic caused rate limits to trigger for multiple customers during the increase in traffic. The configuration changes during the incident allowed acceptable traffic to return to normal rather than being rate limited.
**Next Steps**
We apologize for the challenges this service-wide limitation caused for you. We are continuing to work to ensure we have the right limits in place to ensure continued high levels of service availability. We are taking the following next steps to improve our service level protections:
• Measure and install limits that match current traffic patterns.
• Monitor shifts in traffic and how rate limits should be applied on an ongoing basis.
• Verify all changes to rate limiting go through our standard change management process (done).
• Relax time window for applying rate limits to allow for more traffic. (done)
• Update our rate limiting to properly use originating IPs. (done)
No components marked as affected
Resolved
June 16, 2026 Synthflow RCA
**Increased Rate Limiting (HTTP 429\) in the Synthflow API**
**Summary**
On June 16, 2026, we experienced a significant increase in overall traffic, which triggered Synthflow’s service-wide rate limiting rules. Users of the Synthflow API would have experienced HTTP 429 Too Many Requests errors. Our service-wide rate limits were set too low for historical purposes. We increased our service-wide rate limits and we are planning further improvements to monitoring and rate limiting.
**Customer impact**
From 18:39-20:48 UTC on June 16, 2026, customers experienced intermittent rate limiting when making API requests or using the user interface. Phone calls were unaffected.
**Root cause**
Synthflow’s rate limits have been in place for more than 6 months as-is. When a request is sent to Synthflow, the request routes through Cloudflare, then GCP Cloud Armor, then various internal Synthflow components. Within this pipeline, Cloudflare IPs were treated as the originating traffic IP. As a result, rate limits were applied across the service rather than per IP. Similarly, we relaxed the time windows within our policies to match traffic patterns.
The combination of limiting by Cloudflare IPs instead of originating IPs and an increase in traffic caused rate limits to trigger for multiple customers during the increase in traffic. The configuration changes during the incident allowed acceptable traffic to return to normal rather than being rate limited.
**Next Steps**
We apologize for the challenges this service-wide limitation caused for you. We are continuing to work to ensure we have the right limits in place to ensure continued high levels of service availability. We are taking the following next steps to improve our service level protections:
• Measure and install limits that match current traffic patterns.
• Monitor shifts in traffic and how rate limits should be applied on an ongoing basis.
• Verify all changes to rate limiting go through our standard change management process (done).
• Relax time window for applying rate limits to allow for more traffic. (done)
• Update our rate limiting to properly use originating IPs. (done)
