SUMMARY

On March 22, 2022 from 16:00 UTC to 17:00 UTC, customers using Zendesk Talk on multiple pods experienced calls not connecting between agents and customers.

Timeline

16:25 UTC | 09:25 PT

We are investigating reports of Talk dropped calls and connectivity issues. We will provide an update once we have more information.

16:46 UTC | 09:46 PT

We are continuing to investigate Talk issues with our provider. We will provide more information as it becomes available.

17:05 UTC | 10:05 PT

We are seeing improvements in Talk connectivity and are continuing to monitor performance. We’ll provide another update in 30 minutes or as soon as more information becomes available.

17:41 UTC | 10:41 PT

We are monitoring sustained recovery in Talk connectivity and are awaiting all-clear from our provider on this incident. We will update again once we have that confirmation.

19:16 UTC | 12:16 PT

We’re happy to report that Talk connectivity issues are now fully resolved. Please let us know if you continue to have any issues.

POST-MORTEM

Root Cause Analysis

This incident was caused by an error in our Talk partner’s service that connects the customer’s call with the agent’s call.

Resolution

To fix this issue, the affected service was disabled by our Talk partner, and they spun up a new data cluster to manage new call requests. We started to see reports of recovery, and continued to monitor until full recovery was observed.

Remediation Items

1. Review mechanisms for better alerting if the result provides a good detection means for this type of failure [To be scheduled]
2. Document more clearly via a reference guide which voice provider products/services are used and relied on to allow for quicker and more consistent reviewing of ongoing incidents via their status page. [To be scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out oursystem status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident,please log a ticket with us via ZBot Messaging within the Widget.