Incident Summary: On June 30th at approximately 5:35pm, our network experienced an unannounced outage lasting until approximately 5:55pm. We sincerely apologize for the disruption this caused to your workday and activities.
What Happened: This outage occurred during a planned upgrade to core networking equipment in our secondary network center. This type of upgrade is designed to happen seamlessly and invisibly in the background. However, during the process, our routing and security systems did not automatically adjust and reroute network traffic designed. This unexpected behavior temporarily blocked communication pathways, making a large number of our services unavailable.
How It Was Resolved: Once the upgrade process completed its cycle, network traffic pathways were automatically restored, and all affected services returned to normal operation. The total duration of the disruption was confined to an estimated 20 minutes.
Next Steps: As we take system availability seriously, we are currently conducting a thorough review of why the automated systems failed to adjust as expected. We will take corrective action and test during a planned and announced maintenance window.
Thank you for your patience and understanding as we worked to resolve this issue quickly.