API Service Outgage

Major incident API
2025-10-10 09:00 UTC · 8 hours

Updates

Issue

API Incident Report — Enterprise API Outage

Date: October 10, 2025
Status: ✅ Resolved

Summary

Earlier today, our monitoring systems detected elevated error rates across all Enterprise API endpoints. The issue was traced to a permissioning misconfiguration in our API gateway that inadvertently restricted access to Enterprise routes, resulting in a full outage for Enterprise API consumers.

Impact

  • Scope: All Enterprise API endpoints were unavailable.
  • Customers Affected: Enterprise API customers only (Public API remained unaffected).
  • User Experience: Requests to Enterprise endpoints failed with 500 errors.

Root Cause

An incorrect permission update was propagated to the API permissioning system, removing the expected access policies for Enterprise API routes. This change prevented authenticated enterprise clients from successfully invoking endpoints.

Resolution

The team identified the permissioning error and restored API gateway permissions to their expected values. Following this change, all Enterprise API endpoints resumed normal operation, and monitoring confirmed full recovery.


Current Status: All systems operational.
Next Steps: Post-incident review scheduled to finalize preventive actions and close the incident.

October 10, 2025 · 19:35 UTC

← Back