Resolved -
GitHub Enterprise cloud customers with data residency in the Central US region were unable to access Copilot usage metrics, billing usage reports, Actions usage metrics, and API Insights data between March 12 at 13:09 UTC through March 12 at 16:14 UTC. We fully mitigated the issue on March 12 at 16:14 UTC after an outage that affected our data analytics infrastructure in this region was fixed. Most services recovered by 16:07 UTC, with full resolution at 16:14 UTC. The total impact duration was approximately 3 hours.
Our investigation determined the incident was caused by an infrastructure outage affecting our data analytics infrastructure in the Central US region. Beginning at approximately 13:09 UTC, keepalive signals dropped to zero across our affected clusters. Our services recovered progressively as the affected infrastructure came back online.
We understand this disruption impacted visibility into Copilot usage, billing usage, and Actions metrics for customers who rely on this data to manage their organizations and businesses. We have taken the following immediate steps:
- Engaged our infrastructure provider to improve resilience against similar regional outages.
- Evaluated our monitoring coverage to reduce detection time for infrastructure dependencies.
- Identified graceful degradation options to be implemented to mitigate in the future.
GitHub is critical infrastructure for your work, your teams, and your businesses. We are focused on these mitigations and continued improvements so usage metrics and dependent services remain reliably available.
Mar 12, 16:14 UTC
Update -
Our database infrastructure has recovered and affected services are now operational. Users can access Copilot usage metrics, billing usage reports, action usage metrics, and API Insights. We are continuing to monitor service stability closely.
Mar 12, 16:14 UTC
Update -
Beginning at approximately 1:20pm UTC, users are unable to access Copilot usage metrics, billing usage reports, action usage metrics, and API Insights due to database infrastructure issues. Recovery is in progress. We are monitoring progress closely. We'll post another update by 5:30pm UTC.
Mar 12, 15:52 UTC
Update -
Beginning at approximately 1:20pm UTC, users are unable to access Copilot usage metrics and dashboards. We are investigating the root cause and working to restore service. We'll post another update by 3:30pm UTC.
Mar 12, 15:00 UTC
Investigating -
We are investigating reports of impacted performance for some GitHub services.
Mar 12, 14:58 UTC