Incident with Git Operations, API Requests, Actions, Pages, Issues and Copilot
Incident
Report for GitHub
Update
On April 10, 2024, between 2024-04-10 18:33 UTC and 2024-04-10 19:03 UTC, several services were degraded due to the release of a compute-intensive database query that prevented a key database cluster from serving other queries.
GitHub Actions saw delays and failures across the entire run life cycle and had a significant increase in the number of timeouts in API requests. All Pages deployments failed for the duration of the incident. Git Systems saw approximately 12% of raw file download requests and 16% of repository archive download requests return HTTP 50X error codes for the duration of the incident. Issues experienced increased latency for issue creation and updates. Codespaces saw roughly 500 requests to create and resume a Codespace timeout during the incident.
We mitigated the incident by rolling back the offending query. We are working to introduce measures to automatically detect compute-intensive queries in test runs during CI to prevent an issue like this one from recurring.
GitHub Actions saw delays and failures across the entire run life cycle and had a significant increase in the number of timeouts in API requests. All Pages deployments failed for the duration of the incident. Git Systems saw approximately 12% of raw file download requests and 16% of repository archive download requests return HTTP 50X error codes for the duration of the incident. Issues experienced increased latency for issue creation and updates. Codespaces saw roughly 500 requests to create and resume a Codespace timeout during the incident.
We mitigated the incident by rolling back the offending query. We are working to introduce measures to automatically detect compute-intensive queries in test runs during CI to prevent an issue like this one from recurring.
Investigating
Git Operations, API Requests, Actions, Pages, Issues and Copilot are operating normally.
Resolved
This incident has been resolved.
Update
Git Operations, API Requests, Actions, Pages, Issues and Copilot are operating normally.
Update
Copilot is experiencing degraded performance. We are continuing to investigate.
Update
We're aware of issues impacting multiple services and have rolled back the deployment. Systems appear to be recovering and we will continue to monitor.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Update
Copilot is experiencing degraded availability. We are continuing to investigate.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Update
API Requests is experiencing degraded availability. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for Git Operations, API Requests, Actions and Pages
This incident affected: Git Operations, API Requests, Issues, Actions, Pages, and Copilot.
