Get email notifications whenever GitHub creates, updates or resolves an incident.
Get text message notifications whenever GitHub creates or resolves an incident.
Get incident updates and maintenance status messages in Slack.
Subscribe via Slack
By subscribing you acknowledge our Privacy Policy. In addition, you agree to the Atlassian Cloud Terms of Service and acknowledge Atlassian's Privacy Policy.
Get webhook notifications whenever GitHub creates an incident, updates an incident, resolves an incident or changes a component status.
Visit our support site.
All Systems Operational
Git Operations
?
Operational
API Requests
?
Operational
Webhooks
?
Operational
Visit www.githubstatus.com for more information
Operational
Issues
?
Operational
Pull Requests
?
Operational
Actions
?
Operational
Packages
?
Operational
Pages
?
Operational
Codespaces
?
Operational
Copilot
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Apr 18, 2024
No incidents reported today.
Apr 17, 2024
Resolved -
This incident has been resolved.
Apr 17, 00:48 UTC
Apr 17, 00:48 UTC
Update -
We're continuing to investigate issues with Copilot
Apr 17, 00:30 UTC
Apr 17, 00:30 UTC
Update -
Copilot is experiencing degraded performance. We are continuing to investigate.
Apr 16, 23:59 UTC
Apr 16, 23:59 UTC
Update -
We're investigating issues with Copilot availability
Apr 16, 23:57 UTC
Apr 16, 23:57 UTC
Investigating -
We are currently investigating this issue.
Apr 16, 23:51 UTC
Apr 16, 23:51 UTC
Apr 16, 2024
Apr 15, 2024
Resolved -
This incident has been resolved.
Apr 15, 14:53 UTC
Apr 15, 14:53 UTC
Update -
We have applied mitigation for Copilot in EU region and are working towards the full recovery of the service.
Apr 15, 14:13 UTC
Apr 15, 14:13 UTC
Update -
Due to an outage in one Copilot region traffic is currently being served from other regions. European users may experience higher response times.
Apr 15, 13:35 UTC
Apr 15, 13:35 UTC
Investigating -
We are investigating reports of degraded performance for Copilot
Apr 15, 12:58 UTC
Apr 15, 12:58 UTC
Apr 14, 2024
Resolved -
Beginning at 17:30 UTC on April 11th and lasting until 20:30 UTC on April 14th, github.com saw significant (up to 2 hours) delays in delivering emails. At 14:21 UTC on April 14th, community reports of this were confirmed and an incident declared. Emails most impacted by the delay were password reset and unrecognized device verification, which contain time-sensitive links or verification codes, and are required to be acted on in order for password resets or unrecognized logins to proceed.
Users attempting to reset their password during the incident were unable to complete the reset. Users without two-factor authentication (2FA), signing in on an unrecognized device, were unable to complete device verification. Enterprise Managed Users, users with 2FA, and users on recognized devices or IP addresses were still able to sign in. This impacted 800-1000 user device verifications and 300-400 password resets.
The mailer delays were caused by increased usage of a shared resource pool; a separate internal job queue became unhealthy and prevented the mailer queue from being worked on.
We have made some immediate improvements to better detect and react to this type of situation again. As a short-term mitigation strategy, we have added a queue-bypass ability for time-sensitive emails, like password reset and unrecognized device verification. We can enable this setting if we observe email delays reoccurring, which will ensure that future incidents do not affect user ability to complete critical login flows. We have paused the unhealthy job queue, to prevent impact to other queues using shared resources. And we have updated our methods of detection for anomalous email delivery, to identify this issue sooner.
Apr 14, 21:53 UTC
Users attempting to reset their password during the incident were unable to complete the reset. Users without two-factor authentication (2FA), signing in on an unrecognized device, were unable to complete device verification. Enterprise Managed Users, users with 2FA, and users on recognized devices or IP addresses were still able to sign in. This impacted 800-1000 user device verifications and 300-400 password resets.
The mailer delays were caused by increased usage of a shared resource pool; a separate internal job queue became unhealthy and prevented the mailer queue from being worked on.
We have made some immediate improvements to better detect and react to this type of situation again. As a short-term mitigation strategy, we have added a queue-bypass ability for time-sensitive emails, like password reset and unrecognized device verification. We can enable this setting if we observe email delays reoccurring, which will ensure that future incidents do not affect user ability to complete critical login flows. We have paused the unhealthy job queue, to prevent impact to other queues using shared resources. And we have updated our methods of detection for anomalous email delivery, to identify this issue sooner.
Apr 14, 21:53 UTC
Update -
We are seeing a full recovery. Device verification and password reset emails are delivered on time.
Apr 14, 21:52 UTC
Apr 14, 21:52 UTC
Update -
We are deploying a possible mitigation to the delayed device verification and password change emails.
Apr 14, 21:34 UTC
Apr 14, 21:34 UTC
Update -
We continue to investigate issues with delays of email deliveries which is preventing users without 2FA enabled from verifying new devices. We will provide more information as it becomes available.
Apr 14, 19:54 UTC
Apr 14, 19:54 UTC
Update -
We are continuing to investigate issues with the delivery of device verification emails for users without 2FA.
Apr 14, 15:50 UTC
Apr 14, 15:50 UTC
Update -
We are continuing to investigate issues with the delivery of device verification emails for users without 2FA on new devices.
Apr 14, 15:01 UTC
Apr 14, 15:01 UTC
Update -
Device verification emails for sign-ins for users without 2FA on new devices are being sent late or not at all. This is blocking successful sign-ins for these users. We are investigating.
Apr 14, 14:27 UTC
Apr 14, 14:27 UTC
Investigating -
We are currently investigating this issue.
Apr 14, 14:21 UTC
Apr 14, 14:21 UTC
Apr 13, 2024
No incidents reported.
Apr 12, 2024
No incidents reported.
Apr 11, 2024
No incidents reported.
Apr 10, 2024
Update -
On April 10, 2024, between 2024-04-10 18:33 UTC and 2024-04-10 19:03 UTC, several services were degraded due to the release of a compute-intensive database query that prevented a key database cluster from serving other queries.
GitHub Actions saw delays and failures across the entire run life cycle and had a significant increase in the number of timeouts in API requests. All Pages deployments failed for the duration of the incident. Git Systems saw approximately 12% of raw file download requests and 16% of repository archive download requests return HTTP 50X error codes for the duration of the incident. Issues experienced increased latency for issue creation and updates. Codespaces saw roughly 500 requests to create and resume a Codespace timeout during the incident.
We mitigated the incident by rolling back the offending query. We are working to introduce measures to automatically detect compute-intensive queries in test runs during CI to prevent an issue like this one from recurring.
Apr 10, 19:03 UTC
GitHub Actions saw delays and failures across the entire run life cycle and had a significant increase in the number of timeouts in API requests. All Pages deployments failed for the duration of the incident. Git Systems saw approximately 12% of raw file download requests and 16% of repository archive download requests return HTTP 50X error codes for the duration of the incident. Issues experienced increased latency for issue creation and updates. Codespaces saw roughly 500 requests to create and resume a Codespace timeout during the incident.
We mitigated the incident by rolling back the offending query. We are working to introduce measures to automatically detect compute-intensive queries in test runs during CI to prevent an issue like this one from recurring.
Apr 10, 19:03 UTC
Investigating -
Git Operations, API Requests, Actions, Pages, Issues and Copilot are operating normally.
Apr 10, 19:03 UTC
Apr 10, 19:03 UTC
Resolved -
This incident has been resolved.
Apr 10, 19:03 UTC
Apr 10, 19:03 UTC
Update -
Git Operations, API Requests, Actions, Pages, Issues and Copilot are operating normally.
Apr 10, 19:03 UTC
Apr 10, 19:03 UTC
Update -
Copilot is experiencing degraded performance. We are continuing to investigate.
Apr 10, 19:01 UTC
Apr 10, 19:01 UTC
Update -
We're aware of issues impacting multiple services and have rolled back the deployment. Systems appear to be recovering and we will continue to monitor.
Apr 10, 18:55 UTC
Apr 10, 18:55 UTC
Update -
API Requests is experiencing degraded performance. We are continuing to investigate.
Apr 10, 18:53 UTC
Apr 10, 18:53 UTC
Update -
Copilot is experiencing degraded availability. We are continuing to investigate.
Apr 10, 18:45 UTC
Apr 10, 18:45 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Apr 10, 18:42 UTC
Apr 10, 18:42 UTC
Update -
API Requests is experiencing degraded availability. We are continuing to investigate.
Apr 10, 18:42 UTC
Apr 10, 18:42 UTC
Investigating -
We are investigating reports of degraded performance for Git Operations, API Requests, Actions and Pages
Apr 10, 18:41 UTC
Apr 10, 18:41 UTC
Resolved -
Between 2024-04-09 21:35 UTC and 2024-04-10 19:03 UTC, creation of new Codespaces was degraded by an image upgrade to the virtual machines of new Codespaces. During the incident, approximately 7% of new Codespaces were created but never became available to their owning end users.
We mitigated the incident by reverting to the previous image version. We are working to improve deployment confidence around image upgrades to reduce the likelihood of recurrence.
Apr 10, 18:07 UTC
We mitigated the incident by reverting to the previous image version. We are working to improve deployment confidence around image upgrades to reduce the likelihood of recurrence.
Apr 10, 18:07 UTC
Update -
We have applied a fix and are continuing to monitor. This incident will remain open for now until we have confirmed that the service is fully restored.
Apr 10, 17:31 UTC
Apr 10, 17:31 UTC
Update -
We believe we have identified the root cause of the issue and are working to fully restore the Codespaces service. We will provide another update within the next 30 minutes.
Apr 10, 16:56 UTC
Apr 10, 16:56 UTC
Update -
We’re seeing issues related to connecting to Codespaces impacting a subset of users. We are actively investigating and will provide another update shortly.
Apr 10, 16:20 UTC
Apr 10, 16:20 UTC
Investigating -
We are investigating reports of degraded performance for Codespaces
Apr 10, 16:12 UTC
Apr 10, 16:12 UTC
Resolved -
Between 8:18 and 9:38 UTC on Wednesday, April 10th, customers experienced increased error rates across several services due to an overloaded primary database instance, ultimately caused by an unbounded query. We mitigated the impact by failing the instance over to more capable hardware and shipping an improved version of the query that runs against read replicas. In response to this incident, we are also working to make performance improvements to the class of queries that most frequently resulted in failed requests during this timeframe.
Web-based repository file editing saw a 17% failure rate during the incident with other repository management operations (e.g. rule updates, web-based branch creation, repository renames) seeing failure rates between 1.5% and 8%. API failure rates for these operations were higher.
Issue and Pull Request authoring was heavily impacted during this incident due to reliance on the impacted database primary. We are continuing work to remove our dependence on this particular primary instance from our authoring workflows for these services.
GitHub search saw a 5% failure rate throughout this incident due to reliance on the impacted primary database when authorizing repository access. The majority of failing requests were for search bar autocomplete with a limited number of search result failures as well.
Apr 10, 09:38 UTC
Web-based repository file editing saw a 17% failure rate during the incident with other repository management operations (e.g. rule updates, web-based branch creation, repository renames) seeing failure rates between 1.5% and 8%. API failure rates for these operations were higher.
Issue and Pull Request authoring was heavily impacted during this incident due to reliance on the impacted database primary. We are continuing work to remove our dependence on this particular primary instance from our authoring workflows for these services.
GitHub search saw a 5% failure rate throughout this incident due to reliance on the impacted primary database when authorizing repository access. The majority of failing requests were for search bar autocomplete with a limited number of search result failures as well.
Apr 10, 09:38 UTC
Update -
Issues and Pull Requests are operating normally.
Apr 10, 09:38 UTC
Apr 10, 09:38 UTC
Update -
The mitigation rolled out has successfully resolved the issue. We have seen failure rates reduce and normal service return across all affected features.
Apr 10, 09:38 UTC
Apr 10, 09:38 UTC
Update -
We are aware of impact across a number of GitHub features. This is primarily seen to be impacting write actions for Issues, Repositories and Pull Requests. Additionally we are seeing increased failure rates for search queries.
Our team has rolled out a mitigation and is monitoring for recovery.
Apr 10, 09:30 UTC
Our team has rolled out a mitigation and is monitoring for recovery.
Apr 10, 09:30 UTC
Investigating -
We are investigating reports of degraded availability for Issues and Pull Requests
Apr 10, 09:22 UTC
Apr 10, 09:22 UTC
Apr 9, 2024
Resolved -
On April 9, 2024, between 18:00 and 20:17 UTC, Actions was degraded and had failures for new and existing customers. During this time, Actions failed to start for 5,426 new repositories, and 1% of runs for existing customers were delayed, with half of those failing due to an infrastructure error.
The root cause was an expired certificate which caused authentication to fail between internal services. The incident was mitigated once the cert was rotated.
We are working to improve our automation to ensure certs are rotated before expiration.
Apr 9, 20:17 UTC
The root cause was an expired certificate which caused authentication to fail between internal services. The incident was mitigated once the cert was rotated.
We are working to improve our automation to ensure certs are rotated before expiration.
Apr 9, 20:17 UTC
Update -
We continue to work to resolve issues with repositories not being able to enable Actions and Actions network configuration setup not working properly. We have confirmed a fix and are in the process of deploying it to production. Another update will be shared within the next 30 minutes.
Apr 9, 19:43 UTC
Apr 9, 19:43 UTC
Update -
We continue to work to resolve issues with repositories not being able to enable Actions and Actions network configuration setup not working properly. We will provide additional information shortly.
Apr 9, 19:06 UTC
Apr 9, 19:06 UTC
Update -
We are aware of issues with repositories not being able to enable Actions. We are in the process of restoring full functionality and will provide additional information shortly.
Apr 9, 18:36 UTC
Apr 9, 18:36 UTC
Investigating -
We are investigating reports of degraded performance for Actions
Apr 9, 18:36 UTC
Apr 9, 18:36 UTC
Resolved -
On April 9, 2024, between 04:32 UTC and 05:10 UTC, an outage occurred in Github Packages, specifically impacting the download functionality of NPM Packages. All attempts to download NPM Packages failed during this period. Upon investigation, we found a recent code change in the NPM Registry to be the root cause. The customer impact was limited to users of NPM Registry, with no effects on other registries.
We mitigated the incident by rolling back the problematic change. We are following up with repair items to cover our observability gaps and implementing measures in our CI process to detect such failures early before they can impact customers.
Apr 9, 05:10 UTC
We mitigated the incident by rolling back the problematic change. We are following up with repair items to cover our observability gaps and implementing measures in our CI process to detect such failures early before they can impact customers.
Apr 9, 05:10 UTC
Update -
We are investigating reports of issues with downloading NPM packages. We will continue to keep users updated on progress towards mitigation.
Apr 9, 04:51 UTC
Apr 9, 04:51 UTC
Investigating -
We are currently investigating this issue.
Apr 9, 04:32 UTC
Apr 9, 04:32 UTC
Apr 8, 2024
No incidents reported.
Apr 7, 2024
No incidents reported.
Apr 6, 2024
Resolved -
On April 6, 2024, between 00:00:00 UTC and 02:20:05 UTC, access to Private Pages on the *.pages.github.io domain was degraded while the deployed TLS certificate was expired. Service was restored by uploading the renewed certificate to our CDN. This was due to a process error and a gap in our alerting. While the certificate was renewed and updated in our internal vault, it was not deployed to the CDN.
We are working to reduce potential for errors in our certificate renewal process as well as adding the *.pages.github.io domain to our existing TLS alerting system.
Apr 6, 02:22 UTC
We are working to reduce potential for errors in our certificate renewal process as well as adding the *.pages.github.io domain to our existing TLS alerting system.
Apr 6, 02:22 UTC
Update -
We are investigating issues with private pages due to an expired certificate
Apr 6, 01:52 UTC
Apr 6, 01:52 UTC
Investigating -
We are investigating reports of degraded performance for Pages
Apr 6, 01:52 UTC
Apr 6, 01:52 UTC
Apr 5, 2024
Resolved -
On April 5, 2024, between 8:11 and 8:58 UTC a number of GitHub services were degraded, returning error responses. Web request error rate peaked at 6%, API request error rate peaked at 10%. Actions had 103,660 workflow runs fail to start.
A database load balancer change caused connection failures in one of our three data centers to various critical database clusters. The incident was mitigated once that change was rolled back.
We have updated our deployment pipeline to better detect this problem in earlier stages of rollout to reduce impact to end users.
Apr 5, 09:18 UTC
A database load balancer change caused connection failures in one of our three data centers to various critical database clusters. The incident was mitigated once that change was rolled back.
We have updated our deployment pipeline to better detect this problem in earlier stages of rollout to reduce impact to end users.
Apr 5, 09:18 UTC
Update -
Pull Requests is operating normally.
Apr 5, 09:17 UTC
Apr 5, 09:17 UTC
Update -
Issues is operating normally.
Apr 5, 09:17 UTC
Apr 5, 09:17 UTC
Update -
API Requests is operating normally.
Apr 5, 09:17 UTC
Apr 5, 09:17 UTC
Update -
Codespaces is operating normally.
Apr 5, 09:17 UTC
Apr 5, 09:17 UTC
Update -
Actions is operating normally.
Apr 5, 09:17 UTC
Apr 5, 09:17 UTC
Update -
Pages is operating normally.
Apr 5, 09:17 UTC
Apr 5, 09:17 UTC
Update -
Actions is experiencing degraded performance. We are continuing to investigate.
Apr 5, 09:17 UTC
Apr 5, 09:17 UTC
Update -
We've reverted a change we believe caused this, are seeing initial indications of reduced errors, and are monitoring for full recovery
Apr 5, 09:00 UTC
Apr 5, 09:00 UTC
Update -
Pages is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:59 UTC
Apr 5, 08:59 UTC
Update -
We're seeing connection failures to some databases in two of three sites and are investigating.
Apr 5, 08:51 UTC
Apr 5, 08:51 UTC
Update -
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:50 UTC
Apr 5, 08:50 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:49 UTC
Apr 5, 08:49 UTC
Update -
API Requests is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:49 UTC
Apr 5, 08:49 UTC
Update -
Codespaces is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:49 UTC
Apr 5, 08:49 UTC
Investigating -
We are investigating reports of degraded availability for Actions
Apr 5, 08:33 UTC
Apr 5, 08:33 UTC
Resolved -
This incident has been resolved.
Apr 5, 08:53 UTC
Apr 5, 08:53 UTC
Investigating -
We are currently investigating this issue.
Apr 5, 08:31 UTC
Apr 5, 08:31 UTC
Resolved -
This incident has been resolved.
Apr 5, 08:48 UTC
Apr 5, 08:48 UTC
Update -
Issues, API Requests, Pull Requests and Codespaces are operating normally.
Apr 5, 08:48 UTC
Apr 5, 08:48 UTC
Update -
Codespaces is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:36 UTC
Apr 5, 08:36 UTC
Update -
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:34 UTC
Apr 5, 08:34 UTC
Update -
API Requests is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:32 UTC
Apr 5, 08:32 UTC
Investigating -
We are investigating reports of degraded performance for Issues
Apr 5, 08:28 UTC
Apr 5, 08:28 UTC
Apr 4, 2024
Resolved -
Between April 3rd, 2024 23:15 UTC and April 4th, 2024 01:10 UTC, GitHub Actions experienced a partial infrastructure outage that led to degraded workflows (failed or delayed starts). Additionally, 0.15% of Webhook deliveries were degraded due to an unrelated spike in database latency in a single availability zone. SLOs for Actions were 90% during the incident, but this was not evenly distributed across customers. We statused green after a long stretch of recovered SLOs, starting at April 4th, 2024 00:35 UTC. During this incident, we also had issues with incident tooling (https://www.githubstatus.com/) failing to update the public status page and occasionally not loading.
The incident was resolved after the infrastructure issue was mitigated at 2024-04-04 04:27 UTC.
We are working to improve monitoring and processes in response to this incident. We are investigating how we can improve resilience and our communication with our infrastructure provider, and how we can better handle ongoing incidents that are no longer impacting SLOs. We are also improving our incident tooling to ensure that the public status page is updated in a timely manner.
Apr 4, 01:10 UTC
The incident was resolved after the infrastructure issue was mitigated at 2024-04-04 04:27 UTC.
We are working to improve monitoring and processes in response to this incident. We are investigating how we can improve resilience and our communication with our infrastructure provider, and how we can better handle ongoing incidents that are no longer impacting SLOs. We are also improving our incident tooling to ensure that the public status page is updated in a timely manner.
Apr 4, 01:10 UTC
Update -
API Requests is operating normally.
Apr 4, 01:09 UTC
Apr 4, 01:09 UTC
Update -
Actions is operating normally.
Apr 4, 01:07 UTC
Apr 4, 01:07 UTC
Update -
We are seeing recovery in Actions workflows creation and accessing Actions statuses via the API.
Apr 4, 00:46 UTC
Apr 4, 00:46 UTC
Update -
Webhooks is experiencing degraded performance. We are continuing to investigate.
Apr 4, 00:25 UTC
Apr 4, 00:25 UTC
Update -
We are investigating Actions workflows failures and delays.
Apr 4, 00:12 UTC
Apr 4, 00:12 UTC
Update -
API Requests is experiencing degraded performance. We are continuing to investigate.
Apr 4, 00:06 UTC
Apr 4, 00:06 UTC
Investigating -
We are investigating reports of degraded performance for Actions
Apr 3, 23:59 UTC
Apr 3, 23:59 UTC
