GitHub Status

Degraded performance for some Copilot users

Update - We believe we have identified the root cause of the issue and are monitoring to ensure the problem does not recur.
Sep 26, 2024 - 02:34 UTC

Update - We are continuing to investigate the root cause of the latency previously observed to ensure there is no reoccurrence, and better stability going forward.

Sep 26, 2024 - 01:46 UTC

Update - We are continuing to investigate the root cause of the latency previously observed to ensure there is no reoccurrence, and better stability going forward.
Sep 26, 2024 - 01:03 UTC

Update - Copilot users should no longer see request failures. We are still investigating the root cause of the issue to ensure that the experience will remain uninterrupted.
Sep 26, 2024 - 00:29 UTC

Update - We are seeing recovery for requests to Copilot API in affected regions, and are continuing to investigate to ensure the experience remains stable.
Sep 25, 2024 - 23:55 UTC

Update - We have noticed a degradation in performance of Copilot API in some regions. This may result in latency or failed responses to requests to Copilot. We are investigating mitigation options.

Sep 25, 2024 - 23:40 UTC

Investigating - We are investigating reports of degraded performance for Copilot
Sep 25, 2024 - 23:39 UTC

Git Operations Operational

Webhooks Operational

Visit www.githubstatus.com for more information Operational

API Requests Operational

Issues Operational

Pull Requests Operational

Actions Operational

Packages Operational

Pages Operational

Codespaces Operational

Copilot Partial Outage

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Past Incidents

Sep 26, 2024

Unresolved incident: Degraded performance for some Copilot users.

Sep 25, 2024

Incident with Actions Runs

Resolved - This incident has been resolved.
Sep 25, 19:19 UTC

Update - We're seeing issues related to Actions runs failing to download actions at the start of a job. We're investigating the cause and working on mitigations for customers impacted by this issue.
Sep 25, 19:14 UTC

Investigating - We are investigating reports of degraded performance for Actions and Pages
Sep 25, 19:11 UTC

Incident with Git Operations

Resolved - This incident has been resolved.
Sep 25, 16:03 UTC

Update - We are investigating reports of issues with both Actions and Packages, related to a brief period of time where specific Git Operations were failing. We will continue to keep users updated on progress towards mitigation.
Sep 25, 15:34 UTC

Investigating - We are investigating reports of degraded performance for Git Operations
Sep 25, 15:25 UTC

Sep 24, 2024

Incident with Codespaces start and creation

Resolved - This incident has been resolved.
Sep 24, 21:04 UTC

Update - Codespaces is operating normally.
Sep 24, 21:04 UTC

Update - We have successfully mitigated the issue affecting create and resume requests for Codespaces. Early signs of recovery are being observed in the impacted region.
Sep 24, 21:01 UTC

Update - Codespaces is experiencing degraded performance. We are continuing to investigate.
Sep 24, 21:00 UTC

Update - We are investigating issues with Codespaces in the US East geographic area. Some users may not be able to create or start their Codespaces at this time. We will update you on mitigation progress.
Sep 24, 20:56 UTC

Investigating - We are investigating reports of degraded availability for Codespaces
Sep 24, 20:54 UTC

Sep 23, 2024

No incidents reported.

Sep 22, 2024

No incidents reported.

Sep 21, 2024

No incidents reported.

Sep 20, 2024

No incidents reported.

Sep 19, 2024

No incidents reported.

Sep 18, 2024

No incidents reported.

Sep 17, 2024

No incidents reported.

Sep 16, 2024

Incident with Pages and Actions

Resolved - On September 16, 2024, between 21:11 UTC and 22:20 UTC, Actions and Pages services were degraded. Customers who deploy Pages from a source branch experienced delayed runs. Approximately 1,100 runs were delayed long enough to get marked as abandoned. The runs that weren't abandoned completed successfully after we recovered from the incident. Actions jobs experienced average delays of 23 minutes, with some jobs experiencing delays as high as 45 minutes. During the course of the incident, 17% of runs were delayed by more than 5 minutes. At peak, as many as 80% of runs experienced delays exceeding 5 minutes. The root cause was a misconfiguration in the service that manages runner connections, which caused CPU throttling and led to a performance degradation in that service.

We mitigated the incident by diverting runner connections away from the misconfigured nodes. We are working to improve our internal monitoring and alerting to reduce our time to detection and mitigation of issues like this one in the future.
Sep 16, 22:08 UTC

Update - Actions is experiencing degraded performance. We are continuing to investigate.
Sep 16, 21:55 UTC

Update - The team is investigating issues with some Actions jobs being queued for a long time and a percentage of jobs failing. A mitigation has been applied and jobs are starting to recover.
Sep 16, 21:53 UTC

Update - Pages is operating normally.
Sep 16, 21:52 UTC

Update - Actions is experiencing degraded availability. We are continuing to investigate.
Sep 16, 21:37 UTC

Investigating - We are investigating reports of degraded performance for Actions and Pages
Sep 16, 21:31 UTC

Disruption with Git SSH

Resolved - This incident has been resolved.
Sep 16, 14:28 UTC

Update - We are no longer seeing dropped Git SSH connections and believe we have mitigated the incident. We are continuing to monitor and investigate to prevent reoccurrence.
Sep 16, 14:27 UTC

Update - We have taken suspected hosts out of rotation and have not seen any impact in the last 20 minutes. We are continuing to monitor to ensure the problem is resolved and are investigating the cause.
Sep 16, 14:11 UTC

Update - We are seeing up to 2% of Git SSH connections failing.

We have taken suspected problematic hosts out of rotation and are monitoring for recovery and continuing to investigate.
Sep 16, 13:38 UTC

Update - We are investigating failed connections for Git SSH. Customers may be experiencing failed SSH connections both in CI and interactively. Retrying the connection may be successful. Git HTTP connections appear to be unaffected.
Sep 16, 13:30 UTC

Investigating - We are currently investigating this issue.
Sep 16, 13:29 UTC

Sep 15, 2024

No incidents reported.

Sep 14, 2024

Incident with Pull Requests

Resolved - This incident has been resolved.
Sep 14, 22:43 UTC

Update - Pull Requests is operating normally.
Sep 14, 22:43 UTC

Update - we believe we have mitigated and are confirming recovery.
Sep 14, 22:41 UTC

Investigating - We are investigating reports of degraded performance for Pull Requests
Sep 14, 22:10 UTC

Sep 13, 2024

Processing delays to some Issues, Pull Requests and Webhooks

Resolved - On Sep 13, 2024, between 05:03 UTC and 07:13 UTC, the Webhooks and Actions services were degraded resulting in some customers experiencing delayed processing of Webhooks and Actions Runs. 0.5% of Webhook deliveries were delayed more than 2 minutes during the incident. 15% of Actions Runs started between 05:03 and 05:24 UTC saw run start delays or failures. At 05:24 UTC, we implemented a mitigation to shift traffic to healthy infrastructure and new Actions Runs resumed normal operations. During the rest of the incident window, Actions runs started before 05:24 UTC continued to see delays publishing logs or job results. No Actions runs or Webhook deliveries were lost, only delayed.

We mitigated the incident by immediately shifting traffic to a healthy cluster while investigating. The incident was caused by an erroneous configuration change on our eventing platform. A permanent fix was deployed at 06:22 UTC after which services began to recover and burn down their backed up queues, with full recovery by 07:13 UTC.

We are working to reduce our time to detection and develop test automation to prevent issues like this one in the future.

Sep 13, 07:13 UTC

Update - We are seeing improvements in telemetry and are monitoring the delivery of delayed Webhooks and Actions job statuses.
Sep 13, 06:49 UTC

Update - We've applied a mitigation to fix the issues being experienced in some cases with delays to webhook deliveries, and the delayed reporting of the outcome of some running Actions jobs. We are monitoring for full recovery.
Sep 13, 06:23 UTC

Update - Actions is experiencing degraded performance. We are continuing to investigate.
Sep 13, 05:59 UTC

Investigating - We are investigating reports of degraded performance for Issues, Pull Requests and Webhooks
Sep 13, 05:42 UTC

Sep 12, 2024

No incidents reported.

Product

Platform

Support

Company