GitHub Status

Sep 6, 2023

Resolved - This incident has been resolved.
Sep 6, 11:41 UTC

Update - We identified the issue affecting Git Operations and applied the mitigation. We expect service to fully recover in 30 min.
Sep 6, 10:07 UTC

Update - Operations that download Git archives and raw files are seeing highly elevated response times (up to 4+ seconds) for users. Other Git operations are not impacted. We are working on mitigation.
Sep 6, 09:03 UTC

Update - We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Sep 6, 08:53 UTC

Investigating - We are investigating reports of degraded performance for Git Operations
Sep 6, 08:21 UTC

Sep 5, 2023

Incident with Actions, Pages, Pull Requests, Codespaces, API Requests, Issues, Webhooks and Packages

Resolved - From 16:24-16:43 UTC, multiple GitHub services were down or degraded due to an outage in one of our primary databases.

The primary host for a shared datastore for GitHub experienced an underlying file system write error which affected availability for the majority of public-facing GitHub services. SAML login was affected, as was access to Actions, Issues, Pull Requests, Pages, API, Webhooks, Codespaces, and Packages. In this case, our automatic failover was unable to handle the partial file system failure mode. We mitigated by manually performing a forced failover, initiated 17 minutes after our first alert and completed 2 minutes later.

With the incident mitigated, we are working to assess more detailed impact and resilience improvements to each impacted service while also improving the automated failover mechanism to support this scenario.
Sep 5, 17:01 UTC

Update - Pages is operating normally.
Sep 5, 17:01 UTC

Update - Actions is operating normally.
Sep 5, 17:00 UTC

Update - Issues is operating normally.
Sep 5, 16:51 UTC

Update - We have performed a mitigation affecting write traffic across services and are seeing recovery for affected customers.
Sep 5, 16:51 UTC

Update - Packages is operating normally.
Sep 5, 16:50 UTC

Update - Webhooks is operating normally.
Sep 5, 16:49 UTC

Update - API Requests is operating normally.
Sep 5, 16:49 UTC

Update - Codespaces is operating normally.
Sep 5, 16:48 UTC

Update - Packages is experiencing degraded performance. We are continuing to investigate.
Sep 5, 16:45 UTC

Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Sep 5, 16:44 UTC

Update - We are investigating an issue that is impacting a small percentage of requests across all services including authentication and are working on mitigating impact.
Sep 5, 16:40 UTC

Update - Issues is experiencing degraded performance. We are continuing to investigate.
Sep 5, 16:39 UTC

Update - API Requests is experiencing degraded performance. We are continuing to investigate.
Sep 5, 16:35 UTC

Update - Codespaces is experiencing degraded availability. We are continuing to investigate.
Sep 5, 16:33 UTC

Update - Pull Requests is experiencing degraded performance. We are continuing to investigate.
Sep 5, 16:31 UTC

Update - Pages is experiencing degraded performance. We are continuing to investigate.
Sep 5, 16:31 UTC

Investigating - We are investigating reports of degraded performance for Actions
Sep 5, 16:30 UTC

Incident with Git Operations and Actions

Resolved - Between 06:59 UTC to 13:18 UTC GitHub customers experienced intermittent increased latency when using our git archive and raw file downloading capability. This capability is typically used for larger CI/CD systems and bulk download of repository data. This also caused intermittent periods of Actions workflow failures.

During this incident we saw bursts of increased unique traffic on specific endpoints that overloaded our caching capability for this system. In addition, the traffic caused saturation of the service that exposed a bug in downloading archived files at high load which caused some archives to be downloaded with incomplete data. This caused intermittent failure of workflows for GitHub Actions, which primarily downloads actions data via these endpoints.

We mitigated this through improvements to our rate limiting and changes to get to a healthy state across our backend service and cache . We initially relied on automated communication messages during this incident and will be working to ensure future incidents have more relevant status updates with specific customer impact.
Sep 5, 14:19 UTC

Update - Recovery is in progress. We are still taking additional mitigation steps and will status back to green when those are completed and we are confident they are effective.
Sep 5, 14:03 UTC

Update - We have mitigated the impact on download and raw file operations and are seeing recovery on response times but are continuing to monitor.
Sep 5, 13:18 UTC

Update - Operations that download Git archives and raw files are seeing highly elevated response times (up to 4+ seconds) for users. Other Git operations are not impacted. We are working on mitigation.
Sep 5, 12:48 UTC

Update - We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Sep 5, 12:15 UTC

Update - We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Sep 5, 11:37 UTC

Update - We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Sep 5, 10:52 UTC

Update - Actions is operating normally.
Sep 5, 10:39 UTC

Update - We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Sep 5, 10:17 UTC

Update - We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Sep 5, 09:35 UTC

Update - We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Sep 5, 09:03 UTC

Update - We are investigating reports of issues with service(s): Git Operations, Actions. We will continue to keep users updated on progress towards mitigation.
Sep 5, 08:32 UTC

Update - Actions is experiencing degraded performance. We are continuing to investigate.
Sep 5, 08:04 UTC

Update - We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Sep 5, 08:01 UTC

Update - We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Sep 5, 07:30 UTC

Investigating - We are investigating reports of degraded performance for Git Operations
Sep 5, 06:59 UTC

Incident with API Requests, Pull Requests and Actions

Resolved - This incident has been resolved.
Sep 5, 05:10 UTC

Update - API Requests is operating normally.
Sep 5, 05:09 UTC

Update - Actions is operating normally.
Sep 5, 05:09 UTC

Update - We are investigating reports of issues with service(s): Actions, API Requests, Pull Requests. We will continue to keep users updated on progress towards mitigation.
Sep 5, 04:52 UTC

Update - Actions is experiencing degraded performance. We are continuing to investigate.
Sep 5, 04:51 UTC

Investigating - We are investigating reports of degraded performance for API Requests and Pull Requests
Sep 5, 04:17 UTC

Incident with Actions

Resolved - This incident has been resolved.
Sep 5, 03:27 UTC

Investigating - We are investigating reports of degraded performance for Actions
Sep 5, 03:18 UTC

Sep 4, 2023

Incident with Actions

Resolved - This incident has been resolved.
Sep 4, 21:56 UTC

Investigating - We are investigating reports of degraded performance for Actions
Sep 4, 21:28 UTC

Incident with Actions

Resolved - This incident has been resolved.
Sep 4, 14:48 UTC

Investigating - We are investigating reports of degraded performance for Actions
Sep 4, 14:21 UTC

Sep 3, 2023

No incidents reported.

Sep 2, 2023

No incidents reported.

Sep 1, 2023

Incident with Codespaces

Resolved - This incident has been resolved.
Sep 1, 15:06 UTC

Update - A limited number of users in the UK and Western Europe are experiencing issues starting or reconnecting to GitHub Codespaces. We are investigating the issue.
Sep 1, 14:51 UTC

Investigating - We are investigating reports of degraded performance for Codespaces
Sep 1, 14:35 UTC

Aug 31, 2023

No incidents reported.

Aug 30, 2023

We are investigating reports of degraded performance.

Resolved - This incident has been resolved.
Aug 30, 21:04 UTC

Investigating - We are currently investigating this issue.
Aug 30, 20:56 UTC

Incident with Webhooks

Resolved - This incident has been resolved.
Aug 30, 20:53 UTC

Update - Webhook delays have been reduced significantly. We anticipate returning to nominal service soon.
Aug 30, 20:42 UTC

Update - The remediation for Webhook delivery delays has been implemented. We will continue to keep users updated on Webhook latency improvements.
Aug 30, 19:13 UTC

Update - We are addressing the source of Webhook delays, and will continue to keep users updated on progress towards mitigation.
Aug 30, 18:10 UTC

Update - We have identified the source of delay in webhook deliveries and are in the process of implementing a fix.
Aug 30, 17:37 UTC

Update - We are investigating increased delivery time issues with Webhooks. We will continue to keep users updated on progress towards mitigation.
Aug 30, 16:49 UTC

Investigating - We are investigating reports of degraded performance for Webhooks
Aug 30, 16:45 UTC

Aug 29, 2023

Incident with Webhooks

Resolved - This incident has been resolved.
Aug 29, 21:58 UTC

Update - GitHub webhooks are fully functional but can take up to 20 minutes to deliver. We are working on identifying the root cause of the increased latency.
Aug 29, 21:52 UTC

Update - The Webhooks backlog continues to process, and we anticipate continued latency while it clears.
Aug 29, 20:55 UTC

Update - The Webhooks backlog continues to process, and we anticipate continued latency while it clears.
Aug 29, 19:38 UTC

Update - We’ve mitigated the issue causing Webhooks latency and are monitoring processing of the backlog of queued requests.
Aug 29, 18:24 UTC

Update - We are investigating increased delivery time issues with Webhooks. We will continue to keep users updated on progress towards mitigation.
Aug 29, 17:31 UTC

Investigating - We are investigating reports of degraded performance for Webhooks
Aug 29, 17:29 UTC

We are investigating reports of degraded performance.

Resolved - This incident has been resolved.
Aug 29, 16:16 UTC

Update - Intermittent failures with our resource provider prevented around 200 Codespaces from launching or resuming in Europe West and Southeast Asia. We have mitigated impact and continue to monitor.
Aug 29, 16:05 UTC

Investigating - We are currently investigating this issue.
Aug 29, 16:05 UTC

Incident with Codespaces

Resolved - This incident has been resolved.
Aug 29, 15:35 UTC

Update - We are investigating issues with Codespaces in the Europe West and Southeast Asia geographic areas. Some users may not be able to connect to their Codespaces at this time. We will update on progress.
Aug 29, 15:25 UTC

Investigating - We are investigating reports of degraded performance for Codespaces
Aug 29, 15:13 UTC

Incident with Actions, Pull Requests and Webhooks

Resolved - This incident has been resolved.
Aug 29, 02:36 UTC

Update - Webhooks is operating normally.
Aug 29, 02:34 UTC

Update - Pull Requests is operating normally.
Aug 29, 02:34 UTC

Update - Actions is operating normally.
Aug 29, 02:34 UTC

Update - GitHub webhooks are delayed, and required pull-request checks and reviews are not triggered.
Aug 29, 02:15 UTC

Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Aug 29, 02:14 UTC

Update - Newly triggered Actions workflows are stalled. We identified the problem, and we are working on a mitigation.
Aug 29, 02:07 UTC

Update - Pull Requests is experiencing degraded performance. We are continuing to investigate.
Aug 29, 02:02 UTC

Investigating - We are investigating reports of degraded performance for Actions
Aug 29, 01:59 UTC

Aug 28, 2023

No incidents reported.

Aug 27, 2023

No incidents reported.

Aug 26, 2023

No incidents reported.

Aug 25, 2023

No incidents reported.

Aug 24, 2023

No incidents reported.

Aug 23, 2023

Incident with Webhooks

Resolved - This incident has been resolved.
Aug 23, 16:23 UTC

Update - We are investigating and working towards mitigating slight delays in Webhook deliveries.
Aug 23, 15:56 UTC

Investigating - We are investigating reports of degraded performance for Webhooks
Aug 23, 15:38 UTC

Product

Platform

Support

Company