GitHub header
All Systems Operational
Git Operations ? Operational
API Requests ? Operational
Webhooks ? Operational
Visit www.githubstatus.com for more information Operational
Issues ? Operational
Pull Requests ? Operational
Actions ? Operational
Packages ? Operational
Pages ? Operational
Codespaces ? Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Nov 20, 2023

No incidents reported today.

Nov 19, 2023

No incidents reported.

Nov 18, 2023

No incidents reported.

Nov 17, 2023

No incidents reported.

Nov 16, 2023

No incidents reported.

Nov 15, 2023
We are investigating reports of degraded performance.
Resolved - On 2023-11-15, from 09:44 to 10:42 UTC, some GitHub customers experienced increased latency or errors accessing repo data.

High concurrent access to a specific git object exposed a bug that forced a backend service to perform excessive calculations, overloading the service. Access to this repo was paused while load was re-rerouted, mitigating the problem.
The conditions that triggered the expensive operations have been identified and refactored.
Nov 15, 11:34 UTC
Update - Error rates and performance have returned to normal.
Nov 15, 11:33 UTC
Update - We have identified the source of the issue and have removed the additional load from the service. Sporadic delays in pull request experiences and intermittent 500s are still occurring and impacting a very small percentage of traffic. Next update is expected within 30 minutes.
Nov 15, 11:21 UTC
Update - We are seeing connectivity issues between some of our systems and git backend services. This is causing intermittent error responses and delays in pull request experiences for a very small percentage of traffic. We are investigating mitigations and expect to provide another update within 30 minutes.
Nov 15, 11:04 UTC
Investigating - We are currently investigating this issue.
Nov 15, 09:50 UTC
Nov 14, 2023

No incidents reported.

Nov 13, 2023
We are investigating reports of degraded performance.
Resolved - Between 20:35 and 21:38 we experienced up to a 20 minute delay delivering around 30,000 notifications due to side effects of some planned maintenance on supporting systems. We have noted the unexpected user impact of this type of maintenance and will address it in future maintenance planning.
Nov 13, 21:38 UTC
Update - An issue related to notifications has been resolved. Users should again be seeing their notifications.
Nov 13, 21:38 UTC
Update - We're seeing issues related to notifications.
Nov 13, 21:15 UTC
Investigating - We are currently investigating this issue.
Nov 13, 21:13 UTC
Nov 12, 2023

No incidents reported.

Nov 11, 2023
Incident with Pages, Webhooks and Actions
Resolved - On November 11, 2023, at 1:00 UTC, GitHub background jobs encountered delays lasting up to 50 minutes. This delay affected various services utilizing background jobs, including Actions, Webhooks, Pull Requests, and Pages. The impact persisted for approximately one hour until 2:10 UTC.

During the incident, some customers experienced delays in starting Github Actions workflow runs and Pages builds. We estimate that about 10% of Actions workflow runs were delayed during the impact window and 99% of Pages builds failed from 1:00 UTC to 1:20 UTC. Users may have experienced a delay in seeing recent pushes reflected in pull request views. This delay averaged between 5 and 10 minutes and affected up to 30% of pull request page views during the incident. 1% of pull request page views experienced delays of up to 60 minutes. Finally, 30% of webhook deliveries in this window missed our target of being delivered within 1 minute of the triggering event.

This incident was caused by excessive rebalancing in our Kafka consumer group that feeds our background job system. We have altered our Kafka configuration to reduce the likelihood of this issue, created diagnostic tools to identify future causes, and will be breaking up this relay into multiple groups to limit the blast radius if the problem does reoccur.
Nov 11, 02:14 UTC
Update - Pages is operating normally.
Nov 11, 02:14 UTC
Update - Actions is operating normally.
Nov 11, 02:13 UTC
Update - Rebalancing completed and job queues are improving. We continue to monitor for full recovery of Webhooks, Actions, and Pages workflows.
Nov 11, 01:53 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Nov 11, 01:42 UTC
Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Nov 11, 01:41 UTC
Update - Pages builds, webhooks, and other workflows were delayed starting at 1:00 UTC. We have failed over the service that was contributing to the delays and see successful processing. We are continuing to monitor for full recovery
Nov 11, 01:40 UTC
Investigating - We are investigating reports of degraded performance for Pages
Nov 11, 01:26 UTC
Nov 10, 2023

No incidents reported.

Nov 9, 2023

No incidents reported.

Nov 8, 2023

No incidents reported.

Nov 7, 2023
Incident with API Requests and Issues
Resolved - Our internal search infrastructure experienced increased latency and timeouts between 13:15 and 14:05 UTC leading to some end user timeouts and slow responses for any requests that made use of that subsystem. This included but was not limited to: user search, repository search, releases, and audit logs.

We mitigated the issue by migrating traffic to an older version of our search clusters and are investigating what caused the performance issues in our new clusters.
Nov 7, 14:25 UTC
Update - API Requests is operating normally.
Nov 7, 14:25 UTC
Update - Response times stabilized back to normal at 13:58 UTC. We are continuing to monitor the slow dependency to ensure it's stable before resolving this incident.
Nov 7, 14:15 UTC
Update - We're seeing intermittent spikes in latency of API requests and page loads. We are investigating but do not have an ETA at this time.
Nov 7, 14:03 UTC
Update - We are investigating reports of issues with service(s): Issues, API Requests. We will continue to keep users updated on progress towards mitigation.
Nov 7, 13:50 UTC
Update - Issues is experiencing degraded performance. We are continuing to investigate.
Nov 7, 13:46 UTC
Investigating - We are investigating reports of degraded performance for API Requests
Nov 7, 13:44 UTC
Nov 6, 2023

No incidents reported.

← Incident History Powered by Atlassian Statuspage