GitHub header
All Systems Operational
Git Operations
Operational
API Requests
Operational
Webhooks
Operational
Visit www.githubstatus.com for more information Operational
Issues
Operational
Pull Requests
Operational
Actions
Operational
Packages
Operational
Pages
Operational
Codespaces
Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Oct 18, 2023

No incidents reported today.

Oct 17, 2023
Incident with Codespaces
Resolved - This incident has been resolved.
Oct 17, 13:49 UTC
Update - Codespaces is experiencing degraded performance. We are continuing to investigate.
Oct 17, 13:46 UTC
Update - We are continuing with efforts to mitigate Codespaces issues and are beginning to see some Codespace creations succeed.
Oct 17, 13:24 UTC
Update - We have identified an issue impacting most Codespaces operations and are working on a mitigation.
Oct 17, 12:18 UTC
Update - Codespaces is experiencing degraded availability. We are continuing to investigate.
Oct 17, 11:18 UTC
Investigating - We are investigating reports of degraded performance for Codespaces
Oct 17, 11:14 UTC
Oct 16, 2023

No incidents reported.

Oct 15, 2023

No incidents reported.

Oct 14, 2023

No incidents reported.

Oct 13, 2023

No incidents reported.

Oct 12, 2023

No incidents reported.

Oct 11, 2023

No incidents reported.

Oct 10, 2023

No incidents reported.

Oct 9, 2023
Incident with Pull Requests
Resolved - On October 9, 2023 at 14:24 UTC, a noticeable delay in commits appearing in pull requests was detected. During the incident, approximately 9% of pull requests (less than the 20% first reported) experienced staleness of up to 7m. The root cause was identified to be an increase in the latency of a downstream dependency causing pull request workers to saturate their available capacity, resulting in delayed updates to PRs - no data was lost during this incident.

We mitigated this by adding additional capacity to the affected worker pool at 15:02 UTC. This allowed our background jobs to catch up with the backlog of updates and provide relief to our customers. Additionally, we have significantly increased the performance of the downstream service to prevent recurrence
Oct 9, 15:18 UTC
Update - We are investigating delays for commits showing up on Pull Requests page loads in the web UI. As a result of this, about 20% of pull requests are currently showing stale data of up-to 7m. We are currently investigating contributing factors right now.
Oct 9, 14:52 UTC
Investigating - We are investigating reports of degraded performance for Pull Requests
Oct 9, 14:51 UTC
Oct 8, 2023

No incidents reported.

Oct 7, 2023

No incidents reported.

Oct 6, 2023
Incident with API Requests
Resolved - We deployed a new configuration to improve our network availability. This resulted in a small percentage of user traffic getting incorrectly blocked, but missed by our automated detections. We mitigated this with a change to the configuration, rolled out slowly over the last hour of this incident time for safe deployment. Beyond the learnings related to the config, we are analyzing how we can more quickly detect this kind of impact as part of future configuration rollouts.
Oct 6, 02:59 UTC
Update - We have confirmed that the fix has resolved the issue in the subset of regions where it has been deployed. We are now continuing the deployment to the remaining regions.
Oct 6, 02:42 UTC
Update - We are monitoring the rollout of the fix and are beginning to see signs of improvement. We will send another update shortly.
Oct 6, 02:03 UTC
Update - A small number of customers are experiencing 403 errors when attempting to access repository data via the API. We have found what we believe to be the cause of the issue and are deploying a fix.
Oct 6, 01:31 UTC
Investigating - We are investigating reports of degraded performance for API Requests
Oct 6, 01:12 UTC
Oct 5, 2023
Incident with Pull Requests
Resolved - On October 5, 2023 at 13:40 UTC, our monitoring systems observed an increase in the time it was taking for Git pushes to become visible when viewing commits in Pull Requests. Under normal operating conditions, a series of asynchronous jobs runs in response to every push and within a few seconds applies a number of side-effects in Pull Requests such as requesting reviews, marking Pull Requests as merged, and showing new commits. During the incident, jobs were entering the queue faster than we could process them, resulting in processing delays as high as 75 seconds on average, and as much as 15 minutes in the worst case. About 10% of all Pull Request page loads were showing out-of-date data during this time.

We had recently created a dedicated worker pool for processing these side-effects, with the goal of improving isolation between services and providing product teams with more direct control over critical parts of the system. We mitigated the incident by increasing capacity of the worker pool, which fully processed the backlog of delayed jobs, and returned everything to normal by 16:07 UTC. In response to this incident, we have adjusted our monitoring thresholds and improved our procedures for scaling up worker pools in response to increasing utilization.
Oct 5, 16:42 UTC
Update - We've implemented a fix for Pull Request web UI delays and are seeing recovery. We are monitoring and will send another update in a few minutes.
Oct 5, 16:12 UTC
Update - We are still investigating delays of up to 15 minutes in commits showing up on Pull Requests in the Web UI for all customers.
Oct 5, 16:01 UTC
Update - We are continuing to investigate the delay of commits showing up on Pull Requests in the Web UI. Pull Requests will still function normally otherwise.
Oct 5, 15:20 UTC
Update - We are investigating delays for commits showing up on Pull Requests page loads in the web UI. As a result of this, about 15% of pull requests are currently showing stale data. We are currently investigating contributing factors right now.
Oct 5, 14:34 UTC
Investigating - We are investigating reports of degraded performance for Pull Requests
Oct 5, 14:18 UTC
Oct 4, 2023

No incidents reported.

← Incident History Powered by Atlassian Statuspage