Get email notifications whenever GitHub creates, updates or resolves an incident.
Get text message notifications whenever GitHub creates or resolves an incident.
Get incident updates and maintenance status messages in Slack.
Subscribe via Slack
By subscribing you acknowledge our Privacy Policy. In addition, you agree to the Atlassian Cloud Terms of Service and acknowledge Atlassian's Privacy Policy.
Get webhook notifications whenever GitHub creates an incident, updates an incident, resolves an incident or changes a component status.
Visit our support site.
All Systems Operational
Git Operations
?
Operational
Webhooks
?
Operational
Visit www.githubstatus.com for more information
Operational
API Requests
?
Operational
Issues
?
Operational
Pull Requests
?
Operational
Actions
?
Operational
Packages
?
Operational
Pages
?
Operational
Codespaces
?
Operational
Copilot
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Aug 28, 2024
No incidents reported today.
Aug 27, 2024
Resolved -
This incident has been resolved.
Aug 27, 23:26 UTC
Aug 27, 23:26 UTC
Update -
We are no longer seeing any traffic going through the affected routes and this issue should be resolved.
Aug 27, 23:25 UTC
Aug 27, 23:25 UTC
Update -
All traffic should be rerouted by now; and we have seen a complete drain from the affected provider. We are performing final validations that networking traffic is back to normal.
Aug 27, 23:20 UTC
Aug 27, 23:20 UTC
Update -
We have drained connections that would be running through the affected routes and are waiting for caches to expire in order to validate that issues are resolved.
Aug 27, 23:03 UTC
Aug 27, 23:03 UTC
Update -
We have identified some potential connectivity issues among public Internet transit routes and are attempting to reroute.
Aug 27, 22:50 UTC
Aug 27, 22:50 UTC
Update -
Git Operations is experiencing degraded performance. We are continuing to investigate.
Aug 27, 22:46 UTC
Aug 27, 22:46 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Aug 27, 22:45 UTC
Aug 27, 22:45 UTC
Update -
We are experiencing some issues related to TLS/SSL encrypted connections and are currently investigating.
Aug 27, 22:38 UTC
Aug 27, 22:38 UTC
Investigating -
We are currently investigating this issue.
Aug 27, 22:37 UTC
Aug 27, 22:37 UTC
Aug 26, 2024
No incidents reported.
Aug 25, 2024
No incidents reported.
Aug 24, 2024
No incidents reported.
Aug 23, 2024
No incidents reported.
Aug 22, 2024
Resolved -
On August 22, 2024, between 16:10 UTC and 17:28 UTC, Actions experienced degraded performance leading to failed workflow runs. On average, 2.5% of workflow runs failed to start with the failure rate peaking at 6%. In addition we saw a 1% error rate for Actions API endpoints. This was due to an Actions service being deployed to faulty hardware that had an incorrect memory configuration, leading to significant performance degradation of those pods due to insufficient memory.
The impact was mitigated when the pods were evicted automatically and moved to healthy hosts. The faulty hardware was disabled to prevent a recurrence. We are improving our health checks to ensure that unhealthy hardware is consistently marked offline automatically. We are also improving our monitoring and deployment practices to reduce our time to detection and automated mitigation at the service layer for issues like this in the future.
Aug 22, 17:28 UTC
The impact was mitigated when the pods were evicted automatically and moved to healthy hosts. The faulty hardware was disabled to prevent a recurrence. We are improving our health checks to ensure that unhealthy hardware is consistently marked offline automatically. We are also improving our monitoring and deployment practices to reduce our time to detection and automated mitigation at the service layer for issues like this in the future.
Aug 22, 17:28 UTC
Update -
We are investigating issues with failed workflow runs due to internal errors. We are seeing signs of recovery and continuing to monitor the situation.
Aug 22, 17:21 UTC
Aug 22, 17:21 UTC
Investigating -
We are investigating reports of degraded performance for Actions
Aug 22, 16:49 UTC
Aug 22, 16:49 UTC
Aug 21, 2024
Resolved -
On August 21, 2024, between 13:48 UTC and 15:00 UTC, Actions experienced degraded performance, leading to delays in workflow runs. On average, 25% of workflow runs were delayed by 8 minutes. Less than 1% of workflow runs exhausted retries and failed to start. The issue stemmed from a backlog of Pull Request events which caused delays in Actions processing the event queues that trigger workflow runs.
We mitigated the incident by disabling the process that led to the sudden spike in Pull Request events. We are working to improve our monitoring and deployment practices to reduce our time to detection and mitigation of issues like this one in the future. We are also identifying appropriate changes to rate limits and reserved capacity to reduce the breadth of impact.
Aug 21, 15:11 UTC
We mitigated the incident by disabling the process that led to the sudden spike in Pull Request events. We are working to improve our monitoring and deployment practices to reduce our time to detection and mitigation of issues like this one in the future. We are also identifying appropriate changes to rate limits and reserved capacity to reduce the breadth of impact.
Aug 21, 15:11 UTC
Update -
We have seen recovery and Actions workflow runs are now running as expected.
Aug 21, 15:10 UTC
Aug 21, 15:10 UTC
Update -
We are seeing reduced delays for Actions workflow runs to get triggered. We are continuing to investigate how to further reduce impact on customers and recover more quickly.
Aug 21, 14:51 UTC
Aug 21, 14:51 UTC
Update -
We are investigating reports of users seeing delays for Actions workflow runs to get triggered.
Aug 21, 14:15 UTC
Aug 21, 14:15 UTC
Investigating -
We are investigating reports of degraded performance for Actions
Aug 21, 14:09 UTC
Aug 21, 14:09 UTC
Aug 20, 2024
No incidents reported.
Aug 19, 2024
No incidents reported.
Aug 18, 2024
No incidents reported.
Aug 17, 2024
No incidents reported.
Aug 16, 2024
No incidents reported.
Aug 15, 2024
Resolved -
On August 15, 2024, between 13:14 UTC and 13:43 UTC, the Actions service was degraded and resulted in failures to start new workflow runs for customers of github.com. On average, 10% of Actions workflow runs failed to start with the failure rate peaking at 15%. This was due to an infrastructure change that enabled a network proxy for requests between the Actions service and an internal API which caused requests to fail.
We mitigated the incident by rolling back the change. We are working to improve our monitoring and deployment practices to reduce our time to detection and mitigation of issues like this one in the future.
Aug 15, 13:59 UTC
We mitigated the incident by rolling back the change. We are working to improve our monitoring and deployment practices to reduce our time to detection and mitigation of issues like this one in the future.
Aug 15, 13:59 UTC
Update -
Approximately 10-15% of customers may be experiencing problems executing new GitHub Actions runs. The problem is currently being investigated by our teams.
Aug 15, 13:45 UTC
Aug 15, 13:45 UTC
Investigating -
We are investigating reports of degraded performance for Actions
Aug 15, 13:35 UTC
Aug 15, 13:35 UTC
Resolved -
On August 14, 2024 between 23:02 UTC and 23:38 UTC, all GitHub services on GitHub.com were inaccessible for all users.
This was due to a configuration change that impacted traffic routing within our database infrastructure, resulting in critical services unexpectedly losing database connectivity. There was no data loss or corruption during this incident.
At 22:59 UTC an erroneous configuration change rolled out to all GitHub.com databases that impacted the ability of the database to respond to health check pings from the routing service. As a result, the routing service could not detect healthy databases to route application traffic to. This led to widespread impact on GitHub.com starting at 23:02 UTC.
We mitigated the incident by reverting the change and confirming restored connectivity to our databases. At 23:38 UTC, traffic resumed and all services recovered to full health. Out of an abundance of caution, we continued to monitor before resolving the incident at 00:30 UTC on August 15th, 2024.
To prevent recurrence we are implementing additional guardrails in our database change management process. We are also prioritizing several repair items such as faster rollback functionality and more resilience to dependency failures.
Given the severity of this incident, follow-up items are the highest priority work for teams at this time.
Aug 15, 00:30 UTC
This was due to a configuration change that impacted traffic routing within our database infrastructure, resulting in critical services unexpectedly losing database connectivity. There was no data loss or corruption during this incident.
At 22:59 UTC an erroneous configuration change rolled out to all GitHub.com databases that impacted the ability of the database to respond to health check pings from the routing service. As a result, the routing service could not detect healthy databases to route application traffic to. This led to widespread impact on GitHub.com starting at 23:02 UTC.
We mitigated the incident by reverting the change and confirming restored connectivity to our databases. At 23:38 UTC, traffic resumed and all services recovered to full health. Out of an abundance of caution, we continued to monitor before resolving the incident at 00:30 UTC on August 15th, 2024.
To prevent recurrence we are implementing additional guardrails in our database change management process. We are also prioritizing several repair items such as faster rollback functionality and more resilience to dependency failures.
Given the severity of this incident, follow-up items are the highest priority work for teams at this time.
Aug 15, 00:30 UTC
Update -
We have fully rolled-back the changes to database infrastructure and mitigated the impact. All services are now fully operational.
Aug 15, 00:26 UTC
Aug 15, 00:26 UTC
Update -
Git Operations is operating normally.
Aug 15, 00:25 UTC
Aug 15, 00:25 UTC
Update -
Copilot is operating normally.
Aug 15, 00:21 UTC
Aug 15, 00:21 UTC
Update -
Codespaces, Packages and Pages are operating normally.
Aug 15, 00:20 UTC
Aug 15, 00:20 UTC
Update -
Webhooks is operating normally.
Aug 15, 00:19 UTC
Aug 15, 00:19 UTC
Update -
Actions is operating normally.
Aug 15, 00:19 UTC
Aug 15, 00:19 UTC
Update -
Actions is operating normally.
Aug 15, 00:19 UTC
Aug 15, 00:19 UTC
Update -
Actions is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:19 UTC
Aug 15, 00:19 UTC
Update -
Actions is operating normally.
Aug 15, 00:19 UTC
Aug 15, 00:19 UTC
Update -
Pull Requests is operating normally.
Aug 15, 00:19 UTC
Aug 15, 00:19 UTC
Update -
Issues is operating normally.
Aug 15, 00:18 UTC
Aug 15, 00:18 UTC
Update -
Copilot is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:12 UTC
Aug 15, 00:12 UTC
Update -
Git Operations is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:11 UTC
Aug 15, 00:11 UTC
Update -
Codespaces is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:10 UTC
Aug 15, 00:10 UTC
Update -
Webhooks is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:10 UTC
Aug 15, 00:10 UTC
Update -
Actions is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:10 UTC
Aug 15, 00:10 UTC
Update -
Pages is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:09 UTC
Aug 15, 00:09 UTC
Update -
Service health continues to improve, and we are working to stabilize all services. Some services may experience delays in updates and notifications as we work through a backlog of events.
Aug 15, 00:09 UTC
Aug 15, 00:09 UTC
Update -
Packages is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:09 UTC
Aug 15, 00:09 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:09 UTC
Aug 15, 00:09 UTC
Update -
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Aug 14, 23:50 UTC
Aug 14, 23:50 UTC
Update -
The database infrastructure change is being rolled back. We are seeing improvements in service health and are monitoring for full recovery.
Aug 14, 23:45 UTC
Aug 14, 23:45 UTC
Update -
We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.
Aug 14, 23:29 UTC
Aug 14, 23:29 UTC
Update -
Codespaces is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:22 UTC
Aug 14, 23:22 UTC
Update -
Webhooks is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:20 UTC
Aug 14, 23:20 UTC
Update -
Issues is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:19 UTC
Aug 14, 23:19 UTC
Update -
Git Operations is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:19 UTC
Aug 14, 23:19 UTC
Update -
Packages is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:18 UTC
Aug 14, 23:18 UTC
Update -
We are investigating reports of issues with GitHub.com and GitHub API. We will continue to keep users updated on progress towards mitigation.
Aug 14, 23:16 UTC
Aug 14, 23:16 UTC
Update -
Copilot is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:13 UTC
Aug 14, 23:13 UTC
Update -
Pages is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:12 UTC
Aug 14, 23:12 UTC
Investigating -
We are investigating reports of degraded availability for Actions, Pages and Pull Requests
Aug 14, 23:11 UTC
Aug 14, 23:11 UTC
Aug 14, 2024
