Resolved -
On August 14, 2024 between 23:02 UTC and 23:38 UTC, all GitHub services were inaccessible for all users.
This was due to a configuration change that impacted traffic routing within our database infrastructure, resulting in critical services unexpectedly losing database connectivity. There was no data loss or corruption during this incident.
We mitigated the incident by reverting the change and confirming restored connectivity to our databases. At 23:38 UTC, traffic resumed and all services recovered to full health. Out of an abundance of caution, we continued to monitor before resolving the incident at 00:30 UTC on August 15th, 2024.
We will provide more details as our investigation proceeds and will post additional updates in the coming days.
Aug 15, 00:30 UTC
Update -
We have fully rolled-back the changes to database infrastructure and mitigated the impact. All services are now fully operational.
Aug 15, 00:26 UTC
Update -
Git Operations is operating normally.
Aug 15, 00:25 UTC
Update -
Copilot is operating normally.
Aug 15, 00:21 UTC
Update -
Codespaces, Packages and Pages are operating normally.
Aug 15, 00:20 UTC
Update -
Webhooks is operating normally.
Aug 15, 00:19 UTC
Update -
Actions is operating normally.
Aug 15, 00:19 UTC
Update -
Actions is operating normally.
Aug 15, 00:19 UTC
Update -
Actions is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:19 UTC
Update -
Actions is operating normally.
Aug 15, 00:19 UTC
Update -
Pull Requests is operating normally.
Aug 15, 00:19 UTC
Update -
Issues is operating normally.
Aug 15, 00:18 UTC
Update -
Copilot is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:12 UTC
Update -
Git Operations is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:11 UTC
Update -
Codespaces is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:10 UTC
Update -
Webhooks is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:10 UTC
Update -
Actions is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:10 UTC
Update -
Pages is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:09 UTC
Update -
Service health continues to improve, and we are working to stabilize all services. Some services may experience delays in updates and notifications as we work through a backlog of events.
Aug 15, 00:09 UTC
Update -
Packages is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:09 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Aug 15, 00:09 UTC
Update -
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Aug 14, 23:50 UTC
Update -
The database infrastructure change is being rolled back. We are seeing improvements in service health and are monitoring for full recovery.
Aug 14, 23:45 UTC
Update -
We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.
Aug 14, 23:29 UTC
Update -
Codespaces is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:22 UTC
Update -
Webhooks is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:20 UTC
Update -
Issues is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:19 UTC
Update -
Git Operations is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:19 UTC
Update -
Packages is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:18 UTC
Update -
We are investigating reports of issues with GitHub.com and GitHub API. We will continue to keep users updated on progress towards mitigation.
Aug 14, 23:16 UTC
Update -
Copilot is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:13 UTC
Update -
Pages is experiencing degraded availability. We are continuing to investigate.
Aug 14, 23:12 UTC
Investigating -
We are investigating reports of degraded availability for Actions, Pages and Pull Requests
Aug 14, 23:11 UTC
