GitHub header
All Systems Operational
Git Operations ? Operational
API Requests ? Operational
Webhooks ? Operational
Visit www.githubstatus.com for more information Operational
Issues ? Operational
Pull Requests ? Operational
Actions ? Operational
Packages ? Operational
Pages ? Operational
Codespaces ? Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Mar 27, 2024

No incidents reported today.

Mar 26, 2024

No incidents reported.

Mar 25, 2024

No incidents reported.

Mar 24, 2024

No incidents reported.

Mar 23, 2024

No incidents reported.

Mar 22, 2024

No incidents reported.

Mar 21, 2024

No incidents reported.

Mar 20, 2024

No incidents reported.

Mar 19, 2024

No incidents reported.

Mar 18, 2024

No incidents reported.

Mar 17, 2024

No incidents reported.

Mar 16, 2024

No incidents reported.

Mar 15, 2024
Incident with Actions and Pages
Resolved - This incident has the same root cause as this incident.. Please follow the link to view the incident summary.
Mar 15, 20:28 UTC
Update - Actions is operating normally.
Mar 15, 20:27 UTC
Update - Pages is experiencing degraded performance. We are continuing to investigate.
Mar 15, 20:09 UTC
Investigating - We are investigating reports of degraded performance for Actions
Mar 15, 20:07 UTC
Incident with Codespaces and API Requests
Resolved - On March 15, 2024, between 19:42 UTC and 20:24 UTC several services were degraded due to a regression in calling the permissions system.

New GitHub Codespaces could not be created, as were Codespaces sessions that required minting a new auth token.

Actions saw delays and infrastructure failures due to the upstream dependency on fetching tokens for the repository for runs to successfully execute.

GitHub Pages were affected due to the impact on Actions, resulting in 1266 page builds failing, which at the low point represented 33% of page builds failing. This resulted in page edits not being reflected on those impacted sites.

We deployed an application update that included a newer version of our database query builder. The new version uses a newer MySQL syntax for upsert queries that is not supported by the database proxy service we use for some of our production-environment database clusters. This incompatibility impacted the permissions cluster specifically, causing requests that attempted such queries to fail.

We responded by rolling back the deployment, restoring the previous query use, and thus mitigated the incident.

We have identified and corrected a misconfiguration of the permissions cluster in our development and CI environments that will ensure queries utilize the proxy service to prevent future syntax additions causing issues in production.
Mar 15, 20:24 UTC
Update - Codespaces is operating normally.
Mar 15, 20:21 UTC
Update - API Requests is operating normally.
Mar 15, 20:20 UTC
Update - We rolled back the most recent deployment and are seeing improvements across all services, and will continue to monitor for additional impact.
Mar 15, 20:17 UTC
Update - API Requests is experiencing degraded performance. We are continuing to investigate.
Mar 15, 20:11 UTC
Update - Codespaces is experiencing degraded availability. We are continuing to investigate.
Mar 15, 20:03 UTC
Update - API Requests is experiencing degraded availability. We are continuing to investigate.
Mar 15, 20:03 UTC
Update - API Requests is experiencing degraded performance. We are continuing to investigate.
Mar 15, 20:00 UTC
Investigating - We are investigating reports of degraded performance for Codespaces
Mar 15, 19:55 UTC
Mar 14, 2024

No incidents reported.

Mar 13, 2024
Incident with Pull Requests
Resolved - From March 12, 2024 23:39 UTC to March 13, 2024 1:58 UTC, some Pull Requests updates were delayed and did not reflect the latest code that had been pushed. On average, 20% of Pull Requests page loads were out of sync and up to 30% of Pull Requests were impacted at peak. An internal component of our job queueing system was incorrectly handling invalid messages, resulting in stalled processing.

We mitigated the incident by shipping a fix to handle the edge case gracefully and allow processing to continue.

Once the fix was deployed at 1:47 UTC, our systems fully caught up with pending background jobs at 1:58 UTC.

We’re working to improve resiliency to invalid messages in our system to prevent future delays for these pull request updates. We are also reviewing our monitoring and observability to identify and remediate these types of failure cases faster.
Mar 13, 01:58 UTC
Update - Pull Requests is operating normally.
Mar 13, 01:58 UTC
Update - We believe we've found a mitigation and are currently monitoring systems for recovery.
Mar 13, 01:53 UTC
Update - We're continuing to investigate delays in PR updates. Next update in 30 minutes.
Mar 13, 01:18 UTC
Update - We're continuing to investigate an elevated number of pull requests that are out of sync on page load.
Mar 13, 00:47 UTC
Update - We're continuing to investigate an elevated number of pull requests that are out of sync on page load.
Mar 13, 00:12 UTC
Update - We're seeing an elevated number of pull requests that are out of sync on page load.
Mar 12, 23:39 UTC
Investigating - We are investigating reports of degraded performance for Pull Requests
Mar 12, 23:39 UTC
← Incident History Powered by Atlassian Statuspage