Quantix Foundation - Git operations for users in the west coast are experiencing an increase in latency – Incident details

All systems operational

Git operations for users in the west coast are experiencing an increase in latency

Resolved
Degraded performance
Started about 2 months agoLasted about 8 hours
Updates
  • Resolved
    Resolved

    On March 19, 2026 between 16:10 UTC and 00:05 UTC (March 20), Git operations (clone, fetch, push) from the US west coast experienced elevated latency and degraded throughput. Users reported clone speeds dropping from typical speeds to under 1 MiB/s in extreme cases. The root cause was network transport link saturation at our Seattle edge site, where a fiber cut affecting our backbone transport resulted in saturation and packet loss. We had a planned scale-up in progress for the site that was accelerated to resolve the backbone capacity pressure. We also brought online additional edge capacity in a cloud region and redirected some users there. Current scale with the upgraded network capacity is sufficient to prevent reoccurrence, as we upgraded from 800Gbps to 3.2Tbps total capacity on this path. We will continue to monitor network health and respond to any further issues.

  • Update
    Update

    We have reached stability with git operations through our changes deployed today.

  • Update
    Update

    We are seeing early signs of improvement. We are working on one more small change to further improve traffic routing on the west coast.

  • Update
    Update

    We have completed the rollout of our new network path and are monitoring its impact.

  • Update
    Update

    We are beginning the rollout of our new network path. During this change, users will continue to see higher latency from the west coast. We will provide another update when the rollout is complete.

  • Update
    Update

    We are working to enable a new network path in the west coast to reduce load and will monitor the impact on latency for Git Operations

  • Update
    Update

    We are still seeing elevated latency for Git operations in the west coast and are continuing to investigate

  • Update
    Update

    We are redirecting traffic back to our Seattle region and customers should see a decrease in latency for Git operations

  • Investigating
    Investigating

    We are investigating reports of degraded performance for Git Operations