Following the routing change on August 2, we have completed two full weekday verification windows, the periods in which the degradation previously recurred. External download performance remained normal throughout both, and the upstream providers have confirmed via reverse path traces that return traffic to our European DC now takes the intended route. We are marking this incident resolved. We thank customers for their patience during this incident.
Resolved
Following the routing change on August 2, we have completed two full weekday verification windows, the periods in which the degradation previously recurred. External download performance remained normal throughout both, and the upstream providers have confirmed via reverse path traces that return traffic to our European DC now takes the intended route. We are marking this incident resolved. We thank customers for their patience during this incident.
Investigating
Maintenance complete — monitoring
The scheduled routing change at our European DC completed successfully at 00:12 UTC, ahead of schedule. Observed impact was a momentary interruption on the order of seconds during route convergence; download performance remained normal throughout and after the window. Because the original degradation followed a weeknight pattern, we will continue close monitoring through early next week before marking this resolved, and will post a final update then.
Investigating
We're continuing to work the degraded connectivity between our European DC and some external services (notably PyPI and parts of Cloudflare). Our monitoring showed the impact followed a nightly pattern on recent weeknights, with large downloads slowing severely or stalling roughly between 23:00 and 07:00 UTC. Overnight into Saturday, that window stayed clean: downloads completed at normal speeds throughout. Workloads should currently see normal external download performance. Because the improvement is recent and the pattern has been intermittent, we are not yet treating this as resolved and are continuing to monitor closely, particularly into the new week.
Working with the datacenter's network team and the upstream providers, the investigation has isolated the cause to return traffic from some external networks taking a congested transit path rather than the preferred one. To correct this, the datacenter's network team will perform a routing change during a scheduled maintenance window on Sunday 2026-08-02, 00:00 to 01:00 UTC (Saturday 5:00 to 6:00 PM Pacific). The change covers all address families and includes automatic rollback. Existing connections to external endpoints may be briefly interrupted at points during the window, and we recommend avoiding sensitive long-running external transfers during this hour. GPU compute and internal cluster networking (including the compute fabric) are not affected. We will confirm results here after the window.
Investigating
We're continuing to investigate intermittent slowness on connectivity between our European DC and some external services, notably PyPI and parts of Cloudflare. Overnight monitoring shows the impact was more pronounced than previously reported: during the late evening (UTC) window, many large downloads from affected services ran at heavily reduced speeds or stalled for extended periods, with intermittent windows of normal throughput. Traffic to other destinations remains unaffected.
The degradation is confirmed to sit within an upstream transit network outside our DC, and severity does not track business-hours load, so this is not simple peak congestion. The transit provider has acknowledged the issue and is actively investigating, and we are engaged directly with the affected CDN operators in parallel. For time-sensitive workloads we recommend caching or mirroring frequently pulled packages until the upstream issue is resolved. We will post further updates as the investigation progresses.
Investigating
We're continuing to investigate intermittent slowness on connectivity between our European DC and some external services, notably PyPI and parts of Cloudflare. Traffic to other destinations remains unaffected. This is consistent with the report we posted on Friday.
Testing over the past 24 hours shows that impact fluctuates with time-of-day load on the upstream network, and the underlying condition has not fully cleared. We are actively engaged with our upstream network partners on a resolution.
Investigating
We are investigating degraded download speeds from external sources affecting clusters in Europe. Customers may see slow or stalling downloads from package registries and CDN-served endpoints, including PyPI, GitHub, and content served via Cloudflare and Fastly. The impact is intermittent: some transfers complete at full speed while others slow significantly. Evidence points to intermittent congestion at an upstream transit provider, and the datacenter's network team is escalating with the provider. GPU compute and internal cluster networking (including the compute fabric) are unaffected.