IL-0549 Atlassian · 2023-05-15networking
Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removed
To improve security by hardening the partitions between network segments, Atlassian migrated an internal service to a new network segment, using blue/green deployment. A DNS record that pointed to the old network segment was not updated during the migration, and adequate verification steps were not run before the old stack was decommissioned. When the old network stack was removed, the service failed for Atlassian products hosted in us-west-1. Moving services across network segments is not a regular activity at Atlassian and is hard to reproduce in a test environment, so existing testing did not catch the problem.
Automated monitoring detected the failure within three minutes. Atlassian rolled the change back, which returned its systems to a known good state, and the disruption lasted one hour and 32 minutes, from 02:36 to 04:08 UTC.
- Severity
- –
- Scope
- Storing and retrieving media, attachments, Git LFS objects and build artifacts in Bitbucket, Confluence, Jira Align, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery and Atlas; Bitbucket Pipelines self-hosted builds failed and cloud-hosted builds were delayed
- Users affected
- Customers with services hosted in the us-west-1 region
- Region
- us-west-1
- Time to detect
- 3m
- Time to resolve
- 1h 32m
Symptoms
From 02:36 UTC on 15 May 2023, Atlassian products with services hosted in the us-west-1 region could not store or retrieve data assets such as media, attachments and build artifacts. Automated monitoring detected it within three minutes, and the status page reported intermittent errors.
In Bitbucket, self-hosted Bitbucket Pipelines builds failed, access to Git LFS failed and cloud-hosted builds were delayed. In Confluence, Jira Align, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery and Atlas, users could not upload, download or view media attachments or files, including images, videos, documents and audio.
Root cause
Root cause Configuration change
- Trigger
- An internal service was migrated to a new network segment as part of an initiative to harden the partitions between network segments, and the old network stack was then removed.
Detection
Automated monitoring detected the incident within three minutes.
Remediation
- Rolled back the network migration change, returning Atlassian systems to a known good state
Action items
- Review the systems that decommission service stacks and check that customer traffic is no longer served before a stack is decommissioned
- Add steps to the service network migration process that identify associated DNS records needing attention
Lessons
Atlassian said the migration used blue/green deployment to limit this kind of risk, but adequate verification was not run before the old stack was decommissioned, and moving services between network segments is rare and hard to replicate in test.
Sources
- https://bitbucket.status.atlassian.com/incidents/10ws2wytyfl2 · official-status · archived copy
Related cases
IL-0512xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedMIT Open Learning · 2026 · sev1same root cause · shared Authoritative DNS servers and records
IL-0502The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster2i2c · 2025same root cause · shared Authoritative DNS servers and records
IL-0533Names in chinacloudapp.cn and chinacloudsites.cn stopped resolving as DNS caches expired, breaking Azure China servicesMicrosoft Azure · 2024same root cause · shared Authoritative DNS servers and records
IL-0254Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hoursSquare · 2023same root cause · shared Authoritative DNS servers and records
IL-0363static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project · 2023same root cause · shared Authoritative DNS servers and records
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0549
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0549/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.