In this blog post we will walk through how you can implement performant and robust data migrations. There are existing orchestration solutions — Apache Airflow is widely used at Atlassian — but in migrations we chose to build our own engine, executor and admin UI, giving us the flexibility to meet the diverse needs of our customers.

Core components

• Executor (orchestrator) — decides which components run and when
• Source — where data is exported from, usually a database or file storage
• Transformer — transforms data entity by entity. The most common transformation is remapping ids, since they change between source and destination if you use incremental ids
• Sink — where data is persisted, usually to a database table

Binary vs database

Binary data is usually much simpler to move around: you don’t have to do any transformations, and it’s easy to cache with hashes, so you don’t need a complicated migration pipeline with event streaming. If you need to transform data, this is where things get complicated. The two key problems are dependencies and distributed systems. Dependencies force constraints on how you import data — if a parent entity fails, all of its child entities fail with it. Think of a Jira issue and its comments: you can’t import a comment before its issue exists. This means you have to import data in phases.

Why you can’t just export and import full DB

This would make our platform much simpler. However, the data model differs slightly between the server and cloud products, and we use auto-incrementing ids rather than GUIDs, so ids need to be remapped during migration. Customers also don’t migrate everything at once — these are large, complex organisations that usually prefer to migrate business unit by business unit, and mergers add to the mix. Our platform has to meet that need and be far more fine-grained about how data moves: migration happens on an entity-by-entity basis, like a single Jira issue or comment.

Solving performance

Delta vs point in time

Migrations still take hours because of the sheer amount of data. The obvious solution is delta (incremental) migrations instead — attempting to keep source and destination in sync. This is a much harder problem than naively migrating everything: you need to detect what changed at the source and persist it at the destination, remember to handle deletes, and then reconcile the data because flaky network conditions mean you may have missed some. I would recommend doing deltas from the start, because enterprise datasets are just too large and you will end up having to implement this anyway.

Export once

Another performance optimisation is to export once, because customers run test migrations, so the same data may be exported multiple times. This works well for binary data like images, since they have a stable hash, but it doesn’t work well for database tables. We implemented a data cache in front of S3: if the hash of an object about to be uploaded matches an existing hash, we return a reference to the existing S3 object. However, we did not see much performance gain from this approach.

Parallelism

This is a simple yet powerful optimisation. You can parallelise most things — the canonical example is running multiple S3 uploads at once. You can also choose a top-level entity to migrate in parallel, for example creating multiple Jira projects at a time. Getting the parallelism right is an art, because every environment is different, with different network bandwidth and hardware resources. This is why we make it a dynamic config with a default value — say 32 parallel uploads — and lower it if we are getting rate limited too much, or raise it if the migration is progressing too slowly. It’s a fine balance, and we’ve seen many incidents caused by parallelism — another point in favour of delta migrations, as you don’t have to stress the system to improve performance.

Solving reliability

Distributed systems

At Atlassian’s scale there are thousands of microservices, and it’s very difficult to guarantee reliability when any downstream microservice failing can fail our platform. This is a problem, since you don’t want to retry a migration that takes hours. One simple mitigation is to fail fast, so recovery is faster. Another is robust retries, for example via an SQS queue — but use this with caution, as the queue can easily build up; we add rate limits to keep it in check. It’s impossible to have 100% reliability, so it’s important to have deep observability into failures when they do happen.

Safe coding

Following safe coding practices goes a long way towards a reliable service: alerting, try/catches, feature flags, extensive end-to-end tests, closing streams and avoiding unbounded data-fetching calls. If you are a new service, chances are you are the failure point — mature services have gone through the pain of growth to become reliable. So for anything new, be it a service or a feature, expect reliability to be lower than you’d hope. Think of major Apple software version updates, or the outages Anthropic had when Claude Code first launched. As an industry we still have a lot to improve on reliable software — and it matters even more as AI writes more of our code, widening the gap between our software and our understanding of it.

Conclusion

Data migrations look simple from the outside — export here, import there — but at enterprise scale they mean confronting id remapping, entity dependencies, flaky distributed systems and datasets that never stop growing. The lessons that have served us best: migrate entity by entity, invest in delta migrations early, treat parallelism as a dial you tune rather than a constant, and assume failures will happen — then build the observability to understand them quickly. None of this is glamorous, but it is what turns a migration platform customers fear into one they trust. In the next post in this series we look at how we test a system like this.