Real Engineering 2: Solving data migrations

In this blog post we will walk through how you can implement performant and robust data migrations. There are existing solutions and one we use is at Atlassian is Apache airflow however at migrations we have chosen to build our own engine, executor and admin UI to move data providing us the most flexibility to meet the diverse needs of our customers.

Core components

• Executor or orchestrator
• Source where data is exported from usually a database or file storage
• Transformer where entity by entity it is transformed most common transformation are ids as they change between source and destination if you use incremental ids
• Sink where data is persisted usually to a database table

Binary vs database

Binary data is usually much simpler to move around as you don’t have to do any transformations and it’s easy to cache with hashes.

Why you can’t just export and import full DB

This would make our platform much simpler however the data model is slightly different from server and cloud products and we use auto incrementing ids not guids this means we need to remap the ids during migration. Customers don’t migrate everything at once these are large complex organisations that usually prefer to migrate in business units and they could also have mergers. So our platform has to meet this customer need and be more fine grained in the way data is migrated and actually done on an entity by entity basis like a Jira issue or jira comment.

Solving performance

Migrations still take hours because of the large amount of data. The obvious solution is to do delta or incremental migrations instead. Attempting to keep source and destinations in sync. This is a much harder problem instead of naively migrating everything as you need to detect what changed in source and persist it in destination. You need to remember to do deletes as well. Then you need to reconcile the data as you may have missed some data due to flaky network conditions.

Another performance optimisation is to export once because customers will do test migrations so same data maybe exported multiple times. This works well for binary data like images since they have a stable hash but database tables this doesn’t work well. We implemented a data cache in front of S3 where if the hash of an object about to be uploaded matches existing hash we return a reference to the existing S3 object. However we did not see much performance gains from this approach.

Solving reliability

Due to the scale of Atlassian there are thousands of Microservices it’s very difficult to guarantee reliability as any downstream Microservice fails our platform will fail. This is a problem since you don’t want to retry a migration that takes hours. One naive solution we have is to fail fast so recovery is faster. Another is simply to have robust retries like using an sqs queue but you need to use this with caution as the queue can easily build up. To solve that we have to add rate limits to avoid a large queue. It’s impossible to have 100% reliability so it’s important to have a lot of observability into failures when they do happen.

Following safe coding practices goes a long way to having a reliable service. Things like alerting, try catches, feature flags, extensive end to end tests, closing streams and avoid unbounded data fetching calls go a long way. If you are a new service chances are you are the failure point, mature services have gone through the pain of growth to become reliable.