Data migrations at Atlassian

Introduction

I work on a team of over 15 engineers focusing on a specific part of the data migration platform at Atlassian, which is part of a business unit of around 200 engineers building and maintaining the migrations platform. The core mission of the department is to meet the data movement needs of our customers. You may wonder why we need so many engineers to achieve this — couldn't we use some existing ETL tool? There are many forms of data movement, such as movement to isolated cloud, self-hosted to cloud, and mergers and acquisitions, which is why it can end up being so complicated. You can dive into some technical details of the platform in Atlassian's post on Lithium — this is the technology that enables us to migrate data entity by entity, like Jira issues and Jira comments. The platform has many more components and dozens of services to support it, like API gateways, an ID mapping service, media APIs, and an orchestration engine.

What makes it interesting

You get a large breadth across Atlassian, working with teams in Confluence, Jira and networks. If you are in the third-party apps team, you additionally get to work with external teams who build on top of Atlassian's public APIs to provide extra features to customers, like test management or diagrams. There is a constant tension, however, as third-party apps may be concerned that Atlassian will just build their app into the product. It's not that easy for Atlassian to just make a product — it needs support from the CEO or CTO, otherwise it won't get enough buy-in. Most likely Atlassian will acquire them instead, like ProForma.

Hard technical problems

Each customer has large amounts — think terabytes — of precious data they want to move out of their servers and into Atlassian's. Doing this in a performant and reliable way is challenging because long-running tasks will almost always fail: servers could crash, network requests could fail, and data may get duplicated due to request retries or multi-node instances doing duplicate processing.

It's business critical

It's the engine that drives revenue, so there is huge investment in this area. The reason it drives revenue is that when customers move from datacenter to cloud, we can 2x our revenue from the same customer. Atlassian already has most of the largest companies paying, so it's an easy way to increase revenue. This makes migrations business critical, and most services are classified as tier 1 — they need to meet certain criteria for security, support and customer data protection.

How much data is moved across in our platform

The volume of migrations is not high — around a few hundred a week — however each migration is very large, with terabytes of data and billions of entities to migrate. By number of users, it's actually still split around 50/50 between server and cloud, so we still have a lot of work to do, and our largest and most complex customers are yet to come, like Apple and Uber.

What are the core technologies of the platform

Atlassian uses a lot of open source software — the key ones are Spring Boot, Kafka, TiDB and React. These are all fairly low-level frameworks, and Atlassian builds its own abstractions and frameworks on top of them.

Should you join migrations

If you have a job offer for this area at Atlassian, you will have this question for me. I think you have to be prepared that it has a busy oncall, and most projects have a heavy testing component — it's difficult to actually do a manual test, and you will spend a lot of time on that. This means there are a lot of opportunities to have a big impact at Atlassian, but it will require a bit of effort from you.