Real Engineering Series 1: Introduction to data migrations at Atlassian

Introduction

Hey I am David, you haven’t heard from me for some time since I have been busy with my day job for the last four years.

I work on a team of over 15 engineers focusing on a specific part of the data migration platform at Atlassian which is then part of a business unit of around 200 engineers building and maintaining the migrations platform. The core mission of the department is to meet the data movements needs of our customers. You may wonder why we need so many engineers to achieve this, couldn’t we use some exisiting ETL tool. There are many forms of data movement such as movement to isolated cloud, self-hosted to cloud, mergers and acquisitions which is why it can end up being so complicated. You can dive into some technical details of the platform here https://www.atlassian.com/blog/atlassian-engineering/lithium this is the technology that enables us to migrate data entity by entity like jira issue and jira comments. The platform has many more components and dozens of services to support it like API gateways, id mapping service, media APIs, orchestration engine

What makes it interesting

You get a large breadth over Atlassian working across teams in Confluence, Jira and networks. If you are in the third party apps team you get to additionally work with external teams who build on top of Atlassian public APIs to provide additional features to customers like test management or diagrams. There is a constant tension however as third party apps maybe concerned Atlassian will just build their app in the product. It’s not that easy for Atlassian to just make a product it needs support from CEO or CTO otherwise you won’t get enough buyin. Most likely Atlassian will acquire them instead like proforma.

Hard technical problems

Each customer has large amounts like terabytes of precious data they want to move out of their servers and into Atlassian’s. To do this in a performant and reliable way is challenging because long running tasks will almost always fails, server could crash, network requests could fail and data may get duplicated due to request retries or multi node instances doing duplicate processing.

Its business critical

It’s the engine that drives revenue so there is huge investment in that area. The reason it drives revenue is because when customers move from datacenter to cloud we can 2x our revenue from the same customer. Atlassian already has most of the largest companies paying so it’s an easy way to increase revenue. This makes migrations business critical and most services are classified as tier 1 they need to meet certain criteria for security, support and customer data protection.

How much data is moved across in our platform

The volume of migrations is not high around a few hundred a week however each migration is very large. There will be terabytes of data and billions of entities to migrate. The number of users it’s actually still split around 50/50 from server and cloud so we still have a lot of work to do and our largest and most complex customers are yet to come like Apple and Uber.

What are the core technologies of the platform

Atlassian uses a lot of open source software the key ones are, Spring boot, Kafka, TiDB, React. All fairly low level frameworks and Atlassian will build their own abstractions and frameworks on top of these.

Should you join migrations

If you have a job offer for this area in Atlassian you will have this question for me. I think you will have to be prepared that it has a busy oncall and most project have a heavy testing component to it. In that it’s difficult to actually do a manual test and you will spend a lot of time on that. This means there is a lot of opportunities to have a big impact on Atlassian but it would require a bit of effort from you.