Series 3: The non-technical side of engineering
Difficult oncall rotation
Distributed systems inherently will face transient problems and the likelihood of failure increases when you move large amounts of data through it like millions of files per migration. A failure is a huge problem when migrating data as it could cause a cascading problem failing the entire migration. Customers have to place their datacenter instance in read only mode to not miss any data so can only do it in a weekend. This means that if a migration fails they will have to wait at least another week and there is a lot of work to rollback the migration by deleting data in the destination cloud site. Migrations also take a very long time often days. Finally customers will do multiple rounds of test migrations before the production one and these migrations are treated just as important. This results in a terrible time for oncall particularly for the Australia team (large customers are usually in Europe or North America), the worst rotations will have multiple pages usually from support a day often at 12am-4am times and to debug and resolve the problem may take hours if it is a genuine platform problem.
Reactive instead of proactive
The majority of pages are done by support who directly communicate with customers instead of automated alerts. The reason being its very difficult to seperate noise from a genuine problem. A failed migration is a very generic error particularly for our team who provide a platform for third party apps to migrate their data. So its hard to tell where the fault really lies. Root cause analysis can take hours in large teams and its even worse when you depend on a team not even apart of your company. This doesn’t mean we can’t make effort to solve it but projects tend to be top down instead of bottom up. Executives don’t tend to prioritise projects on observability or UI/UX they like to target major platform changes. The consequence of being reactive is that a certain class of bugs are caught too late. We do have end to end test coverage and they catch many golden path related bugs.
No one knows what is happening
Probably the hardest part of enterprise software is that they have too many features that no one knows about. The original authors tend to swap teams or leave the company. Often there is no documentation and even if there was documentation it takes effort to understand a feature as its not just the code you have to understand but how it fits into the broader system. For example you want to add a new status for migrations. How would you know the downstream impact of this change. There are so many services that read status, you wouldn’t know the exact impact of each service that operated on the status field. The impact of not know anything is huge you see it play out it the wrong projects being created, large amount of regressions introduced, project timelines are missed and it compounds because an engineer isn’t likely to invest a large amount of effort to try to change the state and add a lot of tooling and documentation around it to make it super easy to onboard, its not their main job. You will find projects written in Atlassian and likely other bigtech have a much worse developer experience compared to open source projects.
What does all culminate into
It really wears someones mind out with all these problems faced I am not sure how well they will transition after the leave bigtech. Maybe they will be too burnt out maybe they would excel after have the chains removed.