Real Engineering Series 2: The challenges of our shift to cloud

Shifting from on-premise to the cloud moves data off the customer’s own infrastructure and onto the software provider’s, and with it come new and more complicated ways to store and process data.

Why customers move away from self-hosting

If you have ever tried to host your own software, like Jira Data Center or Tableau Server, you will deeply appreciate why it is worth paying extra for a cloud product.

You essentially have to become an SRE just to keep this one app alive for the business. Most days it runs smoothly, and then on some random day everything breaks and the server goes down. You scramble to fix it, but you don’t know the application well enough to understand why, so you escalate to the software provider. The whole thing can take hours, if not days, after which you never want to self-host again.

Then there is maintenance. Upgrading across major versions is not easy, but staying on an old version opens you up to security vulnerabilities, so either way you lose days to it. All of these costs motivated us to move off self-hosting and onto the cloud, pushing responsibility back to the makers of the product, who know best how to host it.

The trade-off

Like most things in software, it is not black and white. Handing control of your data to another party is scary. You are now completely dependent on them to do the right thing.

Practical problems

Some of the key practical concerns you need to work through:

• Is there appropriate data access and traceability in place, e.g. principle of least privilege?
• Data is not being logged
• Are there soft deletes and backups?
• Is your data sent off to another company and if they handle data correctly there e.g. analytics platform
• Do they have a security team?

Even after working through that list, you are still completely exposed to their engineers’ mistakes. A single bug in their authorisation code could let people read data they should not. That is very hard to protect against, so you need to carefully evaluate the quality of the engineering team. Some signs of a strong one:

• Extensive end-to-end test coverage
• They can debug issues quickly
• The product has good UI/UX and few bugs

Compliance problems

This is less practical in my opinion and less likely to have issues in practice. The main concern is if data is stored in a country where the government can easily request any data. I think the only major country that could happen is China. I think other countries like US they would still have to go through a process. Note however certain scenarios companies will automatically turn over data to the authorities if it clearly violates universal laws like CSAM.

Common compliance problems

• Data residency ie storing and processing data in the right region
• Data transiting wrong regions
• Data isolation ie not storing in the same database tables as other customers

The most intense requirements I have seen is from the US government where all data and services need to be hosted in US and access needs to be via us servers so workers need to Remote Desktop into one of those computers.

Isolated cloud environments

These are environments with dedicated infrastructure for a given customer. This means services and databases are never shared. This is a great data protection model as it’s very difficult to leak data between customers. Using our analogy before with incorrect auth the blast radius would be limited perhaps an employee has elevated privileges but on one outside the organisation would have access.

The way it works is it’s literally different hardware running the cloud service between customers. But also crucially the domains are different so data can’t easily transit out of the customers environment. So all the API endpoints of a customer in isolated cloud would be myowncloud.coolsaas.com vs coolsaas.com. This means the engineers maintaining the services cannot mistake the environments too since they will have different access points. A good way to think about it is isolated cloud is like staging and production the two are completed isolated and can’t call one another.

Pitfall of microservices

It’s very difficult for a Microservices architecture to meet these data requirements. There are thousands of Microservices and you need to shift them all to a specific environment is very difficult. That means getting hundreds of teams to adopt a framework that enables standardised deployments to new environments and fixing their services to work in a new environment requiring a lot of testing and maintenance. Common things that would break are hardcoded routes and handling services that don’t exist in environment. Some services must be across environments like marketplace which show available third party apps. Customers would want the same catalogue of apps they can use. Often the product is worse in the environments due to lack of support or missing features.