Series 3: Testing

Hard to test bugs

End to end tests are amazing and would encourage all teams to invest heavily in this area. However its difficult to cover every scenario and edge case. You may not want to do this either as it would be too expensive to implement and maintain for your team. Some example bugs which we have seen in practice are unexpected responses and status codes from remote calls. An example of this is that we have retries with exponential backoff for our API calls but only for 5xx and 429s. However we started returning 404s from our API due a race condition which can be resolve from retries. This was a bug in both the endpoint and status code being returned from that endpoint causing a migration failure.

Another example is not covering customer scenarios, there are a huge number of ways you could use an enterprise application and if you don’t encode these rules in your test suite regressions can easily happen. Developers tend to use the product in a certain way and those paths naturally end up getting well tested. But customers will use it in a different way often with much more data which is rarely covered.

I will leave you with one more class of bugs that are missed we have already seen race conditions, transient problems from remote calls (this includes db calls) so you should pay extra attention in these areas of the code. That is storing state in memory instead of a persistent store. Depending on in memory stores and caches for mission critical data is disastrous for long running jobs like migrations. It should be fault tolerant to server restarts and app version updates. This can be done by using a persistent store like the disk or a database. I have seen many times a server restart put an app in a weird state which would require manual user intervension to get it out of its stuck state.