Yonatan Zunger’s Blog

This is another post about design. It’s about a principle which can apply fairly broadly; it could equally be about how to structure an API in a software system, or about how to handle a requirement in a business. Here it is in two flavors:

The software version

If your system’s dependency on another system cannot be expressed through a narrow, stable API, don’t depend on that external system — instead, reimplement it yourself.

The business version

If your business depends on some core function, and you care about the details of how the job is done, rather than just whether it’s done to some simple standard, don’t outsource that function. (i.e., FedEx needs to fly its own planes)

At first glance this may sound extreme; “reimplement it yourself” / “do it in-house” is a tall order for many things you may rely on. But in practice, this sort of decision can be life or death for your system. The reason is that, if you care about how a job is done in detail, you’re going to want to probe into it in depth; you’re going to want detailed controls over the individual steps of the task; you’re going to want to be involved in the day-to-day of the operations to make sure it’s done to your particular need. In terms of software, this means that you won’t be communicating with this system just via a narrow API like “write this data to a file;” you’ll be using complex API’s, bypassing API’s altogether to get internal state, and so on.

As this progresses, you gradually move from using the system to being intimately involved in it, debugging it, and ultimately needing to modify it to your particular needs. But crucially, if you don’t control that system, you can’t do that.

Now, this doesn’t mean that you shouldn’t consider outsourcing the job at first, and moving to in-house when your need to mess with the details grows. But if you’re going to do that, you need to recognize that the design constraints of working with this external system are going to shape your own design from the get-go, and even once you go in-house, the legacy of those decisions will be with you forever. If you are confident enough in the API that you believe that these design choices will be correct even afterwards, and that the changes you’ll make as you go in-house will simply be extensions to that initial relationship, great;  but if you suspect that your needs are going to end up being fundamentally different from the external system, you may want to bite the bullet and do it yourself from the get-go.

There’s an obvious risk in doing this, of course; it’s more expensive, takes more time and money, and doesn’t give you an immediate advantage over a competitor who outsources. But this risk can pay off if you know that you’re going to hit that transition point reasonably soon — that way, a competitor who built around the wrong outsourcing is suddenly going to find themselves in need of a massive redesign, while you’re revealing wonderful new features to the world.

While reviewing some code today, a principle of software design somehow distilled itself to clarity in my head.

When designing your system, think of every major system¹ upon which your own system directly depends² as a bug.

By “think of it as a bug,” I mean that sooner or later, you are going to come to truly hate this dependency. It won’t do what you want, or it will turn old and crufty, or it will get outdated, or your system will outgrow it. Perhaps it already stinks. And therefore, think about what you are going to have to do to take it out and replace it with something better, and possibly not even having similar API’s.
Yes, you should have your code sufficiently factored and modular that such a replacement will be minimally invasive. But more importantly: if that replacement requires any change in the API’s³ by which the outside world is using your system, then there is something wrong with your design. Stop and fix that immediately.

¹Both external dependencies and major subsystems of your own code. Both will suck in time, I promise you.
²If the systems upon which you directly depend have done this properly, you don’t need to worry about your indirect dependencies. If they haven’t, then you should consider replacing them now, because you are obviously dealing with the work of madmen.
³Or UI’s, if your software is at the top of its software stack. UI’s are just API’s for communicating efficiently with humans. (Or perhaps API’s are just UI’s for communicating with computers?)