This is another post about design. It’s about a principle which can apply fairly broadly; it could equally be about how to structure an API in a software system, or about how to handle a requirement in a business. Here it is in two flavors:
The software version
If your system’s dependency on another system cannot be expressed through a narrow, stable API, don’t depend on that external system — instead, reimplement it yourself.
The business version
If your business depends on some core function, and you care about the details of how the job is done, rather than just whether it’s done to some simple standard, don’t outsource that function. (i.e., FedEx needs to fly its own planes)
At first glance this may sound extreme; “reimplement it yourself” / “do it in-house” is a tall order for many things you may rely on. But in practice, this sort of decision can be life or death for your system. The reason is that, if you care about how a job is done in detail, you’re going to want to probe into it in depth; you’re going to want detailed controls over the individual steps of the task; you’re going to want to be involved in the day-to-day of the operations to make sure it’s done to your particular need. In terms of software, this means that you won’t be communicating with this system just via a narrow API like “write this data to a file;” you’ll be using complex API’s, bypassing API’s altogether to get internal state, and so on.
As this progresses, you gradually move from using the system to being intimately involved in it, debugging it, and ultimately needing to modify it to your particular needs. But crucially, if you don’t control that system, you can’t do that.
Now, this doesn’t mean that you shouldn’t consider outsourcing the job at first, and moving to in-house when your need to mess with the details grows. But if you’re going to do that, you need to recognize that the design constraints of working with this external system are going to shape your own design from the get-go, and even once you go in-house, the legacy of those decisions will be with you forever. If you are confident enough in the API that you believe that these design choices will be correct even afterwards, and that the changes you’ll make as you go in-house will simply be extensions to that initial relationship, great; but if you suspect that your needs are going to end up being fundamentally different from the external system, you may want to bite the bullet and do it yourself from the get-go.
There’s an obvious risk in doing this, of course; it’s more expensive, takes more time and money, and doesn’t give you an immediate advantage over a competitor who outsources. But this risk can pay off if you know that you’re going to hit that transition point reasonably soon — that way, a competitor who built around the wrong outsourcing is suddenly going to find themselves in need of a massive redesign, while you’re revealing wonderful new features to the world.
In software there can be an intermediate step of writing your own API which wraps a 3rd party API – this lets you design the interaction framework you need without actually having to implement all the details right away. I’m wonder what the business analogue is – perhaps a details spec sheet or certification requirements?
Something like that, or more generally an internal arms-length policy where you treat one business unit as being “as separate as possible” from the other units. It’s a good way to test if the two functions are really independent or if they need to be integrated with one another.
I think that most people would love to do this, but doing something *better* internally becomes a possibility only when you’re relatively big compared to other people who might be doing it. (In the FedEx example, flying your own planes only makes sense if you can afford an airline rather than three planes).
As a software engineering example from my job, we use embedded Linux at work. Linux is not a narrow API, nor is it a stable API when you have to build your own distro, constantly adapt it, write and rewrite kernel code/drivers, and maintain mutually incompatible releases according to which kernel your customers are capable of working with.
However, we can keep our Linux hacks up to date with a handful of engineers, less than it would take to develop a suitable OS.