Goldratt’s Reliability & Effectiveness in B2C SaaS
There is a problem with Goldratt’s derivation of Reliability measure that when applied to B2C SaaS has disastrous consequences.
Reliability
The Reliability measure comes from Goldratt’s description¹ of two non-financial measures of performance: Reliability and Effectiveness. Reliability is defined as
Things that should have been done but were not
The end result that follows is
You do not fulfill the commitments to the external world
Which is then rephrased as
You don’t ship on time
And this becomes a measure of due date performance.
The above makes perfect sense in manufacturing. It also makes sense in software development where requirements are set and known ahead of time. These are run more like projects and the Critical Chain method applies. But this derivation becomes problematic in some contexts.
In a complex software development environment, the above line of reasoning is wrong. By complex software development environment, I mean that development is in a tight feedback loop of continuous discovery with customers where discovery and implementation happen simultaneously through customer interviews, design testing, prototypes, etc. B2C SaaS is often an example of such conditions. In such an environment, Goldratt’s original reasoning breaks down in the following way.
Things that should have been done but were not
No issues here. The end result that follows is
You do not fulfill the commitments to the external world
No issues here. This is then rephrased as
You don’t ship on time
False! This third step does not follow from the others in a complex software development environment context! We don’t know what we are shipping. This is a constraint that leads us to use methods like continuous discovery where we learn what to ship while shipping it. We cannot commit to something we don’t know. Only once we ship something the customer finds valuable, are we making a commitment to the external world. But in this context, time is not what we commit to.
Notice that in most software development we nevertheless estimate how long the unknown will take, we treat those estimates as commitments, and we go out of our way to try to improve our measurement of the perceived problem of not shipping on time. We even measure velocity because flow is important and we optimize software delivery according to these, without considering overall Throughput or build up of Inventory (as defined later).
In a complex software development environment we need a different rephrasing. Consider rephrasing “You do not fulfill the commitments to the external world” as
The service is unavailable
With the above rephrasing, the concept of Throughput Dollar Days (TDDs) can still be applied as a measure of Reliability. However, in the case we have here, TDDs accumulate not according to any lateness criteria. Instead, they accumulate due to unavailability of services. More specifically, TDDs accumulate according to the time a subscriber is unable to access a feature they are paying for via their subscription.
Effectiveness
Goldratt defines Effectiveness as
Things that were done that shouldn’t have been done
The end result that follows is
Inventory
In a complex software development environment, what should we consider Inventory? To come up with an answer, it is easier to reference the original definition of “things that were done that shouldn’t have been done”. These are unused features of the delivered software.
Measurement of which features are used by which subscriber is something that’s now possible in modern product metric systems. And this measurement can be our Inventory. An example may help here.
If subscriber Potato uses features A & B from a set of available features: A, B, C, D that Potato’s subscription has access to during a subscription term, then features C & D are unused inventory. Furthermore, a different subscriber Tomato uses features B & C from the same set of available features, then features A & D are unused inventory.
We can apply the usual Inventory Dollar Days (IDDs) to Effectiveness here, however we will quickly run into a problem. We will notice that IDDs grow unbounded. To address this problem, a better measure may be Inventory Dollar Days per subscription period (e.g., IDDs/month). This allows us to look at the rate instead of the cumulative sum. IDDs/month also retains the desirable property that it will grow if the number of subscribers (that don’t use features) grows.
Context Matters
I didn’t fully grok Goldratt’s methods until I read The Goal and Critical Chain. The main benefit was to see the Five Focusing Steps applied in different contexts. Seeing the manufacturing and project examples clarified what is generally applicable, and what is context specific.
Here, I hope I highlighted for you what is general and what is specific when it comes to Reliability and Effectiveness. The general methods remain useful. Understanding specific context, in this case B2C SaaS, is as important.
- Eliyahu Goldratt. Beyond the Goal: Eliyahu Goldratt Speaks on the Theory of Constraints (Your Coach in a Box). Chapter 4. Narrated by Eliyahu Goldratt, Gildan Media, LLC, 2005. Audiobook.
