A Plea for Lean Software[1] is a classical paper that presents us with some hints of why software increase in complexity and give us some advice on how to avoid or minimize complexity. Here I present my interpretation of Niklaus Wirth's ideas in a non linear way adding some personal reflections on top of it.
The amazing thing about this paper is that it was written in 1995, when programs where only a few KB or MB big. And Niklaus Wirth[2] was already clamming that software growth was getting out of control. It amazes me how accurate his observations still are even 25 years later. As an Engineering Manager, I see a lot of what he's talking about in my mundane activities and I also see myself acting in a lot of ways similar to what he points as crucial for software complexity growth.
This paper is famous for the following quote: "Software is getting slower more rapidly than hardware becomes faster". Wirth attributes it to Martin Reiser.
Wirth starts by pointing two factors for ever-growing software:
- Rapidly growing hardware performance
- Customers' ignorance of features that are essential-versus-nice to have
The first factor is very easy to grasp and makes total sense. But the second one I had a little of trouble understanding it. After some reflection, what I could come up from this claim is that the customers' lack of understanding of what brings value and what does not bring value provides software vendors the wrong kind of information on what to build. And that happens because the userbase is always making requests about the product that they claim are useful but not necessarily is (some of them may be). If the software vendors don't think critically on what the user is requesting, it will lead to software complexity. This behavior incentives a feature factory schema that don't necessary adds value to the customer. I'd would like to add that not only the customers' ignorance leads to complexity but also the software vendors' ignorance to recognize customers' ignorance and have a clear understanding of what adds value to customers' life. As a software vendor myself, only the fact of being aware of my ignorance and limitation may be a great start for reducing complexity or avoiding it to grow. But that requires a lot of humility.
Wirth's talks about what he calls monolithic design. You build a single system to assist a diverse userbase that pays for all features but only use a few or if I may add, each user segment uses a small set of features that is different from the other user segment. That leads to his recommendation that we should be building a thoughtful well designed system core that is extensible, which I think is indeed a good thing. However we all know that software development comes with some constraints. And greatest of all constraints is time. Wirth puts a lot of criticism on time pressure declaring that it discourages careful planning and improving acceptable solutions and on the other end, encourages quickly software additions and corrections.
But time may be in fact a real constraint and the pressure sometimes makes sense. The problem is that we don't know how to identify when it as a real constraint versus we're just making this up. I know that there are cases that the constraint is real, like developing a website for next World Cup. But in most occasions, the time constraint comes from managers' anxiety (as manager I recognize this), not well thought sprint timeboxes or simply because we have to. Since it may very hard to distinguish a real time constraint from a fantasy one, a good way of stop fooling ourselves it is to use time as an indicator of when we should be ready shipping a next increment of improvement. An that's agile. Unfortunately it very hard to understand what the next increment of improvement should be and people don't really understand the increment part. Wirth's also elaborates on the importance of iterative improvements, not as an escape from the time pressure as I have elaborated, but as an escape from the hardship that is building software:
Initial designs for sophisticated software applications are invariably complicated, even when developed by competent engineers. Truly good solutions emerge after iterative improvements or after redesigns that exploit new insights, and the most rewarding iterations are those that result in program simplifications
Another point made by Wirth is that complexity promotes customer dependence on the vendor. So there is an incentive to make things complex in order to create a dependency of the customer generating a more stable stream of income. In a way it makes a lot of sense. A good simple product in which the customer can solve all its problems by yourself without any sales or customer support might not lead to a great engagement. But I don't really think that this practice happens consciously though. Most of the time, I would guess it happens accidentally. Even because customer support and a sales team may not be very cheap or scalable. But I do recognize that some vendors may consider (consciously) complexity as a strategy for customer lock-in.
The paper goes on and details more practical technical advice on how to keep complexity low by presenting a practical case called project Oberon and it finishes with 9 lessons learned from this project. Those are very useful and it is worth checking out. I'd love to know more about software complexity and its root causes and I will probably research more about the topic. For now I'll finish with what I think is best phrase of the paper:
Good engineering is characterized by a gradual, step-wise refinement of products that yields increased performance under given constraints and with given resources.