Understanding software purposes meta-rationally
“This is obviously impossible, which is why rational requirements analysis doesn't attempt it.”
Here we illustrate the themes of the previous section, on the meta-rational approach to purposes in general, with software development specifics.
Most or all of the points in this section also apply to any large rational project, such as a scientific research program or organizational management. You can ignore any unfamiliar software details without missing much, and supply equivalents from your own area of expertise.
This is a draft chunk from my meta-rationality book. If you are newly arrived, check the outline to get oriented. (You might want to begin at the beginning, instead of here! And at minimum, it would be good to have read the previous two sections.) If you run into unfamiliar terms, consult the Glossary.
The best explanation of rational project purposes I know is a blog post by Einar W. Høst, “Into the Tar Pit.” He summarizes the issues thus:
Where did the problem [i.e., purpose] come from? Who defined the problem? How is the problem articulated and communicated, by whom, to whom? Is there agreement on what the problem is? How many interpretations and formulations of the problem are there? Why this problem and not some other problem? Who are affected by this problem? Who has an interest in it? Who owns it? Why does the problem matter? Who determined that it was a problem worth solving? Why does it need to be solved? How badly does it need to be solved? Is time relevant? Has it always been a problem? How long has this problem or similar problems existed? Could it cease to be a problem? What happens if it isn’t solved? Is a partial solution viable? How does the problem relate to other problems, or to solutions to other problems? How often does the problem change? What does it mean for the problem to change? Will it still need solving? What forces in the real world could potentially lead to changes? How radical can we expect such changes to be?
… The curse of software development is that we can never fully answer all of these questions, yet they are crucial to our enterprise!
These meta-rational questions are good to ask about all rational project purposes, not just in software.
Office work is a mess
In software development, extensive meta-rational work is useful only when it’s not obvious what to build. Often it is clear what’s needed: when the activity the program supports is either quite simple, or already thoroughly understood. Then you can proceed mainly rationally; you have a technical Problem and need only routine engineering to produce a Solution. However, it is usually not obvious what an “enterprise system” (software that supports office work across multiple departments) needs to do; and it takes meta-rationality to find out.
A company is a mess, always: an assemblage of anomalies in a misty marsh of nebulosity. Its operation mostly cannot be understood rationally. Any rational representation of its processes, of its purposes, of the work done in it, must be inaccurate—often to the point of fantasy. Nevertheless, it is necessary to make such representations in the course of building enterprise software. In doing so, one must bear in mind the gap between representation and reality, and remain curious about what you may be overlooking or distorting, and what the consequences of that may be.
As with all meta-rational activity, there can be no definite method for understanding purposes. However, as is usually true of more specific, oft-repeated meta-rational activities, there is much to say about the particular type of task. That is: understanding purposes for software, in this case.
Understanding software’s purposes cannot be separated from understanding the work it supports, in its context of use. This should be obvious, but it is often neglected or overlooked entirely in rational requirements analysis. That may take stakeholder representatives’ requests literally and unquestioningly, without consideration of how they relate to broader purposes and their broader business context.
Understanding of software purposes can only emerge gradually from the mess, in part through an iterative process of creating preliminary, partial, candidate software designs, and opening them for critique by anyone who will be affected.
Understanding of a company’s work processes is distributed throughout the workforce. Understanding of the consequences of a company’s work is diffused among people affected. These understandings can never be fully centralized, so people in as many different roles as possible should be involved in figuring out what software needs to do and how.
The formulation of software purposes cannot be separated from preliminary understanding of architectural design decisions. Those are the general form of a Solution to the Problems (“requirements”). This contrasts with rational requirements analysis, which is often expected to complete before architectural work begins.
When these recommendations are violated, it’s common for requirements analysts to produce nonsense. Since their department may consider the job complete, redoing the work is likely to fall by default to the software architect. This is likely to cause interdepartmental conflict and to require difficult social negotiations—or the project will fail.
It is better for the architect to get involved in purpose discovery from the beginning, or nearly so. Requirements are inseparable from questions of feasibility and cost, which only engineering analysis can answer. A limited-edition Lamborghini may serve your purposes better than a Kia, but only if you ignore the price tag and the years on the waiting list.
Software purposes change frequently, as the business changes. Development must anticipate changing purposes with flexible development methods and with inherently flexible software. It must react to them with mid-course development process corrections, and with modifications or additions to the software itself.
Changes in software also drive changes in people’s work—for better or worse. Consequences should be considered throughout the process.
The rest of this section elaborates on these bullet points.
This is a part-paid post; the first half is free, and there is a paywall coming up at the midpoint. Please consider becoming a paying member, to support and encourage my work, and for full access to the paid posts! (Most are free, because mainly I care about sharing understanding.)
This is a long post, and some email programs may also cut it off part way through. In that case, you can read it on the web.
Nothing here is a radical new proposal. Software development is one of the disciplines in which meta-rationality is both most necessary and most common, so I didn’t need to invent this from scratch. Aspects of the approach I advocate have long been considered best practice by many academics, and by some industry pundits, under names such as “user-centered design,” “iterative design,” and “agile development.”
It’s not typically employed in industry, because it requires meta-rationality:
It is not understandable in rational terms, and so is difficult to quantitatively value or administrate, which is what merely rational managers want to do.
The meta-rational approach is much more work, so it is often considered too costly. However, getting an accurate understanding of purposes is enormously less expensive than a large failed software development project. “Measure twice, cut once!” A cost-vs.-risk analysis may strongly recommend it. In that case, senior management should mandate it, applying as much torque as required to make it happen.
Clarifying purposes introduces delay before “starting the real work,” meaning writing the program. (Figuring out what the program should do is not considered “real work.”) Delay may be unacceptable to managers who feel time pressure and want to cut corners.
Since it is not rational, it is often explicitly considered irrational woo by both technical people and upper management: just vague stuff made up based on feels rather than analysis.
The people best qualified to do it often have degrees in design or in social science (urk!), and may even be female. “We don’t have people like that, and we don’t know how to evaluate their performance, and what would we do with them after they are finished?”
What would you do with your programmers after they are “finished”? Serious software is never finished; it’s continually revised and extended as new purposes are found for it. But the mainstream culture of the industry supposes that requirements analysis should get finished as quickly as possible, and before programming gets started.
So taking the meta-rational approach, however ideal in theory, may be infeasible in practice: due to institutional rigidity and lack of understanding on the part of higher-ups. It sometimes winds up getting done on a covert guerilla basis, working around institutional restrictions and departmental boundaries, because the development project’s leaders realize it will fail otherwise.
Success may depend on adequate organizational support (and therefore has unavoidably political aspects), but it can fail no matter how much of that you get, too. Meta-rationality is not a magic bullet, no matter how good you are at it.
Purposes emerge gradually from nebulosity
Software purposes are inherently nebulous; but eventually must get formalized, because software is formal. This is what rational requirements analysis attempts. The meta-rational recommendation is not to not formalize, but to do it well. That implies:
Formalizing purposes only once you have an adequate informal understanding of them.
Formalizing only to the extent necessary to guide software construction, and only in ways that are actually helpful, avoiding rationality theater.
Maintaining ongoing awareness of points where your formalization fails to reflect some aspects of the nebulous purposes (as it always must).
Considering possible consequences of this distortion.
We can gain an informal understanding of purposes only gradually, and not through any systematic process. They emerge from improvisational exploration of the context.
At first there is only thick fog over the marsh. Plodding forward in no particular direction, we see dark vague shapes hulking in the distance. Plunging ahead, they come into view, still misty, as houses, a church, the village clock tower. We reach solid ground, kick the mud off our boots, and walk into the town square. The sun has come out, and we see the busy shops of the butcher, the baker, and the candlestick maker. We step into the bakery, buy a baguette, and chat with the proprietor. How does she keep her sourdough starter going? Where does she buy flour? What makes better or worse flour for baguettes? For croissants? (Mental note: visit the miller to get a sense of how he keeps track of how finely ground different customers want their sacks full.)
Software purposes are in part created through this process. They are not independently pre-existing objective entities.
When you talk to the warehouse manager, the forklift operator has convinced him that barcode scanners are a must, for logging arrivals of packages without having to type everything in.
You: Suppose we put a ton of cameras around the loading dock, we could actually log everything without anyone having to do anything; it could see each package as workers moved it.
Warehouse boss: Is that possible?
You: I’m not sure, but I think current tech is up to it. Would that be better than hand-scanning?
Boss: Well, yeah, as long as it’s reliable!
The purpose “track the movements of goods without human involvement” has emerged. Is that a discovery? Or as a co-creation of you, the forklift operator, the warehouse boss, and a hypothetically-feasible technology? Both. Neither. This is not a meaningful question, or a meaningful distinction.
For meta-rationality, recognizing that purposes are created, as much as discovered, is a prerequisite to radical innovation. It enables opening entirely new opportunities for progress, not just Solving known or obvious Problems.
We do not have a way of talking about how new things come about which recognizes that “discovery” and “creation” are both inadequate ways of describing what is often a single, unified process. Lacking any better term, I will sometimes use the awkward description “discovery/creation.”
What precisely is the purpose “track the movements of goods without human involvement”? That is, as yet, highly nebulous. It seems like a good idea, but endless questions remain. Which goods can feasibly get tracked by camera, and which (if any) can’t? Do we track them only as they come off the truck at the loading dock, or is it desirable and feasible and cost-effective to track further? Into warehouse storage? Into retail stores’ holding areas? Onto the shop shelves? Into customers’ carts? Are there privacy implications? Are those legal, regulatory, ethical, or practical matters of labor or customer relations? And so on, down to much finer details.
And what sort of thing is “track the movements of goods without human involvement”? Earlier, I said that for meta-rationality, purposes are “interactional opportunities immanent in broad context.” Does “track the movements of goods without human involvement” fit that description?
And how do we know whether it is a purpose? There can be no definite answer to that. With further investigation, our meta-rational judgment may be that it is, or isn’t. That judgment may depend on unenumerable factors. Certainly technical feasibility (at what cost? and in what time frame?) is one; how much labor it would save, and how much more accurate it would make inventory records, are two more; I mentioned privacy issues as a less obvious one, and doubtless many more would emerge with investigation. There is no Correct way of summing these considerations to make a rational determination.
Involve everyone affected, throughout the process
Initially, you don’t understand the context, because it’s a vast mess, most of which you have never encountered. Enterprise software purpose understanding involves characteristically meta-rational tasks, attitudes, and abilities that we’ve previously discussed more abstractly.
To get a broad sense of what factors are relevant, you need to develop a panoramic overview of the business context. Open-ended curiosity is the necessary stance.
What actually is the business overall? What is it for, what are the major moving parts, and how actually does it work? Often, before you do this work, no one in the business knows.1 (And, as in the study mentioned previously in “Why meta-rationality is invisible,” they might experience profound trauma if they found out.)
You need to stand outside and above the company as a whole. That is the standpoint for meta-rational inquiry and reflection. You will act on business systems, as objects; not within systems—the standpoint for rational work, and not at the margin of systems—the standpoint for circumrational work.
Corporations, despite theory, are never actually profit-maximizing engines, and that’s not the main goal of most people in them. They may have all sorts of other purposes: self-interested employee and social-group ones; plus positive external effects, even beyond providing customer value. In any case, companies mostly don’t operate rationally, despite any attempts by senior management to force them to do so.
You will, eventually, also need to understand how masses of intricate details of the work of software system’s users fit into to this big picture. (We’ll come back to that soon.)
You can never know most of the details. The panoramic view helps you judge which details you need to know and understand, and which are irrelevant to constructing your software. (Judging relevance by reflecting on unenumerable considerations is a characteristically meta-rational task.)
For meta-rationality, the holder of purpose is the entire context. To gain an adequate understanding, you need to collaborate with people in many different roles, who partially understand different parts of the mess. This is what “stakeholder requirements elicitation” tries to do. However, it wrongly imagines this is a finite process, once-and-done; and it’s usually rushed and half-baked; and it wrongly assumes people know their purposes and can tell you; and “stakeholder representatives” are often chosen for their organizational clout, rather than their understanding of the work the software will support.
The point of involving everyone significantly affected is not to be “nice” by being “inclusive” (although that may be a happy side-effect). Who should be included (“relevance of participants,” in the table) is a matter for meta-rational judgment. The point is not that everyone feels heard. It’s that everyone’s purposes (which they may not even know!) get adequately served. The point is to ensure a good outcome broadly; including financial returns for whoever is paying for the software—but not only that, either.
“Everyone involved” will have radically different expertises, with corresponding ontologies and technical vocabularies. Analysis of purpose needs to communicate with all of them effectively. This entails the typically meta-rational quality of ignoring boundaries between expertise domains. This may reveal broad properties of the organization that weren’t previously recognized by anyone.
You cannot take time to learn most of the expertise systems in the context, but with sorcery you may develop an adequate “pidgin in the trading zone.” That is, a good-enough subset of technical vocabulary that enables communication across domains. (I introduced this idea earlier, and will discuss it in more detail later.)
As much as possible, you need to keep your collaborators-in-understanding involved as long as possible, because purposes emerge only gradually, and continue to do so throughout the process. This is a central intuition in the original Agile Manifesto. One of its four maxims is “Customer collaboration over contract negotiation.” (In that context, “customer” means “the organization that is paying for the software.”) One of the twelve Agile principles is “Business people and developers must work together daily throughout the project.” These insights have mainly been ignored and actively negated, along with the rest of the Manifesto, by groups claiming to “do Agile.”
Software purpose understanding is inseparable from architectural decisions
Initially, we have only a vague idea of what purposes a proposed software project will serve. We also do not know what functionality is realistically feasible to build with available resources. Software development inevitably involves large technical uncertainties, because we mostly only write software to do things that have never been done before.2
These two unknowns—desirable purposes and feasible technical approaches—each constrain the other. This is a chicken-and-egg problem, which cannot be addressed by constructing a Problem and only then Solving it. The Solution—working software—forces details of the Problem, because not all possible purposes can be satisfied.
Purposes and architecture must be coaxed in tandem from nebulosity. The overall shape of the project gradually becomes clearer, more specific, more accurate, and more relevant both to user needs and architectural implications. This has to be an ongoing process of discovery/creation.
This is an instance of the meta-rational maxim:
Recognizing what general form the Solution must take often goes much of the way toward Solving a Problem. Recognizing the general form of the Problem often goes much of the way toward knowing the form of the Solution.
Design theory is relevant here, which is why I said the people best qualified for the task may have a background in that field. The distinction between engineering and design is that engineering starts with a relatively formal Problem, but in design you often don’t know what the Problem is until you are done. You proceed by constructing a series of design sketches that seem relevant in some more-or-less nebulous way, and figuring out quite what purposes each might satisfy.
This is the halfway point of the post!