Meta-rationality illustrated: software architecting

What is it, and why must it be done meta-rationally?

A random software architecture diagram (CC-SA-4.0)

This free post is the first section of a chapter illustrating meta-rationality with examples from software architecting.

It is a draft chunk of my meta-rationality book. If you are newly arrived, check the outline to get oriented. (You might want to begin at the beginning!) If you run into unfamiliar terms, consult the Glossary.

---

This chapter illustrates meta-rationality using software architecting as an example domain. It aims to increase your understanding of meta-rationality, not of software architecting. No background in software development is necessary to understand this chapter. I will explain the few technical concepts needed in general terms, which will suffice.

The chapter explains why this kind of work has to be meta-rational, how we do some of the meta-rational tasks it entails, and why attempts to architect rationally may fail. In sum: we design, build, and use software as rational systems with nebulous purposes in nebulous contexts that include other rational systems. Success demands a meta-rational overview understanding of the context and purpose, as well as detailed rational understanding of software technology.

This chapter’s explanation simplifies or passes over complex issues in the field. A chapter in Part Five aims to increase understanding of software development by applying meta-rationality, and treats the same material in greater detail, depth, and breadth.

I hope this chapter is concrete enough that at the end you think: 

Oh, so that’s what this “meta-rationality” stuff is! OK, now I get it.

I chose software architecting as a prototypical illustration of meta-rationality because:

• All the meta-rational themes show up in this domain. Meta-rationality is critical because architecting bridges the gap between the nebulosity of the real world context and the seeming formal rationality of software systems.

• Software architecting sounds boring and unmysterious. That is good because it’s hard to romanticize. Thousands of people do it, thousands of times per year. It appears to be relatively routine and straightforward and well-understood. Scientific discovery, by contrast, is often described as creative, intuitive, or requiring genius. These vague notions impede investigation of how we do it.

• I have done software architecting professionally, so I have a pretty good insider’s understanding, which I hope to communicate.

This post is the first chunk of a long chapter: the whole of Part Four in miniature. The chapter illustrates, in simplified practical terms, concepts that the rest of the Part treats more theoretically, in more detail, and in greater generality.

What is software architecting?

The architecture of a software system is its big-picture design: what the main parts are, what they do, how they relate to each other, and the technologies relied in each. Architecting is designing the architecture.

We’ll mostly discuss software complex enough to take a good-sized team to build and maintain it. Architecting such a system involves a series of decisions that commit a software development team for many years (the typical lifetime of a software system). These decisions are nearly irrevocable; it is difficult to change the architecture part way through building a system, much less after people start using it. If you get it wrong, you probably have to throw the whole thing away and start over. This can be an enormously expensive mistake. In extreme cases, badly architected software systems have cost billions of dollars, never worked, and were abandoned after a decade of failed effort.¹ More commonly, software systems are unreliable and hard to use, leak confidential data, cost much more to build and maintain than budgeted, and after a few years succumb to “software rot” and have to be completely replaced. Bad architecture is not the only cause of these woes, but it is a major one.

Architecting is a task, not necessarily a job. Depending on the size of a project and a company’s management practices, it can be the responsibility of different job roles. Some projects have a person specifically designated as the software architect, which might or might not be their full-time job, and might or might not be their job title.² There may be a team of several people whose full-time responsibility is architecting a single large project. On a small project, it may be no one’s assigned responsibility. In that case, it usually gets done by the most senior technical person on the project.

Most frequently, the designated architect is a technical manager, but architecting is not just (or even mainly) a technical activity, nor a people-management one. It requires meta-rational domain, business, and organizational understandings, which are skill sets most technical managers don’t need or use. For that reason, architecting is also often done in close collaboration with people in the marketing department, who may sometimes force critical architectural decisions despite limited technical understanding.

Why must software architecting be meta-rational?

Software architecting figures out what sort of software system to construct. How a system should work depends on what it should do. What a system feasibly can do depends on how it works. Skilled architecting finds a practical resolution to these interlocking dependencies.

• Figuring out what a system should do requires understanding its context and purpose. Those are inherently nebulous, meta-rational considerations.

• The extreme complexity of software systems makes them impossible to reason about entirely rationally, despite their perfect formality. Purely technical questions about what’s practical require meta-rational consideration as well.

Software architecting bridges many organizational roles. Each requires specialized rational expertise.

• Business analysis: To count as successful, software development projects must deliver more value than they cost. Value is often measured financially, but non-quantifiable benefits usually count too. Business analysis relies on rational methods for constructing financial projections, market strategy, and sales strategy.

• Requirements analysis: This means “finding out what people want software to do.” It relies on rational methods for business process documentation, market research, and user experience research. It also has its own specialized rational methods, which we’ll explore in some detail later.

• Technical management: Architecting must take into account the abilities and agendas of the available developers. The technically best architecture may be unworkable because the engineering staff is unable or unwilling to build it. Technical management includes both project management and line management, which are separate specific bodies of expertise.

• Software technology expertise: The architect must understand what is technically feasible—what designs are practical—leaving aside issues about the people doing the work. This requires orienting to the landscape of current software development practices, and to the capabilities of other software systems that the new system may depend on.

Ideally a software architect would be expert in all these fields, with full knowledge of all the relevant specifics. That is impossible, so you must rely on the fluid competence that comes with meta-rationality.

Each field of expertise involves nebulosity and unknown unknowns. In this chapter, we’ll look especially into “requirements.” 

• Requirements are purposes: what the system is meant to do. Unfortunately, the people who want the system—and will use or pay for it—usually have only the vaguest ideas about what they want. That makes requirements exceptionally nebulous. Requirements analysis is mostly figuring out specifically what tasks users would perform using a system. Unfortunately, mostly they can’t tell you even what they already do now—much less what they want the software to enable them to do in the future. Successful requirements analysis requires understanding some aspects of their work better than they do, and how it could be improved.

• A software system operates in a context that includes its users, but goes far beyond just them. It helps users get work done that often involves other non-user people, other software systems, non-software business processes, and non-human objects (for example, the eggplants tracked by a supermarket inventory system). People, institutional processes, and eggplants are nebulous.

I would love to hear what you think!

A main reason for posting drafts here is to get suggestions for improvement from interested readers.

I would welcome comments, at any level of detail. What is unclear? What is missing? What seems wrong, or dubious? How could I make it more relevant? More practical? More interesting? More fun?

1. Wikipedia has a “List of failed and overbudget custom software projects” article. Outstanding examples include the attempt to replace the US air traffic control system (project scrapped after twelve years and $3–6 billion spent) and the UK’s electronic healthcare records system (mostly scrapped after nine years and £12bn spent). Many different things have to go wrong to create disasters of this size, but failed system architecture is usually a root cause.

2. Many companies with software development teams have no “Software Architect” job title. Also, someone with that title may do no architecting. Sometimes it’s just an honorific for “programmer who has been around for a long time.”