A Gentle Introduction to Robotics

Agile: Functional Decomposition, Cadence, and Exploration

What Is Agile?

When agile was first introduced, a great many software developers rejoiced. “We don’t need no stinkin’ project plans!” A great many software development managers also rejoiced, because agile was advertised as a silver bullet which management (and many agile advocates) took to mean “rapid and inexpensive.” This was unfortunate because agile, as commonly implemented through what are described as agile processes, is just about the opposite of both.

Agile per se is actually process-free. Agile is defined by the Agile Manifesto; the manifesto does not describe any process steps at all. “Agile” is commonly, and with some accuracy, viewed as a simple rejection of the supremely detailed, triple-checked, massively labor intensive, massively expensive, military weapons systems development contract style software development methods common at the time the manifesto was published. PERT and earned value analysis, you may recall, were invented for the Polaris missle submarine program, although admittedly Gantt charts were developed about 50 years prior.

Agile in the abstract is essentially a rediscovery of IBM’s “chief programmer team” idea. The chief programmer team idea was, basically: “find a world-class genius, give the world-class genius a regular genius companion or two and a few skilled but normal people gophers, give the bunch of them a goal, and get out of their way.” (Seriously. Look it up.) When you have even a regular genius, process is counter-productive. Regular genius software developers — the type that don’t know how to use a debugger because they’ve never needed one — typically have a productivity rate at least 10 times that of normal folk like me and you. World-class genius software developers, in contrast, simply breathe out major subsystems: they don’t have a productivity rate because you can’t divide by zero. World-class genius software developers can envision world-changing deliverables that pierce to the heart of a goal, explain those deliverables to the regular genius, and lead the troop into implementing them. The result is that a chief programmer team can turn your organization from a minor player into a dominant force. At least until the world-class genius gets bored with playing in your sandbox and wanders off somewhere else.

But the supply of world-class geniuses is somewhat limited. In the software world, there are on the order of 20 to 50 of them on the planet, and, if you think about it, you even know many of their names. Do those specific people work for you? Then the chief programmer team, or agile in the raw, is not available to you: you are left with process. Process gets something useful out of normal people — the ones you can actually find.

Fortunately, agile, even agile as commonly implemented through agile processes, in fact is a silver bullet. Unfortunately, what almost everyone developing software has is a vampire infestation, and silver bullets help only against werewolves. This section will, among other things, talk about what agile processes look like, what advantages they have to various audiences, when they should be sought out, and when they should be avoided like the plague.

So What Is An Agile Process?

There is, unfortunately, no Agile Process.

There is no Agile Process, but there are agile processes. Although there are exceptions, as a family, agile processes characteristically:

In this they contrast sharply with historical project management and control processes which characteristically:

In short, agile processes throw out the project planning paradigms that have been in successful use since the pyramids were built, and replace them with something entirely new.

Agile processes change everything. For good or for ill, nothing of prior ways of thinking and doing applies to agile processes.

Effective agile processes are tuned to do the absolute minimum of non-deliverable work. Anything unnecessary to producing working software has been omitted. An important, but not necessarily obvious, implication of this is that anything remaining in an effective agile process is necessary. Agile processes have been described, with an uncomfortable kernel of truth, as an assemblage of steps, each of which is broken, and each of which patches a break in a different step so the whole thing does not come crashing down. Agile processes are a system: an arrangement of components with emergent properties. The emergent property in this case is that the arrangement emits software. Missing or misplaced process pieces in an agile process destroy the emergent property of software emission, just as missing or misplaced pieces is an airplane destroy the emergent property of flight. Trying to pick and choose “just the really necessary few” of the activities that make up an effective agile process is like trying to pick and choose “just the really necessary few” of the girders that make up a bridge: all either produces is a pile of debris. The best you can hope for in either case is to avoid fatalities.

Although there are exceptions, what is commonly meant by an “agile process” is a way of adapting to ongoing discovery of organizational needs. The widespread and archetypical agile process, Scrum, has the overall outline:

There are a number of Scrum variants which differ from one another and from Scrum itself in the exact method by which each of these steps is performed, and possibly in the vocabulary used to describe the various process components and actors. In general, however, the vocabulary of Scrum is adhered to:

Any particular agile process will change some or possibly all of this vocabulary, spend a great deal of attention on exactly how one or another of the steps is to be performed, invent additional vocabulary to describe exactly how the particular agile process differs from other apparently identical agile processes, and possibly involve additional steps to those described above.

Peculiarities of Agile Processes

As mentioned before, agile processes violate almost all expectations we have formed about the way projects in general work. Agile process properties of particular note include:

These properties have several curious consequences. For example:

The issue of out of process acceptance criteria is an interesting one. Everyone gets one, or even two, bites at the apple; no one minds a little grift. But repeated instances of the product owner using out of process acceptance criteria will be identified by the developers as a problem. The engineering reaction to a problem is to solve it, definitively, and engineers have long attention spans: they routinely work on projects where the payoff is years away. As it happens, competence in gaming systems is significantly enhanced by designing systems for a living. Also, as it happens, most user stories are in fact terrible and the developers implement what the user stories intended to say and not what they actually said. Either eternal malicious compliance, or repeated dances where the product owner repeatedly modifies a narrative trying to force a particular solution and the product developers repeatedly find solutions to the narrative as stated that are demonstrably and obviously superior to what the developers know is envisioned, will occur; the repeated process derailments will attract management curiosity. Management will eventually realize fixing the repeated development disasters requires either replacing one product owner who is manifestly incapable of performing the major component of their job, writing an effective user story; or replacing an entire software development team that quickly, enthusiastically, competently, and fully implements the user stories presented to it. The outcome is predictable. And if not, screw it, non-H-1B visa engineers are perfectly aware they can find a new job in a week.

Business Benefits (and One Huge Drawback) of Agile Processes

This participation of the business in the sequencing decisions via the product owner is in fact the major difference between front-loaded processes such as waterfall and body-loaded processes such as agile. (If the business participates in sequencing decisions in waterfall, it becomes a rear-loaded process as everything that should be analysis, design, planning, and sequencing gets pushed off to repair and rework instead. This works about as well as you’d expect: the first 80% of the project gets done in the first 95% of the schedule and budget, and the remaining 175% of the project gets done in the remaining 250% of the schedule and budget. You hired experts, let them expert: quit trying to “help” the surgeon by shoving random members of the surgical team “out of the way” and getting rid of the anesthesiologist because “they’re neither cutting nor sewing and so not contributing to the deliverable.” The surgeon may be the star of the show, but the anesthesiologist is the person in charge of the patient still being alive at the end of the surgery. That is not a “deliverable,” true; but it is frequently regarded as a success criterion.)

Participation of the business in sequencing decisions has both business benefits and technical benefits. Short- and mid-term business benefits including quickly abandoning directions found to be fruitless, and very quickly — weeks rather than years — changing direction to accomodate changes in the environment, at least once those environmental changes are acknowledged. A major, unexpected, and welcome long-term business benefit is that fit and finish, in addition to function, become recognized deliverables. A house with just subfloors over joists, walls without drywall, raw joists for ceilings, with electrical switches hanging out of outlets nailed into visible studs and joists, and with windows and doors and plumbing rough fit but not finished fit, does everything we expect a house to do: it is functional. But we easily and quickly recognize we really don’t want to live there: it lacks fit and finish. One can sequence project tasks to get full functionality (“code complete”) about one-third of the way through the overall project cost and duration, but the project is not done at that point; recognizing this has very powerful long-term organizational benefits.

But likely the greatest business benefit of agile processes is almost accidental: agile development processes are financially predictable. More expensive than you can believe possible, but predictable. Software development is a skilled design activity, so the good-fast-cheap triangle has two ratio scales and one ordinal scale. The run rate (“cheap” axis) is constant: the team has a fixed size and that isn’t going to change without major disruption. The process is going to burp out something every sprint cycle and deployment of that something will cost about the same every time. The quality (“good” axis) is constant: the team has adopted its definition of “done” and works to that. If conformance to the definition of “done” starts to slip, management must intervene to get the process back on track: two ordinal scales in play characterizes research, not development, and doing research when you think you’re doing development is a solemn promise of an “Oh, the humanity!” moment later. (Incidentally, all three good-fast-cheap axes being ratio scales means you’re doing manufacturing, and you even get to set the value on each axis by capitalizing the process; all three being ordinal scales means you’re doing art.) With “good” and “cheap” nailed down, that leaves “fast” — duration — as the ordinal scale.

However, the primary business benefit of agile processes, financial predictability, is a direct result of the primary business disadvantage of agile processes: temporal unpredictability. The content of the something that gets burped out every sprint cycle is predictable only in sequence, not in location on the calendar. Time in agile processes is an ordinal scale: no numbers, just order.

Q: How many sprints until the feature is delivered?
A: All of them.

This is probably the single most surprising, and certainly most infuriating, attribute of agile processes: they are inherently unschedulable. Further, this is not a defect of the process: this is by necessity. Agile processes nail down predictability in run rate and predictability in quality level; they do this by completely sacrificing predictability in duration. This does not mean scheduling is difficult, or requires additional information, or needs to be fixed by working smarter not harder. With agile processes, scheduling is literally impossible.

The only schedule information available, or even possible, with agile processes is “already done” and “not yet done.”

But don’t underestimate the business benefit of financial predictability. It sure beats the alternative.

Technical Benefits of Agile Processes

There are two possible, but typically mutually exclusive, major technical benefits to the participation of the business in the sequencing decisions, and they have to do with refactoring. Typically the development team introduces “refactoring” tasks into the backlog to address what is called technical debt: situations where the conservative, minimum cost, minimum process risk steps taken to obtain the current state have resulted in exposure to excessive product risk. Refactoring is typically desirable even in front-loaded development processes, where it occurs as part of “version 2,” but is necessary on an ongoing basis given the “just in time” planning done in agile processes.

The optimum technical benefit arises when management lets the developers address these developed risks with improvements that typically leave product functionality unchanged but mitigate the developed exposures and avoid or remove the developed risks. As a rule of thumb, refactoring tasks should average 5% to 10% of available effort, and this effort is distinct from any “hardening” effort around major scheduled releases. If more effort than this is needed, something in the process is broken; if less effort than this is requested, what is broken is either the personnel on the project or (more likely) the metrics by which the personnel are measured. You get the behavior you reward: why are you rewarding building broken software?

A non-optimal but still substantial technical benefit arises when management does not let the refactoring tasks trickle into the sprints. This visibly demonstrates to the development team that long-term viability requirements such as adaptability and maintainability are being ignored because all those “refactoring” “user stories” the “prima donna programmers” put into the “product backlog” didn’t “progress” the “feature list” or “quiet the complaints” about how “hard to use” the software is. In this case, the form the technical benefit takes is that the technologists get to say “told ya so, ya dipstick” as they bail out of the doomed organization and go onto the next.

The greatest single cause of software abandonment is the software becoming too brittle to further modify and shattering against a changed environment. As one might expect, this wastes the entire development cost of the software as well as forcing cessation of whatever operations the software supported. Organizations to which this happens typically simply cease to exist, generally by being absorbed into some less ossified organization or sometimes by disappearing into a blizzard of lawsuits; or in some few cases radically change form and become much smaller. The process of restoring flexibility and resiliency to software is called refactoring, and yes it actually is the technologists trying desperately to save the organization from its own stupidity. Most organizations die of suicide via self-inflicted wounds, typically because they reward those who hide costs in the future and move on, and punish those who grit their teeth and deal with the pile of ordure left behind by the first group.

You get used to seeing this after a while. They call this recognition of human nature and organizational dynamics “cynicism,” and I highly recommend it.

Organizational Benefits of Agile Processes

The fundamental organizational needs software addresses are pretty universal. Exactly how these needs interact depends on whether the software is internal — supporting the organization’s own operations — or external — a product or service being provided to customers, clients, or other external parties. The needs themselves in these two cases are similar, but the emphasis on the needs differs between the two. In internal software development, the primary need is to effectively support operations, a secondary need is to be adaptable to a changing environment, and a tertiary need is to be maintainable. In product development, the primary need is to be profitable — which over the long term typically reduces either to the product being maintainable, or to moving the customer base from purchasing durable product to purchasing subscriptions to product — a secondary need is to visibly (not necessarily effectively) support the operations of the customer or client, and a tertiary need is to be adaptable to a changing environment. The distinctions made here are important. Microsoft Office, for example, is insanely profitable exactly because its customers are not its users: its customers are organizations, whose needs for data creation, communication, and consumption are visibly supported with shiny eye candy, while its users are the unfortunate souls trying to do their jobs while forcing a data-destroying, bug-ridden, user-hostile mass of misfeatures into some semblance of utility for the job at hand. Other office suites failed largely because they valued the user over the customer. Agile processes, taking place at the coal face as they do, permit software development effort to focus on actually supporting operations rather than pretending to support operations. Agile processes do not guarantee this focus — that depends on aligning managerial and organizational goals with operational goals, which is much easier to say than to do — but they manage to avoid preventing it.

For the few organization willing to acknowledge that their high-priced experts actually have expertise, agile processes ensure that at any point in time:

The current partial solution is delivered and potentially deployed at the agile cycle cadence of once every two or three weeks, or (in a continuous delivery model that drops the “sprint” concept in favor of anytime task starts, completions, and approvals) at the development build cadence of several times a day. The cycle of identifying, costing, scheduling, planning, and meeting needs, and of occasionally paying down technical debt when the carrying costs of that debt uncomfortably impede progress, never actually ends. Needs are never ending, so meeting needs is never ending. Carrying technical debt is income and expense statement money while paying it down is balance sheet money; those different moneys come out of different pockets (which is not usually an actual issue) with different owners (which always is an actual issue and typically an intractable one at that). Agile processes simply deal with the never ending nature of needs directly, rather than as a series of discrete projects.

Fundamentally, the organizational function of agile processes is to reframe software development as operations. After all, the primary difference between a project and operations is that a project has an end.

And this is why agile processes are so different from traditional project planning and management processes: agile processes do not actually do projects. A project has a fixed deliverable, which when delivered can be treated as a capital asset, which transforms the project development costs into capital investment. This is what capitalism does: it replaces running costs with capital investment. That is the definition of capitalism. Agile processes do the opposite: they replace capital investment with operational expense. Agile processes do not produce software durable assets; agile processes produce software furnishings. Agile processes turn software development from projects into operations; and this in turn implies software development will never be “complete,” just as business operations themselves are never “complete.” “Ceasing business operations” is an euphemism for “going out of business;” this is typically not the goal of an organization. Starting an agile software process is a commitment to keeping it up forever.

Now, breaking the existing business paradigm in this way in turn enables “business at the speed of thought,” a phrase of some fame. The limberness it yields permits an organization to “pivot” — to reinvent itself — with breathtaking rapidity. But when considering the “at the speed of thought” phrase, it is wise to recall that William James, one of the outstanding observers of human psychology, remarked “A great many people think they are thinking when they are merely rearranging their prejudices.”

Agile processes help organizations meet changing demands in a changing environment. But the organization as a whole, not just the development process it uses, needs to be flexible. Consider that when the concept of the “layoff” finally made its way into Japanese corporate culture, it was widely regarded as an admission of abject management failure. Laying off workers was regarded as an admission that management was too stupid to figure out a way to use its major asset — skilled and enthusiastic people — to make a profit. That view may have been uncomplimentary, but it was also absolutely accurate. If the organization defines itself by an activity it performs, the organization will fail when the environment changes to make the activity unnecessary or unprofitable. Being a car manufacturer is different than being a transportation manufacturer: the latter includes shoes, roller skates, backpacks, motorcycles, busses, trucks, autonomous cargo aircraft, spacecraft launch systems, and teleportation devices; but a car is just a car. Defining the organization as a distributor of goods or services others manufacture or provide dooms the organization to bankruptcy when the Internet disintermediates that supply chain as it has thousands of others: the spiffiest of customer order self-service web sites and the highest of sales commissions isn’t going to help a “business plan” that boils down to “I hope no one starts an Amazon storefront in this product category.” For an organization to be capable of pivoting, the organization must define itself in terms that allow it to surprise itself with what it does. This is, shall we say, uncommon.

But that uncommonness is not necessarily a bad thing. I have consulted at an organization for which the financials said it was a banking services company running a race track on the side, but the Stewards in charge of it all didn’t think of their organization that way. Amazon’s financials say it is a cloud service company running a gift shop on the side, but I very much doubt Amazon thinks of itself that way. In contrast, General Electric was once the model conglomerate: it was willing to do anything, as long as it was first or second in that anything’s market; it discovered every morning what it happened to do that day. Amazon is doing fine — currently — but General Electric is not — currently. There is passion for a thing to be done, and there is passion for the doing of a thing; an any organization, one of these will be paramount, making the other discardable. Either attitude can bring success. But the attitude has implications, and all things change.

Agile As An Adaptation To Humans

In one sense, agile processes are a correct, adaptive, and arguably close to optimal response to human capabilities. Humans are typically unable to envision something truly new: humans are much more capable of responding to an object than of responding to a concept, of responding to something that exists in front of them than of responding to an immaterial idea of what could be. Humans are typically unable to envision long-term future activity: humans are much more capable of planning a handful of short-term activities meeting a small goal than of planning a sequence of tens or hundreds of thousands of activities meeting a large goal. Humans are typically unable to recognize risk and risk factors in the abstract: humans are much more capable of recognizing immediate threat and after the fact unintended consequences. Humans are well suited to dealing with variations on a known theme, and poorly suited to dealing with developments that change the way the world works. Limiting the planning horizon to two weeks removes software development from the realm of only world-class geniuses, visionaries, and disciplined engineers and puts it in the realm of more typical, more available humans.

There is a reason we honor “inventors:” the ability to conceive the truly new is rare. And there is also a reason the phrase “unintended consequences” exists: the ability to anticipate the impact of the truly new is close to non-existent. The global social and economic changes of the latter half of the 20th century are traceable in large part to (among only a very few other things) the creation of the Interstate highway system in the United States. I doubt anyone anticipated roads in the United States would contribute to a 50 year long revolution in global economic activity and world geopolitics, the disappearance of artisan (skilled) labor as a primary human activity, a change in family structures, and a global environmental crisis. Humans typically have profoundly poor judgment about massive changes, and agile processes adapt to that. Humans also typically have very good judgment about incremental changes, and agile processes exploit that.

Perhaps the sadest response a development organization can get from its users is the common refrain: “that’s exactly what I asked for, and nothing at all like what I want.” Humans are horrible at envisioning and communicating about what does not yet exist. Agile does nothing to address this problem — and it cannot, because humans are humans — but it does limit its impact. Every sprint, an agile process holds up the something it produced and says, “you mean like this?” The answer in response is infrequently “yes,” almost always “not quite,” and sometimes “oh Hell no.” But in any case, the response comes after two weeks’ work, not two years’ work. Agile processes iteratively produce stake in the ground approximations until finally the product owner gets what is actually desired, or at least something acceptable, and all the immediate unintended and unanticipated consequences have been seen and dealt with. It is messy and expensive; but when dealing with the new, it is unavoidable. As an additional bonus, eventually the product owners get used to asking themselves the question, “what will I do if the developers actually give me what I ask for?” Over time, this tends to make the user stories more concrete.

Agile As An Adaptation To Business

Agile processes are, by design, intrinsically short sighted. In this, they are a natural match with current business practices, which are also intrinsically short sighted. The major contributor to the short sightedness of agile processes is the use of functional decomposition rather than structural decomposition: a solution is imagined as a series of incremental functions individually recognized as needed, rather than a system of inter-supporting capabilities.

All development processes need to identify needs (requirements), envision ways to meet those needs (specifications), figure out ways to implement the envisioned methods (design), sequence the work (project planning), do the work (implementation), test that the work was done right (verification), and test that the right work was done (validation). Agile processes are no exception; agile processes do all of these. But agile processes do these steps piecemeal, as desired functions are identified and produced, and do these under a magnifying glass, for each individual function rather than for the system as a whole.

The vocabulary and style of agile processes is intended primarily to conceal that these steps take place at all and secondarily to implicate management in their having taken place. Agile uses vocabulary such as “user story” instead of “requirement,” “backlog grooming” instead of “requirements analysis,” “mock-up” or “wire-frame” instead of “specification,” “unit testing” and “acceptance criteria” instead of “verification,” “product owner demonstration” instead of “validation,” and “follow-on user story” instead of “rework.” Agile processes deliberately spread each of these process functions out over as many individual work units as possible to make the functions themselves as invisible as possible, and deliberately involve management as participants in as many of these as possible to keep management frantically active with reactive administrivia so it does not have a chance to notice how much planning and analysis activity is actually going on behind the appearance of frenetic chaos. It does this out of self-defense: identifying the process steps provides names for necessary but non-deliverable work going into work product, which in turn makes this necessary work targets for management to suppress as non-deliverable and hence “useless.” After all, it is far easier to measure activity than effectiveness, and once management has convinced itself, for example, that the deliverable is a concrete pour they will tenaciously resist the calculations, site preparation, and form building that goes into making the concrete pour useful. Tons of concrete on site is the deliverable: all the rest is “overhead.”

The Dunning-Kruger effect is a major player in post-industrial production, where the “deliverable” is a system — an arrangement that produces emergent behavior, remember that definition? — rather than a thing, and management not only cannot see and touch the product but the entire production process is both incomprehensible and largely invisible to them. Software is produced by typing into a computer, right? How hard can typing into a computer be? My ten year old does that all the time!

An Example Triumphant Success

But, as you might expect, this focus on details that can be delivered in the short term has rather profound business implications. Consider, if you would, building a house in an agile manner.

Do you detect some cost, duration, and quality implications in that sequence? It looks to me that you built that house three and a half to four times. And every one of the operations described was the minimum cost, minimum process risk way to get from the then current state to the newly required state.

The sequence as laid out above would be an absolute triumph of a successful house build with agile processes. That is what spectacular success looks like for agile processes. We don’t build houses that way. There is a reason for that.

Agile Processes As Exploration

The problem with agile development is that “agile development” is actually a misnomer. Agile processes do not develop software — or houses — into existence. Agile processes maintain software into existence, and maintenance is five to ten times as expensive as is development, for a variety of fundamental and unavoidable reasons. Agile processes replace capital expense — a project that produces a capitalized work product that is a solution — with much larger operational expense that winds up producing a “solution.”

Despite the expense of producing it, the software produced is always tentative, trial, and ultimately disposable. It never becomes a capital asset, a durable something to be preserved: it starts as and stays furnishings, what the organization happens to have in place right now. This is actually a necessary attitude when using functional decomposition, that is, feature driven development: at any moment, any of the previously produced software may need to be drilled through, reshaped, pushed aside, completely replaced, or entirely discarded as the next feature makes the previously produced software non-viable. Agile development, by moving software from the result of a project to an element of operations, ensures the software is never done: the software is always scaffolding around current business process, never the definitive embodiment of that process. This dethroning of “the computer system” has benefits, profound ones, for the adaptability of the business itself. It is arguably the correct positioning of the software anyway: we want operational support software to liberate the organization, not constrain it. But it also has costs, profound ones. Software development has always been a money pit; agile software development is a money crater. The adoption of agile processes often implies another zero to the “software development” line in the budget; this is a source of profound sticker shock to most organizations, and is enough to sink an organization unprepared for it.

Despite their costs, agile development processes are incredibly valuable — when the organization is discovering its needs and is willing to explore alternatives in meeting those needs.

Humans are short-sighted creatures: most of our anticipation of what the future holds is simply wrong, most of our preparations for future problems are wasted. Our militaries are always preparing to fight the last war, not because generals are stupid but because they are human. Humans always prepare for what they know, from experience, is important: the past. We are much better at figuring out what would have helped in the past, or what would help right now, than we are at figuring out what will help in the future. We are much better at saying “exactly like that, but with this minor tweak” than we are at describing a desired abstraction.

We can easily recognize a desire: say, a desire to quickly make detailed, face-to-face business arrangements between organizations on different continents. But most solutions envisioned by humans will be incremental: faster passenger aircraft, for example. Some times the environment will push us in another direction: a pandemic (a topic of recent interest) might make us realize high-fidelity telepresence is a viable alternative to physical presence. And eventually some crazy world-class genius visionary who actually needs immediate actual presence on the other side of the planet, instead of just wants it, will imagine and implement an impossible solution: upload ourselves into a there-local agent, physical teleportation, personal ballistic missiles, or something else close to inconceivable.

But I can promise you now: whatever we actually wind up with in 30 years, be it psychically linked clones or telepresence or AI duplicate “us”es or teleportation or personal ballistic missles or blockchain or smart contracts or a radioactive mile thick ice sheet over the entire planet, no one will give a rat’s ass about some incremental improvement faster passenger aircraft once made to intercontinental business dealings. Intercontinental business dealings, if they exist at all, will have been simply solved in some entirely unexpected manner. Most humans — certainly me and 7.8 billion to maybe 50 odds you — simply lack the capacity to imagine actual solutions rather than tweaks to the current state.

A Word On Organizational Scale Agile Processes

A recent innovation in agile processes, as the management consultants instead of the technologists got a hold of it, is the “organizational scale agile process.” In this innovation, the entire organization becomes agile, and the organization synchronizes agile management schedules and agile software schedules and agile service schedules and agile infrastructure schedules into “release trains” where all the simultaneous and parallel agile efforts burp out their coordinated deliverables at the appointed time.

Now, you might wonder how a process where the content available at a specific future date cannot be predicted — ordinal scale, remember! — manages to coordinate its available content at a specific future date with another process where the content available at a specific future date cannot be predicted. The answer is smoke, mirrors, distraction, and self-delusion. Agile processes get their power from not having a schedule: not having a schedule is the single, central, most distinctive attribute of agile processes. No matter how well you can herd cats, you can’t herd half a dozen cats that don’t actually exist.

Now, since agile processes keep their product potentially shippable at all times, you in fact can bundle up the results from a number of agile processes at a scheduled time and ship the bundle. You just can’t predict what will be in the bundle.

If the phrase “scheduled delivery date” exists in the lexicon, the organization is not doing agile, no matter what the management consultants are saying; it is instead doing either chaos, or waterfall at ten times the expense waterfall needs. “Agile” and “schedule” are antonyms. Pretending they can be reconciled is not a recipe for success.

Summary: Appropriate and Inappropriate Applications for Agile Processes

Agile processes are invaluable when you have no idea where you are going and what you will do when you get there. They let the organization identify immediate pain points at any time, pivot to meet them (in an expensive and clumsy way, perhaps, but a way that takes two weeks and not two years to start), and quickly start getting symptomatic relief. After experience has shown what is a good idea and what is not and gives hints as to what the actual problem is, symptomatic relief can be extended into, or replaced by, an actual solution to the actual problem, typically through an actual project. Agile is wonderful for voyages of discovery: it fact, if you were to reduce “voyage of discovery” to a process, the result would almost define agile. With time and experience and advances in what is feasible, you discover you do not actually want what Lewis and Clark imagined as the solution to their need, a better way to portage canoes across a continent: you discover you actually want a railroad, or a vehicle road and a vehicle, or commercial air travel. Or teleportation.

On the other hand, humans are, let’s face it, pretty boring: there is little new under the sun. Most things we humans do have been done before. Maybe your situation is different: it might actually be that no one has ever loved like this before. It’s possible. But, you know, the odds are actually pretty good that just about everyone has loved exactly like this plenty of times and you can get the full rundown and a load of hilarious anecdotes besides just by asking around. When you have a pretty good idea of the end state — an order system, a customer relationship management system, a stock management system, a payroll system, a threat tracking system, a materials handling system, a house, or anything else with a widely understood name and a typical feature list you can look up on Wikipedia — choosing to get there with agile processes is batshit insane.

So let’s hear it for boredom, and thank you for coming to my TED rant.