Select Page

Institutional Failure as Surprise

by | April 24, 2018

Realizing an institution is near failure is a difficult epistemic problem. There are many outwardly visible pieces of institutions that do not reflect their actual health.

Before the collapse of financial institutions starting in 1929, naive observers were optimistic on the basis of soaring stock prices. Even after the Black Tuesday stock market crash, most observers expected a normal depression and recovery. Instead, the system continued to deteriorate, bank failures wiped out savings, the gold standard was abandoned internationally, and the Great Depression ensued.

Particularly in mature organizations, many automated systems handle tasks. Such systems can persist and even fulfill their function, while the institution as a whole is failing.The default is decay, maintenance of old abilities is difficult, and growth of new abilities is rare. One must look at what features of an institution indicate the current health of the core organization itself, while carefully distinguishing these from features reflective of past health and support from outside institutions.

From these signs, it’s possible to discover whether an institution has the ability to face new threats or is merely trudging through a slow process of decay. If an institution is unable to adapt to meet new challenges, it will lose again and again. Enduring defeat can only last for so long, no matter how large or well established the retreating organization. Eventually the inability to win dooms all institutions.

Robots Outlive Their Makers

Institutions often proceduralize tasks; that is, they create sets of instructions for completing tasks. This process yields bureaucracies; bureaucratization is proceduralization. If you’ve ever worked in or with an institution of some size, you’ve encountered proceduralization. Getting a driver’s license at the DMV is a great example. You must follow a rigid set of instructions to do so.

The DMV’s procedures are annoying, but they get the job done — millions of people have gotten driver’s licenses. However, proceduralization delivers very effective results at the cost of fragility. Human intelligence is a general process capable of solving problems. Applying your mind to any given task produces an approximate, context-appropriate solution. You can greatly improve this solution by adapting it more and more to the particular context in which it is used.

However, as you adapt your solution to fit the case at hand, it is nearly impossible to also have the solution remain generalizable, let alone contain the full set of instructions necessary to fit it to all situations. As a result, proceduralization tends to sacrifice much of the adaptability and context fit that intelligence can bring to particular cases.

The basic structure of proceduralized systems makes it difficult for the people working inside them to deviate in order to adapt to a new context, even when doing so would be beneficial. Poor incentive structures and lack of employee knowledge are the two main factors that prevent adaptation.

A basic building block of bureaucracy is the creation of incentive schemes and responsibility distributions that allow you to have many people reliably follow a procedure of some kind. This kind of incentive-backed proceduralization pervades much of the modern workplace and institutional landscape. Because it is in the basic nature of such institutions to motivate with incentives and constraints, it is exceedingly difficult to change or adapt them from the inside, lest you incur punishment or fall behind your less innovative co-workers.

One specific aspect of the incentive structure further solidifies the un-adaptability of bureaucracies: Knowledge of the principles on which the institutions were built will inevitably fade, because the employees don’t need to understand these principles in order to complete their tasks. Understanding beyond what is needed to play your role is not necessarily penalized, but it certainly isn’t rewarded.

Over time, this incentive structure will result in a bureaucracy with no remaining understanding of the principles which generated it. Absent these models, it will be difficult to change the system or adapt the proceduralizations to better fit new contexts.

To make matters worse, the institutional stasis established by bureaucratic incentive structures and lack of principled knowledge will decay slightly over time. Systems of incentives often do not incentivize their own preservation. This fact results in a kind of erosion, as resources are extracted and minor things changed here and there at the expense of the institution’s functionality.

Other times systems succeed in partially incentivizing their own preservation, which extends their life; however, even then they always align the incentives imperfectly. As a result, some parts of the system bloat over time, rendering it unfit for the original function.

In programming there is a kind of program called a quine, a program that takes no input and as its output produces its own source code, replicating the code perfectly. There is no such thing as an institutional quine, a self-contained institution with no inputs, that perfectly replicates itself. A system of procedures tied to a system of incentives requires active maintenance to perform the task it was designed to perform, to counteract the inevitable decay that ensues as individuals fight to turn the organization’s resources to their own ends.

This is the fundamental problem of bureaucracy: a system devoid of human judgement and oversight results in constant politicking, and constant politicking results in decay. This decay produces something worse than just an unadaptable system: an unadaptable system that fails to perform even its original limited function.

It is best to think of such institutions as machines with human parts. They can be constructed and designed by humans who are at the helm, but can easily outlast the humans that created them, even with no replacement at the helm. In this situation, they will not automatically fail, but will shamble along less and less effectively in the preordained direction, sometimes continuing to accumulate material wealth or even ever greater numbers of employees. Their agility and adaptability will vanish, however, as will their ability to achieve their original goals.

In this way, a powerful institution can be brought down by attacks or changing circumstances which it cannot adapt to. For example, major newspapers are still struggling to adapt to the internet and the subsequent rise of online news. They have not recovered their previous profitability or effectiveness at shaping opinion.

The proceduralized actions such rigidified institutions perform, even if they are functioning well and not diminished by the usual transformations and distortions that arise in bureaucracies, are powerful but context dependent. As such, the institution as a whole is powerful but context dependent. Those that generate such institutions are powerful and not context dependent.

Leaning on the Outside

Some automated systems are not truly part of a given institution itself at all, but rather are an interface with an outside institution.

An interesting example might be the simple sign that a given institution appears to be keeping the lights on in the office. To do so requires members of the organization to work in a well-maintained building that is connected to a functioning power grid, while keeping up with payments for the service.

The building can be maintained by an appropriate service provider. That the provider is doing their job is a sign of the health of the provider, not the organization hiring them. That the power grid is functional also doesn’t reflect the health of the organization under consideration, unless it is the city or national government. That the payments are being made is in itself a weak or moderately strong sign depending on the size of the organization. Generally if it is a very large or established one, it is a weaker sign. When large entities go bankrupt, they keep the lights on until the end.

Thinking about the example, you should generalize it to include all the relevant ways in which an institution relies on others to maintain its appearance. If it is using simple contracts to acquire visible resources (such as reliable lighting), do not consider these elements signs of competence beyond whatever competence is needed to acquire adequate funding.

This insight is especially important, because there are several types of institutions that will reliably have enough funding until their very end. Notable examples are large companies and government institutions. In these cases, signs like reliable electricity won’t provide essentially any evidence of organizational flourishing.

If the institution is relying on non-monetary agreements, such as perhaps other institutions being legally required to provide them with a relevant service, you should ask yourself whether the organization could survive or at least oppose an attack on these services. Could the institution maneuver itself into having such guarantees, if it didn’t already have them? If the answer is no, this means the institution has lost an important ability. It can no longer negotiate new deals. That the deal continues to endure is not strong evidence that the ability to create or even permanently secure the resources on which they are dependent endures.

When seeking signs of institutional failure, you must carefully filter out evidence that primarily indicates the success of other organizations, making sure to account for success-independent funding sources or unstable contracts that an institution would be unable to re-establish.

Official Trappings are Easy to Maintain

Under conditions of widespread institutional dysfunction, formal trappings can be disconnected from the core competence they are supposedly associated with. Sometimes they can even begin to anti-correlate. But assuming the institution in question doesn’t exist in such a dysfunctional context, the formal trappings of an organization actually do indicate competence.

A crucial consideration is that such trappings are in general easier to maintain than set up anew. Naive intuitions are easily misled on this point. It is tempting to equate the difficulty of setting up a new, well-positioned organization with that of keeping an existing organization well-positioned, when in reality it is much more difficult to do the former than the latter.

When labor unions were established in the early 20th century, they organized striking workers to endure near-starvation levels of hardship and violent reprisals from factory owners, and eventually achieved a stable position. Now, unions maintain that position with bureaucratic and legalistic tactics, and strikes are resolved with contracts instead of truncheons and pipe bombs.

Reputation is another good example of this divergence in difficulty levels. Reputations generally remain, unless spoiled. A very easy way to avoid spoiling a reputation is never failing at a task. An easy way to never fail at an externally visible task is to never engage in a task. In this way, an institution that is notably inactive and perhaps incapable of new or effective action can maintain its prestige long after demonstrations of the power, ability, or knowledge that earned this prestige in the first place are beyond its reach. NASA relies heavily on the reputation it earned from the moon landings. This mostly persists today, even though the last manned moon landing was in 1972.

In many human endeavors, the most legibly valuable thing you can bring with you is a past track record of achievement. Sometimes you are only allowed entry into such a domain if you already have a track record. Barriers to entry enforced by assessment based on track record sometimes arise naturally and rationally, as there are no good alternative signs to judge relevant competence. Other times they are the result of cartel-like rent seeking, intended to protect incumbents.

Certain permits have harsh entry conditions but lax inspection for compliance. When this is the case, the barriers to entry very likely exist for their own sake and not as a form of quality control. A regulatory environment relying on track record is the most direct way to protect incumbent organizations from competition. Once such credentials are gained, they are hard to lose. These formal trappings show the organization was capable of acquiring the permits at the time of acquisition, but not later.

Unless recent, past success should not be taken as evidence of an organization’s future endurance.

Fighting Institutions Do Not Fail

An organization engaged in ongoing conflict is surprisingly likely to be healthy, simply because surviving attacks requires some degree of health. Under conditions of real opposition, even retaining past resources, like prestige, should be understood as a sign of activity.

After all, should opposition be serious in pursuing its conflict, it will attempt to disrupt, attack, sabotage, or disable crucial automated processes and individuals. It will also attempt to wear out, destroy, or steal notable accumulated resources.

If the institution does not degrade, there is someone repairing the damage, and that someone has to be effectively working with the reality of the institution under repair. There are two important considerations that must be considered before accepting this read in a given case, however: ‘How real is the conflict?’ and ‘How big is the besieged organization?’

How real is the conflict?

Not all apparent opposition is real opposition, as is frequently the case with cartels. Cartels are vehicles for reaping some of the benefits of a monopoly, without being a single organization. Some are like OPEC, the alliance of oil exporting countries, in overtly attempt to fix prices along their shared interests. However, many of these cartels have an incentive to disguise their coordination.

A recent example was Apple, Intel, Adobe, and Google making a secret agreement to not poach each other’s employees with job offers. This arrangement gave all the companies a better negotiating position with their skilled engineers, enabling the companies to pay them lower salaries. The state of California doesn’t allow for non-compete clauses in their contracts.

In such circumstances it is an asset rather than a liability for a set of companies if the public or crucial decision makers are under the impression that the companies are in conflict. As the arrangement came to light, they were sued and eventually had to settle, paying $415 million dollars.

At least sometimes the defeat of competitors isn’t desirable for a given company or organization. The appearance of competition or opposition can be good optics. In Communist Yugoslavia, there existed toothless parties such as the Christian Socialists that were bound in a permanent coalition with the League of Communists of Yugoslavia. While the reality of this arrangement was a one-party state, the form was that of a multiparty state. The fig leaf of being a democratic society was preserved, at least internally. To eliminate these toothless parties would not be advantageous to the ruling party.

If the defeat of the other side isn’t desired, then the attacks and counter-attacks can be, despite appearances, quite benign. Beyond politics with its staged political debates and occasional show trials, professional wrestling made an industry of producing performative feuds between its wrestlers for entertainment value. The pretense was that the industry was a sport, the reality was that it was show business. They even had an established term for keeping up pretenses of feuds, kayfabe. Since long standing fake conflicts of these kind can be proceduralized, they don’t constitute strong evidence of an institution’s vitality. Fake conflicts don’t require much adaptability.

How big is the besieged organization?

A very large institution can survive real opposition, even if its organization is mostly hollow. It absorbs organizational damage, never truly recovering, but still persisting. As it is unlikely to simply outlast a determined opponent, in order to survive it must have some automated defense mechanism in place that can permanently disable or deter opposition.

A security organization’s ability to launch investigations finding compromising material on their opponent is an example of this phenomenon. This ability is part of their core functionality and can easily be deployed. Such automated counter-attacks will not be innovative, but rather merely exercising one of the many organs the organization developed long ago.

Despite being more vulnerable to destruction by greater powers, fighting, self-contained, and young organizations are far likelier to be active. Where do we see these today? Overall, large and proceduralized institutions dominate the landscape in industry after industry. Even in Silicon Valley, companies like Yahoo and Facebook are best understood as mature media companies rather than young upstarts.


Peaceful, integrated, and long-lasting institutions are often seen as healthy and likely to endure. However, precisely these conditions are what allow their gradual hollowing out and descent into dysfunction to remain unnoticed.

Their ancient nature might signify a fully automated machine. Their integration with the rest of society and other institutions can signal they are getting by on the health of their environment rather than their own remaining functionality. And finally a lack of serious conflict means their resources and positions aren’t honest signals of their current abilities. The difficulty of assessing these factors makes it clear that organizational failure often comes as surprise not just to outsiders, but to insiders as well.