Intellectual Dark Matter

by Samo Burja | July 16, 2019

Knowledge that we can show exists, but cannot directly access, rests at the foundations of society and technology.

Missing mass, missing knowledge

Many galaxies would fly apart if they had as much mass as estimates based on their visible signature suggest. Although some have posited alternative theories of gravitation to explain this discrepancy, most physicists now hypothesize the existence of mass-bearing particles that are not detectable through emitted radiation such as visible light. We call these particles dark matter, and it is estimated to compose about 85% of all matter in the observable universe.

In analyzing the functional institutions of our society, we are not able to see for ourselves most of the knowledge that created them. Knowledge of this sort includes trade secrets, tacit technical knowledge, private social networks, private intelligence-gathering operations, management and persuasive skill, cooperation and collusion among founders and their allies, and founders’ long-term plans for their institutions.¹

This knowledge has profound effects on the social landscape. We must understand it if we hope to understand society. We therefore must examine intellectual dark matter: knowledge we cannot see publicly, but whose existence we can infer because our institutions would fly apart if the knowledge we see were all there was.² Such intellectual dark matter rests at the foundations of our society, dwarfing in scope and importance the accessible, shareable, visible knowledge on which we normally focus.

There are many forms of intellectual dark matter, but the three principal ones are lost, proprietary, and tacit knowledge.

Lost knowledge

A body of understanding becomes lost knowledge when the tradition of knowledge maintaining it ceases to exist. At the dawn of the modern era, during the Renaissance, there was a clear understanding of the importance and scope of lost knowledge. This led to an ambitious intellectual effort in which scholars unearthed and attempted to understand ancient wisdom.

The recovered works of thinkers and scholars such as Cicero, Livy, and Thucydides were closely analyzed, and as a result now serve as the intellectual underpinning of many Western political systems. The unearthed texts of ancient Greek geometers and natural philosophers developed over centuries into modern mathematics and physics in the following scientific revolution. Our modern prosperity is arguably downstream of this discovery of lost knowledge.

The echo of this early modern period can be found in the popular conception of the Dark Ages. If a dark age is an age that has forgotten most of what was learned, we are still living in one.

The relearning of Greek and Latin works was left fundamentally incomplete. Just as physicists are only able to observe 15% of matter in existence, today we possess written fragments from only 13% of the ~2,000 ancient Greek authors known to us by name. This does not account for the authors we do not know, and only a small portion of the 13% figure consists of complete works: while we have recovered Aristotle’s Politics, we only have fragments of his Economics.

Our core philosophical, political, and theological works are conceived in dialogue with Greek and Roman thought. Medieval and modern thinking so vital to the creation of our largest and most important institutions, such as that of St. Thomas Aquinas or Montesquieu, rests on the preserved works of antiquity. Those works in turn are themselves written in dialogue with further works that remain lost to us. We therefore cannot even see the intellectual foundations of our most important religious, academic, and political institutions.

Lost knowledge is not just ancient. Strategic actors of the present understand the advantage of locating and revitalizing recently lost traditions of knowledge. If you aspire to build world-class rocket engines today, you might go to great depths, even deep-sea depths, to understand rocket construction during the golden age of American space exploration. In 2013, Jeff Bezos recovered two Apollo 11 rockets from the bottom of the ocean. Do you believe that he donated them to the Smithsonian without having the team at Blue Origin, his aerospace company, reverse-engineer them first? It seems unlikely.

Proprietary knowledge

The next large chunk of intellectual dark matter is proprietary knowledge. The use and spread of such knowledge is restricted by an institution guarding its monopoly.

Companies use legal means such as non-disclosure agreements and information security practices to guard against industrial espionage and secure economic advantage. For example, the Medallion Fund managed by Renaissance Technology has returned an average of 40% annually since its inception, including a 100% return in 2008, making it by far the best-performing hedge fund in history and netting its investors tens of billions of dollars. Its mathematical underpinnings are kept secret not only via non-disclosure and non-compete agreements, but also very high compensation and a carefully crafted, unique company cultures that disincentivizes interaction with the outside world.

RenTech’s offices, for example, are on Long Island near Stony Brook University, about 60 miles from Manhattan, where the finance community is concentrated. This makes sense — were other quant funds to learn of its methods, Renaissance’s ability to exploit market discrepancies would become far less profitable. RenTech is not unique in these practices. Bridgewater Associates, another premier hedge fund with a very unique company culture, has its offices at a secluded riverside location across the Long Island Sound in Connecticut, 20 miles from the nearest hedge fund cluster in Greenwich.

Deeply networked professions can also develop a culture of restricting access to and limiting rights to use information. The purest form of such professional cultures are the guilds of medieval Europe. Only a handful of modern guilds are granted equivalent legal recognition.

The term guild is archaic but apt. Recognizing a well-coordinated contemporary guild is important for understanding economic processes. Such communities carry significant clout in our social and economic landscape, and are likelier than a single company to punish transgressions, be it through legal, economic, or reputational attacks.

Guilds also protect information through formalized training and apprenticeship. While this training may not be necessary to master the relevant skills, its first purpose is to ensure commitment from those trying to access information. One of the ways costs are imposed is through obscurantism, deliberately conveying information unclearly. By making a subject appear much more intellectually demanding than it actually is, you discourage people from attempting to learn or compete with you. The pretense of intellectual rigor allows you to overstate proprietary knowledge and thereby further increase your authority or extend it to domains beyond your expertise. The overuse of mathematics in economics is a good example of this.

Law provides an example of a field where such guilds thrive in practice, if not in name. Nearly all knowledge of how to achieve favorable judicial outcomes is local and informal. Ranked in importance for judicial outcome, understanding of what will and will not be admitted in court comes first, second comes the construction of plausible legal arguments, and only third the discovery of relevant precedent cases. This first sort of information is guarded by a particular network or law firm to help secure an economic niche, hence the high degree of specialization within law. This is also why it is possible to outsource the routine and labor-intensive task of searching for favorable precedents to junior partners, assistants, and even machine learning algorithms — the firm is not seeking to protect that information.

Guilds and companies are not the only kind of institutions that guard proprietary knowledge. States regulate the use of information for political advantage. They make use of legal means and information security practices — much as guilds and companies — as well as all the capacities afforded by their surveillance, security, and defense apparatuses.

Tacit knowledge

Tacit knowledge is knowledge that is not transmitted in written form. For example, a blacksmith learns to craft well-balanced swords through direct practice and correction from a master — not by reading a textbook.

Most practical knowledge is tacit, and for good reason. Explicit instructions become far too complicated and cumbersome to describe even moderately difficult tasks. Moreover, people learn far more easily from practice than from books, making practice a more reliable means of teaching large numbers of people, for example workers, how to perform a task. People become fluent in languages by speaking them, not by reading language textbooks.³

Tacit knowledge is the most significant and widespread form of knowledge that allows institutions, and thus the economy, to function. Many professions, such as a cashier or an Uber driver, are very simple and function near the edge of automation. But those most critical to society, such as statesmen, skilled industrial workers, or engineers, require large bodies of tacit knowledge to perform well, and the best performers are highly compensated.

The Bessemer process illustrates the critical economic value of tacit knowledge and the difficulty of making it explicit. In 1856, Henry Bessemer patented a new process for making steel that was much less expensive than existing methods. He licensed the patent to several manufacturers, but they weren’t able to get the process to work based on his explanations, and eventually sued Bessemer over it. Bessemer took matters into his own hands, started his own steel company, and implemented the process with great success.

Enormous efforts are made by strategic actors to secure and protect tacit knowledge, especially when that knowledge provides an adversarial advantage. In World War II, Germany’s Wernher von Braun developed the V-2 rocket — the first guided ballistic missile — with devastating success. Neither Germany’s rivals, nor its allies possessed the technology, and Germany was sure to keep this knowledge close to its chest.

As the war was ending, the SS closely guarded von Braun and his team with orders to execute them if the Allies approached. Nonetheless, the scientists managed to escape and surrender to Allied forces, after which they were immediately sent to the U.S. to train teams of American engineers and military personnel in rocketry. This was not an isolated case — the U.S. and the U.S.S.R. both carried out large special operations to capture German personnel with valuable tacit knowledge, and both benefited immensely from it.⁴

Since it cannot be easily transferred via texts, tacit knowledge must be taught via direct practice and extensive interaction with a skilled practitioner. Traditional master-apprentice relationships are the gold standard for these training relationships, though other arrangements are feasible so long as there is an economic incentive for the skilled practitioner to spend a large amount of time with his student. Otherwise, the knowledge simply isn’t transferred, and with many crafts, is lost forever.

The quest for dark matter

We are standing on top of a vast system of institutions powered by intellectual dark matter. Some of this matter can be made visible — proprietary and tacit knowledge function in private but can be uncovered. Much of this matter is lost, never to be seen again.

Institutions dependent on lost knowledge are running on autopilot and will fail to adapt or renew themselves. Western countries continue to have high living standards and drive the bulk of innovation, but this should not be taken for granted. Many failed institutions were once highly functional, and they can maintain the appearance of health even as the late stages of decay set in. On the eve of the financial crisis, Lehman Brothers looked as strong as ever. This is especially concerning considering how pervasive inflexible bureaucratic institutions are in our society.

We cannot predict and guide the trajectory of our society if we do not understand the importance of intellectual dark matter and so fail to locate and preserve it. The sum of public information available to us may be less important than even a small fraction of this knowledge. If we find this information and assemble it into a coherent understanding, we stand a chance of dramatically changing the world’s course for the better.

Footnotes

[1] As a very concrete example of the final item on this list, consider the founding of Amazon.com in 1994. One could infer at the time from Bezos’ previous employment, an article in his high school newspaper, and reports from his ex-girlfriend that he planned for Amazon to take over all of e-commerce to net enough money to start a space tech company. For many years, however, Amazon was only branded as a bookseller, and Bezos made sure to obscure the company’s long-term plans in quarterly earnings calls. Knowledge of this intellectual dark matter would have informed your strategic outlook, either as an Amazon competitor or a prospective investor. This email chain from Paul Graham is another example of an object of this form for Airbnb.

[2] I first introduced this concept in August 2018 at a talk for the Foresight Institute.

[3] Learning a new language to fluency appears to be one of the rarest intellectual achievements compared to the amount of merely written education that is widely available for achieving it in both institutional — academic or governmental — and commercial contexts. Language immersion is functionally practice of a craft with a master — the fluent native speaker — and appears to be the most likely route to achieve fluency. Marko Jukic discusses this in an article that is available here.

[4] One might argue that the Allies sought to seize primarily proprietary rather than tacit knowledge in the above example. But this is not correct: the vast majority of the US & USSR’s payoff came decades after they seized German scientists and industrial workers, rather than from any immediate proprietary knowledge they were able to take. For example, von Braun went on to become the architect of NASA — a project whose returns greatly exceed those of any 1945 V-2 blueprints.

Published with The Long Now Foundation.