Based on my lecture at Topos House, San Francisco on February 29, 2020.

Not all fields of knowledge exist yet. If you tried to study biochemistry in 1820, you’d have a lot of trouble: the field had yet to cohere. Do we think that all the biochemistries of the world have been discovered? If we did back then, we’d be wrong. For those seeking a safe career, sticking to the established fields is probably the right move. But for those interested in pushing forward the frontiers of knowledge, it will sometimes be better to work in a field that has not yet cohered, or in a field on the cusp of crystallizing. Often the best intellectual opportunities — if you want to be a breakthrough researcher — and the largest economic opportunities — if you want to build a great company — are going to be precisely in those areas about to crystallize into a new field.

If you went into biochemistry when biochemistry was emerging, your name might be in a textbook today. You didn’t have to be brilliant; you just had to pick your problem well. The greatest difficulty would be in figuring out who to learn from. Before you figure out where and how to learn, you have to decide who has the knowledge you seek. When the field of biochemistry was on the cusp of crystallizing, for example, you would have looked to experts in the fields of biology and chemistry. But as an outsider, who are you to evaluate the quality of a field? Just because a field claims to exist, doesn’t mean it exists.

Evaluating Existing Fields

One thing you can do is look at how a field performs on its own criteria. Take the replication crisis in academic psychology for example. In academic science, replicability has been one of the gold standards both for internal bureaucratic targets and also for the layman’s understanding of the philosophy of science. Philosophy of science might not seem very important, but it is key to figuring out how we know what we know, and what is good science as opposed to bad science. There is, in fact, no consensus philosophy of science, which means that there is no canonical science of science.

But surely scientists must know how science works? Well, scientists might know how science works in the way that birds know how aerodynamics works. They know it, but not at all on a conscious level — they just fly. Even if birds could speak, their answers might be quite useless, even to baby birds. Their knowledge is not formatted for scientific understanding; it remains locked away as a type of intellectual dark matter. If you are deciding whether to enter a field in which you are not yet an expert, you need a more precise epistemic foundation than a bird’s intuition, especially when deciding where expertise truly resides.

A thought experiment: say you had five experimental planes in front of you. How do you decide which one to board? Is it the wooden one, the bamboo one, the steel one, the large one with smoke-puffing engines, or the modest one made by the bicycle shop owners? The Wright brothers were bicycle shop owners, and they built a janky-looking machine. Plenty of the other early flying machines looked vastly more impressive than theirs, but they didn’t fly. In the case of deciding which of these early flying machines to board, relying on institutional claims to epistemic authority would not work—not even the ones made by Harvard professors could fly.

When making such decisions, you cannot assume that the members of prestigious institutions of your society are experts simply because they claim to be experts. They would claim to be experts whether they were or weren’t! In any period of human history, if you examine how institutions portray their own expertise, they always portray themselves as tremendously knowledgeable with impeccable foundations. In the rare cases where they do admit to not knowing something they propose it to be either unknowable or, to be knowable—but only if you give them more funding. The Catholic Church would claim this when it came to metaphysics and the question of salvation. Today, the Church of Scientology claims that Scientology is at the productive frontier of psychology, that they’ve disproven psychiatry, and that they have these cutting-edge, electronic brain-measuring devices. And so would a university cognitive science department.

We all make choices using dumb heuristics, but importantly, they are often good enough for everyday life. If you choose to study history at Oxford because the buildings look old, you’ll likely do pretty well. Rules of thumb such as “Are the buildings old?” can have a valid core. The heuristic isn’t a bad one for the prospective historian to use: old buildings often come with libraries well-stocked with old books. But does this heuristic hold in other cases?

Say you are deciding between Harvard and MIT for studying astrophysics, and you view Harvard as the better pick. Why do you believe this? Is it because Harvard has older-looking buildings? We can imagine that some prospective astrophysicists, too, are swayed by Harvard’s historic appearance. Much of academia, in fact, functions on such cargo cult heuristics. Maybe one should study astrophysics at Harvard, rather than MIT. But if this is so, then it’s right by accident: the “old buildings” heuristic doesn’t apply to astrophysics. It is thus a mistake to rely on it—a broken clock is right for two moments a day, and wrong in all others. Your life has many important moments.

Because there is no consensus philosophy of science—no science of science—if you want to go into a more established field, you have to rely either on institutions, or your own evaluation of individual researchers. What you must do, then, is form your own judgments of the claims to intellectual authority made by particular institutions, or the quality of thinking of possibly exceptional individuals. But what if you want to go into a less established field?

Communities of Practice

For those considering entering a new field, there are several ways to acquire deep expertise. Firstly, there is the community of practice—imagine a hobbyist society or hacktivist collective, for example. People gravitate towards these communities chiefly for friendship and community, and proceed to enthuse together over their community’s mechanism of practice, channeling this shared energy into competition for status, acceptance, and love. Thus, communities of practice tend to have a social pressure towards excellence. This often makes joining one one of the best ways to acquire knowledge.

History is full of examples of communities of practice, from the circles of philosophers of Ancient Greece, to the guilds of medieval Europe, to Meetup.com. The Royal Society was originally a group of bored 17th century British aristocrats who wanted to stay far away from politics in the aftermath of the English Civil war to pursue knowledge of nature together. One can imagine them showing off their fanciful etchings and astronomical instruments, or one-upping each other with exotic mineral samples brought from far-flung corners of the globe. A side effect of this competition was science: we learned things about geology, astronomy, entomology, and so on. Thanks to this, Charles Darwin had access to a dataset describing the taxonomy and habitats of insects and plants from all over the world, which provided much of the evidence for his theories of natural selection. Were that prior work not done, Darwin’s theories would have been mere conjecture, difficult to establish as authoritative contributions to our understanding of the mechanisms of nature.

So seek out communities of practice; find out who is excited by what you want to learn. This community may or may not be very well-connected, but above all it should be relatively narrowly focused on the practice of some activity that its members constantly relate to. Without this focus, the community won’t have highly-trained internal heuristics—better heuristics than the age of buildings!—to identify who really knows what they’re talking about.

Communities of practice are not perfect, but you can almost always rely on them in some way. Some of them may be fraudulent or confused, but you’re certainly better off joining one than attempting to be a pure individualist. There are likely entire communities out there centered on what you wish to study. Find them. Think about who would have the socioeconomic leisure to run such a community, and what they’d call themselves. Maybe you will find them at a university among college students, maybe on the internet. Perhaps you will find them among the modern equivalent of bored aristocrats—angel investors, perhaps. Smart, bored rich people will always find a hobby. It’s only a question of what that hobby is—is it what you’re searching for?

Master-Apprentice Relationships

Aside from communities of practice, another indispensable mechanism of transferring deep expertise is the master-apprentice relationship. In medieval European guilds, the master-apprentice relationship formed a strong contract. Instead of owing $90,000 of student debt and receiving a diploma, you instead would sign yourself up for service to a particular master for around seven or eight years. And at the end, they would essentially grant you their business. Their retirement plan would be you. Can you imagine how different your relationship with your doctoral advisor would be if you personally constituted their retirement plan? Their incentives would be much better aligned, to say the least.

A weaker, non-economic version of the master-apprentice relationship does still apply for professors and thinkers who care about their legacy. And if a field has clearly identifiable experts, it is extremely valuable to work with the best person you can, in the hopes of forming a relationship with them. You should approach them and offer to proofread their papers, or even offer to make their coffee. Getting to work with them for a few hours a day while they talk about their field could be an education well worth unpaid or underpaid labor, especially if they understand you are there because you truly care about the field, and want to learn.

You may find semi-retired prominent figures more approachable than you think. Try asking around at their institution, whether a university or a firm, to feel out your chances of meeting them. Devote some time to going to as many of their institution’s relevant events as you can—maybe even crash some happy hours. As always, a cold email with a thoughtful response to one of their papers will most likely garner a response.

A little applied anthropology can go a long way. Say you wanted to figure out who Paul Graham hangs out with in London, in the hope of meeting and learning from him. In this case you should think: if I were Paul Graham, who would I hang out with in London? This is a more realistic task than it might seem. You are only a few degrees removed from everyone else on the planet. Once you find the right social circles, the key is finding someone who knows your target contact and giving them a reason to introduce you to them. At this point, your job becomes to align the incentives of the gatekeepers with your own. Show up at the right parties until you are able to go up to Paul Graham and ask, “Can I make you coffee?”

Prospective apprentices should be careful, as they are placing themselves in a position that can be economically exploited. Currently, there is no mechanism to make your mentor reward you financially. But if your payoff is knowledge, you will soon be able to reason for yourself if the relationship is worth it. It may not be. Perhaps the communication gap is too big, or the relationship too awkward. But finding a good master is worth the effort.

Functional Institutions

Finally, in addition to communities of practice and master-apprentice relationships, well-functioning institutions present you with a third option. Note that it’s important to discover whether the field you’re entering already exists, or whether it’s about to crystallize. Only if it already exists will there be the option of joining a functional institution.

I may appear skeptical about the functionality of large institutions such as universities or newspapers at scale—my thoughts in this regard are best summarized in How To Use Bureaucracies, Functional Institutions Are The Exception, Institutional Failure As Surprise, and Intellectual Dark Matter—but I do think functional institutions exist. Some institutions work extremely well at scale. I would consider Microsoft Research to be fairly functional. Even though Microsoft is a large company, they do a pretty good job of maintaining a professional environment.

When I say professional, I do not mean “behaves properly.” What I mean is “adheres to an expected social role.” When I show up to Dr. Bob’s office, I don’t expect to deal with Bob; I expect to deal with The Doctor. The doctor is not going to behave as Bob, Alice, or Caroline might act, even if that’s who the doctor is. The doctor is going to behave as The Doctor! This is essentially a human user interface. I’m not Samo. I am The Patient. And Alice is The Doctor. Professionalism is a LARP—and an important one. It might seem as natural as gravity that there are doctors and patients, but these are social roles. And we had to be educated to fit those social roles.

If you are thinking about entering an institution, you should seek a very specific culture of professionalism, one that seems to match the task at hand, rather than what amounts to mutual sabotage. If the professionalism of a large at-scale institution is well-designed, the overall incentives of institutional leadership will be aligned with the overall mission. And if such an institution deploys efficient bureaucracies, it can function well as a productive research organization. But again, functional institutions are the exception. So only institutionalize yourself—that is, enter a mostly bureaucratic environment—if you believe the institution that you are entering to be a functional one.

One way to determine if the bureaucratic mechanism is efficient is whether or not they expect you to do all of the paperwork. Inefficient, decaying bureaucracies are a little bit like dying stars: they eject most of their mass. Their internal cancerousness generates reams of excess paperwork, and they are thus incentivized to push paperwork onto the user. For example, if a public official likes you, they may ask you to fill out a few lines and send you on your way, but if they dislike you, they may give you a mound of paperwork for the exact same task. I have some experience of this growing up in Eastern Europe, but Americans can easily experience this at the local DMV.

If you experience low bureaucratic burden in a highly bureaucratized organization, that bureaucracy is working very well. It’s like a machine that hums in the background, versus a machine that screeches and puffs smoke into your face. It’s going to be a very visible experience, even if you don’t have much other information.

Finally, check the institution’s leadership. Even if it is a well-oiled machine, make sure that its roles are very clearly delineated towards the mission, rather than towards internal conflict. The leadership might not want your output and may actively steer away from your output. A good example of this dynamic can be found in Richard Feynman’s critique of NASA’s administrative process. After the Challenger explosion, NASA called on Feynman to find out what had gone wrong with their bureaucratic process. The issue, he found, was that the leadership didn’t want to hear bad news from the engineers, because they wanted to push through as many flights as they could in order to score PR wins. And the Challenger mission, with a schoolteacher on board in an effort to demonstrate that space was to be for everyone, was an important PR flight. Leadership didn’t want to hear the truth, even though the engineers themselves were still acting professionally and some parts of the NASA bureaucracy were still functioning well. The Challenger went up in flames, and the Space Shuttle remained expensive.

Conclusion

Entering a field for the first time as an autodidact, you should decide whether you want to enter into a community of practice, a master-apprentice relationship, or a functional institution. Try to figure out for each of those whether it fits the criteria. Seek out an energetic community of practice, or a good master. If you are considering joining an institution, make sure that its bureaucracy is working well, its professionalized roles are functional, and its leadership is aligned with correct output.

Leaps of faith like these can be harrowing, but with planning they need not be. Tracking existing sources of prestige and economic stability may be rational in a narrow sense, but the world of institutions in which they are embedded is prone to dysfunction. Recognizing when dysfunction has overtaken an institution is key to ensuring that we remain able to generate new knowledge about the world. History is littered with examples of collapsed civilizations that failed to do so. New scientific fields carry the promise of civilizational advancement, but they are not discovered automatically. It’s up to us to find them.

More essays