October 6, 2020

On the Organization of Scientific Work

If we are to have scientific revolutions, we must have scientific revolutionaries.

On the Organization of Scientific Work

The historical course of science has been punctuated by the concentrated works of small groups and individuals. In our era, science operates at a dehumanizing scale, resulting in a metrics-driven mechanization of inquiry from which nothing original can ever hope to emerge, yet consuming enormous resources to do so. Scientific work must be decoupled from the external forces introduced by dehumanizing scale, and the work must be driven foremost by the norms, tastes, relationships and talents of small groups.

Saints may emerge from the docile and humble, but rarely scholars.

Scale & Community in Science

Communities break down with scale, losing the vitality they had when small and ultimately becoming an undifferentiated mass with an enormous diffusion of focus, to the point that any given group of sufficient scale is not really differentiable from any other. One example of this effect is the eternal September experienced by online communities during the wide proliferation of the internet and subsequently the web.

Machine Learning (ICML) conference paper submissions over time

This effect also exists in scientific communities. Physics enrollment ballooned after 1945, after Nagasaki and Hiroshima. In our time, machine learning conference attendance and submission rates continue to compound, often outstripping the doubling of Moore's law the discipline relies on to make any forward progress. The quality of results in these and other fields since the vast expansion of their communities has not increased.

Why?

One often-repeated reason is that low hanging fruit is picked early and so increasing total effort results in decreasing returns. This view casts scientific work as a purely incremental process, advanced by fungible labor, when in fact science has historically advanced as punctuated equilibria, long processes of digging at the details before a synthesis of unexplained observations with a new theory results in explosive change. Progress in a scientific field has never been a function of total effort.

In fact, the problem lies in the scaling of scientific work.

Science operating at scale destroys the capacity for the required digging and the possibility of any novel synthesis. Science at scale requires a mechanization of scientific work through the application of metrics to scientific output. In an attempt to turn the full complexity of science at scale into something comprehensible, attempts to quantify the importance and 'impact' of every published scientific work in terms of h-index or other measure have the inevitable result that the purpose of science at scale is to increase h-indices and more efficiently allocate resources to those who are best at it. This environment reduces the scientist to an increasingly specialized automaton or else skilled politician.

In fact this is not a new observation, in "Advice to a Young Investigator" (1897), Ramon y Cajal says;

School discipline reigns with such tyranny in the universities of France, Germany, and Italy that at times even the greatest initiative is suffocated, along with the flowering of original thought... Now that scientific research has become a regular profession on the payroll of the state, the observer can no longer afford to concentrate for extended periods of time on one subject, and must work even harder.

What room is there for human qualities under such regimes? Where can the heroism, the vigor, the boldness of inquiry possibly survive? Where is the fame of the individual scientist, even in intellectual circles, outside the possible commercial success of his work?

Scale, inevitably leads to power-law distributed outcomes, leading to the inevitable concentration of talent and resources among a few investigators pursuing a few lines of inquiry, and their pale second-rate imitators. Through this mechanism science at scale reinforces (and in fact, under sufficient political capture imposes) consensus, further annihilating the possibility of the necessary revolutionary synthesis of ideas.

There is little incentive to pursue heterodox work in the scaled science world. Your success within it depends on your ability to get your work recognized and funded, which means your work must fit into an abstract reviewers can skim for understanding. There is no true peer review at scale, certainly not in the sense of deep evaluation and criticism, how can there be when each reviewer is confronted with hundreds if not thousands of manuscripts? Indeed there are recent efforts to still further mechanize the production of scientific work by increasing automation of the review process itself.

How can this hope to be anything but incrementalism on the consensus? How can this result in anything but the subversion of earnest scientist into the hyperspecialized grant writer eking out an ecological niche in the bowels of the mechanism?

Revolutionizing science means reorganizing Science

Efforts to mitigate the problems of science at scale by attempting to adapt scientific institutions to better handle them cannot hope to succeed, if the effects of scale are even recognized as problems at all. In some ways we are now in an era of what can be thought of as 'scale realism' sometimes called 'biggerism', the frequently repeated belief that science can only progress further if still larger scales are achieved, still more resources dedicated.

Hope lies with opposing scale altogether by constructing resilient, sustainable architectures for carrying out scientific work in small groups.

In fact, such small effectively closed scientific communities built on interpersonal relationships already exist to some extent, and operate invisibly today either as loose networks of informal collaborators or else as tightly bound groups resembling families or clans. Their work is the equal and often the better of anything being produced by science as practiced at scale.

In small groups the mechanisms of openness and peer review fulfill their intended purposes, and are further reinforced by shared community norms, tastes, and interpersonal relationships. One has a much stronger sense of the quality of the work of the person across the desk from you than one does for the work of a different group on a different floor, let alone on another continent. There are much higher stakes in disappointing ones immediate peers than disappointing the anonymous infamous reviewer 2.

This form of organization is historically prevalent, in part because far fewer people could afford to become scientists than in our era. Groups of the appropriate size and social structure still form the bedrock of most scientific activity - the smallest unit of research work is still the lab, consisting of a dozen or so grad students, perhaps a few more experienced postdocs, and one or two tenured faculty. Unfortunately this organization is vestigial. The capacity of such groups to perform revolutionary work is hindered precisely by their material and spiritual coupling to the mechanism of science at scale, leading each lab to look much like the next, and working toward the same ultimate purpose.

Scientific work must therefore be spiritually, organizationally, and materially decoupled from the forces of science at scale. The way to achieve this is to give primacy to the organization of small groups and the space for those groups to develop their own norms.

Throughout history people have organized themselves to exist separately from a prevailing social context in many ways, whether in the form of monastic orders, artist's collectives, or secret societies. Indeed, scientists themselves have used whatever social advantages they had to decouple themselves from ruinous external forces.

More recently we may examine the rapid hot-housed evolution of online subcultures as they dealt with a newly scaling web, through the development of radical and explicitly transgressive norms or else inscrutable private languages.

Let a hundred schools of thought contend

Shifting the production of scientific work away from the institutions of science at scale and toward the science-monasteries is desirable, and it may soon become necessary. The concentration of resources required for the continued function of science at scale may not be possible for much longer. The sooner the necessary experimentation with new scientific organization is carried out, the more the risk of catastrophic loss is reduced.

This experimentation must seek to answer many questions about how scientific work can be profitably organized, which until now have not been examined in any depth, except at times as historical curiosities or from an analytical rather than a synthetic or instrumental point of view.

It is unlikely that a single set of norms would prevail over all such organizations. Rather it will be the jobs of their founders to determine and codify their own. There will be many failures, people haven't tried to seriously attempt these things for a long time, and certainly never under the current circumstances, or with the tools now available. Ways will be found to make communities sustainable, not least because scientific work is often commercially valuable and capital remains a great lever. Inter-organizational norms will also be necessary, but not for a long time yet.

The organizational and spiritual decoupling should be our first points of attack. To contend with a resource or economic decoupling requires resources of our own, which do we do not yet possess. It is certain that once shown an exemplary path, many who are sensitive to the same conditions will follow. We should however avoid placing the spiritual developments, much less the aesthetic considerations, at the core of our endeavors. Instead we must produce exemplary scientific work that can stand alone, and imbue its dissemination with the spiritual context under which it was developed.

I do not seek to resurrect the ghost of science as it was practiced before the rise of science at scale, that would be a fundamental mistake. I certainly have not read many old books. What is certain and ultimately needed is the development of something fundamentally new.