Why Great Innovations Can't Be Planned

Source: Intellectuals

Excerpted from Chapter 8 of "Why Greatness Cannot Be Planned", with deletion

Image source: Generated by Unbounded AI tool

In contemporary society, we have never been more respectful of "goals".

In large companies, the series of work goals represented by KPI and the inspection of the completion of the goals have almost become the only benchmark for measuring the work of employees. In the field of education, standardized tests are not only used to evaluate students' academic performance, but also to evaluate the success of school education. Even middle schools in the United States are under pressure to pursue excellent performance.

In the scientific community, scientists have to go through numerous assessments and supervisions, ranging from strategic direction to research progress. Funding applications should be reviewed to see if they belong to key areas and national interests. Scientists must state their research goals in the application form, and review the completion of the goals every few years.

In Kenneth Stanley's view, this kind of goal-oriented thinking is a "goal myth". It seems that all pursuits can be disassembled into specific goals one by one, and then mechanically and gradually promoted, and finally can be achieved. Reap the rewards. But Stanley believes that great discoveries often come from creative free exploration, rather than mechanical completion of goals.

Stanley is an artificial intelligence and machine learning scholar. The company he and his partner Joel Lehman (Joel Lehman) started was later absorbed by Uber's artificial intelligence laboratory and OpenAI's Open endless team, supporting the Chat GPT in recent years. to the development of the most sensational innovations. The two scholars have written their insights into how innovation is produced in the book "Why Greatness Cannot Be Planned".

The two scholars believe that the greater the achievement, the harder it is to rely on goal-oriented thinking, and free exploration often lays the foundation for great discoveries. Great achievements are always born in unplanned and unexpected places. No one would think that the advancement of lithium battery technology driven by the hot sales of electronic products will eventually lead to Tesla, which revolutionized the auto industry. The high-performance graphics card generated by the demand of the game industry will become the basis for the fierce competition of AI large models in the future.

Kenneth and Joel also extrapolated this discovery to the daily social and cultural fields, thinking that scientific research, business, artistic innovation, and even life choices can all use this principle as a reference.

In this excerpt, the two authors talked about the failure of goal-oriented thinking in research funding. The state-led grand science and technology plan, whether it is the cancer war led by the U.S. government, or Japan's fifth generation computer development plan, are far from achieving their intended goals. Most of the projects that the scientific community reached a consensus in the project review did not produce innovative results, but they could get unexpected surprises by funding interesting research. Despite such a failure of goal-oriented scientific research investment, most governments still insist on dividing priority research and non-priority research according to the goals of the project, which may have a detrimental effect on the development of science.

Scientists seeking new explorations and discoveries start by raising funds for experimental projects. It turns out that the decision to fund a scientific experiment is often heavily influenced by goal-oriented thinking.

This is a critical issue, as wrong investment decisions can hinder scientific progress and development, with potential societal impacts. In the long run, it's easy to see where the deceptive influence of scientific goals manifests itself.

Intuitively, it would be wiser to invest in science projects if the researchers in the grant proposals lay out clear goals and clearly state what ambitious discoveries will be made when the project is completed. But the lesson we have learned from the image incubator website is that the most interesting discoveries are often unpredictable in advance, so we have reason to believe that non-target (divergent) thinking may also reveal the fundamental nature of the current way of investing in scientific projects question.

Again, advancing science is an interesting example. Unlike the field of education, the field of science is one that is integral to the drive for new exploration and discovery, and where individual failures do not carry high stakes. On the whole, the activities of scientific exploration should be especially suitable for non-targeted exploration. But we shall see that even when occasional failures are acceptable, activities in the scientific field are often deceptively bound by purpose.

Consensus is often the biggest obstacle to innovation

In many countries, including the United States, most research projects are funded by grants from government funding agencies. Such official grants are critical to advancing basic science because they support research that is not yet commercially viable. Of course, much of the scientific research that gets funded fails, because groundbreaking ideas often carry a high risk of failure. Therefore, although some funded scientific research projects will eventually succeed, more projects will fail. This means that research funding agencies such as the US National Science Foundation (NSF) and the European Science Foundation (ESF) need to take a certain amount of risk when making investment decisions in order to hope to promote the most innovative ideas to reality. It would be interesting, then, to study how funding decisions are made by funding agencies for research projects, as we may again face the problem of deceptive and restrictive goals.

The general process of applying for research funding is as follows: scientists submit applications to funding agencies and provide proposals that illustrate scientific ideas; the proposals are then sent to a review panel composed of expert peer reviewers. A senior scientist in a field, such as biology or computer science; the reviewers then give a rating, ranging from poor to excellent. In general, proposals with the highest average ratings are the most likely to be funded.

At first glance, this seems like a very reasonable screening process. Ideally, the best idea in a field should be able to convince a panel of professional scientists and rate it as excellent. However, behind this superficially reasonable common sense, there are also hidden troubles, because the main function of this review system is to support consensus. In other words, the more the reviewer community agrees that the proposal is excellent, the more likely the institution will provide funding. The problem, however, is that consensus is often the biggest obstacle to stepping stones to success.

The problem here is that when people with opposite or different preferences are forced to vote, the winner often does not represent anyone's preferences or ideals (which may explain why people are generally frustrated with political outcomes). Seeking consensus will prevent people from moving along interesting stepping stones, because different people may not agree on what is the most interesting stepping stone. Resolving differences in the preferences of different groups of people often leads to compromises between opposing stepping stones, just as mixing contrasting black and white results in a dull gray.

The product of this compromise often only dilutes the color of the two original ideas in the end. For scientists writing proposals, the best way to win funding is to come up with the perfect compromise, the softest shade of gray — enough to please all eyes, but unlikely to be highly novel or interesting. Therefore, when people try to find consensus in exploration, the whole system does not allow everyone to discover their own stepping stone chain, but compresses various opinions into a stable average.

Perhaps sometimes it makes more sense to favor maximum disagreement, rather than unanimous opinion. Going against the consensus has the potential to be more interesting than the prosaic "agreement". After all, attracting unanimous votes is nothing more than a sign of "following what others say and what they say". If you follow the trend to do popular research, and follow the trend like a parrot, you may be able to get widespread recognition and support; on the contrary, a really interesting idea may cause controversy. On the frontiers of what we currently know and the unknown, there are questions for which there are still uncertain answers, which is why in the uncharted realms of science, expert opinion should remain divided and divergent. We should let the greatest minds of mankind explore the "wilderness" border area between human beings, instead of "indulging in pleasure" in the comfort zone of the greatest consensus.

Think about which project might be more revolutionary: the one rated "mixed" or the one rated "generally positive"? Experts who disagree may be more capable of driving great achievements than experts who always agree.

Of course, this doesn't mean that proposals that were badly rated by all staff should be funded, and if all experts agree that an idea is bad, such as all giving a "poor" rating, then there's no evidence that it's worth pursuing. But when experts fundamentally disagree with each other, something interesting happens.

Darwin's theory of evolution was rejected by many experts when it was first published - which is actually a good sign! As the American historian of science Thomas Kuhn (Thomas Kuhn) put forward the concept of paradigm shift, the existing scientific framework began to appear cracks. At these moments, dissonance of opinion is the prelude to revolutionary subversion. For all these reasons, some of our resources should be used to reward disagreement rather than consensus.

This idea also has a connection to goals, since the basis for reward consensus is goal-directed thinking. In a goal-directed view, the more experts agree that a certain path is worth pursuing, the more people should choose that path. A consensus path is a goal-based choice because people agree on the path's destination. And the amount of consensus among experts provides a measure of the best destinations – a type of goal-based evidence.

If your goal is to seek an idea that tends toward general agreement, then consensus is certainly a laudable ally. This is why, in goal-driven search, the focus is always on the final destination rather than on the fun and novelty of the current stepping stone. This makes goal-based search impossible to be a "treasure hunter". Untargeted search discourages people from ending up on the same path or destination, and only then can interesting ideas attract resources and funding.

At this point, it's good to recall the difference in search behavior between following fun and following purposeful performance. Science is one of humanity's greatest quests, and reaching consensus before deciding what to do next is tantamount to stifling creative endeavors in science. Of course, we're not suggesting that only divisive scientific proposals should be funded, but some of society's resources should be used to support interesting exploration. Exploration in the field of science also needs to uphold the concepts of "treasure hunter" and "stepping stone collector".

Of course, reaching consensus makes sense for certain types of decision-making, but not for creative exploration. We make the point that "disunity" between research groups and within the field of scientific inquiry as a whole can sometimes drive progress. The power of disunity can help us better organize scientific exploration and other creative endeavors.

Only invest in key research areas and scientific research projects with ambitious goals, which will not bring innovation

In addition to driving consensus, goal-based thinking may affect research investment decisions in other ways. For example, assuming you are a believer in teleology, you might think that the framework for scientific progress is predictable. In other words, according to the purposeful thinking of "where there is a will, there is a way," the stepping stones to great discoveries will be arranged in an orderly and predictable fashion.

Under this kind of thinking orientation, it seems that the key innovation to cure cancer should be the improvement or perfection of the existing cancer treatment method, or at least should come from the research field directly related to cancer. Yet, as we see time and time again throughout this book, the stepping stones to great results are unpredictable. Therefore, if we want to cure cancer, only focusing on the field of cancer may not allow us to achieve this ambitious goal. But even when a piece of research fails to achieve its original goals, its by-products can lead to unexpected groundbreaking discoveries in seemingly unrelated fields.

In fact, governments around the world have invested huge amounts of research funds and launched many such key research projects in order to solve some specific scientific problems. For example, Japan's Ministry of International Trade and Industry launched a 10-year-long large-scale research project in 1982, the "Fifth Generation Computer System Project", which aimed to promote Japan's computer technology to the world's leading position.

Although the Japanese government invested heavily in directed research and development, it is widely believed that the program did not achieve its goal of developing a product with the potential for commercial success, although the program did produce a new generation of promising Japanese computer researchers for Japan . Likewise, the "War on Cancer" launched by U.S. President Nixon in 1971 (to eradicate cancer as a high-mortality disease) has not been successful, despite targeted research into the development of more effective cancer treatments , and deepen our understanding of tumor biology. In fact, seemingly unrelated scientific research projects such as the Human Genome Project hold more promise of discovering better cancer treatments.

Of course, sometimes ambitious scientific exploration projects can be successful. For example, the US-Soviet moon landing race in the 1960s was initiated by President Kennedy. To achieve this goal, ten years later, humans will land on the moon in a spacecraft and return safely." But this uncertain declaration was later realized because it was right on the edge of technological possibility (and also In other words, this ambitious goal was only one step away from being realized at that time).

However, potentially misleading conclusions about the power of goals drawn from these success stories often feed naive goal optimism—the belief that any goal can be firmly established and It must be possible. For example, a former president of the American Cancer Society once said: “We are very close to curing cancer, we just lack the will, the funding and the comprehensive planning to put a man on the moon.”

Finally, even in the success stories of these magnificent scientific enterprises, the technologies that ultimately have the most profound impact on human society are often unanticipated. For example, the space race has brought us innovations like cochlear implants, memory foam mattresses, freeze-dried food, and improved emergency blankets.

While these ambitious research projects are clearly driven by goal thinking, they also offer some more subtle implications. A similar line of thought is that there is also a predictable framework for how scientific projects affect the world.

In other words, we may be able to continue to rely on investment to continuously optimize the scientific research projects that currently seem to have the most impact, and eventually some scientific research projects with breakthrough impact will be born. The logic behind it is that moderately influential scientific research projects will lead to more more influential scientific research projects, and ultimately enable scientific exploration and discovery to bring disruptive changes to the world.

According to this logic, another manifestation of goal-driven thinking in the field of scientific research funding is to judge whether it is worth investing based on the importance of the expected impact of a scientific research project. In fact, one of the main criteria that funding agencies such as the National Science Foundation evaluates applications for research grants is the impact scope of the proposed research project. Therefore, scientific research projects that are considered to have low impact potential have a low probability of receiving funding.

The same logic is behind the tendency of politicians to scoff at scientific research with seemingly fanciful goals—research that clearly leads to nothing important—as a waste of money. Behind these examples, there is a very tempting reasoning process, that is, before the research project is carried out, we may classify the research project and its results as important or unimportant projects according to whether they have broad social impact.

Reading this, you may be able to see that this kind of thinking is too arbitrary-because many important discoveries are made by accident or unexpected. Therefore, predicting the impact of scientific research projects is not always feasible, but it will lead us to ignore the important role of chance. Furthermore, even if we could evaluate most scientific research projects in advance and predict their impact in a reliable way, it would not be wise to fund only the most important of them.

The point is that it may be short-sighted to judge a single stepping stone by criteria more appropriate to the system as a whole. Ultimately, the goal of science as a whole is to discover profound and transformative truths. But in the process, it may not matter at all whether any particular research project is transformative. In fact, a scientific research project that is very interesting and can further generate more interesting or unexpected experimental results is perhaps more worthy of attention than its own importance.

One such example is the image incubator website, which as a whole system ended up generating images of alien faces and cars that would be difficult for a single user to complete. The case of novelty search follows the same logic, as a system of exploration, it might find a robot that can traverse a maze, but only if the robot will not be ranked according to its ability to traverse the maze. Such a result.

For this reason, if we accept that stepping stones in scientific inquiry are unpredictable, then "importance" may also be an implicitly deceptive criterion in scientific inquiry. Does a scientific achievement of certain importance necessarily bring about a breakthrough that is closer to transformative?

In other words, in the realm of scientific research, importance is just another broken compass of goals. Because the stepping stones to the most important scientific discoveries may not matter, and the stepping stones to the most disruptive technologies may not show any signs of being transformative.

In the field of science, another way to decide whether to support major projects, or to judge whether projects are worth investing in based on estimated impact, is to use the degree to which scientific research projects meet specific interests as the criterion for investment. Without getting into too much politics, this means that the government only wants to fund research agendas it deems important at the time, or research projects that provide clear short-term benefits to the country.

For example, according to the High Quality Research Act introduced by U.S. Representative Lamar Smith in 2013, before deciding to fund any scientific research projects, the chairman of the National Science Foundation must Issue a statement certifying that the program "(1) is in the fundamental national interest of the United States by advancing scientific advancement to advance the health, prosperity, or welfare of the nation, and to ensure national defense security; and (2) is of the highest quality, groundbreaking, capable answer or address questions of greatest importance to society as a whole; and (3) do not duplicate other research projects being funded by the foundation or other federal scientific agencies".

The assumption behind the second provision is that it is possible or advisable to judge whether scientific research projects are worthy of funding based on their importance, while the first provision assumes that scientific research can only be carried out along the lines of direct interest to the country. Directions are expanded narrowly without a broader search.

Although the bill is unlikely to be passed and implemented in the United States, Canada has already implemented a similar policy. In 2011, the National Research Council of Canada (NRC) began diverting research funding toward economic development at the expense of basic research.

Only 20% of the total budget ended up being spent on fundamental research areas such as "curiosity and exploratory activities," explained NRC president John McDougall at the time. By 2013, the NRC announced it was "opening its doors to research in the commercial field" and focused its funding on 12 "industry-themed entry points." The council claims it is "reinventing itself to support the growth of Canadian industry...all with one ultimate goal: providing quality jobs, increasing commercial R&D activity, achieving greater commercialization, and Building a prosperous and productive Canada."

This apparent shift means that the focus of government investment has shifted away from "basic scientific research that has no direct practical value," and instead has narrowly shifted to research activities consistent with national goals.

Most importantly, the shift itself is not political, but a cross-cutting warning that the wishful application of goal-oriented thinking to "high-minded" scientific research is dangerous.

Of course, the idea that "fundamental breakthroughs in specific and important research areas can be reliably produced as long as a large amount of money is invested" is very attractive, but narrowly framed key research areas and ambitious goal-driven scientific research projects are actually not advisable. Because, whether or not the underlying assumptions are attractive enough, the structure of scientific inquiry doesn't really work that way.

Who can be sure where the next great, commercially viable technology will come from? So the crux of the matter is that aimless exploration, which may sound gloomy, can make the world of science more interesting. There are many interesting and important discoveries waiting to be discovered, but unearthing them requires continuous intellectual investment and an open mind, not simple brute force with a goal.

So we're not saying that scientific progress is impossible in general, but that we don't know what would lead to important scientific discoveries. Just as disunity is surprisingly important to science, it might be wise to invest in seemingly insignificant but obviously interesting scientific experiments. Although this means that we may need to go through many unrelated steps first, following interests rather than narrow ambitions may better reveal the stepping stones to disruptive scientific discoveries and massive economic growth.

"Nowhere to go" is exactly how information collectors operate, how treasure hunters hunt treasures, collect stepping stones, and the right path to get anywhere is the way to the future. "I don't know where the road leads to" is the reason why human beings can create great things. Consensus, the importance of predictability, alignment with national interests—these are all derivatives of goal thinking that only lead us further and further away from what we want as we march toward the unknown.

Fund interesting discoveries that lead to effective innovation

The notion that "disunity" or "unimportance" has some value sounds weird, whereas purpose-driven systems seem perfectly reasonable on the surface. For example, another goal-related criterion when assessing whether research projects are worthy of funding is that reviewers base their decisions on the project's likelihood of success. In other words, the application for scientific research funding must explain the goals of the research project and then be submitted to the reviewers for evaluation. Many research proposals are rejected because the reviewers consider the goals set to be unrealistic or not clear enough. But, given that goals are in any case a broken compass, perhaps the likelihood of success shouldn't always be the focus of the review.

What we want to say is that not all scientific research projects need to set an objective or a research hypothesis. Some scientific research projects are also worth a try even if they are only considered from the perspective of fun.

We may even have no hesitation in funding researchers who have a track record of interesting discoveries, in the same way that the MacArthur Awards provide large sums to highly creative people. Of course, the MacArthur Foundation is not sure where these people's ideas will lead them, and the foundation's approach of "directly issuing blank checks" may also make you feel irrational.

After all, no one knows what these researchers intend to accomplish or how they hope to accomplish it, but the true meaning of scientific research lies in exploring places full of unknowns and uncertainties. If we fail to accept this point of view, then all "stumble upon" paths that do not have a clear purpose may be rejected from the start. However, as mentioned earlier, goals that are too "big" are almost never achieved. Therefore, forcing researchers to state their goals in funding applications only leads them to come up with mediocre goals.

The fear of risk is one of the main reasons why people cling to their goals so tightly. While a certain degree of risk is the price that must be paid for exploration and progress, those responsible for paying the bills generally do not want to take excessive risk lest resources are simply wasted on impractical, whimsical projects.

But our fears do not change the fact that risk is an integral part of scientific inquiry, which requires us to cross many unknown stepping stones over a long period of time. Because we want to go further, a risk-averse goal mindset will limit and constrain our progress.

For example, how many people predicted that advances in consumer electronics would lead to the Tesla Roadster, the world's first production-ready all-electric sports car? However, simply by integrating thousands of lithium batteries for laptops, it is possible to create practical electric vehicles that are lighter and more powerful.

Few discoveries are more surprising than the sudden realization that we are just one step away from some unrealized potential. Achievements that once seemed impossible are suddenly brought into the realm of achievability through previously undiscovered connections. Stepping down seemingly unforeseen dead ends can sometimes help us reap huge rewards.

It is the accumulation of these stepping stones that leads to the greatest innovations in the long run. When every small step of discovery is a revelation, this chain of exploration itself is no less than a revolution. So while betting on a revolutionary discovery may be risky, over time it will come. Revolutionary discoveries, as in the course of all great discoveries, are rarely the goal set by the stepping stones leading to them. Investors have long recognized this principle, even if it is not explicitly stated. In short, if you want to invest in visionaries, look to those who are prowling and exploring in nearby areas of uncertainty.

There is indeed a group of innovators who have somehow seen through the deceptiveness of the goal. For artists and designers, the philosophy behind an idea is often more important than its purpose (if there is one).

Art is often more concerned with creative exploration than with meeting a particular concrete goal. Ask any artist, and he'll tell you that in art-making, it's better to follow the winding path of inspiration than to commit to painting the next Mona Lisa.

Of course, goals do sometimes come into play when art and design collide. In construction, for example, the roof must provide protection from rain, while the foundation must be solid and stable. It turns out that these types of goals share an intriguing parallel with the constraints placed on organisms in natural evolution. Every creature in nature must live long enough to survive and reproduce. But different organisms have a variety of ways to meet this goal, which is reflected in the rich and huge species diversity on the earth.

Thus, rain-proof roofs and stable foundations in architecture are more like constraints on creativity than typical goals in themselves. Just as all living things must be able to reproduce, buildings must be both functional and safe. Innovation in these fields often means finding new ways to work within constraints. However, the overall search in these fields is still advancing into the unknown space.

Looking at the history of art and design, we can easily find examples of stepping stone chains full of drama and chance. For example, in painting, Impressionism begets Expressionism, which begets Surrealism. Great new directions in art are often discovered precisely because they are not the artist's goal.

There are some exploratory steps along the way that negate historical steps, while others redefine or modify steps. But the important point is that no artist tries to predict future changes in the first place, so as to determine or plan what kind of masterpiece he should create. Regardless of the possible consequences, every artistic innovation has its own significance. At the same time, the potential to lead people to novel areas is often the hallmark of effective innovation.

In the current mainstream culture, the idea that progress is primarily driven by rigid goals has influenced all areas of education, science, art, and more. The way we organize most of our work doesn’t seem to escape the illusory comfort of goal thinking.

While non-goal exploration itself is not a panacea, it is best to be soberly aware that blindly trusting goal-based exploration and evaluation often leads to mediocre results and ruts that lead to stagnation. While exploring this world is not easy because of the way it works, at least we know that there is a path that can lead us out of the shackles of a given goal outcome.