It's perplexing how individuals with limited scientific understanding, particularly within the economic and financial sectors, confidently project the trajectory of scientific innovation using shaky theories from their fields. These days, we hear how, like with most things in economics, the benefits of larger models are set to decrease with the same level of additional data. Centuries-old "Scaling Law" has been oft-discussed by some to herald the peaking of GenAI. The latest news about development struggles coming out of one or more GenAI firms provides the perpetual doomers and skeptics with the newest tool to predict the demise. The reasonable-sounding arguments draw parallels from the economic theory of marginal utility to predict the plateau.
As compelling as these may be, the reality is that such attempts to map out the future of science ignore the complexities and serendipitous breakthroughs that defy simplistic economic models, revealing a disconnect that does a disservice to both fields. As Thomas Kuhn argued, scientific progress occurs through paradigm shifts rather than a steady, logical progression. These shifts represent radical changes in the fundamental concepts and experimental practices of a scientific discipline, and they are often unpredictable and cannot be forecasted by existing theories.
Since their start, transformer models have defied predictions and expectations in abilities. There was no logical reason for them to figure out almost all human languages and much more with one set of equations. So, their long list of unpredictable abilities may have reached an absolute end at the current data level, as the Scaling Law believers seem to suggest confidently. There is a better path to rely on when considering investments than such arbitrary guesses. For this, it is better to discuss this through some examples.
Back to Basics: Innovation Beyond Linear Expectations
Unlike the modern cadence of scheduled tech product launches or app upgrades, breakthroughs in the physical, material, and biological sciences are a journey through uncharted territory. It cannot be willed into existence or neatly plotted on a corporate timeline. Breakthroughs emerge from a confluence of relentless effort, open-minded exploration, and often, serendipity—the unexpected discovery of something valuable not initially sought. This element of unpredictability was less prevalent in the applications realm until the emergence of GenAI. In other spheres, the role of serendipity offers vital lessons, reminding us that GenAI models do not follow the predictable trajectories of, say, successive iPhone releases.
Case Study: SK Hynix and MR-MUF Technology
A striking example from recent years is SK Hynix's development of the Modified Resin Multi-Layer Underfill (MR-MUF) technology. While industry giants like Samsung focused their resources on other promising, although more conventional, technologies for advanced chip packaging, SK Hynix forged a new, risky path. Initially overlooked by competitors, the MR-MUF technology eventually proved to be a game-changer in the highest-end memory manufacturing, without which today's GPU abilities would be a fraction of what it is.
Analysts and media pundits are quick to blame Samsung's management for allowing SK Hynix to seize dominance in a field Samsung once ruled unchallenged. Yet, they conveniently overlook the role of chance factors inherent in innovation. It's too easy for non-scientist financial writers lacking technical expertise to pontificate what Samsung should have done differently. While some of their critiques might hold merit, they dismiss the crucial impact of serendipity. When different laboratories and teams take varied paths to discover what might work in material sciences, success doesn't necessarily favor the biggest name or the most renowned organization.
Certainly, SK Hynix's success was not merely a fortunate accident but the result of unwavering dedication to an innovative approach that others dismissed. However, had their strategy not yielded the desired results, they might have been criticized for their stubbornness. The best way to see is to study a case where boldness failed.
Case Study: Unity Biotechnology's Pursuit of Anti-Aging Therapies
A few years ago, Unity Biotechnology embarked on an ambitious mission to combat aging-related diseases by targeting senescent cells—cells that have stopped dividing but refuse to die, contributing to tissue degeneration. Their bold approach centered on developing senolytic drugs designed to eliminate these cells and rejuvenate tissues. Unity's lead candidate, UBX0101, aimed to treat osteoarthritis by clearing senescent cells from the joints. Despite a solid scientific rationale that appeared ironclad to many in the investment community and significant investment, the drug failed to demonstrate efficacy in phase II clinical trials in 2020. The company's stock plummeted, and they had to pivot their focus to other programs. Unity's journey underscores how a logical and promising scientific strategy can still lead to unforeseen setbacks.
This example highlights that even with sound logic, robust research, and ample resources, success in innovation is never guaranteed. While we rightly celebrate the victors who achieve groundbreaking advancements, it's crucial to acknowledge that failures are intrinsic to those who pursue bold paths. Presuming success or failure based solely on logical analysis overlooks the complex interplay of factors influencing innovation—including chance, timing, and unpredictable challenges. It's a reminder that the path of discovery is fraught with uncertainties, and serendipity often plays a pivotal role in determining outcomes.
Case Study: Quantum Computing's Uncertain Trajectory
It's usual for nearly two decades for some financial experts, often armed with nothing more than a superficial understanding of terms like "entanglement"—and sometimes not even that—to boldly predict the imminent revolution of quantum computing. Their presentations are filled with charted technological timelines and investment strategies squeezing quantum mechanics to conform to their linear development models. Yet quantum computing, grounded in principles that defy conventional logic, has not progressed along any neat paths envisioned.
The inherent complexity of manipulating quantum phenomena and the fragility of quantum states have presented formidable challenges. While there have been undeniable advancements in qubit technology and algorithm development, the road to practical, fault-tolerant quantum computers has been longer and more arduous than anticipated at every stage. Disappointments have arisen as certain technical milestones proved more elusive than predicted, and now there is an additional problem of "rivalry."
A recent article from the MIT Technology Review (titled "Why AI Could Eat Quantum Computing's Lunch" https://geninnov.ai/s/hoNMxG) has some experts questioning whether we will need quantum computing given the advances in transformer methods. This provocative thesis challenges the prevailing narrative of quantum computing's inevitable dominance. While the article's conclusions remain contested, it serves as a valuable reminder that technological progress is not only non-linear but also subject to unexpected breakthroughs in parallel fields.
Case Study: Emergent Abilities in Transformer Models Through Scaling
When toying with individual Lego blocks, it is impossible to see how they can come together to give an impression of a small house, a car, or even a working robot. Similarly, In the world of artificial intelligence, scientists have found that increasing the size of transformer models (which are like the Lego blocks of AI) and training them with more data can lead to unexpected new abilities. When scaled up, these models begin to exhibit emergent properties—new features or behaviors that weren't directly programmed but appear when the system becomes sufficiently complex.
One example is how large language models like GPT-3 began to understand and generate human-like language far better than smaller models. As researchers added more layers (think of layers like pages in a book) and fed the model more text from the internet, the AI didn't just get better at tasks it was already doing—it started doing things nobody anticipated. It could write essays, translate languages, and even solve math problems, all without being specifically taught to do so.
This phenomenon highlights an important lesson about innovation: Sometimes, entirely new and valuable properties emerge by simply pushing the boundaries of size and complexity. Or they may not for a long while or forever. Different from everything in programming until these models, the designers of the GenAI models have little visibility into what they are likely to get when they stuff their equations with more data.
Conclusion: Embracing the Unpredictable Path of Innovation
The journey of GenAI, much like the broader history of scientific discoveries, underscores the limitations of linear predictions and the importance of embracing uncertainty. While the pursuit of ever-larger models may yield diminishing returns or encounter unforeseen hurdles, the exploration of alternative approaches, such as post-transformer architectures, smaller specialized models, and hybrid systems, holds immense potential. The unexpected emergence of capabilities in transformer models through scaling serves as a powerful reminder that technological advancements often arise from pushing the boundaries of what's possible, even when the path forward is completely dark.
Habitually and for various business, investment, or other reasons, all of us will often resort to predicting the path of various innovation drives - like whether Samsung is now a fallen angel forever, when GenAI models will stop improving, or when the first mass applications of Gene editing may emerge. Yet, it's crucial to maintain humility and honesty in our predictions. Those who don't try anything are sure to stagnate, but this does not mean those with bold visions and ambitious goals will definitely succeed. The talking heads in our fields will vax eloquently on when Singularity will come or cars will fly, but real life does not owe anything to us to emerge in any particular way, and this is most true in the fields of innovation.
