必要なのは... (r)進化!?

This is just an opinion of what I experience and am witnessing, but looking at how LLMs scale feels like I've seen it before: with CPUs trying to outrun Moore's Law and break the rules of physics. Heat, power leakage, and diminishing returns made it increasingly expensive to squeeze out even small gains in clock speed. The GHz race shifted because it had to. For LLMs, more compute, more data, more parameters, and everything just keeps getting better? That curve seems to hit a ceiling and innovation needs to succeed the scaling race now. History does not repeat itself, but it rhymes. What learnings can we make from history to "predict" a potential future? History In the early 2000s, CPUs ran into a wall, a very physical one ^^ So makers adapted. Instead of crunching every single watt out of a single core, multi-cores became common. Athlon 64 x2, Pentium D, PS3 with its heavy Cell approach. From linear to parallel. From sequential to multi-threaded (and funny race conditions ;). Talks of distributed systems, SIMD/MIMD and new benchmarking spawned into what we have today. We still use CPUs, but differently. We still have Memory, but think about Cache, RAM, GPU or Unified. Same same, but different. Innovation because of limitation. Present I feel something similar is about to happen to gen AI. Yes, there are improvements in different areas, some in scaling, some optimisation, some performance, but the slope is becoming slippery. The last 12 months went from "Opus 4.5 is the pinnacle" to "What the hell is wrong with Claude?". The perfect (business) storm of scaling execution! But the low-hanging fruits have been eaten and the crops don't grow as fast anymore. Costs rise quickly, latency becomes a constraint, and even large context windows feel more like extensions than breakthroughs. What remains is more incremental, more expensive, and more complex. You could argue the whole venture of "agents" is the same multi-core experience repeating itself. A different kind of orchestration layer helping out. But I don't think it's the same. In the sense described by Russell and Norvig, an agent has a notion of perception, state, decision-making, and action under uncertainty. What we have instead are structured loops: prompt a model (often the same!), call a tool, reflect on the result, and repeat. These systems are useful, sometimes impressively so, but they are still closer to imperative programming wrapped around a single reasoning engine than to true autonomous entities. It feels less like a new paradigm and more like writing a scheduler for a very peculiar kind of processor. Future? To head into speculative territory: instead of building one increasingly large model, we might move toward systems composed of many models. Not just replicas, but differentiated components with distinct strengths, "biases", and roles. Systems that can disagree, verify each other, and converge on better answers through interaction rather than scale alone. No 10x Claude Opus arguing with endless pre-prompted skill markdowns but still the same network. Real domain-specific, "biased" networks for art, code, language, economy... coexisting and connected via a common world ontology, to create something like Claude Machiavelli (joking ^^). To make this viable, (agent) orchestration alone is not enough. These systems would need a shared structure, some form of explicit or semi-explicit ontology that allows them to reason about the same world in compatible ways. There are initiatives in a similar direction and to me it starts looking more like a brain. Repeat Multi-core development was messy in the beginning. There is a reason "Can it run Crysis?" did not age well coming from heavy single-core into a multi-core environment. And similar might happen for agents, coding and more. Distributed databases, multi-threading programming, catching all those free-running threads... a shift away from "just scaling" may introduce complexity before it delivers clarity. In hindsight, everybody will have known upfront, have seen the obvious. I don't know... It is entirely possible that this is not how things will unfold. Larger models might still have another leap. New architectures beyond transformers could reset the curve (Google, where are thou?). What we currently call agents might evolve into something much closer to actual agency. But as "All You Need Is Attention" is approaching 10 years since an idea sparked a revolution, maybe a decade is enough for a new paradigm to emerge, peak, and repeat... This is just an opinion of what I experience and am witnessing, but looking at how LLMs scale feels like I've seen it before: with CPUs trying to outrun Moore's Law and break the rules of physics. Heat, power leakage, and diminishing returns made it increasingly expensive to squeeze out even small gains in clock speed. The GHz race shifted because it had to. For LLMs, more compute, more data, more parameters, and everything just keeps getting better? That curve seems to hit a ceiling and innovation needs to succeed the scaling race now. History does not repeat itself, but it rhymes. What learnings can we make from history to "predict" a potential future? History In the early 2000s, CPUs ran into a wall, a very physical one ^^ So makers adapted. Instead of crunching every single watt out of a single core, multi-cores became common. Athlon 64 x2, Pentium D, PS3 with its heavy Cell approach. From linear to parallel. From sequential to multi-threaded (and funny race conditions ;). Talks of distributed systems, SIMD/MIMD and new benchmarking spawned into what we have today. We still use CPUs, but differently. We still have Memory, but think about Cache, RAM, GPU or Unified. Same same, but different. Innovation because of limitation. Present I feel something similar is about to happen to gen AI. Yes, there are improvements in different areas, some in scaling, some optimisation, some performance, but the slope is becoming slippery. The last 12 months went from "Opus 4.5 is the pinnacle" to "What the hell is wrong with Claude?". The perfect (business) storm of scaling execution! But the low-hanging fruits have been eaten and the crops don't grow as fast anymore. Costs rise quickly, latency becomes a constraint, and even large context windows feel more like extensions than breakthroughs. What remains is more incremental, more expensive, and more complex. You could argue the whole venture of "agents" is the same multi-core experience repeating itself. A different kind of orchestration layer helping out. But I don't think it's the same. In the sense described by Russell and Norvig, an agent has a notion of perception, state, decision-making, and action under uncertainty. What we have instead are structured loops: prompt a model (often the same!), call a tool, reflect on the result, and repeat. These systems are useful, sometimes impressively so, but they are still closer to imperative programming wrapped around a single reasoning engine than to true autonomous entities. It feels less like a new paradigm and more like writing a scheduler for a very peculiar kind of processor. Future? To head into speculative territory: instead of building one increasingly large model, we might move toward systems composed of many models. Not just replicas, but differentiated components with distinct strengths, "biases", and roles. Systems that can disagree, verify each other, and converge on better answers through interaction rather than scale alone. No 10x Claude Opus arguing with endless pre-prompted skill markdowns but still the same network. Real domain-specific, "biased" networks for art, code, language, economy... coexisting and connected via a common world ontology, to create something like Claude Machiavelli (joking ^^). To make this viable, (agent) orchestration alone is not enough. These systems would need a shared structure, some form of explicit or semi-explicit ontology that allows them to reason about the same world in compatible ways. There are initiatives in a similar direction and to me it starts looking more like a brain. Repeat Multi-core development was messy in the beginning. There is a reason "Can it run Crysis?" did not age well coming from heavy single-core into a multi-core environment. And similar might happen for agents, coding and more. Distributed databases, multi-threading programming, catching all those free-running threads... a shift away from "just scaling" may introduce complexity before it delivers clarity. In hindsight, everybody will have known upfront, have seen the obvious. I don't know... It is entirely possible that this is not how things will unfold. Larger models might still have another leap. New architectures beyond transformers could reset the curve (Google, where are thou?). What we currently call agents might evolve into something much closer to actual agency. But as "All You Need Is Attention" is approaching 10 years since an idea sparked a revolution, maybe a decade is enough for a new paradigm to emerge, peak, and repeat...