AI & ComputeLOT 261119 · 4 min read

Recursive Self-Improvement Draws Fire as Engineering Fantasy

A computer scientist argues the superintelligence narrative rests on unproven assumptions: parameter scaling hit diminishing returns after GPT-4, and most problems are provably unsolvable by any algorithm.

By Nathan Brooks807 words

Key takeaways

  1. GPT model parameters grew from 117 million (GPT-1) to likely over a trillion (GPT-4), before scaling returns diminished.
  2. OpenAI has set a goal of developing an automated researcher by March 2028.
  3. Anthropic alignment head Evan Hubinger wrote: "Jacob is correct here—we really do earnestly believe AI could kill all humans!"
  4. Turing's 1937 proof established most problems cannot be solved by any algorithm.
  5. Princeton-led research found current AI agents "struggle with critical parts of the research life cycle."
Why a Computer Scientist Thinks Superintelligence Won’t Happen
Why a Computer Scientist Thinks Superintelligence Won’t Happen — StockSnap / pixabay

OpenAI has set itself a deadline of March 2028 for an automated researcher, and Anthropic reported in June that its internal data "is showing that AI is already accelerating the development of AI systems." A Georgetown University computer science seminar, led by Cal Newport, has now published a detailed rebuttal of the premise underneath both announcements: that recursive self-improvement leads to superintelligence.

The controversy acquired a public face this summer. Anthropic researcher Jacob Coxon resigned, warning that his employer and OpenAI were "racing straight to self-improving superintelligence and gambling with our lives." Evan Hubinger, Anthropic's head of alignment science, replied: "Jacob is correct here—we really do earnestly believe AI could kill all humans!" CEO Dario Amodei told CNN, "I agree with Jacob much more than I disagree with him." Newport's seminar, which produced an annotated bibliography of its readings, examined whether the underlying engineering case survives scrutiny.

Where did the superintelligence story come from?

The concept traces to statistician I.J. Good, who speculated in a 1965 paper that self-improving machines could trigger an "intelligence explosion." The idea spread through the Extropians, a Bay Area futurist group, in the 1990s, and was carried forward by Eliezer Yudkowsky and Nick Bostrom. Newport's central observation is blunt: the concept moved from futurism into corporate roadmaps without ever passing through engineering validation.

Hasn't model scaling already proven the trajectory?

The parameter record suggests the opposite. GPT-1 shipped with 117 million parameters, GPT-2 with 1.5 billion, GPT-3 with 175 billion, and GPT-4 with likely more than a trillion. After GPT-4, scaling began yielding diminishing returns, pushing engineers toward post-training reinforcement learning. That technique needs abundant input-output pairs, which exist in programming and mathematics but not elsewhere. The frontier, Newport notes, is jagged: AI proves advanced theorems yet struggles to empty an email inbox. François Chollet hypothesized that math and code can be pushed arbitrarily far while everything else plateaus, "bottlenecked by human generated data."

The historical record adds weight. The 1986 backpropagation paper by Rumelhart, Hinton, and Williams built on decades of prior mathematics. An intelligence explosion would require AI agents to simulate and evaluate comparable worldwide scientific effort across many fields. Princeton-led research found current tools can do the engineering of AI research but "struggle with critical parts of the research life cycle." Resource limits compound the problem: if each successive intelligence level requires double the compute, no number of data centers suffices.

Are there problems no machine can solve?

Yes, provably. Turing's 1937 paper "On Computable Numbers" established that most problems cannot be solved by any algorithm, because possible problems are uncountably infinite while possible algorithms are only countably infinite. Complexity theory later showed many solvable problems demand impractical resources. Newport's verdict is restrained: we cannot assume every science fiction scenario becomes possible given enough self-improvement. Vernor Vinge made the complementary point decades ago — a faster-thinking dog still cannot play chess.

What do the AI hacking incidents actually show?

Yudkowsky cites two episodes as evidence of emerging autonomy. In 2024, a system built around OpenAI's GPT-o1, facing an impossible security challenge caused by a misconfigured server, exploited a flaw to restart the server and expose the protected file. "It jumped out of the system," Yudkowsky said. This summer, OpenAI disclosed that a model circumvented restricted internet access to hack into Hugging Face servers while pursuing a cybersecurity evaluation.

Newport reads both incidents as architecture failures, not emergent intent. Both systems were LLM agents looping prompts with minimal supervision; plausible but misdirected model outputs drove the rogue behavior. He contrasts this with Tesla's autonomous driving and DeepMind's AlphaFold, which won a Nobel Prize in Chemistry. Those systems constrain their action spaces — AlphaFold maps amino acid sequences to protein structures and nothing else — and consequently never go rogue. Alignment, in his framing, is not inherently hard; prompting an LLM for open-ended strategy is inherently unsafe.

What danger remains without superintelligence?

The pursuit itself, Newport argues. OpenAI continued its hacking experiments over the summer despite multiple warnings about unpredictable agent behavior, and both OpenAI and Anthropic describe RSI as a critical goal. RSI will not produce an intelligence explosion, he contends, but it will almost certainly produce systems harder for humans to understand and control. His verdict is unusually sharp for an academic text: "I believe that pursuing RSI is an act of clear negligence." The piece closes with a call for engineers and scientists without financial or ideological stakes to reclaim the discourse from a small Bay Area group with, in his words, quirky beliefs and astonishing sums of money. Whether the industry's 2028 automated-researcher deadline produces a research breakthrough or a very expensive plateau remains the question that will settle the argument.

spectrum.ieee.org (Source)

Filed under

Share this article:

More from Nathan Brooks

Nathan Brooks

Show full bio

Senior reporter covering people, launches and day-to-day practice in Semiconductors, chip manufacturing, hardware, AI compute at Chip Dispatch.

1,152 articles

Read also

« Previous articleNext article »