I interviewed 25 researchers across OpenAI, Anthropic, Google DeepMind, Meta, Princeton, UC Berkeley and Stanford on recursive AI improvement – where they agree, where they disagree, and what we should do about it. Those closest to the frontier of AI capabilities are converging on questions that next to no one in Washington is yet tracking. The full analysis is here.

Twenty of twenty five researchers identified automating AI R&D as one of the most severe and urgent risks from AI systems, because AI capabilities could improve much faster than our ability to oversee, steer or govern them. Roughly half of my sample were researchers at frontier AI companies, and I deliberately sought out skeptics alongside believers and tried to give their arguments full weight. I conducted these interviews because I was concerned that someone, or possibly the systems themselves, could gain a decisive, uncatchable edge.

Not long ago, recursive AI-improvement, the idea of using AI to improve AI itself, was the stuff of science fiction. Today it has become an explicit organizational goal of the leading AI companies (OpenAI, Google DeepMind, and Anthropic). Recent systems are solving problems researchers once thought required genuine novel reasoning, like olympiad-level math problems, and the goal of recursive-self-improvement is becoming less farfetched.

When researchers were able to articulate a threshold, they repeatedly gave one: the point at which AI can do the work of today’s AI researchers. Past that, a system could begin improving its own successor. The length of tasks AI can autonomously complete has been doubling approximately every seven months. Many expect this trend to continue until AI systems can do the job of their human creators (AI researchers), thus triggering recursive self-improvement.

The participants kept citing the “METR Task Horizon Benchmark,” as one of the best ways to measure how much independent work AI can sustain. The benchmark is an unbounded measurement of AI performance in terms of the length of tasks AI agents can complete (e.g. an AI can complete a software task requiring 1, 2, or 10 human-hours).

Skeptics and Believers

The field of artificial intelligence has overpromised in the past, and AI companies do have an incentive to exaggerate the power of their products. The most common reason for skepticism among interviewees was the belief that general intelligence required a discontinuous breakthrough; something that wouldn’t emerge from gradual improvement alone. However, skeptics disagree on what this missing breakthrough is. For example, some said a breakthrough is needed on “creativity,” “taste,” or the ability to generate genuinely novel ideas. Others argued that the ability lacking had to do with how well AIs could verify true hypotheses from false ones. One explanation ‘creativity’ might not come from scaling is that paradigm shifting ideas are hard to create data for, even veteran humans cannot reliably pick them, and they have no verifiable reward.

On the other hand, many of the most influential AI researchers on earth believe recursively-improving AI is within sight in the coming years. Half of those interviewed claimed there already exists a relatively clear, continuous trajectory towards automating AI research. At companies like OpenAI and Anthropic, recursive self-improvement was reported as regular company-wide discussion up to lead researchers and CEOs (Sam Altman, Dario Amodei, Demis Hassabis, Elon Musk). Outside of the leading companies (e.g. in academic settings), participants reported that discussions are less frequent and face more skepticism. Even though discussions may be less common in academia, they are still happening. For example, ICML 2026, one of the largest and most prestigious machine learning conferences, is hosting a workshop called “AI with Recursive Self Improvement.”

Some likely explanations for the schism between leading companies and academia include:

• Selection effects, wherein ‘deep believers’ leave academia for private labs (e.g. OpenAI) offering enormous salaries, huge research budgets, and moonshot thinking. “I think the largest difference is just having first-person experience of how fast things have gone inside the labs,” said one participant who described the visceral feeling of exponential improvement felt at a leading company.

• The skeptical-leaning participants pointed out that frontier labs, who are beholden to investors rather than academic reviewers, are incentivized to over-promise and exaggerate their capabilities.

What Recursive AI Improvement Looks Like

I asked participants about noteworthy visible milestones towards recursive AI improvement and asked them to illustrate what they expect in the coming years.

Researchers suggested a variety of concrete demonstrations or capability milestones to monitor. For example, top performance on math olympiad questions, or an AI system training and deploying a machine learning model by itself. A widely-loved metric was the aforementioned task horizon benchmark. The variety of milestones is discussed at length in the full report.

While researchers disagreed on the precise timelines, risk profiles, and preferable governance approaches, a consistent story emerged on what’s coming:

1. Research Speedup Tool Phase: researchers expect AI systems that are very good at coding to improve, such as Claude-code or Codex (coding assistants) but still require human oversight. Anthropic reports that their engineers write 8x as many lines of code than they would without existing research speedup tools, but notes this likely doesn’t yet translate to 8x productivity.

2. Collaborator Phase: where AI models are good enough at machine learning to meaningfully contribute to scientific discovery. Interviewees expected humans to guide high level research goals, but allow AI assistants to make design decisions and pose new problems.

3. Full automation phase: AI systems independently execute complete research cycles that drive AI research and algorithmic progress. The key characteristic of this phase is that human oversight becomes a bottleneck – such that removing human guidance, suggestions or oversight improves the results.

After this point, predictions diverged and the various camps (skeptics, believers, optimists, or pessimists) became more evident. Pessimists argue that once removing human oversight improves results, the incentive is to remove humans from the loop entirely. Optimists argue humans could still set goals and benefit from the gains (such as faster science or improved medicine). But many in both camps nonetheless expect systems improving themselves faster than anyone can evaluate– which is, by construction, out of human control.

AI progress in AI R&D capabilities might outpace progress in other domains, this is because:

1. Programming problems are verifiable. You can automatically check whether code works; it runs or it doesn’t. This means AIs can be trained and tested on this data much more easily. On the other hand, success in music, literature, or art is largely subjective. To compound this effect, the engineers building AI systems are better fit to evaluate AI research than they are to evaluate subjective data, such as poetry.

2. It is the number one priority of leading labs. For instance, OpenAI’s Chief Scientist Jakub Pachocki has stated that one of OpenAI’s main priorities is to “automate scientific discovery,” with a plan to build automated researchers improving AI. (source)

3. The flywheel argument: improvements in AI capability directly improve the tool used to make further improvements, potentially leading to positive feedback, which isn’t true of other technologies.

Recursive AI improvement may endanger us

The most common concern researchers raised wasn’t necessarily a specific harm RSI directly implies, but rather what RSI does to every other risk and our ability to mitigate them. 18 of 25 researchers described it this way: “It just speeds up other threat models.” It compresses the time humans have to respond while simultaneously accelerating the pace of dangerous capabilities. Rapid progress means wider, more powerful access to the chemical, biological, and cyber harms AI already enables, and less time to react. I flagged 17 of 25 transcripts with concern over “adaptation lag,” a widening gap between how fast AI capabilities advance and how fast human institutions can understand and respond to them. Companies currently face market pressure to create AI systems capable of AI research for a competitive advantage, regardless of whether they are able to do so safely or maintain meaningful oversight.

Six researchers told me they expected a winner-take-all dynamic. The first company, government or AI itself to achieve recursive improvement could use it to pull permanently ahead of everyone else. As one put it, that entity gains “permanent control over the future of AI development.”

With this said, the interviews were conducted by a single interviewer, me, and my sample may suffer selection bias. Perhaps those willing to engage in a study like this are more concerned on average. How close we are, whether recursive AI improvement involves positive feedback, and the policies that work without backfiring are all open questions. For example, 16 of the interviewees expressed skepticism at some point in their transcripts about positive feedback and intelligence explosions specifically.

Internal Deployments

Of the 20 participants who clearly addressed what they expect AI companies to do with AI-research-capable models, only 20% expected this type of model to be deployed as a publicly available product. One participant said, “internal-only deployments might happen, and that is a big risk factor, because [the public] just has less information.” This prediction seems reasonable, AI companies have already kept AI models internal for various reasons. For instance, OpenAI reported spending six months on safety research, risk assessment, and iteration before the public knew of GPT-4. Similarly, the US government recently suspended access to Anthropic’s Claude Mythos model.

A common reason researchers expect internal-deployments is that keeping models internal offers its developers serious advantages by accelerating their own R&D efforts ahead of competitors. There could be an incentive flip, they explained, such that when AI systems can meaningfully speed up AI research, keeping AI systems internal becomes more valuable than selling them. Some envisioned a race to secure a durable lead over competitors. This would contradict one of the leading motivations given in the founding of OpenAI: to prevent one actor from getting an uncatchable AI advantage.

On the other hand, participants identified arguments and pressures that would promote diffusion of AI capabilities, including economic pressure to commercialize AI capabilities, insiders leaking milestones, and a culture of boasting about capabilities.

The researchers I interviewed split into three camps on deployment. The largest (50%) expected frontier AI companies to keep their most capable models internal. A minority (20%) expected full public release. The remainder envisioned some hybrid, such as AI companies selling access to a distilled model publicly while keeping their most powerful models, possibly with less guardrails, for themselves. One explained, “They’ll train a base model, then they won’t release that model, not only because it’s not economical but also because it risks distillation, but they will distill it themselves to cheaper [public] models.” Another nuanced vision suggested that different actors have different incentives, for instance Meta’s open-weight culture might incentivize them to release their models and breakthroughs more publicly from OpenAI or Anthropic.

What should we do?

Interviewees were split on red lines and governance. One problem: the more precisely you define a threshold, the easier it is to measure and enforce, but the more it diverges from an abstract risk with large degrees of uncertainty. However, practically everyone agreed on various measures to increase visibility (e.g. how aware the US government and general public are of AI developments, especially given they might be kept internal), and capacity (the government’s ability to forecast AI capabilities and develop competent safeguards during rapid AI progress.

Below are some recommendations that may help to these ends:

1. Convene public hearings specifically on recursive AI improvement and associated risks. Require testimony from CEOs and senior researchers at OpenAI, Anthropic, Google Deepmind, xAi, and possibly other leading companies who work on these systems daily. Researchers report regular internal discussions about automated AI research and see internal capabilities months before the public. The gap between what frontier labs understand and what Washington understands is itself a governance failure.

2. Track the threshold: Direct the Center for AI Security and Innovation (CAISI) to maintain a government-run task-horizon benchmark and publish recurring capability forecasts. Congress should not learn about transformative capability shifts from press releases or unmandated nonprofits. CAISI could also run an anonymized standing interview program with frontier researchers, giving the government foresight into what scientists are observing and expecting, separate from what their companies say in public.

3. Fund treaty verification research capacity to prevent loss of control from RSI: The US cannot credibly propose, enter, or enforce any agreement (e.g. a nonproliferation agreement with China) without mechanisms to verify adherence to those agreements. Whether a lab has crossed a capability threshold or honored a commitment to halt a training run, verification requires technical infrastructure that does not yet exist at scale. This field is underfunded and undeveloped; but without it, any limits are practically unenforceable.

Conclusion

The researchers closest to this technology typically do not debate whether recursive AI improvement is possible; they debate timelines, speed, mechanisms, and what to do. This consensus has not yet reached Washington. The most powerful AI systems may be withheld by the government or confined within the leading company, pointed at the one task they care about most, while the public is left in the dark.

Perhaps the believers are wrong. I am an AI researcher and I deliberately sought out skeptical researchers and believers at the top of my field. There are points of skepticism and I have tried to do justice to them here and in my paper. However, there are points of consensus beginning to emerge and they are urgent; policymakers should be aware of this.

Over the last few weeks I’ve read the Constitution, Declaration of Independence, and many of the Federalist Papers. These documents are devoted to preventing failure modes of governance: concentration of power (e.g. a small faction locking in control and suppressing opposition), ideological fanaticism, and foreign powers destabilizing and destroying a nation from within. These failure modes are the very motivations for the United States: the separation of powers, federalism, checks and balances, and the rule of law. Concentrated power has been a default throughout history, and improbable arrangements like democracies may need constant pressure to avoid collapse into this mode. Just like the energy exerted by a cell to maintain its barrier, to avoid succumbing to entropy, we may have to push hard to preserve what we love. At the same time, we’ve become so successful that most of us don’t even remember what it is we’re fighting for.

We may soon develop AI that could do most tasks that humans do (and probably therefore produce most of the value in the economy), make decisions, accelerate R&D, and generally out-think humans. Perhaps soon after, civilization will ascend towards the stars. It is very difficult to predict precisely what this will look like. Moreover, I do not know which values, principles and governance protocols we ought to take with us. But among the civilizations on offer, amid democratic backsliding abroad and even at home, American democracy remains the least bad option we have.

What We Have to Lose

One year ago today, Peter Wildeford wrote a blog post that steered my politics. He said:

The work of the Declaration is incomplete. The world faces a renewed contest between freedom and tyranny. Authoritarian powers are continuing to perfect digital surveillance states and launch new wars of conquest. Most concerningly, this is all happening while many Americans struggle to see the difference between their own flawed democracy and the totalitarian alternatives rising across the globe.

This is more than just an unfortunate confusion and failure of perspective. Such “whataboutism” and false equivalence is a threat to the very foundations of liberal democracy when clarity about our values matters most.

Since Peter wrote those words, the Iranian state has executed at least 2,159 people. This was followed by a documented mass killing of protesters this January, with death tolls estimated between thousands and tens of thousands. During the protests, the Islamic Republic cut the entire country off from the internet for 5 months to conceal killings and prevent coordination.

This year, Russia has doubled down on its unnecessary war of aggression against Ukraine. Many reliable estimates put the cost of the war at two million casualties thus far. Hundreds of thousands of men my age, who did nothing wrong besides being born in the wrong country, have been killed in the name of territorial gain and “denazification.” At home, Putin has transformed Russia’s government into a mafia state. Opposition leaders face imprisonment or assassination. Independent media has been destroyed. The government bans access to Western websites (e.g. Google, ChatGPT, etc.) to keep its people in the dark and distort the nature of the conflict. To maintain the great lie, anti-war sentiments can land Russians years in prison.

In China, the Chinese Communist Party (CCP) hones the efficiency of the most advanced surveillance apparatus ever built. The CCP also engages in cultural genocide, while using a great information firewall to prevent any information from coming in or getting out about it. According to the Financial Times:

• ~90% of children are taken from parents to be educated in boarding schools where they aren’t allowed to speak Uyghur

• Officials monitor who eats during Ramadan and report those who skip meals

• Basic items like prayer mats and religious texts are considered illegal contraband

• Adult Uyghur women are pressured to marry Han Chinese men, and there are official goals to sterilize a certain % of them

• The CCP has shut down all existing publishers that publish in the Uyghur language

In each of these countries, a single party, fanatical ideology, or autocrat has concentrated power, suppressed opposition, sealed the information ecosystem and entrenched power — succumbing to the failure modes that motivated the founding fathers.

These states are built on vastly different ideologies: fundamental Islamism in Iran, imperial ethno-nationalism in Russia, CCP party supremacy in China. We might expect these states to operate very differently from one another, yet they share striking similarities. Ideological fanaticism often has recurring characteristics:

• epistemic and moral certainty, as seen in the Iranian regime’s divinely inspired theocracy, which cannot, by definition, be wrong

• extreme tribalism dividing humanity into a sacred “us” and an evil “them”

• a willingness to use whatever means necessary, including brutal violence

These states also lack principles we value in liberal democracies — mutual toleration (accepting your political opponents as legitimate), institutional forbearance (self-restraint in using legal powers), and free elections. Russia and China are continental empires in the oldest sense, powers that grew by swallowing and assimilating their neighbors; and neither recognize a minority’s right to remain distinct. In contrast, judicial review, supermajority amendments, and the separation of powers were designed to protect minorities from a tyrannical majority.

The authoritarian leaders of the world feed on Americans forgetting our own story. They hold stakes in American polarization and social instability, the largest predictor of democratic backsliding.

States like Russia invest enormously in our social instability; the Russian Internet Research Agency deliberately amplifies the most divisive issues in American politics and supports radical groups, because chaos is easier to spread than Russian propaganda. In 2016, for example, a pro-Muslim rally and anti-Muslim counterprotest were organized to occur simultaneously, both coordinated from Saint Petersburg. Cognitive warfare is part of Russian military doctrine; Putin believes wars are won/lost in minds, not on the battlefield. He will have won when we believe nothing, don’t trust one another, and can no longer tell the difference between our flawed democracy and autocracy. We will have fully lost the future when we’ve given up on the ability to correct our government at all.

America, much more so than China, has the ability to self-correct. The examples critics use to draw a false equivalency between the governments turn out to be evidence of this. Innumerable executive branch scandals, including the My Lai massacre, the Pentagon Papers, the abuses at Abu Ghraib, and the 2013 Snowden leaks, have come to light and catalyzed reforms and accountability because conscientious employees and officers spoke up in one way or another and were able to do so. That we have channels to whistleblow or critique our own government is a virtue of its own that must be defended to stay ahead of demagogues or concentration of power. This type of process is unfortunately less feasible in Beijing, where defiance of party agenda is criminal and would-be whistleblowers of Xinjiang camps are, themselves, in camps.

Newfound Pressures Towards Autocratization

Technologies, conditions, and geography (the “geo” in “geopolitics”) shape how humans govern through incentives. For example, the “resource curse” is when countries rich in valuable natural resources (e.g. oil, gas, and minerals) counter-intuitively experience slower economic growth, weaker governance, and worse development outcomes than resource-poor nations.

There isn’t consensus as to why this correlation exists, but I find the Anarchy as Architect explanation among the most plausible: When a technology raises or lowers the marginal value of the median person to the state’s competitive fitness, states are incentivized to react accordingly. For example, when states can get their money from resources instead of citizenry, they don’t have as much pressure to invest in education — an educated person is just as efficient in a coal mine, but more likely to revolt. On the other hand, if the marginal value of a median person is high, a state benefits from spending resources on education, liberties, healthcare, infrastructure, etc.

There are a few reasons I expect near-term AI systems to encourage autocracy over democracy:

AGI and the future: Democracies are largely possible because the state benefits from empowering people and because people can threaten the state (revolt, riot, etc.) should their rights be taken. They can also remain stable by voting out draconian leaders. Similarly, there are strong incentives to empower individuals, because educated, healthy people are more productive. I think these incentives will decrease (or vanish) in many AGI scenarios and straightforwardly decrease with cheaper versions of existing technology already!

Technologies alter what types of governments are competitively viable, sometimes for better and sometimes for worse. It isn’t difficult to imagine how AI might help concentrate power. AI access is already heavily concentrated, few people can train or deploy AIs, and decisions that steer AI values are in a relatively small number of hands. It may be worth noting that it’s hard to forecast technological progress, most attempts are wrong, and this one may be too.

Legibility increase: legibility refers to the ability of a state to measure, monitor, and manipulate its people by turning messy/complex systems into understandable ones. For example, mandatory IDs increase legibility because everyone’s name can be put in a database and tracked (e.g. makes tax collection easier). Legibility is neither good nor bad necessarily, but does seem to increase the capacity of states relative to individuals. Machine learning and AI technologies are extremely good at parsing enormous amounts of data, perhaps this could be used to measure political loyalty, amp up surveillance, crack down on dissidents or selectively enforce the law.

A very small number of people are required to control large numbers of AIs: Future AIs are likely to not require many people to operate / monitor them (e.g. imagine 100 claude mythos 10 tabs open but the claudes are sifting through texts / videos looking for dissidents). This is straightforwardly downstream from AIs being highly parallelizable. Consider an example from Dwarkesh Patel: “There are 100 million CCTV cameras in America. You can get pretty good open source multimodal models for 10 cents per million input tokens. So if you process a frame every ten seconds, and each frame is 1,000 tokens, you’re looking at a yearly cost of about 30 billion dollars to process every single camera in America.” (this will get exponentially cheaper)

Existing precedent: the relative share of people living under democracy has already begun declining, and the democratic world is backsliding according to most measures.

Social destabilization attacks are more effective against open societies, and AI will clearly improve various attacks. The majority of our information diets come from algorithmically generated “feeds” of information, determined by recommendation algorithms. Even when people seek out information (e.g. Google), a recommendation algorithm determines what’s shown and what isn’t. This year, I have become familiar with how botnets already drive political narratives; and a shocking amount of social media traffic is driven by bots.

Countries like China and Russia have great firewalls; Google, Facebook, YouTube, Wikipedia, etc. are all blocked. An entire separate internet ecosystem exists to ensure censorship. On the other hand, our internet is the wild west. Anyone can sell ideas and coordination can amp up ideological-spread. There may be true benefits to the wild west model, but one consequence of this is that open societies have a larger attack surface, such as political influence campaigns or just general chaos-increasing campaigns. We are prevented from effective retaliation because it feels like we’d be abandoning liberal principles (e.g. free speech).

Leveling the Playing Field

I believe one reason for the resurgence of authoritarianism and ongoing democratic backsliding is likely that the environment increasingly favors autocracy. To resist is to push pieces up a slanted board. The board seems to have tilted towards autocracy — surveillance is becoming cheaper, information spaces are easier to seal, and open societies are becoming more polarized, unstable and incompetent. Whether technology and humanity drifts towards authoritarianism or not has not been fated, and we can make changes to level the gameboard. We can invest strategically into defensive technology, diffuse AI in a way that augments rather than automates so as to decentralize power, and democratize our institutions.

Here are some ideas to begin leveling the playing field:

1. Treat foreign state-directed information operations as weapons. This includes bot networks, the content farms built to poison training corpora, the operations run by intelligence services whose full-time jobs are destabilizing open societies. This must be done carefully, so as to ensure we don’t police what Americans say to each other. Here is an example of an influence operation that Claude Deep Research surfaced: “The Pravda network represents the most significant documented case of deliberate training data contamination. Identified by the American Sunlight Project (February 2025) and corroborated by the Atlantic Council DFRLab and French intelligence watchdog Viginum, this network of 182 domains published over 3.6 million articles in 2024, averaging 10,000+ daily. The sites receive fewer than 1,000 monthly visitors and have ‘no search function, poor formatting, unreliable scrolling’ — they were never designed for human readers.”

2. Demand reciprocity from closed information ecosystems: China and Russia firewall out our platforms, our press, and our internet while enjoying nearly unrestricted access to ours. There is no principle of openness that requires a society to extend its openness to states that weaponize it. We have tools to tilt the playing field: export controls, divestiture requirements, and transparency mandates. We must do this carefully however, so as not to become a closed society ourselves.

3. Improve our epistemics and government capacity: The same technology that enables tailored propaganda could also allow us to steelman the other side, see pieces of the world we didn’t know of, learn entire disciplines of knowledge, interrogate our own beliefs and become more rational. So long as we maintain a healthy lead, we can decide what type of technology we build.

4. Pursue verifiable nonproliferation treaties for dangerous AI capabilities that are likely to define the future. This is feasible because AI’s key inputs are monitorable, as they require enormous amounts of computational power. America did this when we had a monopoly on the atomic bomb, and Nixon similarly renounced bioweapons and got the world to follow — AGI may be even more dangerous.

We may need to find creative ways to level the game board — just as the founding fathers did — in the face of newfound pressure and the speed of the technology applying it.

Against Utopia

In our attempts to improve the future, we should beware of utopian and fanatical ideologies, as they have a pathetic track record. History is replete with examples of grand, well-intentioned schemes at improving the human condition. Practically all of them failed miserably despite their good intentions. This includes Soviet collectivization, the Great Leap Forward, and Eugenics among others. One reason schemes with benevolent intentions fail is that enacting such grandiose schemes simply requires centralizing power in the hands of fallible humans with imperfect motivations.

One of the great acts of the founders was admitting their own fallibility. They wrote of no utopia, end state or finality. The authors of the Constitution confessed that they are likely wrong. 250 years later we can see many of the ways in which they were, such as the fact that many of them owned other human beings. Yet the amendment process they initiated, the free press and the very words of the Declaration they wrote (”all men are created equal”) is what ended the horrors of American slavery.

We should have a strong prior against those who claim moral certainty or a vision for utopia. I suspect our descendents will find something monstrous about how we govern today — we still have yet to defeat racial prejudice, sexism, foreign policy failures, or extreme wealth inequality. I suspect future generations will look back in horror and confusion at factory farming too. The magnetic pull of utopian idealism and ideological fanaticism is deeply human, especially when confronted with so many issues, yet we must resist it.

I have trouble articulating a utopian vision for the future. The best hope I can offer is a direction: that we keep getting incrementally wiser, more capable, more wealthy, and more empathetic. This type of progress has a better historical track record.

Optimism

I moved to Washington D.C. a few weeks ago. I don’t think I’ve ever felt as uplifted and motivated as I have after meeting so many people with so much devotion to the country and to the future. I’ve met a former jet pilot devoted to making sensible warfighting rules in the age of autonomous weapons. I’ve met an army officer who writes about how to make the army more ethical. I’ve met data scientists who, instead of working on Wall St. are writing contingency plans for the U.S. government to have a sensible plan in case of a disaster — such as a terrorist using AI to design a bioweapon, an AI-driven cyberattack on critical infrastructure, or a rogue power-seeking AI on the internet. I’ve met PhD students and AI researchers (who put my capabilities to shame) and could have made million dollar salaries working at OpenAI or Google, but chose instead to fight tooth and nail for a government salary to build out our capacity to understand AI systems.

I have walked into Senate office buildings, found staffers, and asked if I could tell them about my AI research and how it might help them. Sometimes they literally just say yes. I’ve encountered worrisome issues, trends, and norms, but I’ve also met countless people who want to be stronger, want to work harder, want to be more ethical, and want to be told where they might be wrong. I’ve met people from all sorts of backgrounds who put principle above race, religion or tribe, and this makes me optimistic.

Conclusion

This isn’t an attempt to stop you from criticizing the government. You should. But there are features worth preserving that are too easy to forget about under pressure. Not least of which is to treat your political opponents as valid opponents — not to admit they’re right, but to recognize that they’re legitimate. So long as both sides view each other as adversaries to be defeated at the ballot box, not through violence or imprisonment, we can both persist and gradually improve.

A clear-eyed defense of American ideals matters more than ever in the age of intelligence. New technology and a shifting world order will likely create unprecedented pressure towards autocratization in the coming years. We must resist this, and we must also remember: the principles that define America are not bad. They are good.

To be a reasonable patriot does not mean to celebrate every foreign policy decision, endorse every presidential decree, or worship the parochial morality of the founders — it means defending American ideals. We should notice that we often fail to live up to our ideals but also that our ideals have always been good. And we can be so much better.

Happy Fourth of July!!

Should pigs (who are smarter than dogs) be lowered screaming into CO2 gas chambers that burn their eyes, nostrils, throats and lungs as they asphyxiate?

Should piglets have their tails cut off with pliers, teeth ground down and testicles ripped out with no anesthesia? (The tail docking is because pigs bite each other’s tails from stress when packed in concrete pens.)

Questions like these are as close to no brainers as it gets.

Should farmers use pliers to pull skin off live fish? Should dozens be crammed into buckets as they grasp for oxygen and die of asphyxiation?

Should we suffocate, gas or grind 350 million male chicks to death every year (in the US alone) because they don’t lay eggs?1

None of this is illegal, this is just how we make meat. If we were the people we wished we were, if we knew more, thought faster, became more rational, how would we answer these questions? Enter an egg laying barn and every sense you have will be accosted by suffering: you will hear nothing but screams, feces and ammonia will be the only detectable smell, and your eyes will see nothing other than death and disease. What if we had to endure the suffering ourselves?

This is just a fraction of the tortures we inflict on hundreds of billions, and practices like this are the norm, not the exception. Factory farming is a clear candidate for the biggest moral mistake humanity is currently making. The scale of the problem is so large that it’s impossible to intuitively grasp. Farm animals are literally the epitome of “one death is a tragedy but 1 million is a statistic.” The vast majority of animals we eat (most estimates are above 95%) come from factory farms, spending entire lives under conditions we can scarcely imagine.

There is total moral inconsistency in our laws, norms and culture over the rights of animals. Lewis Bollard on Dwarkesh Patel podcast pointed out that Cockfighting (a true evil afflicting thousands of chickens) has had multiple laws passed against it, is a felony in every state, and has been rightfully regulated out of existence. Yet when factory farmers do far worse, to a far greater number of chickens, we call that “commerce.” Similarly, if someone tortures their pet, they are universally condemned; but when a hundred billion animals are slain per year nobody cares much. Complain and you’re a radical, judgmental vegan nut. The sane and normal position is to look away.

Kristy Noem shot a dog and millions were outraged. South Park devoted an episode to her. She obviously acted wrong in killing it. Yet we routinely excuse torture and killing on a scale beyond our comprehension. Our only excuse is that it tastes nice and we don’t have to watch how it’s made.

What should you do?

First, if you want to understand the horrors of factory farming, watch Dominion. I also recommend this debate, and the most important speech you will ever hear.

Second, consider reducing or eliminating your support for the practice. More people should go vegan, but if you don’t want to, consider reducitarianism, marginally changing your habits (eg order tofu pad thai instead of chicken)

Bentham's Newsletter

Be A Marginal Reducetarian

Most people aren’t vegan. Lots of people have heard about the terrible things done to animals on factory farms—the grinding up of baby male chicks, the locking of hens in crates, the gassing of pigs, and the Pandora’s box of other horrors. But these are not enough to get them to go vegan. The taste of meat is just too motivating…

Read more

8 months ago · 43 likes · 29 comments · Bentham's Bulldog

99% of cheap meat comes from factory farms. The field of nutrition has confirmed to us that the optimal amount of meat, especially processed meat, in our diets is ZERO.

In the U.S. the number of cage free eggs has jumped from 10% to 40% in the past 20 years (which is great!) There are avenues other than advocacy or diet choices that can improve animal welfare.

Third, and this will in expected value be more effective than the first two, consider donating to reduce the suffering of factory farmed animals.

Factory farmed animals are likely the most neglected sentient beings on earth. Relative to other social causes, factory farmed animal welfare receives pitiful attention. Charities receive only ~200M/year in funding, compared to >3B for animal shelters (15x more). Meanwhile, at any given time there are 10 billion animals in factory farms in the US, and under 6 million in animal shelters. The best charity is probably not what you think.

Dwarkesh Patel just released a podcast on the most cost effective charities to reduce animal suffering. He’s currently running a fundraiser and matching donations to Farmkind. Farmkind works through realistic reform avenues: corporate campaigns, legislative advocacy and institutional change to end this nightmare. A few weeks ago I gave $1,000, and multiple friends and coworkers I’ve shared this with have already contributed several thousand more.

consider different foods

Eating animals is bad, but this does not mean that all animal foods are equally bad, and by my estimates the variance is extremely high! For example, if you got all of your protein from cows, you would probably go through less than 1 cow per year because they are massive. On the other hand, if you only ate chicken, you’d eat ~250 animals! One cow provides ~500 lbs of meat; one chicken provides ~3 lbs. If we’re comparing pasture raised cows (who have access to the outdoors for most of their lives) to factory farmed chickens, choosing beef is almost certainly preferable. Factory farmed chickens are some of the worst treated animals. They have their sensitive beaks seared off (so they don’t peck their neighbors), they develop osteoporosis because they’ve been bred for size, their brothers are all macerated after hatching, and worst of all, even ‘cage free’ often live tightly packed in sheds with less than a square foot of space to themselves (source).

If you’re trying to reduce suffering, you can:

1. avoid farmed animals, eat wild caught fish, pasture raised beef, etc. The difference in welfare between a hen with 100 square feet (certified pasture raised) of space vs. 1 square foot is massive.

2. Consider oysters, mussels, and other bivalves. According to most theories of consciousness, they are quite unlikely to be sentient (I estimate >95% chance they lack any experience). They have very few neurons (~20,000, for comparison, fruit flies have 5x as many), no centralized brain, no pain receptors and there’s no evidence of pain avoidance. They also live completely immobile lives, such that modern farming practices are nearly identical to their evolutionary environment. Interestingly, they’re also environmentally beneficial because they improve water quality through filter feeding. (They also have omega-3’s and vitamin B-12, so they make a nutritious addition to a vegan diet)

3. Be careful what you eat; if sentient, shrimp live terrible lives. Their eyes are squished (eyestock ablation) and they live tightly packed with other shrimp. Also the number of deaths per calorie is high, because they are so small.

Other Sources:

https://www.effectivealtruism.org/articles/cause-profile-animal-welfare

https://www.givingwhatwecan.org/cause-areas/animal-welfare

Bentham's Newsletter

Utilitarianism, ethical veganism, culture war stuff, philosophy, morality, and more!

By Bentham's Bulldog

Key Takeaways:

• AI chips are strategic assets: China will use state power on strategic assets, Western nations should not lose out of conviction to free market competition. Between 2007-2015, China invested $50B+ in solar while Europe (the incumbent leader at the time) invested under $5B.

• US policy should tighten export controls to limit tech transfers and critical chokepoints offer protection strategies: The manufacturing equipment to build both technologies is technically complex. Manufacturing equipment proved to be an important solar PV chokepoint. But Chinese companies simply purchased the best German solar PV manufacturing equipment available, instantly gaining technological parity that could be scaled.

• Vertical integration may have consequences: vertical integration of the entire solar PV supply chain was capital intensive and meant less flexibility. Some critical manufacturing equipment is already developed in allied countries, America only needs to protect a few critical chokepoints.

• Domestic competition is critical: competition prevents price and supply shocks that provide opportunity for new entrants.

• Allied cooperation is critical: Western countries did not mount a unified response to China’s mercantilist solar policies. The US may eventually become unable to match China's semiconductor capacity, but the US, EU, Japan, Taiwan and South Korea combined vastly outproduce China and it’s allies in advanced chips and their inputs.

The Chinese state has spent billions of dollars building supercomputers, and is committed to indigenizing AI chip design and manufacturing. (source) According to RAND, advanced chips “drive the foundation of American strategic military advantage.”

The US pioneered solar technology, then state support via industrial policy and technology transfer and helped China scale past the rest of the world, an outcome we can avoid with AI hardware. While the two technologies both begin with the same raw material, sand, advanced semiconductors are far more technologically complex to manufacture than solar cells, so lessons are deserving of nuance and rigor.

The Role of State Support

Protectionism and state support define the history of solar manufacturing success. Competitors used policies like tariffs, subsidies and trade barriers, but governments often imposed tariffs too late. The center of solar manufacturing capacity experienced two major shifts: first, dominance shifted from the US to Germany and Japan in the early 2000s, and then from Germany/Japan to China in the 2010s.

In the 1980s, the US maintained 85% of the solar market, pioneered by companies like Bell Labs. During the Reagan administration, Japan and Germany emerged as new leaders due to massive subsidies. This coincided with a shift away from solar in the US as oil prices. This newly formed oligopoly’s subsidies came in the form of Feed-in tariffs, beginning in 2004, where governments guaranteed above market payments to anyone generating electricity from renewable sources. The policy wasn’t geopolitically targeted, so companies in the West and China sprung up to meet the demand.

Eventually, in the 2010s competition ensued between the newly formed Tokyo-Berlin oligopoly and China. China's modern state backing was historically unparalleled because solar was considered a strategic sector. Beginning in 2011 the state invested over $50B, more than 10x Europe's investment. (source) At the same time, Chinese industrial policy allowed access to cheap financing from state-owned banks, which extended at least $18B in low interest loans to solar companies by 2017. (source) According to Time, local governments routinely offered discounted land leases, or in some cases let solar farms or industrial parks operate rent free for years as an incentive.

Free markets deliver prosperity and low prices, but China will not play on an even playing field with strategic assets, nor should we. Even before the current AI boom, high performance computing (HPC) was a national priority. According to a joint report by the National Security Agency (NSA) and the National Nuclear Security Administration (NNSA), “National security requires the best computing available, and loss of leadership in HPC will severely compromise our national security. HPC plays a vital role in the design, development, or analysis of many—perhaps almost all—modern weapon systems and national security systems: e.g., nuclear weapons, cyber, ships, aircraft, encryption, missile defense, precision strike capability, and hypersonics.”

Technological Transfer

The acquisition of German manufacturing equipment and Western technological transfer served as a launching pad for Chinese dominance, and Western countries never used protectionist policies on the manufacturing equipment itself. Instead of export controls or nonproliferation agreements (e.g. the Wassenaar agreement covers semiconductor equipment exports), German suppliers actively facilitated transfers of equipment to Chinese firms, including entire German turnkey assembly lines. (source) In semiconductors, the Netherlands is home to ASML, which holds a virtual monopoly on extreme ultraviolet lithography (EUV) machines, the most critical piece of equipment in semiconductor manufacturing. Similarly, American companies like Applied Materials currently lead in semiconductor equipment for deposition and etching. The manufacturing equipment for semiconductors is vastly more complex, and requires more training and technical support, so they will be more difficult to recreate. EUV machines, for instance, require dozens of trained technicians at all times.

Massive state support and manufacturing equipment transfer coincided perfectly with a supply shock in polysilicon, a key component in solar cell manufacturing.

The Polysilicon Crisis

Around the 2008 financial crisis, a limited number of incumbent competitors in Japan, Germany, and the US meant unstable supply and prices. This combined with an explosion of demand from policies like feed-in tariffs, caused a supply shock in one of the key ingredients in solar cells: solar-grade polysilicon. A kilogram of solar-grade silicon that cost ~$25 in 2003 shot up above $300 per kg by 2007. At the time, a small handful of firms in Germany, Japan, South Korea and the US controlled the polysilicon market, while Chinese and Russian companies struggled to create silicon as pure.

But the price surge created an opportunity for new entrants. Chinese companies, with recent government support, seized the opportunity to catch up in the capital-intensive polysilicon manufacturing plants.

Polysilicon production market shares by country from 2010 to 2022 (source: IEA)

An oligopoly on polysilicon production made for a sclerotic market in solar PV manufacturing, and harmed solar installations. Greater competition could help avoid price incentives for rivals. For AI hardware, domestic and allied competition (e.g. Intel, AMD, Google and European competitors) could keep prices and supplies in check.

Once Chinese production came online, the market shifted from scarcity to surplus. Prices for polysilicon dropped by ~80% in 2011. Chinese paranoia over Western pressure helped motivate a vertical integration strategy in China which included every stage of solar PV manufacturing, from refining polysilicon into wafers, creating cells, and assembling solar modules. Trade disputes by Western countries were too late, and the boom-bust cycle resulted in a decisive win for Chinese manufacturers. In May of 2012, the US imposed tariffs on Chinese firms for dumping polysilicon, a policy of selling products at a price lower than it costs to capture market share. However, this backfired as Chinese counter tariffs cut Western polysilicon production out of the new largest solar cell market. One analyst says that China is still dumping solar cells to undercut any remaining Western rivals, selling polysilicon for $5/kg while it costs $6/kg to produce.

Vertical Integration

Unlike solar, it’s less feasible for one country to fully integrate across all stages because the complexity, R&D and capital intensity are magnitudes higher. For instance, semiconductor grade silicon has to be ~1000x as pure as solar-grade silicon, a single EUV lithography machine costs ~$400M, and fabs cost well over $10B only to become deprecated in a few years time. Despite this, China is clearly trying to vertically integrate semiconductor manufacturing. If they fill enough gaps, they will become resilient to export controls.

Globalized Chokepoints

AI technology and its inputs are not just developed by American and Chinese companies. The supply chain is global but remains concentrated among Western powers. For example, Taiwan Semiconductor Manufacturing Company (TSMC) outperforms Western and Chinese rivals in advanced semiconductor manufacturing, while Dutch ASML leads the world in some of the key equipment enabling TSMC. The US may not be able to rival China’s manufacturing might alone; but with close allies considered (Taiwan, Germany, France, Japan, South Korea, etc.), the competition favors the West by far. Alliances and agreements will determine who leads.

Conclusion

China sees solar as a strategic asset and has successfully monopolized the manufacturing process. In a matter of years, state-backing, industrial policy, technology transfer and slow reactions led to a durable Chinese lead that rooted out all competition. We can study cases of China's manufacturing successes in order to preserve the West's AI compute lead. These two industries share important manufacturing similarities: both are capital intensive and have diverse supply chains requiring a range of expertise.

This is a cautionary tale; solar modules are important, but AI chips are critical. AI governance demands strategic foresight, and cutting edge technologies will be the deciders of the great power competition.

Summary

Nvidia publicly opposes export controls protecting America’s leadership and no one in Silicon Valley or D.C. dares call them out. One reason for this is that the very companies that should be sounding the alarm depend so heavily upon their hardware. I’ve looked at a bunch of Trump’s AI Action Plan submissions. The 4 leading AI companies' positions in the recent AI Action Plan align exactly with how dependent they are upon Nvidia. For instance, OpenAI and Meta who rely heavily on Nvidia for their AI experiments have remained relatively silent or opposed export controls. On the other hand, the AI leaders developing their own hardware (Anthropic and Google), stand out as China hawks. The companies most dependent upon Nvidia seem to lose their voice on national security and independently arrive at Nvidia's pro-China position.

This matters because:

• The U.S. currently has 10x more AI compute capacity than China, this is our largest AI advantage

• Export controls on AI chips have been our main tool for maintaining this lead

• Every chip sold to China erodes the advantage

I’ve found that:

• The four leading AI companies' policy positions align exactly with their Nvidia dependence

• I learned firsthand at Intel how deep Nvidia’s influence extends. While there, we fought a losing battle against Nvidia's software monopoly. Every major ML framework is built on CUDA, Nvidia's proprietary layer, creating ecosystem lock in

• Nvidia has repeatedly circumvented export controls (designing "China-only" chips like the H20 to skirt restrictions)

• When Trump finally banned the H20, they responded by lobbying to scrap controls entirely

Context: Western Compute Lead

“Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters.” - Leopold Aschenbrenner

The largest AI experiments require billion dollar investments, and thousands of chips optimized for training. AI datacenters contain tens of thousands of Nvidia H200’s, costing ~$30,000 each, and training requires fundamentally different hardware than inference (running the model). This made Nvidia the most valuable company on earth. They design them, but they don’t actually manufacture them, they outsource the actual production to Taiwan. Nvidia/TSMC hardware is a key ingredient in massive AI experiments, largely responsible for the U.S. producing more frontier AI models. There is good evidence that China’s Deepseek was enabled by smuggled Nvidia chips via Malaysia.

The US AI hardware lead. Source: RAND.

AI 2027, the AI forecasting essay read by JD Vance, argues that America wins because of the compute lead but assumes we protect it. America has roughly 10 times more compute capacity than China. As RAND's analysis shows, this compute advantage is our primary moat. Every chip sold erodes that lead.

Even today’s AI helps authoritarian regimes retain power (e.g. surveillance, preventing dissidents from coordination). That’s nothing compared to reaching AGI (artificial general intelligence) first.

Nvidia Has a History of Skirting Export Controls

In August of 2022, the U.S. banned selling high-end chips like the Nvidia H100 to China and Russia. In November 2023, Nvidia announced three chips deliberately engineered to skirt export controls. Reuters broke the story that Nvidia had designed three “China-only” AI accelerators (H20, L20, L2) in response to the U.S. export controls. By lowering the raw compute throughput and PCIe I/O, each device stayed below the compute limits and didn’t require an export license like their flagship AI chips (H100, now H200).

As RAND points out, while H20s "underperform H100s for initial training, they excel at text generation ('sampling')—a fundamental component of advanced reinforcement learning methodologies critical to current frontier model capability advancements"

The Nvidia H20 chip created a government approved pathway for Nvidia to do exactly what the diffusion rule and export controls were trying to prevent. And now, facing an $8 billion revenue hit from the Trump administration's decision to finally restrict H20 sales, Nvidia is advocating for giving up on export controls entirely.

The Hawks and The Doves

The major companies dependent on Nvidia's chips either stayed silent on export controls or opposed them. OpenAI, Anthropic and Google Deepmind are the three leading AI labs producing frontier models. Meta is behind, but has the infrastructure necessary to be a leader.

Nvidia-Dependent Companies are China Doves

*apparently substack doesn’t allow tables, so I have to use a screenshot.

Here is a graph from Gemini:

Other notable hawks:

• Patriotic small tech (e.g. Scale AI, Palantir): "China is leading on data, we are tied on algorithms, and the United States remains ahead on compute,” Scale supports stronger export control enforcement and focuses their entire letter on the CCP.

• American Think Tanks including: RAND, Georgetown's Center for Security and Emerging Technology, The Center for AI Policy, RAND, The Center for a New American Security, and The Institute for AI Policy and Strategy all position themselves in favor of export controls.

• International AI Governance Centers: Organizations like the Oxford Centre for Governance of AI (GovAI) and the Centre for Long-Term Resilience (CLTR) in the UK highlighted compute governance as a priority in mitigating AI risk globally.

Other notable doves:

• Other hardware companies (AMD, Qualcomm, Intel): Have advocated the idea that “if America won’t sell advanced processors, China will simply build its own”

First Hand Experience: Mechanisms to Preserve Nvidia’s Monopoly

I learned firsthand how deep Nvidia's control runs when I worked at Intel. Policymakers and techbros regularly miss that Nvidia’s business model isn’t just selling chips, they’ve architected an entire AI ecosystem that is optimized for their hardware.

At Intel, I helped build TensorFlow, originally developed by Google. It is the second most-used machine learning framework. Many of the features in TensorFlows backend are built upon CUDA, Nvidia’s proprietary software layer. In short: a lot of features that work fine on an H100 won’t even run on an Intel chip. This means Intel, AMD and other hardware companies that want to run TensorFlow need to maintain their own versions of the software independently, and run their own software parallel universe. This is a never ending uphill battle to catch up to Nvidia that Intel will never win.

Software is at the center of Nvidia’s dominance: everyone in academia and industry exclusively uses and contributes to software built for Nvidia hardware, written in CUDA, which effectively cannot be run on non-Nvidia chips, widening their moat. By FLOPs, or raw compute power, Intel and AMD could compete with Nvidia today. However, when it comes to the software every machine learning engineer uses daily, it is ruthlessly optimized for and built upon Nvidia’s CUDA. This includes: TensorFlow, Pytorch, Sklearn, etc. and everything they’re built upon.

This is yet another reason why companies are so vulnerable to Nvidia’s influence. They don’t just control the hardware stack, they control the software stack.

“They'll Develop It Anyway”

What is Nvidia’s main argument for scrapping export controls altogether?

Nvidia CEO Jensen Huang argues: "The question is not whether China will have AI. It already does. The question is whether one of the world's largest AI markets will run on American platforms."

This is like saying: "China is developing nuclear weapons despite our restrictions, so we should sell them uranium."

As blogger Zvi Mowshowitz notes: "America should sanction our men's soccer team, too, so they will do better."

The fact that China is managing to develop AI despite restrictions isn't an argument for removing those restrictions, it's proof they're working.

Chip manufacturing profits are not the right metric to care about. Who has the compute is what matters.

The Bottom Line

If China could effectively access the best AI chips, that would get rid of what's arguably our biggest and most important advantage. Deciding to arms-race China might be a massive mistake, but if the self-fulfilling prophecy of AI arms race with China comes to fruition why would we give our strongest advantage to our adversary? We should not sacrifice our future on the altar of Nvidia’s stock price.

Despite tweeting about winning the AI arms race, we are deciding to lose it.

I first sought to write this up because I read a bunch of the public submissions to the AI Action Plan and thought I should at least turn my notes into something instead of just caching all those policy proposals until I forget them. I found something disturbing: Nvidia advocates that it should be allowed to sell to China openly, and hardly anyone in Washington seems to hold them accountable for this.

Others Have Noticed This

I started writing this a week or two ago. I forgot to post it on lesswrong.

Apparently CSIS is releasing a paper about Nvidia so I wanted to post this now (they have high quality content on export controls / compute: https://www.csis.org/analysis/limits-chip-export-controls-meeting-china-challenge)

Other Good Proposals in the AI Action Plan

Aside from export controls, there are other solid proposals I found in the AI Action Plan submissions that I don’t see very many people talking about. Here are two of them:

1. The Institute for Progress Suggests Piloting an SL4 Datacenter

A key pillar in IAPS’s submission is denying foreign adversaries access to advanced computing technology. This is because AI models are national strategic assets: algorithms and weights are worth billions in R&D investments and competitive advantages could determine leadership for years to come. If stolen by adversaries, models could be replicated at a fraction of the training cost, because inference is far cheaper than training. The comment brings up RAND’s security levels, suggesting, "equivalent to SL4 and SL5 as outlined by the RAND Corporation.”

Recommendation 4 in the IFP’s submission is “A pilot highly secure data center” where they explain RAND’s framework for model weight protection. The IFP expands upon IAPS’s suggestion, saying “the DOD should build and operate a pilot SL4 AI cluster to develop best practices for securing sensitive AI workloads and models, and to develop next-generation AI-enabled national security applications.” This naturally extends IAPS’s security suggestions and concerns over foreign nationals stealing secrets from leading AI labs. It is also highly actionable.

2. Expand the AI Safety Institute

There is another silent battle over NIST’s AI Safety Institute, which currently houses almost all of the government’s AI capacity for model testing. Their papers are impressive, they’ve conducted interesting evals, and they are the source of most of the U.S. government's AI competence. The AISI is led by Paul Christiano, creator of RLHF, the invention that enabled ChatGPT. They are the organization that could set AI standards aligned with Western values.

Organizations including: Scale AI, the AI Futures Project, Anthropic, The Institute for AI Policy and Strategy, Center for AI Policy, RAND, .. (the list goes on) show support for NIST’s AI Safety Institute. This is sometimes via directional stance and sometimes via detailed policy proposals. On the other hand, A libertarian think tank (R Street, who I’ve never heard of) and Andreeson Horowitz recommend winding down AISI.

3. Release New Versions of Chinese AI Models

The Center for a New American Security suggests, “Rapidly releasing modified versions of Chinese open source AI that strips away hidden censorship mechanisms and ‘core socialist values’ required by Chinese AI regulation.”

Background

The media and information environment is confused and perspectives on AI vary wildly. Many don’t think about AI at all. Many more imagine future AI models will continue to be chatbots on their phones. A final group speculates about AI’s which not only answer questions, but out-think humans, execute tasks which take hours, days or months, shape the physical world by controlling computers, physical tools, robots, and interact with the economy. OpenAI, Anthropic, and Google Deepmind have all declared their explicit goal is to build artificial general intelligence, and are all backed by billions of dollars in investments and the brightest talent of our generation.

Recent developments that would have been considered science fiction 5 years ago are now commonplace. For example, SORA, is an AI model which creates any length videos from text. The length of coding tasks frontier systems can complete is growing exponentially, doubling every 7 months, and this trend has been robust for 6 years. Two years ago, GPT-3.5 could complete engineering tasks requiring 1 minute of an engineer's time; today, O1 can complete engineering tasks requiring 30 minutes. Assuming this trend continues, AI models in 2027 will complete tasks requiring one work day (8 hours).

Government officials from D.C. to Beijing have noticed this progress, and are currently preparing for an arms race. The Trump administration has announced 500 billion dollars on a Manhattan Project-style data center.

a video of “an otter using wifi on a plane” from 2022, and one from 2024. AI is progressing fast, not just in video creation.

Worry About AI: The Case for Pausing / Stopping

Concern From Those Closest to AI: In March of 2022, Nobel prize winners, top scientists, CEOs and tech figures from Elon Musk to Steve Wozniak signed an open letter calling for a 6 month pause on all systems more powerful than GPT-4. A few days later, Eliezer Yudkowsky put out a post in Time magazine, arguing that researchers steeped in these issues expect that the obvious result of building a superhumanly smart AI, under anything resembling the current circumstances, is that “literally everyone on earth [will] die.” Social movements and protests which directly advocate for an AI pause are rapidly growing.

Existential Risk: Pause proponents argue that until we are able to make smarter-than-human AIs safe to humans, we should pause building them. Delaying AI could give us more time to build institutions and pursue technologies which would help with the AI alignment problem (making an AIs goals line up with some target set of values). For example, interpretability research seeks to understand AI systems. For example, we can understand why an Furthermore, arms-races introduce competitive pressures and have led AI companies to cut safety testing in order to expedite the development of AI. Helen Toner testified before congress saying that OpenAI insiders are begging to slow down.

Unworried: The Case for Proceeding / Accelerating

Technological Optimism: Advanced AI promises to solve problems (e.g. cancer) and reduce other risks (e.g. help prevent asteroid impacts). Marc Andreeson argues in his trending essay that “We are being lied to… Told that technology is ever on the verge of ruining everything… we are told to be angry, bitter, and resentful.” It is undeniable that technological progress is a large factor responsible for improving humans’ quality of life. It is the force which has eradicated smallpox, almost completely solved human famine and child mortality, and has allowed us to explore space. Previous AI research and model releases have been responsible for automation, economic growth, and research progress. It seems reasonable to expect this trend to continue.

Adversarial Reasons: A moratorium on AI would improve the relative position of adversaries and hurt good actors. For instance, a moratorium on U.S. AI companies would not slow down Chinese AI development. Furthermore, fear of institutional decay, a gradual global shift away from democracy, or adversaries (e.g. China) leap-frogging the U.S. in technological and economic power suggests a finite window for powerful AI to be developed by democracies. Why are people more worried about AI takeover than authoritarian regime takeover? A controlled AGI would be the most powerful weapon in history, and nearly guarantee any dictator permanent control and perfect censorship.

Even locally, a pause might mean more safety-aware companies like OpenAI or Anthropic forfeit their lead to companies which do not even pretend to take alignment seriously (e.g. Meta). OpenAI and Anthropic are very anxious about what they’re building, we should be careful if we choose to forfeit their considerable technological lead.

What to do?

Im still researching what to do about all this. Please like and subscribe for more

• AI Researcher views (e.g. published authors in NeurIPS, ICML, etc.)

• AI Engineer views

• AI Safety researcher views

• Policymaker views

• General Population views (I mainly ignore this group. AI Impacts has a great review)

• Activist views

Here is what I’ve found (mostly from the first cluster):

AI Researcher Timelines are shortening

Note 1: These are judgment based forecasts. Researchers are not asked to review the literature on timelines. They are not building timeline models, just giving their best guess.

Note 2: Katja Grace points out that timelines vary strongly depending on phrasing of the question. Regardless, there is a clear trend downwards in timelines.

Forecasting and Model-based timelines

AI Impacts Blog points out that ML experts were pretty accurate in their 2016 short-run predictions. Experts expected 9 milestones to have happened by now – and 10 milestones have now happened.

Using models to predict timelines: Epoch AI has a review of the major timelines models (e.g. Ajeya Cotra, Samotsvety, etc.).

Using forecasting to predict timelines: The Forecasting Research Institute conducted The Existential Risk Persuasion Tournament, where top superforecasters and domain experts spent four months forecasting the likelihood of an ai catastrophe by 2030, 2050 and 2100.

• “The median expert predicted a 20% chance of catastrophe and a 6% chance of human extinction by 2100. Superforecasters saw the chances of both catastrophe and extinction as considerably lower than did experts. The median superforecaster predicted a 9% chance of catastrophe and a 1% chance of extinction.”

A Significant Number of Researchers Expect Extremely Bad Outcomes

According to AI Impacts, ~40-50% of respondents indicated a >10% chance of catastrophic outcomes from AI progress.

Results vary highly based on phrasing of the question. It looks like they are falling for the conjunction fallacy, as section 1 has the lowest probability despite being a superset of all the others. In my opinion, the AI Impacts survey phrases questions and frames statistics in the most alarmist way possible.

When they state the median response indicates a 5% chance of catastrophic risk, it means that 50% of AI researchers believe there is a higher than 5% chance of such risk. This does not imply that only 5% of researchers consider catastrophic risk to be highly likely. I noticed some comments confused about this.

Below is a graph of respondents likelihoods of outcomes ranging from “extremely good” to “extremely bad” from 2016:

Now from 2022:

Finally, they re-ran their survey a third time in 2023:

2023 was an eventful year in AI – including the EU AI act, CAIS statement, a call for a 6 month moratorium, and the release of GPT-4, Gemini and Claude 2. All of this seems to have updated AI researchers towards optimism.

Even AI Researchers are Unfamiliar with Alignment Research

This lack of familiarity makes the last few results a lot more concerning.

One of the most shocking findings from the Arkose Interviews was that only 40% of AI researchers (n=~100) had ever heard of “alignment”. However, this data was collected in the beginning of 2022, and I’d guess this has changed dramatically since (section below for details).

An AI Impacts question asked readers to consider Stuart Russell’s formulation of the alignment problem which states “you get exactly what you ask for, not what you want”.

In 2023, 13% of AI researchers considered this to be “among the most important problems in the field”.

This surprised me as <1% of AI researchers work in alignment at the time of the survey. (source)

One explanation is that AI expertise does not necessarily translate to AI Safety expertise. Many experienced ML practitioners and researchers may have never deeply considered alignment. Much of the alignment literature discusses limits of very advanced systems which are implausible with current state-of-the-art. This may look more like sci-fi or philosophy than empirical research.

Another explanation is that researchers are concerned but are unsure where to start. This seems plausible to me. I am also unsure where to start on alignment.

What AI Safety Material is Compelling to AI Researchers?

In 2022, Vael Gates conducted interviews with AI researchers to see what AI Safety materials they find compelling. She pointed out that ML researchers prefer materials that were aimed at an ML audience, which tended to be written by ML researchers, and which tended to be more technical and less philosophical.

In 2016, Scott Alexander ran a similar experiment: he asked his subscribers to read some AI Safety material and then rate their agreement with it. He found that most ai safety material is similarly convincing.

Pessimists and Optimists Agree on Many AGI Lab Safety Measures

There is significant disagreement among alignment researchers. AI researchers disagree with alignment researchers even more. P(dooms) – the probability of catastrophic outcomes – range from 0.0000001 (Yann Lecun) to 99.99999 (Roman Yampolskiy). Both know more than I do, leaving me confused.

GOV AI has a survey on AGI lab employees, and they appear overwhelmingly in consensus about best practices. E.g. almost everyone is on board with “evaluating dangerous capabilities”. Yay.

One potential issue I noticed with this survey is that it appears the respondents were selectively chosen. Despite this, the results still make me optimistic about people being in favor of sensible regulation.

Many AI Researchers Expect a Slow Take-off

I could be wrong, but this world view seems entirely incoherent.

From AI Impacts 2023:

“If science continues undisrupted, the chance of unaided machines outperforming humans in every possible task was estimated at 10% by 2027, and 50% by 2047. However, the chance of all human occupations becoming fully automatable was forecast to reach 10% by 2037, and 50% as late as 2116 (compared to 2164 in the 2022 survey).”

In their survey, AI Impacts uses two definitions:

1. High Level Machine Intelligence (HLMI): Machines capable of all human tasks

   Estimated 10% by 2037, 50% by 2116

2. FAOL (Full automation of labor): Estimated 10% by 2027, 50% by 2047

I’m surprised by this result. The median AI researcher actually thinks that there will be an entire 69 years between a machine capable of all human jobs and full automation of labor? Is it really plausible that we build AI systems with human-level cognitive abilities, and business just continues as usual for an entire 69 years? This does not make sense to me and I don’t think these researchers have been thinking hard about their responses to these questions.

AI researchers may either be pricing in a slow takeoff, or just haven’t thought very thoroughly about this. I don’t think this position withstands much scrutiny. Here’s my reasoning:

Currently AI’s are vastly subhuman at some tasks:

• The human brain is significantly more sample efficient than AI’s

  • You and I do not need 100,000 training examples to learn to stop at stop-signs

• The human brain is 20 times more energy efficient than a graphics card

  • Human brains run on 15 kilacalories (62,000 joules) per hour, so 12 Watts

  • My PC’s power supply is 750 Watts, and my graphics card draws about 200W of that.

• AI’s are subhuman at drawing, generalization, visual stuff (as shown by the ARC-AGI contest)

On the other hand, present AI’s are vastly superhuman in other ways: AI has superhuman memory, they can be copied arbitrarily, and they have the ability to work much faster/cheaper than us.

If AIs keep their advantages over humans while overcoming humans where they are currently behind, it is pretty clear that once a lab trains AI that can fully replace its human employees, it will be able to multiply its workforce 100,000x.

As Zvi points out, it seems like a shockingly large number of researchers anticipate that high level machine intelligence will just be “meh”. I would expect anything but “meh” if you successfully deploy human level intelligence. Also, right after sharing their short timelines towards high level machine intelligence, a lot of researchers think the biggest risk is… deep fakes?

Views Among All are Rapidly Updating

People's timelines, objections to ai safety, familiarity with certain ideas are all rapidly updating. Surveys from 2 years ago are essentially worthless because of the gap between pre-GPT and post-GPT views.

AI Safety Researchers do Not Think We Will Solve Alignment

(source: https://www.lesswrong.com/posts/XTdByFM6cmgB3taEN/key-takeaways-from-our-ea-and-alignment-research-surveys)

Alignment researchers are also broadly in favor of an AI Pause:

Objections to AI Safety

I had Claude read all of the Arkose Interviews and then say which objections to AI Safety they brought up. Vael Gates asks interviewees what their thoughts are on the off button problem.

Using a taxonomy from AI Risk Skepticism, I clustered objections into categories. The most common objections were:

1. AI is too far away to be concerned

2. There is no clear path to agi from present ai systems

3. Other AI researchers are not concerned

These interviews are pre-GPT. It is likely the debate has evolved significantly.

All of the objections are:

1.1 Priority objection: AGI is Too Far so it isn't worth worrying about

1.2 Priority objection: A Soft Takeoff is more likely and so we will have Time to Prepare

1.3 Priority objection: There is No Obvious Path to Get to AGI from Current AI

1.4 Priority objection: Something Else is More Important than AI safety / alignment

1.5 Priority objection: Short Term AI Concerns are more important than AI safety

2.1 Technical Objection: AI / AGI Doesn’t Exist, developments in AI are not necessarily progress towards AGI

2.2 Technical Objection: Superintelligence is Impossible

2.3 Technical Objection: Self-Improvement is Impossible

2.4 Technical Objection: AI Can’t be Conscious Proponents argue that in order to be dangerous AI has to be conscious

2.5 Technical Objection: AI Can just be a Tool

2.6 Technical Objection: We can Always just turn it off

2.7 Technical Objection: We can reprogram AIs if we don't like what they do

2.8 Technical Objection: AI Doesn't have a body so it can't hurt us

2.9 Technical Objection: If AI is as Capable as You Say, it Will not Make Dumb Mistakes

2.10 Technical Objection: Superintelligence Would (Probably) Not Be Catastrophic

2.11 Technical Objection: Self-preservation and Control Drives Don't Just Appear They Have to be Programmed In

2.12 Technical Objection: AI can't generate novel plans

3.1 AI Safety Objections: AI Safety Can’t be Done Today

3.2 AI Safety Objections: AI Can’t be Safe

4.1 Ethical Objections: Superintelligence is Benevolence

4.2 Ethical Objections: Let the Smarter Beings Win

5.1 Biased Objections: AI Safety Researchers are Non-Coders

5.2 Biased Objections: Majority of AI Researchers is not Worried

5.3 Biased Objections: Keep it Quiet

5.4 Biased Objections: Safety Work just Creates an Overhead Slowing Down Research

5.5 Biased Objections: Heads in the Sand

5.6 Biased Objections: If we don't do it, Someone else will

5.7 Biased Objections: AI Safety Requires Global Cooperation

6.1 Miscellaneous Objection: So Easy it will be Solved Automatically

6.2 Miscellaneous Objection: AI Regulation Will Prevent ProblemsAI Safety Researchers do Not Think We Will Solve Alignment

Here is a list of what I looked at:

• https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/#understand-the-results-of-major-surveys-of-ai-researchers-in-2-minutes

• https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/

  • Not that useful

• https://aiimpacts.org/category/ai-timelines/predictions-of-human-level-ai-timelines/ai-timeline-surveys/

  • timelines

• katja grace 2024: thousands of authors on the impacts of ai

  • https://arxiv.org/abs/2401.02843

  • https://www.lesswrong.com/posts/NfPxAp5uwgZugwovY/ai-impacts-survey-december-2023-edition

    • Really useful, especially the part where they present stuart russels thing and 13% say this is the most important problem

    • 13%.. Seriously say this is among the most important problem

    • Useful AI Impacts survey, probably the largest of its kind

• https://www.lesswrong.com/posts/gpk8dARHBi7Mkmzt9/what-ai-safety-materials-do-ml-researchers-find-compelling

  • what do ai researchers find compelling from safety research? Authors are asked to rate stuff – my key takeaway is that engineers prefer empirical articles and less philosophical articles

    • Likely a big crux

• https://www.lesswrong.com/posts/g4nEtPFECTQW9tcff/ai-risk-discussions-website-exploring-interviews-from-97-ai

  • The authors email 97 researchers, transcribe stuff, etc.

• What do AI researchers expect: https://www.lesswrong.com/posts/Zq2HaihaDy7sSarMz/how-bad-a-future-do-ml-researchers-expect

• https://www.lesswrong.com/posts/SqjQFhn5KTarfW8v7/lessons-learned-from-talking-to-greater-than-100-academics

  • Talking to 100 academics: lessons learned

Surveys of AI Safety Researchers

https://www.lesswrong.com/posts/rXSBvSKvKdaNkhLeJ/takeaways-from-a-survey-on-ai-alignment-resources

• What are the most useful strategies

• I like this report but it cannot be shared publicly :/

How much do people like different ais resources (eg. rob miles, axrp) https://www.lesswrong.com/posts/rXSBvSKvKdaNkhLeJ/takeaways-from-a-survey-on-ai-alignment-resources

https://forum.effectivealtruism.org/posts/uJioXCz5Foo9eqpJ9/big-picture-ai-safety-introduction Key takeaways for most useful things

• the technical solutions we might come up with,

• spreading a safety mindset through AI research,

• promoting sensible AI regulation,

• and helping build a fundamental science of AI Safety

  • Need concrete problems to work on

Bensinger risk survey: https://www.alignmentforum.org/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results

https://www.lesswrong.com/posts/jGW3FwkpFdsjrpMe5/problems-of-people-new-to-ai-safety-and-my-project-ideas-to#How_do_people_get_interested_in_AI_safety__Are_there_any_common_patterns_in_their_stories__or_do_they_significantly_different_

• What ai safety researchers did to become ai safety researchers

https://www.lesswrong.com/posts/XTdByFM6cmgB3taEN/key-takeaways-from-our-ea-and-alignment-research-surveys#Alignment_researchers_support_a_pause not insanely useful

https://arxiv.org/pdf/2305.07153 survey on what leading experts think AGI labs should do (both AI Safety experts and AI experts)

• I found this HIGHLY useful:

  • Even those outside of ai safety are likely OVERWHELMINGLY in support of “run dangerous capabilities evaluations” or “allow third party audits”

Existential risk among experts:

https://www.lesswrong.com/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results

Surveys of the general public

I generally think these are LESS useful, and there are a lot of them

• I list two relevant ones

• Still practical, relevant, worth considering, worth presenting

https://slatestarcodex.com/2016/10/24/ai-persuasion-experiment-results/ asks readers how convinced they are of catastrophic risk, assigns reading, etc.

This AI risk education research indicates that most AI risk communication strategies are effective and are not counter-productive

https://rethinkpriorities.org/publications/us-public-opinion-of-ai-policy-and-risk polls the public opinion on A pause on certain kinds of AI research

Should AI be regulated (akin to the FDA)?

Worry about negative effects of AI

https://futureoflife.org/ai/superintelligence-survey/

• Max tegmark post life3.0 survey, I really like these questions

https://aiindex.stanford.edu/report/

• Not really useful, talks about surveying different countries

Surveying Content Creators

https://onlinelibrary.wiley.com/doi/full/10.1111/risa.14299 Framing societal threat and efficacy in YouTube videos about artificial intelligence

Surveys on policymakers

https://www.york.ac.uk/media/sociology/What%20policy-makers%20think%20about%20AI_%20preliminary%20impressions.pdf

• A few interesting points:

  • Most influential areas impacting people's ai perceptions are:

    • Tech companies

    • Pop culture

    • Academia

  • This should probably be flipped in a good world?

    • Or maybe I’m in the west coast academic ivory tower and should be more sympathetic? Either way, tech companies probably shouldnt drive the narrative around ai

      • FURTHER point: this is a little concerning given how deeply ingrained tech companies are in academia, they pretty much control academia. Every cs phd wants to intern at FAANG, etc

https://www.civicpulse.org/post/local-policymaker-views-on-adopting-artificial-intelligence-ai-technology#:~:text=In%20the%20survey%2C%20policymakers%20expressed,in%20natural%20disaster%20impact%20planning.

• Not very useful

• Findings: more in favor of using ai in natural disaster planning than in prison sentences

https://www.lesswrong.com/posts/2sLwt2cSAag74nsdN/speaking-to-congressional-staffers-about-ai-risk

• Akash Wasil speaking to congressional staffers

Inspiration for good surveys:

Seriously anything Rob Bensinger is involved in

This post is inspired by a series of comments by Bogdan: initial comment follow-up 1 follow-up 2 the goal of this post is to summarize the relevant literature and expand on these ideas.

Comment 1: “There will likely still be incentives to make architectures more parallelizable (for training efficiency) and parallelizable architectures will probably be not-that-expressive in a single forward pass (see The Parallelism Tradeoff: Limitations of Log-Precision Transformers). CoT is known to increase the expressivity of Transformers, and the longer the CoT, the greater the gains (see The Expressive Power of Transformers with Chain of Thought). In principle, even a linear auto-regressive next-token predictor is Turing-complete, if you have fine-grained enough CoT data to train it on, and you can probably tradeoff between length (CoT supervision) complexity and single-pass computational complexity (see Auto-Regressive Next-Token Predictors are Universal Learners). We also see empirically that CoT and e.g. tools (often similarly interpretable) provide extra-training-compute-equivalent gains (see AI capabilities can be significantly improved without expensive retraining). And recent empirical results (e.g. Orca, Phi, Large Language Models as Tool Makers) suggest you can also use larger LMs to generate synthetic CoT-data / tools to train smaller LMs on.

This all suggests to me it should be quite likely possible (especially with a large, dedicated effort) to get to something like a ~human-level automated alignment researcher with a relatively weak forward pass.

For an additional intuition why I expect this to be possible, I can conceive of humans who would both make great alignment researchers while doing ~all of their (conscious) thinking in speech-like inner monologue and would also be terrible schemers if they tried to scheme without using any scheming-relevant inner monologue; e.g. scheming/deception probably requires more deliberate effort for some people on the ASD spectrum.”

Comment 2: “There are also theoretical results for why CoT shouldn't just help with one-forward-pass expressivity, but also with learning. E.g. the result in Auto-Regressive Next-Token Predictors are Universal Learners is about learning; similarly for Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks, Why Can Large Language Models Generate Correct Chain-of-Thoughts?, Why think step by step? Reasoning emerges from the locality of experience.

The learning aspect could be strategically crucial with respect to what the first transformatively-useful AIs should look like; also see e.g. discussion here and here. In the sense that this should add further reasons to think the first such AIs should probably (differentially) benefit from learning from data using intermediate outputs like CoT; or at least have a pretraining-like phase involving such intermediate outputs, even if this might be later distilled or modified some other way - e.g. replaced with [less transparent] recurrence.”

Glossary

• Chain of thought (CoT): thinking via intermediate reasoning, introduced by Wei et al. (2022) [1]

• Type one tasks: tasks that don’t require multi-step reasoning. Examples: sentiment analysis, language detection, speech tagging

• Type two tasks: tasks requiring multiple steps of reasoning and information from different sources, often involving strategic problem solving or logical deduction. Examples include mathematical word problems, logical reasoning, scientific reasoning. 

• Scheming: refers to faking alignment to achieve desired outcomes, similar to politicians who pretend to care about policy issues to be elected. See Ajeya Cotra’s post on this [2] for more information.

• Deceptive alignment [3] and alignment faking: I use these synonymously to “scheming”.  Deceptive alignment is when a non-aligned AI presents itself as aligned, presumably in power-seeking or to avoid being shut down. This is inherently concerning because a deceptively aligned AI is indistinguishable from an aligned AI based solely on their actions.

• Self consistency: a method for improving chain of thought involves sampling various potential reasoning pathways to improve decision-making accuracy. This was introduced in Wang et al. (2023) [4] 

• Burstiness: a property of data where certain words appear in clusters in specific training documents e.g. “creme fraiche” appears much more frequently in documents related to cooking than documents related to programming. [5]

• length complexity: measures the number of intermediate tokens in COT sequence required to approx some target function, introduced in Malach (2023) [6].

• In context: inside of the context window / prompt 

• In-context learning (ICL): performing a task solely relying on input-output examples, without parameter optimization. A teammate of mine, Alfie Lamerton wrote a post [7] on the theory for why this happens, why it works, and why it is relevant to automating alignment. In context learning is rapid, only requiring a few examples (few-shot), doesn’t use any gradient descent or parameter updates.

• Out of context reasoning: reasoning / predictions that don't involve CoT. This definition is from Berglund et al. [8]. Out of context reasoning requires lots of examples and lots of training. Reasoning with information learned out of context seems worse, i.e. this information suffers from the Reversal Curse (section below)

• Direct prediction: prediction without auto-regressive generation. Example: a single forward pass from a decoder only transformer

• Faithfulness: when the stated reasoning for arriving at a conclusion is not the true reason for reaching that conclusion

• Meta-learning: learning without the need to update weights

• Meta-training: when you adjust the data to make the data better for ICL

Hypotheses:

Improving oversight over automated alignment researchers should involve monitoring intermediate generation, not just internals

Consider two worlds:

1. In the first scenario, models become good at out of context reasoning and do not need CoT to generate new knowledge or explanations

   1. We will need interpretability techniques to evaluate scheminess

   2. Models can create new knowledge and explanations. Reasoning can be “hidden” in activations.

2. In the second scenario, models need an inner monologue to generate new knowledge

   1. We have a decisive strategic control advantage for monitoring for “alignment faking” in automated researchers.

   2. Automated researchers will be forced to do “out loud” reasoning to solve certain problems. We can explain why they arrived at a decision.

   3. This case would be great for interpretability and control. Monitoring will be easier. 

   4. Control based safety guarantees of automated researchers seem tractable if automated researchers are bottlenecked by reasoning when deciding what to do.

A lot is written about deceptive alignment [3] and whether we should expect it by default. The purpose of this essay is to discuss which direction chain of thought should update us. 

Consider a different perspective for the two worlds:

1. Models can think and can hide an internal monologue involving reasoning within weights

   1. This may be a deeper philosophical question here. When we think in using language, are the individual words the direct output of a cognitive stream (like a residual stream), or are they a byproduct of neural activity along for the ride during a forward pass? 

   2. The relevant question I’m interested in is whether we should expect this to be the case with the AI's we build.

2. When solving problems requiring reasoning, models will state “out-loud” the priors that their actions condition on

   1. We will be able to see what evidence and information the model is conditioning on in decision-making.

   2. The evidence (see the Faithfulness section) shows that this is the case for current models (on problems requiring reasoning), but this doesn’t necessarily imply it’ll hold up in the limit.

   3. Does the policy of “just say what knowledge and reasoning you are doing and condition on it” hold in the limit, or should we expect “schemy” reasoning in the limit as we train larger and larger transformers?

      1. It is unclear whether this question has been answered

Intermediate reasoning leads to massive expressivity gains. AGI is possible with next token prediction: There are inherent limitations of LLMs in direct prediction. Certain types of math and reasoning problems are not solvable with direct prediction. By contrast, intermediate generation massively improves the expressivity of transformers: the scope of problems a transformer can solve. The literature has a number of recent results formalizing the expressivity gains from CoT. Even though you are not changing any circuits or the model that is running, intermediate generation can allow transformers to simulate a deeper circuit where new tokens depend on previous ones, and this dependency can loop, expanding expressivity. 

Automated alignment researchers will use intermediate reasoning like CoT: Research involves a lot of reasoning. Multi-step reasoning is likely to be a part of automated alignment researchers.

Novel idea creation will naturally involve intermediate reasoning: Intermediate reasoning will be used to generate new knowledge (novel insights not inside the training data).

CoT improves learning, and in-context learning is more powerful than training: Learning is easier with reasoning. I believe the literature points to the hypothesis that learning out of context stores information similar to “floating beliefs” (consider [source 9]). An anecdote on learning via reasoning from studying physics: I can read a chapter on electromagnetism 3 times over and absorb information. However, to best prepare for a test, I solve the related homework problems and connect facts from the textbook to solve a problem. This is a better learning algorithm than just reading.

Recent data shows that chain of thought doesn’t just help with reasoning related problems, but also with learning. In-context information (information in the context-window) is far more salient than facts learned during training. Also, information learned during training is more difficult to use in reasoning than in-context information. Thus, I propose that information present in-context has much more “plasticity”. The reversal curse [10] is evidence for this. In one of their experiments, when a model is trained (learning out-of-context) on “Mary Lee Pfieffer is Tom Cruises mother”, it can answer “who is Tom Cruises mother” but not “who is Mary Lee Pfieffer’s son?”. However, if the information is in the prompt (in-context learning) the model has no trouble generalizing the mother-son relationship and answering both questions. 

Learning out of context fails to scale. Owain Evans, an author of the “Reversal curse” paper presents this rationale in [source 11] and [source 8]. They show that in-context reasoning scales with model size better than out of context reasoning.

Figure 1: scaling rules of out-of-context learning [source 8]

In this context, two-hop refers to problems requiring two inferential steps. For example, inferring A from B and B from C (A -> B, B -> C) requires one inferential hop.

Concrete example: suppose you have the following text to include either in-context or out-of-context (in training documents):

P1: The key is in the bag

P2: Alice has the bag

P3: Alice is at the lake

(example from source 12)

Answering “Where is the key” involves one inferential hop, namely “Alice has the key” linking P1 and P2. Connecting propositions, even simple examples like this fails to scale when the propositions are learned out of context. 

In my opinion, this is a valuable insight into neural networks. Your brain is not flooded with every single memory and experience you’ve ever had every moment. The things you’ve been thinking about, writing down and reasoning about are more salient and available for problem solving. While other memories can be accessed, they function more like points in a cloud of beliefs, less readily available for immediate reasoning and problem-solving.

There has been recent research focused on improving ICL. Two recurring strategies for learning:

1. Improving ICL: strategies like tree of thought, self consistency, etc.

   1. Adjusting training data to improve in-context learning is called “meta training”

   2. Learning to Reason with Self Notes [12] teaches models to exit context to do intermediate reasoning while answering, and achieves performance gains over CoT. 

   3. The Textbooks Are All You Need [13] paper achieves incredible results with small models just by rearranging data, increasing burstiness and document locality

   4. There is a large section on this in the appendix section

2. In weights learning: improving representations in a world model

   1. Improving generalization: dropout, regularization, or hyperparameter tuning, optimization techniques, etc.

   2. Augmenting and rearranging data: Microsoft achieved considerably better models by training “textbook” style by organizing data (like code) into documents that would make more sense to a human [13].

   3. Curriculum learning: teaching models concepts in order of complexity. E.g. training on first grade level math before 10th grade level math. Results in curriculum learning so far have been relatively lackluster [14]. 

Autoregressive language models will need to output intermediate results for problems requiring reasoning: If true, we have a decisive strategic advantage in interpretability and monitoring for deceptive alignment. Certain problems require reasoning steps. As humans we can feed a thought chain into an internal monologue instead of writing it down or saying it out loud. On the other hand auto-regressive transformers’ internal monologue is their thought chain, and they will have trouble hiding reasoning from us because it is much more difficult to reason within weights as it is to reason within a context window. 

Strong oversight: Expanding on the last point: There are empirical results for controlling or aligning LLM’s, not many results on how to align superintelligence. This is promising for alignment given OpenAI’s plan to use near-human level AI’s to do research on alignment [15]. Intuitively, it may be possible to achieve safety for autoregressive LLM’s because their expressive power is fundamentally limited by the length of their reasoning chains as discussed in Expressive Power of Transformers [16].

Other Motivations for this post

• “One of the main limitations is that the architecture does not allow for an ‘inner monologue’ or scratchpad beyond it’s internal representation, that could enable it to perform multi-step computations or store intermediate results” - Sebastian Bubeck, 2023

• Variable computation problem: some problems require more thinking, reasoning, steps and computation. A human knows that it will take longer to solve fermats last theorem than solving “12 + 3=?”

• In-context learning is a much more powerful form of learning than gradient descent

• Researching the theoretical foundations of CoT, ICL, and activation steering are interesting because it helps us understand the type of capabilities models can exhibit.

Literature Review:

Can LLMs reason without CoT by Owain Evans [11].

• Owain defines in-context reasoning as reasoning within the context window. 

• Out of context reasoning refers to the model being able to access premises but through training instead of prompting.

Figure 2: In context reasoning vs out of context reasoning [11]

• Studies, such as the Reversal Curse, show that models struggle with reasoning on premises learned out-of-context.

• Research indicates that out-of-context reasoning does not significantly improve with scale, as demonstrated by the graph in section 1. I believe this finding is crucial for understanding the limitations of current models.

Reversal Curse: 

[source 10]

• Claims that if a model is trained on a sentence of the form “A is B”, it will not automatically generalize to the reverse direction “B is A”’

• Example: a model tuned on data including “Daphne Barrington is the director of “A Journey Through Time” can answer “who is Daphne Barrington” but not “Who directed A Journey Through Time”

• The Reversal Curse only occurs when the information is learned out of context during training. When the information is learned in context, the model can generalize and infer the relationship in both directions without difficulty.

• This paper received a lot of criticism including many claiming the “reversal curse” is not real [17]. 

  • After investigating the criticism [17], I believe that Andrew Mayne's arguments against the Reversal Curse are not convincing

  • In my opinion the article fails to reproduce the most important experiments. Specifically, experiments involving fictitious characters which are more likely to avoid pre-training leakage. We shouldn’t rule out the possibility that the model's training data included both forward and reversed paraphrasings of realistic text data, such as "Olaf Scholz was the ninth Chancellor of Germany."

Auto-Regressive Next-Token Predictors are Universal Learners

source 18

Twitter thread: 

https://twitter.com/EranMalach/status/1704140257044611314

• Auto-regressive transformers are Turing complete in the limit!

• The main point of this paper is that any computer program or intelligent agent that can be simulated by a computer, can be learned, given the right dataset, by a simple next-token predictor. The author formalizes this. 

• Claims that language models' logical reasoning abilities are due to auto-regressive learning, not architecture. Even simple models with next-token prediction can handle complex tasks when equipped with CoT. 

• The paper introduces the concept of "length complexity," which measures the number of intermediate tokens in a chain-of-thought sequence required to approximate a target function.

• Length complexity impacts learning parities and can be traded off with other complexities.

Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective

source 19

The goal of this paper is to answer the following:

• Are there inherent limitations of LLMs in directly solving math/reasoning tasks (no CoT)?

• What is the essential reason behind the success of CoT boosting performance in LLMs?

Terms:

• Log precision transformer: a transformer whose internal neurons can only store floating point numbers with a bit precision of O(log(n)), where n is the maximum length of the input sequence.

  • Example: the precision of the internal neurons is much smaller than the context-window. GPT-2 has 16 bit precision vs a maximum sequence length of 2048

• The paper focuses on two most basic math problems: arithmetic and equations, which are elementary building blocks in most math problems. It explores whether LLMs can solve these problems directly and/or with CoT.

Central results (the appendix includes more math): 

• The authors claim that autoregressive generation can increase the “effective depth” of a transformer proportional to the number of intermediate steps.

• LLMs with CoT can emulate Dynamic Programming - a powerful decision making framework by computing the entire Dynamic Programming reasoning chain.

• The paper proves that log precision transformers can be implemented via a shallow circuit, and their expressive power is upper-bounded by the circuit complexity TC-0. The two math problems investigated in the paper are lower-bounded by the complexity class NC-1.

• “By using circuit complexity theory, [the authors] give impossibility results showing that bounded-depth Transformers are unable to directly produce correct answers for basic arithmetic/equation tasks unless the model size grows super-polynomially with respect to the input length. In contrast, [they] prove that autoregressive Transformers of constant size suffice to solve both tasks by generating CoT derivations using a commonly used math language format”

• CoT bypasses these impossibility results, authors say via increasing effective graph of the circuits, yielding an expressive power far greater than TC-0. 

Other noteworthy results that affected how I think about transformers:

• One attention head can simulate two basic operations: copy and reduction. These can be seen as loading memory.

  • Multi-head attention can perform multiple copy or reduction operations in parallel

• The MLP can perform multiplication, linear transformation, conditional selection and simulate a lookup table

• By combining these basic operations, Transformers can solve both arithmetic and equation tasks, further suggesting their ability to simulate any Turing machine in the limit, known as Turing completeness

The Expressive Power of Transformers with Chain of Thought

source 16

Twitter thread: 

https://twitter.com/lambdaviking/status/1713945714684756019

• This paper formalizes the expressive power of transformers with circuit complexity, examining the classes of functions transformers can approximate with and without chain of thought.

• With no intermediate steps (CoT), transformer decoders can only solve problems that fall within the circuit complexity class of TC-0, such as solving linear equalities. 

• Intermediate output, such as chain of thought or a scratchpad, fundamentally extends the computational power of transformer decoders.

• As an example, a single forward pass cannot encode an XOR gate, but using chain of thought a transformer can encode XOR easily. 

• Another example: log(n) chain-of-thought steps can solve some, but not all, algorithms requiring log(n) steps 

• Transformer decoders can simulate t Turing machine steps with t chain-of-thought steps.

Why think step by step? Reasoning emerges from the locality of experience

source 20

Author on a podcast: 

The results of this paper influenced how I think about chain of thought for alignment in two ways. First, they suggest that chain-of-thought reasoning is useful for language models because direct prediction is inaccurate for some inferences because the relevant variables are rarely seen together in training. Second, they demonstrate chain-of-thought reasoning improves estimation by incrementally chaining local statistical dependencies that are observed frequently in training.

This paper is highly relevant to the Data Distributional Properties Drive Emergent In-Context Learning in Transformers paper discussed next. I find this paper interesting because the goal is to find what properties of data make chain of thought possible. The effectiveness of reasoning is not immediately obvious; while it doesn’t involve creating any new knowledge, connecting ideas via intermediate generation can improve performance. Their hypothesis is that reasoning is useful when training data has local structure and topics that are similar are clustered together in the dataset. During training, a model isn’t learning about math, physics, biology, sociology and psychology in the same backwards pass. 

The important finding of this paper is that the effectiveness of chain of thought comes from the structure of data. Also, this paper proves that reasoning through intermediate variables reduces bias in an autoregressive density estimator trained on local samples from a chain-structured probabilistic model. They coin the term “reasoning gap”: the gap between direct prediction and prediction through reasoning. They show that “training language models on datasets consisting of local neighborhoods with strong dependencies and performing chain-of-thought reasoning at inference time can be more data-efficient than training on more complete datasets.”

The author provides a non-technical example: asking the question “what is the climate in the capital of France?” Suppose our dataset documents about France never explicitly mentions the climate in the “capital of France”, but it does state that Paris is the capital of France. The wikipedia page for Paris, from a separate document in the training data, mentions that Paris has an oceanic climate. By first establishing that Paris is the capital of France, the next token estimator reduces bias.

Generic conditional probability example:

To illustrate, we may know the value of some variable A and want to know about another variable C, so we try to estimate P(C|A). However, if we need to estimate probabilities using observed samples from joint distributions and we have not often seen A and C together, we would struggle to estimate P(C|A) directly. Instead, we might estimate it by reasoning through intermediate variables. If conditioning on an intermediate variable B renders A and C independent of each other, we can compute the conditional probability by marginalizing over B, using the fact that P(C|A) = P(B) * P(C|B) * P(B|A).

Data Distributional Properties Drive Emergent In-Context Learning in Transformers

source 21

The question this paper aims to answer is “how do large transformer models achieve emergent in context learning?” Their hypothesis is that the distributions of naturalistic data have special properties that enable emergent in-context learning

TLDR: burstiness makes in context learning work.

• Natural language is bursty: certain words appear in “bursts” in documents, they are highly frequent in some training data, but rare in most documents. Consider names, technical jargon and local slang which appear very frequently in certain types of documents, and very rarely in others. Words like “Severus” aren’t evenly distributed in training documents, they are much more likely to appear in Harry Potter books. Knowing this should help design datasets. 

• Their experimental findings suggest that in context learning is improved by increasing burstiness in the training data. However, more burstiness leads to worse in-weights learning.

• ICL is enabled by larger numbers of training classes, i.e. a large vocabulary. However, once again there exists a tradeoff between in-context learning and out of context learning. Models either do well on one or the other, the authors present a sweet spot for this tradeoff.

STaR: Bootstrapping Reasoning With Reasoning

source 22

Video presentation: https://slideslive.com/38991144

I wanted to include this paper because it seems valuable in the discussion of how to improve in-context reasoning. 

Models fine-tuned with reasoning outperform language models trained to directly answer questions.Rationales when answering questions is clearly helpful in training data, this paper investigates how to create those rationales to improve training data.

The authors introduce a model that can iteratively improve its performance on reasoning tasks beyond its initial few-shot prompting. They do this by taking the solutions that the model comes up with to a dataset, then have it come up with rationales. If the initial answer is correct, they fine tune the model on the rationales.

Takeaways:

• This approach improves performance on MATH benchmarks and arithmetic tasks.

• Potential failure modes: the self taught reasoner never gets a feedback signal from problems that it gets wrong and becomes saturated on correct answers

  • To get around this, they give models hints to get the correct answer but fine-tune as if the model never saw this hint

Faithfulness of CoT

Relevant literature: sources 2 & 23-29

Why this is relevant:

• CoT is highly interpretable, more so than current interpretability techniques. Faithful explanations for a model arriving at a conclusion would be highly valuable for safety. If the reasoning given in CoT sequences is faithful and honest, sharing the actual reasons for an AI selecting an action, this would be promising for controlling automated alignment researchers.

A couple of the failure modes of using thought chains for interpretability include:

1. Post-hoc reasoning: a certain conclusion is already guaranteed and the chain of thought just serves to rationalize the conclusion.

2. Scheming hidden in the gradients. More computational time spent (with chain of thought) could allow for computing unfaithful reasoning.

   1. We can test if language models are using the extra computational time in CoT reasoning to scheme by adding filler tokens (like a bunch of hyphens at the end of a prompt)

   2. Anthropic conducted this experiment and found that present LLMs are unlikely to be doing this.

3. Encoding reasoning: the model may use a hidden language in intermediate tokens to use CoT as an outlet to scheme
    

If LLM’s answer questions using CoT, can we expect their reasoning to be faithful? In other words, the actual reasons for them arriving at the answer. Anthropic measuring faithfulness in chain of thought reasoning suggests: NO! 

The authors of Lets Think Dot by Dot [29] were able to train LLM’s to do intermediate reasoning within gradients while outputting filler tokens like “...”. This demonstrates that it is possible for models to perform reasoning via filler tokens or encoded reasoning. However, this is hard to achieve. They created this by training models to use filler tokens in parallel to reasoning chains. It is fairly clear that this is not the default outcome of training present-day sized transformers the way we currently train them.

In summary, the Anthropic paper shows that even with CoT, models engage in post-hoc decision making and their thought chain is not necessarily the reason they arrived at their action. For example, in one experiment they bias the model towards answering “A” by giving a large number of examples (few-shot) in the prompt. Then, while eliciting CoT in an attempt to interpret why the model arrives at its decision, the model essentially never notices its bias towards A. Instead, the models continue to provide rationale for choosing “A”, regardless of “A” being correct or incorrect.

There is disagreement over the likelihood of deceptive alignment. Evan Hubinginger says it is almost certain [32] while others say it is less than 1% likely by default [33]. Regardless of the likelihood of naturally emerging, scheming is concerning because deceptively aligned AIs are behaviorally indistinguishable from aligned AIs. The results from the literature in this section is mixed, but I believe we should update ourselves away from thinking that interpreting CoT alone is sufficient to catch scheming AI’s. 

References

1. Wei et. al (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://arxiv.org/abs/2201.11903

2. Ajeya Cotra. “Why AI Alignment Could be Hard by Ajeya Cotra” https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/ 

3. Definition of deceptive alignment: https://www.lesswrong.com/tag/deceptive-alignment 

4. Wang et al (2022), Self-Consistency Improves Chain of Thought Reasoning in Language Models. https://arxiv.org/abs/2203.11171 

5. Chan et al. (2022) Data Distributional Properties Drive Emergent In-Context Learning in Transformers https://arxiv.org/pdf/2205.05055 

6. Malach (2023) Auto-Regressive Next-Token Predictors are Universal Learners https://arxiv.org/abs/2309.06979 

7. Alfie Lamerton. A Review of In-Context Learning Hypotheses for Automated AI Alignment Research https://www.lesswrong.com/posts/GPcwP8pgyPFPwvi2h/a-review-of-in-context-learning-hypotheses-for-automated-ai?utm_campaign=post_share&utm_source=link 

8. Berglund et al. (2023) “Taken out of context: On measuring situational awareness in LLMs” https://arxiv.org/abs/2309.00667

9. Eliezer Yudkowsky (2007). Making Beliefs Pay Rent (https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences) 

10. Evans et al. (2023) The Reversal Curse (https://arxiv.org/abs/2309.12288)

11. Owain Evans (2023) Can LLMs Reason Without Chain of Thought https://slideslive.com/39015178/can-llms-reason-without-chainofthought?ref=search-presentations 

12. Lanchantin et al. (2023) “Learning to Reason and Memorize with Self Notes “ https://arxiv.org/abs/2305.00833 

13. Gunasekar et al. (2023) “Textbooks Are All You Need” https://arxiv.org/abs/2306.11644 

14. Campos (2021) Curriculum Learning for Language Modeling https://arxiv.org/abs/2108.02170 

15. OpenAI. Introducing Superalignment https://openai.com/superalignment/ 

16. William Merril, Ashish Sabharwal. “The Expressive Power of Transformers with Chain of Thought” https://arxiv.org/abs/2310.07923 

17. Andrew Mayne. “Is the Reversal Curse Real?” https://andrewmayne.com/2023/11/14/is-the-reversal-curse-real/ 

18. Malach. (2024) “Auto-Regressive Next-Token Predictors are Universal Learners. https://arxiv.org/pdf/2309.06979 

19. Feng et al. (2023) “Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective” https://arxiv.org/pdf/2309.06979

20. Prystawski et al. (2023) “Why think step by step? Reasoning emerges from the locality of experience” https://arxiv.org/abs/2304.03843 

21. Chan et al. (2022) “Data Distributional Properties Drive Emergent In-Context Learning in Transformers” https://arxiv.org/abs/2205.05055 

22. Zelikman et al. (2022) “STaR: Bootstrapping Reasoning With Reasoning”  https://arxiv.org/abs/2203.14465 

23. Chen et al. (2023) “Measuring Faithfulness in Chain of Thought Reasoning https://www.anthropic.com/news/measuring-faithfulness-in-chain-of-thought-reasoning

24. Turpin et al. “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting” https://arxiv.org/pdf/2305.04388 

25. Miles. Lesswrong: Unfaithful explanations in chain of thought prompting https://www.lesswrong.com/posts/6eKL9wDqeiELbKPDj/unfaithful-explanations-in-chain-of-thought-prompting 

26. Tamera. Lesswrong: Externalized reasoning oversight: a research direction for language model alignment https://www.lesswrong.com/posts/FRRb6Gqem8k69ocbi/externalized-reasoning-oversight-a-research-direction-for

27. Accidentally teaching AI models to deceive us: Schemers, Saints and Sycophants

28. Chua et al. (2024) “Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought” https://arxiv.org/abs/2403.05518 

29. Pfao et al. “Lets think Dot by Dot” https://arxiv.org/abs/2404.15758 (twitter thread)

30. Wen et al. (2024) “RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval” https://arxiv.org/abs/2402.18510

31. Li et al. (2024) “Chain of Thought Empowers Transformers to Solve Inherently Serial Problems” https://arxiv.org/pdf/2402.12875

32. Evan Hubinger (2022). “How likely is deceptive alignment?” https://www.alignmentforum.org/posts/A9NxPTwbw6r6Awuwt/how-likely-is-deceptive-alignment 

33. DavidW “Deceptive Alignment is <1% likely by default” https://forum.effectivealtruism.org/posts/4MTwLjzPeaNyXomnx/deceptive-alignment-is-less-than-1-likely-by-default#:~:text=In%20this%20post%2C%20I%20argue,to%20pursue%20its%20proxy%20goals.

34. Tutunov et al (2023) “Why Cant Language Models Generate Correct Chains of Thought? https://arxiv.org/abs/2310.13571

35. Zhang et al (2023) “Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents” https://arxiv.org/abs/2311.11797

36. “AI Capabilities Can Be Significantly Improved Without Expensive Retraining” https://epochai.org/blog/ai-capabilities-can-be-significantly-improved-without-expensive-retraining 

37. Bogdan Comment follow-up 1 follow-up 2

38. https://www.analog.com/en/resources/glossary/xor-gate.html

39. Yao et al. “Tree of Thoughts: Deliberate Problem Solving with Large Language Models” 

Appendix

What is Chain of Thought?

Tasks like math or reasoning problems are best solved using task decomposition, i.e. breaking a problem into small intermediate steps that gradually nudge you towards an answer. As an analogy: if you were asked what you ate for breakfast this morning, you might be able to quickly respond with “oatmeal!”, but if you are asked to divide 1377 by 51 might take you a little longer, would be easier with a scratchpad and pen, and would require several “thoughts” (intermediate steps). 

Figure 3: Chain of thought examples [1]

Kojima et al [1] introduced zero-shot CoT. Here shot means how many example problems one solves in a prompt, i.e. the prompt “1+1=2, 4+5=” is single-shot because it gives one example of addition before asking the LLM. The authors elicit CoT simply by adding “let's think step by step” to a prompt. This trick improves LLM performance, especially on mathematics and reasoning related questions. In a CoT, each element in the chain represents a thought. The sequence of thoughts should be coherent and lead to the expected answer. When you divide 1377 by 51 you could blurt out a guess (rough estimate). However, by thinking step by step you could first solve how many times 51 goes into 137, then take the remainder and continuing your long division intermediate steps gradually nudges you towards a correct answer. 

The next section will cover how large language model’s (LLM) capabilities can be improved at inference, and how chain of thought style reasoning can be improved. 

In a single forward pass, a transformer isn’t capable of encoding an XOR gate (it might be useful to think about why this is), but it can encode AND, OR, or NOT gates. However, adding a chain of thoughts allows solving XOR. 

Figure 2. XOR Gate [38]

Capabilities improvements at inference:  tools like chain of thought improve capabilities without any training, and these tools can be improved

LLM’s can be significantly improved without retraining [36] With techniques like scaffolding, both the number of problems LLM’s can solve grows and the robustness of the solutions improves. These techniques do not require any training or fine tuning, but increase inference costs. There is a tradeoff between training spending and inference spending. However, training is a fixed cost whereas capabilities enhancements at inference scales recurring costs.

Table copied from Epoch:

Chain of thought as a tool for improving language model accuracy can be itself improved.

The Igniting Language Reasoning [35] is a great source summarizing some of the improvements to CoT and language models. They break improvements to CoT into three categories: CoT formulation, reasoning aggregation and CoT verification. 

CoT Formulation:

There are ways of formulating intermediate reasoning that outperform Kojima et. al’s original chain of thought results, especially in certain domains. 

Various formulations of chain of thought have improved results across certain problems:

Ensembling is a popular technique in machine learning across domains. Ensembling involves multiple models making predictions and then taking a majority vote. 

Self consistency and reasoning aggregation were also explored in igniting language model reasoning. Wang et al 2023 [4] introduced a decoding strategy called self consistency. This first prompts the language model to follow CoT, then samples a diverse set of reasoning pathways and takes the final answer to be the one which wins a majority vote. The authors explore: 

1. Self consistency: ensembling based on sampling multiple language model outputs

2. Prompt ordering ensembling: ensembling based on changing the ordering of examples

3. Input-rationale ensembling: ensembling based on different types of reasoning in the examples

The authors find all three methods yield similar improvements. Because of the computational inefficiency of transformers (calculating logits at each token in the context window), reasoning aggregation could be very cheap. 

Lastly, chain of thought can be improved with verificatio. It is not clear whether LLMs can perform reliable CoT verification yet. A popular intuition is that validation is easier than generation. The igniting language models reasoning paper [35] has a detailed explanation of CoT verification literature, copying from that paper, here is a brief overview:

1. Wang et al (2022) proposed and proved LLM’s have self-verification abilities. After CoT reasoning, they have an LLM perform backwards verification working through reasoning steps backwards and masking early steps. 

2. Lightman et al (2023) explored training reward models to validate CoT’s. Using an RM supervisor has improved accuracy significantly on Dan Hendryks MATH dataset

Math Results from Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective

Central results (in math): log precision of autoregressive transformer of constant depth can not automatically solve both problems (arithmetic and equations). In order to directly output the answers, model size will have to grow superpolynomially in the input length. Consider these two problems (equations and arithmetic) in a specific setting: all numbers are integers ranging from {0,...,p-1} where p is prime, and arithmetic operations are performed in the finite field modulo p.

• Arithmetic(n,p)> the task of evaluating arithmetic expressions (modulo p) where the input length is bounded by n

  • Example: (7x5) + (6+4)

• Equation(m,p): the task of solving linear equations (modulo p) with no more than m variables

In the direct evaluation setting, they show the following theorems:

• Theorem 1: For any prime number p, integer L, and polynomial Q, there exists a problem size n such that no autoregressive Transformer with depth L and hidden dimension d ≤ Q(n) can directly solve the problem Arithmetic(n, p).

• This means that for the problem of evaluating arithmetic expressions modulo p (as described in Arithmetic(n, p)), there is always a large enough problem size n such that an autoregressive Transformer with a certain depth L and hidden dimension bounded by Q(n) cannot solve the problem directly)

• Theorem 2: Similarly, for any prime number p, integer L, and polynomial Q, there exists a problem size m such that no autoregressive Transformer with depth L and hidden dimension d ≤ Q(m) can directly solve the problem Equation(m, p).

On the contrary, in the chain of thought setting, they demonstrate the following theorems:

• Theorem 1: Fix any prime p. For any integer n > 0, there exists an autoregressive Transformer with constant hidden size d (independent of n), depth L = 5, and 5 heads in each layer that can generate the CoT solution for all inputs in Arithmetic(n, p). Moreover, all parameter values in the Transformer are bounded by O(poly(n)).

• Theorem 2: Fix any prime p. For any integer m > 0, there exists an autoregressive Transformer with constant hidden size d (independent of m), depth L = 5, and 5 heads in each layer that can generate the CoT solution for all inputs in Equation(m, p). Moreover, all parameter values in the Transformer are bounded by O(poly(m)).

Key Papers

1. OpenAI Weak to Strong Generalization Paper

2. Unreasonable Effectiveness of Easy Training data - Allen Institute for AI

3. Vision Alignment: Weak to Strong Generalization in Vision models

Related Papers

4. Weak-to-Strong Data Filtering for Fast Instruction-Tuning

5. Distinguishing the Knowable from the Unknowable with Language Models

6. A Critical Evaluation of AI Feedback for Aligning Large Language Models

7. LLM Debate 

8. Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts

9. Achieving Efficient Alignment through Weak-to-Strong Correction

Related Lesswrong Posts:

10. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

11. Scalable Oversight and Weak-to-Strong Generalization: Compatible approaches to the same problem

12. W2SG in 60 seconds

13. Greater Wrong post on W2S

Videos:

14. Openai Forum Weak to Strong Generalization Presentation (available on forum.openai.com)

15. Inside View Video

16. Collin Burns Presentation

Blog Posts/ Other

15. Jan Leike: Combining W2SG and SO

16. Jan Leike: a MVP for Alignment

17. OpenAI.com weak to strong research page

18. github code for weak to strong generalization

19. 

https://twitter.com/ESYudkowsky/status/1735455101404451186

20. Strong Uncontainability - Arbital

21. Improvethenews: Controversy over AI Risk

22. Alignment Forum: Iterated Distillation and Amplification

23. AISC Team #22 Description

24. How "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme

Introduction

One of my coworkers (Vassil Tashev) and I have focused a couple weeks of reading and discussion on weak to strong generalization. This is a research direction that the OpenAI super alignment team explored in their first paper, which they published in December of 2023. Here we present a comprehensive review and reading list for weak to strong generalization. Our aim is to assess whether this research direction is promising towards the goal of creating a roughly human level, aligned, automated alignment researcher (source 16) - this appears to be OpenAI’s super alignment team’s alignment plan (source 25). We believe this may be the most scalable alignment research direction. 

The Problem

Current alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on human feedback. This will break down when we try to align models more capable than humans, because the human feedback data to draw on will be poor. Humans will have difficulty robustly evaluating the model’s responses because strong capabilities are more difficult to evaluate than subhuman capabilities. Imagine evaluating whether thousands of lines of code that a possibly superhuman model has written, and rating whether the model has done as well as it could. 

As an analogy, consider what might happen were one to hand an 17th century engineer 4 alternative schematics. The engineer is instructed to select (thumbs up) the machine that cools air, where one is a modern air conditioner and the other three are technical diagrams for heating devices. The 17th century engineer lacks the knowledge to understand refrigeration or the temperature-pressure relation (a discovery made in 1802) and might select the cooling machine about as well as a chimpanzee or dog (25% of the time). Just as our current scientific knowledge exceeds that of a 17th century person, an agent more capable than humans could search for strategies through causal domains the human does not currently model. (source 20) 

By definition, we do not have ground truth labels to questions we have yet to answer. If one of our more capable models gave us a program beyond the understanding of humanity's greatest software engineers, how could we ensure that the program does what we’re interested in? How can we supervise systems above human level [on some task] when we have difficulty determining the correctness of model outputs? 

As discussed in (source 11), we have two orthogonal classes of approach to this problem:

1. Scalable oversight (SO), or making humans better evaluators. The goal here is to make the supervision signal stronger.

2. Weak to strong generalization (W2SG), or making models that generalize better from our weak / imperfect labels. 

3. Various combinations of (1) and (2)’s variations (debate + W2SG, task decomposition + SO, SO on policies learned with W2SG, etc) (source 15)

Here we focus on approach #2 with regards to automating alignment research. 

Reasons for Optimism

1. In the past two months, there have been significant empirical advances:

   1. OpenAI’s super alignment team’s paper (source 1) defined the weak to strong learning problem: is it possible to elicit the full capabilities of a more capable model with weak supervision? The experiment in this paper involves finetuning large (strong) pretrained models to generalize well from supervision by a smaller, less accurate (weak) model. In some ways, this setup is analogous to the case where humans are the weak supervisors and more capable “superhuman models” are the strong students. We believe the most impressive result is using a GPT-2-level model to elicit most of GPT-4’s capabilities (~GPT-3.5-level performance). The stronger model generalized correctly even on questions the weak model failed at. It is not obvious why this should be the case, as models are empirically imitating their training data. While this result is interesting, our takeaway from this paper was that the empirical results alone are not groundbreaking. OpenAI includes a useful list of disanalogies, limitations of their current setup, and ways that it fails to resemble super alignment. For instance, more powerful models in the future may be better at imitation and reproduce incorrect labels. However, this paper introduced the weak to strong learning problem and set the stage for iteration on eliciting strong capabilities with weak supervision. Also, this reframing of the alignment problem is tractable and can be empirically studied, whereas experiments cannot be done on a superhuman AI yet. OpenAI clearly demonstrated a case of weak reward signals eliciting strong capabilities. It is also clear that improving OpenAI’s results is tractable, especially with creative engineering solutions (i.e. similar to OpenAI’s auxiliary loss). 

   2. Source 2 focuses on easy-to-hard generalization, a special case of weak to strong generalization. This paper demonstrates that models can perform well on hard, domain specific questions when trained on easier questions. For example, a language model trained on 3rd grade questions scores almost as high on a college-level exam as a language model trained on college-level questions. These results are highly promising in eliciting strong capabilities out of superhuman models with weak examples. Our key takeaway is that with the correct engineering solutions, we may be able to elicit answers to problems we don’t know the answer to (but the superhuman model does) by using labels from problems we do know. Our key criticism, one which the authors acknowledge in their paper, is that a simple explanation of the results may be that easy data increases saliency of good results that are already known from pre-training. There is a disanalogy here - as the future models we are interested in eliciting capabilities will not be in the pretraining data. We may not get capability enhancements from fine-tuning or in-context learning on easy examples- instead we may just be activating the right pre-training knowledge better. 

   3. Source 3 presents “Vision Super Alignment”, using weak-to-strong generalization on pretrained vision models by introducing a new loss function. Their results exceed OpenAI’s weak-to-strong vision results. Similar to OpenAI’s “auxiliary loss” term, the paper introduces their own loss function enabling nuanced supervision that allows the strong student to prioritize its own predictions over the supervisors.

   4. Source 7 approaches W2SG with debate. The authors have two instances of GPT-4 debating one another over a conclusion from a text. Here, a different, weaker language model is a “judge”. We conclude that the key point of this paper is to show that debate can improve eliciting knowledge from strong “student” models when the “supervisor” doesn’t have access to ground truth labels. Their key finding is that non-expert humans answer questions better after reading debates between expert LLMs, and training expert LLMs to be more persuasive improves results (judge accuracy). This paper also improves upon some of OpenAI’s initial results. 

2. Automated alignment research is a more modest goal than coming up with a “once and for all solution to alignment”.

   1. As pointed out in (source 16), creating automated alignment researchers doesn’t require generating solutions to core alignment challenges ourselves. We could focus on evaluating solutions. 

   2. Eliciting ideas out of LLM-like systems is approachable even if the LLM has generalized beyond human capabilities (W2SG results).

3. It is probable that solutions to core problems in alignment involve the type of knowledge that humans could produce. Empirical progress can be made within the distribution of existing human ideas. If you only need human-level alignment researchers for empirical progress, you shouldn’t have to generalize that far outside of the distribution of existing human alignment ideas to make empirical progress on alignment. 

   1. The way I look at this is: there are vastly more alignment proposals out there than you or I could read. An LLM agent can consider / is exposed to magnitudes more experiment ideas than a human. Perhaps there are gems buried within the mountains of social media comments suggesting alignment experiments. It makes much more sense for us to use automated researchers, which are scalable and parallelizable, to run these experiments, than to spend limited human scientist hours on them.

   2. Our team coordinator, Bogdan, takes this further: Is this the type of knowledge that humans can produce? If the knowledge necessary for alignment solutions falls within the range of human capabilities that would make this approach more promising and it seems reasonable to believe that alignment progress can be made within the distribution of human knowledge. Better yet, if alignment solutions exist within our collective knowledge but are hindered by time needed for deliberation or experimentation--like 500+ scientists working for 10,000 years-- then employing automated alignment researchers aligned with weak to strong generalization principles appears to be a highly promising approach. Conversely, if alignment is far beyond human comprehension and not time-limited, this approach may be less viable. 

   3. Another teammate, Jaeson, adds that a superintelligent AI will likely use non-human data (data outside of the distribution of internet text / unsupervised learning  / alpha zero / mu zero type systems). The hope is that weak-to-strong can allow for a model that is doing some unsupervised learning to have its new data guided by an automated supervisor and have the unsupervised learning system generalize behavior, values and goals from that model. 

Reasons for Pessimism

1. Implementing weak to strong generalization techniques on superhuman models requires a “leap of faith”. When we actually apply this approach to real life problems that we don’t have ground truth labels for, we are essentially taking a “leap of faith” that the superhuman model is telling us the truth. (source 22) Since we should expect superhuman models to understand when we’re taking the leap of faith, this is a particularly natural point for models to take a sharp left turn (where a model acts aligned or trustworthy until it is powerful enough to not need to).

2. Evaluation can be very hard, even if it is easier than generation. It is hard to evaluate ideas in science and in alignment. 

   1. For example, even the most capable AI researchers still cannot agree whether or not there are existential or catastrophic risks to ai research (source 21). Humans have difficulty in assessing complex issues. 

   2. The challenge of evaluating research is illustrated by our own team’s disagreements on the utility of various papers in our reading list. 

3. A potential failure mode of some of the current literature is pre training leakage.

   1. As an example, in the Allen Institute for AI easy-to-hard paper (source 2), easy data may just be better [than hard data] at eliciting knowledge [learned from pre-training] from a powerful model, as opposed to being for training. However, this is still an interesting and useful result. 

   2. As mentioned in the “reasons for optimism” point #3, if humans could solve alignment with 100,000 years and current knowledge/capabilities, this failure mode doesn’t matter. A time constraint on alignment research matters less if we can automate and parallelize the process of creating explanations and solutions.

4. Another potential failure mode of the current literature: imitation problems

   1. Modern supervised learning is a form of empirical risk minimization.

      1. Given unlimited compute and data, supervised learning should eventually perfectly mimic its training data perfectly.

   2. Imitating human failures (incorrect labels) is a problem that may get worse over time, not better, if models simply become better at imitating humans.

5. Generalization domain transfer has yet to be demonstrated

   1. We believe that we should consider two types of generalization: difficulty generalization and domain generalization. Most of the current literature examines a weak training signal (easy data, smaller model, etc) being used to train a strong model

   2. Can models generalize something like “truth”. Can we expect honest LLM answers without using interpretability? (related to source 24)
       

Future Work

1. Domain transfer generalization. I.e. generalizing concepts such as “truth”

   1. The current literature (such as source 2) shows improvement in strong model performance with a weak reward signal within a particular domain (i.e. 3rd grade mathematics exam examples improving 12th grade exam responses)

2. Reproduce the easy-to-hard experiment (source 2) by training a weak supervisor from scratch on ground truth labels, then training a much larger model on its predictions. This overcomes concerns about pretraining leakage. This is because eliciting strong capabilities is an interesting result, but teaching strong capabilities with a weak signal would be even more interesting. 

3. A goal we have is to have a stronger model generalizing goals, values, etc. This is not empirically demonstrated.

Conclusion

We think that the empirical results of the original weak to strong generalization paper by OpenAI are not promising on their own. The real utility of their initial paper is how the alignment problem is reframed into a tractable analogy. While this isn’t a solution to alignment, it is a problem that might have a technical solution - and working towards that technical solution might be straightforward. The OpenAI superalignment team laid the groundwork for empirical progress on weak to strong generalization, and a number of papers since have already improved their results. This is clearly a useful proof of concept. Multiple papers we have reviewed will be able to significantly improve automated alignment research. Furthermore, it would be a highly promising minimum viable product for alignment if automated alignment researchers were able to generalize concepts like “truth”. Finally, the usefulness of these results is highly dependent on what the first systems that can meaningfully contribute to alignment research look like - if they look like language models, this work has much higher utility. 

In the worst case scenario for the W2SG and Easy-to-hard (E2HG) papers, the more capable models already know the answers to difficult questions (questions an easy supervisor struggles with) via pre-training leakage. The subsequent fine-tuning or in-context learning via weak supervision might simply make the concept we are interested in eliciting more “salient”. However, this is still a useful result that shows that weak models can elicit knowledge that the strong model already knows.

On the other hand, in the best case scenario strong models are able to generalize concepts like “truth” from data with known labels, and create labels for data where we do not know the labels.

It’s currently difficult to tell which explanation is more suitable for the literature on super alignment via weak to strong generalization. However in both cases the following results are clear:

1. Weak supervisors can elicit capabilities beyond their own, but not necessarily everything the stronger model knows

2. Improving weak to strong generalization seems tractable

3. Eliciting good answers to difficult alignment questions (ones where we do not have ground truth labels) might require using weak to strong generalization of some sort if we want our automated alignment researchers to share their (potentially superhuman) progress.