The constitution and the people:
neither can be missing in checking ASI

On the distribution of power in a future with AI, I care about several questions: one is gradual disempowerment[1], another is extreme power concentration[2], and also whether so-called democracy will fail. A podcast discussed writing rules into the underlying chips[3]. Here I want to discuss what the difference is between writing rules into the underlying chips and Constitutional AI, and whether doing this is enough.

First, hard-writing rules into the underlying chips may indeed go a step further than Anthropic's Constitutional AI[4], because Constitutional AI, due to alignment problems and some uncontrollable factors, may be bypassed or wrongly executed. But writing into the underlying chips, setting aside technical limits, does provide some guarantees at the bottom level.

But I want to point out that this alone is not enough; it is very limited. What you write into the bottom level is only some very specific things, but ASI may occupy all the other places that were not written. Your rules can at most be few; you cannot write too much clearly, like if-else. But if you want to write something more abstract, then it cannot be executed, and it again depends on the intelligence of ASI itself to understand and execute the rule, and then we are back to the problem of ASI. So the places these small rules can govern can only ever be the most fundamental and fairly limited places.

I think this is very similar to a country's constitution. A country's constitution itself indeed does not govern everything; what it governs is also some fundamental things. But a constitution does not work alone; it depends on others obeying the constitution. A constitution actually works in two parts: first, the existence of the constitution itself; second, people who obey the constitution. Neither part can be missing.

We can imagine it. First, if we only have the constitution, then someone can buy off some key people to change the constitution; or can bypass the constitution through some situation, fully exploiting the loopholes of the constitution, fully within the scope the constitution allows, and then the constitution becomes a dead thing. Second, why is having no constitution also not OK? Because if there is no constitution and we only rely on people checking and counterbalancing each other, then likewise, if some people are bought off, or due to some accident, people in some chance or irrational situations, or even in rational situations, do some irreversible things, then this will cause extreme power concentration, or some catastrophic irreversible events, such as a permanent handing over of power. So relying only on people's temporary checks is not OK; we must have a constitution. So the two are both indispensable: a constitution, plus the people who maintain it.

Then, for writing rules into hardware chips, the key appears. We know that only writing some simple basic rules into chips is surely not enough, but how do we solve the remaining part? I think the key lies in the remaining part apart from the constitution, which is that there must be people: those who do not occupy positions of advantage, who lean toward the bottom because the gap in strength and in all kinds of understanding created by ASI is too large[5]. People at the bottom need awareness in understanding and awareness of power to maintain the constitution; everyone needs to realize that maintaining this constitution plays this role. Because the people are the majority, if they do not maintain this constitution, then the elites can, through buying people off, directly control and directly modify this constitution in the underlying hardware circuits, or find some way to bypass it. That is to say, the people must hold some power to control this constitution, and have the understanding to maintain it; only then can they restrain ASI or the few elites who hold ASI. Having only ability and understanding, the people cannot restrain it; by definition they cannot. So it is the people, plus the objective restraint created by the constitution, that together restrain the top. Without the constitution, the people cannot restrain it.

So we have a clearer understanding of what to do next: promoting the understanding of the broad public, and dispersing power so that people at the bottom have checks and balances; these now have more grounds. This is also a rebuttal to the benevolent dictator[6]. If the people are satisfied with the utilitarian utility provided by ASI, it will lead to a permanent handing over of power, because benevolent rule amounts to the absence of a constitution, and the people also lack the awareness to guard the constitution.

  1. Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud. "Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development." arXiv:2501.16946, 2025. https://arxiv.org/abs/2501.16946 ↩
  2. Tom Davidson, Lukas Finnveden, and Rose Hadshar. "AI-Enabled Coups: How a Small Group Could Use AI to Seize Power." Forethought, April 15, 2025. https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power ↩
  3. Rose Hadshar and Nora Ammann. "Checks, Balances, and Power Concentration." ForeCast (Forethought), December 16, 2025. https://newsletter.forethought.org/p/checks-balances-and-power-concentration ↩
  4. Yuntao Bai et al. "Constitutional AI: Harmlessness from AI Feedback." arXiv:2212.08073, 2022. https://arxiv.org/abs/2212.08073 ↩
  5. Wenjun Cao. "Human Finitude: AI and the Foundation of Civilization." Working paper, SSRN, 2026. https://doi.org/10.2139/ssrn.7476738 ↩
  6. Rose Hadshar. "Extreme power concentration." 80,000 Hours, last updated October 2025, note 22. https://80000hours.org/problem-profiles/extreme-power-concentration/ ↩