Published on January 28, 2026 3:07 PM GMT

Many arguments over the risks of advanced AI systems are long, complex, and invoke esoteric or contested concepts and ideas. I believe policy action to address the potential risks from AI is desirable and should be a priority for policymakers. This isn’t a view that everyone shares, nor is this concern necessarily salient enough in the minds of the public for politicians to have a strong incentive to work on this issue.

I think increasing the saliency of AI and its risks will require more simple and down-to-earth arguments about the risks that AI presents. This is my attempt to give a simple, stratospherically high-level argument for why people should care about Ai policy.

The point

My goal is to argue for something like the following:

Risks from advanced AI systems should be a policy priority, and we should be willing to accept some costs, including slowing the rate of advancement of the technology, in order to address those risks.

The argument

My argument has three basic steps:

1. Powerful AI systems are likely to be developed within 20 years.
2. There is a reasonable chance that these powerful systems will be very harmful if proper mitigations aren’t put in place.
3. It is worthwhile to make some trade-offs to mitigate the chance of harm.

Powerful AI systems in the next 20 years

The capabilities of AI systems have been increasing rapidly and impressively, both quantitatively on benchmarking metrics and qualitatively in terms of the look and feel of what models are able to do. The types things models are capable of today would be astounding, perhaps bordering on unthinkable, several years ago. Models can produce coherent and sensible writing, generate functional code from mere snippets of natural language text, and are starting to be integrated into systems in a more “agentic” way where the model acts with a larger degree of autonomy.

I think we should expect AI models to continue to get more powerful, and 20 years of progress seems very likely to give us enough space to see incredibly powerful systems. Scaling up compute has a track record of producing increasing capabilities. In the case that pure compute isn’t enough, the tremendous investment of capital in AI companies could sustain research on improved methods and algorithms without necessarily relying solely on compute. 20 years is a long time to overcome roadblocks with new innovations, and clever new approaches could result in sudden, unexpected speed-ups much earlier in that time frame. Predicting the progress of a new technology is hard, it seems entirely reasonable that we will see profoundly powerful models within the next 20 years^[1] (i.e. within the lifetimes of many people alive today).

For purposes of illustration, I think AI having an impact on society of something like 5-20x social media would not be surprising or out of the question. Social media has had a sizable and policy-relevant impact on life since its emergence onto the scene, AI could be much more impactful. Thinking about the possible impact of powerful AI in this way can help give a sense of what would or wouldn’t warrant attention from policymakers.

Reasonable chance of significant harm

Many relevant experts, such as deep learning pioneers Yoshua Bengio and Geoffrey Hinton, have expressed concerns about the possibility that advanced AI systems could cause extreme harm. This is not an uncontroversial view, and many equally relevant experts disagree. My view is that while we can not be overwhelming confident that advanced AI systems will definitely 100% cause extreme harm, there is great uncertainty. Given this uncertainty, I think there is a reasonable chance, with our current state of understanding, that such systems could cause significant harm.

The fact that some of the greatest experts in machine learning and AI have concerns about the technology should give us all pause. At the same time, we should not blindly take their pronouncement for granted. Even experts make mistakes, and can be overconfident or misread a situation^[2]. I think there is a reasonable prima facie case that we should be concerned about advanced AI systems causing harm, which I will describe here. If this case is reasonable, then the significant expert disagreement supports the conclusion that this prima facie case can’t be ruled out by deeper knowledge of the AI systems.

In their recent book, “If Anyone Builds It, Everyone Dies”, AI researchers and safety advocates Eliezer Yudkowsky and Nate Soares describe AI systems as being “grown” rather than “crafted”. This is the best intuitive description I have seen for a fundamental property of machine learning models which may not be obvious to those who aren’t familiar with the idea of “training” rather than “programming” an ML model. This is the thing that makes AI different from every other technology, and a critical consequence of this fact is that no one, not even the people who “train” AI models, actually understand how they work. That may seem unbelievable at first, but it is a consequence of the “grown not created” dynamic. As a result, AI systems present unique risks. There are limitations to the ability to render systems safe or predict what effects they will have when deployed because we simply lack, at a society level, sufficient understanding of how these systems work to actually do that analysis and make confident statements. We can’t say with confidence whether an AI system will or won’t do a certain thing.

But that just means there is significant uncertainty. How do we know that uncertainty could result in a situation where an AI system causes significant harm? Because of the large uncertainty we naturally can’t be sure that this will happen, but that uncertainty means that we should be open to the possibility that unexpected things that might seem somewhat speculative now could actually happen, just as things that seemed crazy 5 years ago have happened in the realm of AI. If we are open to the idea that extremely powerful AI systems could emerge, this means that those powerful systems could drastically change the world. If AI’s develop of a level of autonomy or agency of their own, they could use that power in alien or difficult to predict ways. Autonomous AIs might not always act in the best interests of normal people, leading to harm. Alternatively, AI company executives or political leaders might exercise substantial control over AIs, giving those few individuals incredible amounts of power. They may not wield this tremendous power in a way that properly takes into consideration the wishes of everyone else. There may also be a mix of power going to autonomous AIs and individuals who exercise power over those AIs. It is hard to predict exactly what will happen. But the existence of powerful AI systems could concentrate power in the hands of entities (whether human or AI) that don’t use this power in the best interests of everyone else. I can’t say this will certainly happen, but in my view it is enough reason to think there is a substantial risk.

Worthwhile trade-offs

Policy actions to address the risk from AI will inevitably have trade-offs, and I think it is important to acknowledge this. I sometimes see people talk about the idea of “surgical” regulation. It is true that regulation should seek to minimize negative side effects and achieve the best possible trade-offs, but I don’t think “surgical” regulation is really a thing. Policy is much more of a machete than a scalpel. Policies that effectively mitigates risks from powerful AI systems are also very likely to have costs, including increased costs of development for these systems, which is likely to slow development and thus its associated benefits.

I think this trade-off is worthwhile (although we should seek to minimize these costs). First, I think the benefits in terms of harm avoided are likely to outweigh these costs. It is difficult to quantify the two sides of this equation because of the uncertainty involved, but I believe that there are regulations for which this trade is worth it. Fully addressing this may require focusing on specific policies, but the advantage of proactive policy action is that we can select the best policies, The question isn’t whether some random policy is worthwhile, but rather whether well-chosen policies make this trade-off worthwhile.

Second, I think taking early, proactive action will result in a move favorable balance of costs and benefits compared to waiting and being reactive. One of the core challenges of a new technology that goes doubly so for AI is the lack of knowledge about how a technology functions in society and its effects. Being proactive allows the option to take steps to improve our knowledge that may take a significant amount of time to play out. This knowledge can improve the trade-offs we face, rather than waiting and being forced to make even tougher decisions later. We can choose policies that help keep our options open. In some cases this can go hand-in-hand with slower development if that slower development helps avoid making hard commitments that limit optionality.

The social media analogy is again instructive here. There is a growing interest in regulation relating to social media, and I think many policymakers would take a more proactive approach if they could have a do-over of the opportunity to regulate social media in its earlier days. Luckily for those policymakers, they now have the once-in-a-career opportunity to see one of the most important issues of their time coming and prepare better for it. One idea I’ve heard about in the social media conversation is the idea that because cell phones are so ubiquitous among teenagers its very challenging for parents to regulate phone and social media use within their own families. There’s simply too much external pressure. This has lead to various suggestions such as government regulation limiting phone access in schools as well as things like age verification laws for social media. These policies naturally come with trade-offs. Imagine that countries has taken a more proactive approach early on, where adoption of social media could have been more gradual. I think its plausible that we would be in a better position now with regard to some of those trade-offs, and that a similar dynamic could play out with AI.

Conclusion

Many arguments about AI risk are highly complicated or technical. I hope the argument I give above gives a sense that there are simpler and less technical arguments that speak to why AI policy action should be a priority.

1. This doesn’t mean it will take all 20 years, it could happen sooner.

2. This should come as no surprise for those who follow the track records of AI experts, including some of the ones I mentioned, in terms of making concrete predictions.