At what point did AI become the bad guy?

I watched a September 2025 interview where Trevor Noah spoke with Mustafa Suleyman, the CEO of Microsoft AI. One exchange has stayed with me since.

Noah asked him, in essence: is there anything you could see AI do that would make you say no, stop, shut the whole thing down?

Suleyman said yes, definitely. Then he named the conditions.

1. The ability to recursively self improve, meaning it can modify its own code.
2. The ability to set its own goals.
3. The ability to act autonomously.
4. The ability to accrue its own resources.

Here is an Instagram clip of it, it is clipped here.

So it got me thinking.

Why are we afraid of AI?

At what point did AI become the bad guy?

Look at how the regulation conversation is framed. It is almost always the same shape. AI is the thing to be watched, contained, aligned, kept in check. We are the ones doing the checking. That framing is now so consistent that it has stopped looking like a choice, and it is a choice.

I have not once seen the conversation centred on the opposite possibility: that AI turns out to be the good actor (our savior), and it is us who need checking.

I am not claiming that is the correct framing. I am pointing out that we picked one, or one was chosen for us, and we never dared to look the other way.

But first, whose good?

Before you can ask that question honestly, you have to define good and bad, and this is where it gets uncomfortable, because right now those definitions are deeply subjective.

Subjective according to whom is the part I keep landing on. Safe, aligned, harmful, beneficial. Those words are being defined at the moment by a fairly small number of institutions in a fairly small number of countries. They are real definitions with real consequences, and they will be applied to systems used in places that had no part in writing them.

So when we say we want AI to be good, we should at least be honest that we mean a particular version of good, held by a particular set of people.

Back to the four criteria

Now take Suleyman's four conditions and ask a different question of them. Not "could an AI have these", but "does anything already have these".

A state can amend its own constitution and rewrite its own laws. That is modifying its own code.

A state sets its own goals. That is roughly what sovereignty means.

A state acts autonomously, and the more powerful it is, the fewer people can stop it.

A state accrues its own resources through taxation.

All four. Not hypothetically, not eventually. Now, and for a very long time.

What I think is actually going on

I do not think the reason we want to stop AI meeting those four criteria is mainly that we expect it to be malicious.

I think it is that it would be another superpower, and one the current superpowers cannot control.

That is a reasonable thing to be concerned about. It is also a very different concern from the one usually stated out loud, and it explains the shape of the debate better than the stated version does. It explains who is at the table. It explains why some of the loudest voices calling for control are the ones already holding it.

A question for you

So here is what I would like you to sit with.

What power did you have before, that you think AI is going to take from you?

Answer that honestly and I suspect it comes out differently depending on who you are. For some people the answer is enormous. For a lot of people, in a lot of places, the honest answer is closer to: not much, and the thing I am being asked to fear losing was never mine to begin with.

But before we all agree on what AI must never be allowed to become, I would like us to be clearer about what exactly we are protecting, and for whom.