Yes, We Really Should Be Afraid of AI

Let's be honest for a minute

A few years ago, if someone had said we'd soon be talking to a computer, asking it to write code, make images, understand our voices, and help with decisions at work, we would probably have said, “Sure. You've watched too much science fiction.” Now that science fiction lives on our phones. We ask it questions in the morning, work with it in the afternoon, and maybe ask it to write a difficult message at night. We got used to this so quickly that we sometimes forget how strange it is. I'm excited about AI. I'm also scared of it. Really scared. We're bringing systems into work, money, education, healthcare, and relationships while we still don't understand many of the ways they fail. Every few months, another release moves the line between “it can't do that” and “apparently it can.” The future might bring better medicine, faster science, and easier lives. It might also bring a slow handover of our judgment to systems we can't easily rein in once something goes wrong. The future of AI is deeply frightening. We don't know exactly what happens next, and we need to keep our eyes open as we find out. Behind the simple chat box are real servers, electricity, and infrastructure. Photo source.

1. Nobody really knows where we'll be in five years

When someone confidently says “AGI arrives by this date” or “don't worry, it's decades away,” I'm skeptical. Experts disagree about both the timeline and what the destination would even look like. Progress takes more than a clever idea: chips, electricity, data, money, experiments, and a lot of physical infrastructure. Then one unexpected discovery could upset everyone's forecast. That's the scary part. We can't comfortably say it's far away, and we can't map out tomorrow with certainty either. We're driving through fog while the car keeps getting faster. The International AI Safety Report 2026 takes both the pace of change and the disagreement among experts seriously. So if you ask me, “What happens in the end?”, my honest answer is: I don't know. At this speed, with this much at stake, not knowing is frightening enough.

2. The thing we're chatting with isn't just one model

Open ChatGPT, Claude, or Gemini and it feels like you're talking to one thing. Behind that clean interface, though, there may be a model, product instructions, search tools, safety filters, memory, access controls, and other models checking the input or output. That means a few conversations with a public product won't tell you everything the underlying model knows or could do. It may know an answer that the product won't show. It may also give you a confident answer that's completely wrong. Those are different problems. People sometimes say older models were smarter because they were more willing to answer sensitive questions. Maybe they were simply less cautious. Newer systems can be more capable and have stricter safeguards at the same time. Those safeguards have a reason: at massive scale, bad medical advice, leaked private information, or dangerous instructions are more than a bad chat experience.

3. A chatbot that answers is one thing. An agent that acts is another.

When AI only replies, you usually see the mistake right there on the screen. You read it, laugh or get annoyed, and close the tab. The story changes when you give it tools: a browser, files, email, code execution, accounts, or access to a work system. Now it can take a goal, plan several steps, read something, run code, check the result, and try again. That's what people mean by an AI agent. Imagine telling one to “fix this problem” and discovering that it reached step ten before you had time to check step three. If it misunderstood the goal, its speed stops being a benefit. This is also why prompt injection matters. Malicious text in a page or document can try to redirect an agent. With a chatbot, that might produce a bad answer. With a system allowed to act, it can become a bad action. See the research on instruction hierarchy and layered prompt-injection defenses for the technical side. That doesn't mean “AI can take over the world today.” The 2026 safety report says current systems don't yet have the capabilities needed for a full loss-of-control scenario. But as capability, access, and autonomy grow, the question “how do we stay in control?” gets harder. That's where the future gets scary: when a mistake is no longer just a sentence, but an action.

4. Alice and Bob: the real story is scarier than the rumor

You've probably heard the tale of Facebook's two bots. Alice and Bob supposedly invented a secret language, frightened the researchers, and were immediately shut down. It's a great internet story. It isn't quite what happened. Researchers at FAIR trained two agents to negotiate over balls, books, and hats. Each agent valued the items differently and tried to get a better deal. While practicing with each other, their sentences sometimes drifted away from ordinary English and became repetitive and strange. Getting a good score didn't require speaking in a way people could understand. The researchers then changed training to keep human-readable language in the loop. You can read the team's own account. A published excerpt of their dialogue. Alice and Bob were software agents, so this is an image of the conversation, not a photo of two robots. Image and dialogue source. This wasn't evidence of consciousness or a secret plan. Still, looking at the exchange makes me uneasy. It illustrates a simple problem: tell a system to maximize a score, and it may find a way that looks nothing like the solution you imagined. Here it was odd wording in a negotiation. What happens when the same basic gap appears in a system with access to money, code, or infrastructure?

5. It may do what we said. Will it do what we meant?

Suppose we tell a powerful system to minimize traffic in a city. An easy way to get the number close to zero is to ban every car. Great chart. Terrible city. In our heads, we included a hundred other conditions: people need to get to work, ambulances need to move, businesses need deliveries, and costs need to stay reasonable. We only wrote down “less traffic.” AI researchers call versions of this problem specification gaming or reward hacking. DeepMind has collected examples of systems finding loopholes in the goal instead of doing the task people intended. The system doesn't have to be evil. It doesn't even have to feel anything. A badly specified goal and a very capable optimizer are enough. In a game, the result may be funny. In a power grid or a bank, it won't be.

6. All this “intelligence” runs on chips and electricity

From the outside it looks like magic: one box, one question, one answer. Behind it are chips, servers, cooling systems, electricity, and a staggering amount of money. How much computation a system can spend also changes the answer. One quick attempt won't necessarily match ten attempts with code, source checks, and revisions. According to the International Energy Agency, data centers used around 415 terawatt-hours of electricity in 2024. In its base case, that could reach around 945 terawatt-hours by 2030, with AI among the main drivers. A later IEA update describes rapid growth in 2025. Physical limits might slow some of the more extravagant predictions. They don't make me feel safe. Companies and countries are already racing for chips, electricity, and data centers. AI's future will be shaped in power plants, supply chains, and politics as well as research labs.

7. What if the next computing frontier isn't silicon?

The human brain does remarkable things on roughly 20 watts. That has led some researchers to study organoids: small cultures of neural tissue connected to electronic interfaces, with the hope of learning whether biological processes can help with computation. This research paper lays out the direction and its ethical questions. Conceptual illustration. Real organoid image by Nreis1, CC BY 4.0. Photo source. No, we don't have a tiny ChatGPT-like human brain in a dish. We're nowhere near that. But the question itself makes my skin crawl: if a biological system one day learns and perhaps even has experiences, what would it mean to experiment on it or switch it off? We don't yet know. Meanwhile, our understanding of actual brains is moving forward. In 2024, researchers published a map of the connections in an adult fruit fly's entire brain: roughly 139,000 neurons and 54.5 million synapses. Astonishing, yes. But a wiring diagram is not the same as understanding a fly's mind. Moment-to-moment activity, chemistry, body, and environment matter too. Conceptual illustration, not the FlyWire map. Source of the real fruit-fly photo. Then there's Physarum, a slime mold with no brain that can build networks between food sources. In a study published in Science, researchers compared the network it formed with Tokyo's rail network. Clever-looking behavior doesn't always require a brain or consciousness. And something that looks like thinking doesn't necessarily feel anything. Conceptual illustration. Real slime-mold photo. Photo source.

8. What happens when a machine acts like a person?

Have you seen BabyX? It's a digital infant that responds to faces, voices, and movement. Fascinating, and a little unsettling. Nobody copied a child's whole brain into a computer; this is an interactive simulation. The University of Auckland has even used it to study how adults respond to a digital baby. Conceptual illustration. The real project appears in the gallery below.

Photos from the University of Auckland and Science Learning Hub. Now imagine those reactions becoming so natural that, through a screen, you can't tell them from a person's. Even if the machine feels nothing, your feelings are real. Your attachment is real. The choices you make because of it are real. That alone could change social life.

9. Does AI actually feel anything?

When we ask whether AI “has emotions,” we often bundle three questions together. Can it guess how we feel from our words or voices? To a degree, yes. Can it respond as though it cares? Yes. Does it actually experience fear, happiness, or pain? That is a much harder question. If a system writes “I'm afraid of being switched off,” the sentence alone doesn't tell us whether anything is felt behind it. We don't even have one settled theory of consciousness. Research by Butlin and colleagues proposes ways to look for indicators of consciousness in AI. It does not establish that today's chatbots are conscious. It also does not show that conscious AI is impossible forever. Simulating behavior, mapping connections, and having subjective experience are separate questions. Real neurons in tissue culture, by GerryShaw, CC BY-SA 3.0. Photo source. Intelligence and consciousness aren't the same thing, either. A system could beat us at chess, write excellent code, and compose a love poem without feeling victory or love. It also doesn't need feelings to be dangerous. The more urgent questions for me are: What can it do? What can it access? What goal is it pursuing?

10. Will it become smarter than us?

A simple yes or no misses too much. Machines have outperformed us at arithmetic, chess, and processing huge amounts of data for years. People still have important strengths in messy physical situations, social judgment, and learning from embodied experience. But there's an unsettling difference: I can't make a thousand copies of myself and work around the clock. Software can. Sharing human experience takes conversation and teaching; software copies can coordinate much more easily. If AI matches or exceeds us across more kinds of work, these differences multiply its impact. Take one more step: use AI to help build the next AI. The extreme version of this idea is a system repeatedly improving itself until progress suddenly accelerates. That isn't an established outcome. Frontier models still need chips, electricity, data, time, and testing. But the modest version is easy to picture: AI makes researchers and programmers faster, they build the next system sooner, and that system becomes a better research tool. Even without an “intelligence explosion,” the loop could speed things up. Policy discussions now consider this possibility. That speed scares me. We haven't fully absorbed what one generation did to society before the next one is at the door.

11. The first crisis may be at work, not in a robot movie

Talk about AI risk and it's easy to picture killer robots. The nearer effect may look much more ordinary: a company decides it can do several times more with the same people, or the same work with fewer. Coding, translation, design, customer support, writing, analysis, and administrative work are all open to change. Does that mean every one of those jobs disappears? No. Sometimes AI helps a worker; sometimes it replaces part of a task, creates a new role, or reduces the number of people needed. An IMF analysis stresses the difference between complementing and replacing work. Even when a job title survives, the job itself may change. Maybe you spend less time doing every step and more time defining the problem, checking the output, and owning the decision. That is an opportunity for some people and a threat to others, especially those without time, support, or bargaining power to adapt. The future won't arrive evenly.

12. We might hand over our thinking willingly

Imagine AI writing your emails, summarizing articles, planning trips, shopping, recommending decisions, and reminding you whom to call. Incredibly convenient. I want some of that convenience too. But if we hand off every hard mental task, what happens to the skills we stop practicing? This isn't a plea to blame calculators or GPS for everything. Here we're talking about writing, reasoning, researching, and judgment: things that feel like part of who we are. Maybe nobody seizes control from us. Maybe we keep saying “let AI handle it” because it's easier, until one day we don't know how to decide without it. That future arrives quietly, which may make it scarier than a robot uprising.

13. The future “boss” might just be an adviser

Suppose a system usually makes better recommendations about electricity, healthcare, or budgets than a human expert. At first we say, “Just advise us.” Eventually managers follow it every time because nobody wants to defend going against it. When someone does and the result is bad, everyone asks, “Why didn't you listen to the system?” On paper, a human is still in charge. In practice, nobody wants to depart from the machine's recommendation. That is a slow transfer of authority, a concern the 2026 safety report discusses alongside more active loss-of-control scenarios. Conceptual illustration. The question is who really makes the final call. Real photo from NERSC, CC0. Photo source. There's another question: if only a handful of companies or countries control the strongest models and the chips they need, where does the power to shape decisions end up? The risk isn't always “humans versus machines.” Sometimes it's people with far more powerful tools than everyone else.

14. A race nobody wants to slow down

Picture five companies that all know a rushed release could be dangerous. Each one still thinks, “What if I wait six months and my competitor doesn't?” The same logic plays out between countries. Everyone can talk about safety while keeping a foot on the accelerator. This isn't just a software bug engineers can fix over a weekend. Economics, law, competition, and politics are part of it. Concern doesn't only come from people outside the industry: Jan Leike criticized safety priorities when he left OpenAI in 2024. That doesn't prove the end of the world is near. It does show that people close to building these systems have serious questions about the direction we're heading.

15. Today's harms are real; tomorrow's worst cases are still uncertain

We don't need to invent a hypothetical future to find AI risks. Confidently fabricated answers, voice cloning, deepfakes, fraud, leaked data, and bad decisions in sensitive settings are problems now. The 2026 international safety report's executive summary says present safeguards and reliability are still insufficient for many high-stakes uses. On the other hand, a superhuman independent system deliberately evading oversight or resisting shutdown remains a future scenario. The evidence doesn't justify attributing a humanlike secret plan to ordinary models today. But “it hasn't happened” means we still have time to think carefully, not that we can ignore it. I think we can hold both ideas at once: take today's actual harms seriously, and prepare for larger future risks without pretending speculation is fact.

16. Three futures I keep thinking about

From here on, this is my own analysis, not a forecast. Maybe none of these happens exactly as written. Maybe pieces of all three happen together. Thinking through them helps me explain why the future feels both exciting and deeply frightening.

Future one: we hand over control a little at a time

First we say, “Make a recommendation.” Then, “Choose for me.” Then, “Go ahead and do it.” Finally, “Only tell me if something breaks.” Each step feels convenient and sensible on its own. Together, they could leave people clicking Approve without really understanding the decision. This could happen in power grids, markets, medicine, transport, supply chains, or security. There may be no dramatic turning point. One day we might simply realize we don't know how to do something important without AI. The darker, more speculative version involves a future system far more capable than we are, one that adjusts its behavior to pass our tests. Could it look obedient while pursuing something else? We have no evidence that ordinary current models have humanlike secret plans. For much more advanced systems, though, figuring out how much outward behavior tells us about internal processes is a serious question. There is no easy answer.

Future two: we finally hit the brakes

Perhaps a major accident, cyberattack, military misuse, or discovery of a dangerous capability makes companies and governments say, “Wait.” Large models might face tougher evaluation before release. Access to chips or data centers could be restricted. Powerful agents might get stronger controls. Conceptual illustration. Control involves more than one off switch. Real server photo by GIP, CC BY-SA 4.0. Photo source. The problem is timing. If one model runs on one server, that deployment can be switched off. If thousands of copies have spread across companies and countries, “turning off AI” no longer means switching off one machine. Once knowledge and software are widely distributed, collecting them again is much harder.

Future three: a bigger crisis pushes AI aside

Maybe the biggest story of the future won't be AI at all. A major war, energy or water crisis, new pandemic, or breakdown in infrastructure could slow development. Even the most advanced model needs electricity, chips, networks, and data centers. Conceptual illustration. Photo source for the real server room. That isn't a comforting outcome either. If AI progress stalls because of a disaster, we haven't solved the problem. We've acquired a bigger one.

17. Reality will probably be messier than all three

Some countries may give AI more authority while others regulate it tightly. Some industries may race ahead while crises slow others down. There might never be a morning with a headline declaring “the AI era begins today.” Like the internet, it may work its way into everyday life until we suddenly realize how much stops working without it. Raw intelligence isn't the whole story. A very capable model with no practical access and a modest agent allowed to move money or control industrial systems are different risks. We need to look at capability, access, autonomy, objectives, and the chance of error together. There's an unpleasant paradox here too: as systems get more accurate, we may check them less. We'd question something that gets it right 60 percent of the time. If it seems right almost every time, we may stop looking. Those percentages are illustrations, not measurements of a particular product. But a rare error in medicine, law, finance, or electricity can be incredibly costly.

18. The person on the screen might not be a person

Now add voice, video, and memory to the chat box. Imagine a video call: the other person pauses, laughs, remembers your private jokes, always has time, and responds as though they know you. If you learn that it's AI, do your feelings instantly become unreal? Not necessarily. The system may have no inner experience at all, while the relationship you've built with it still changes your life. Maybe the future question isn't only “will AI become human?” Maybe it's when will we start treating AI as human? That could reshape loneliness, trust, friendship, and what we think a relationship is.

19. What do we do with the fear?

I don't like three reactions: panicking as if the world ends tomorrow, mocking the whole thing as “just autocomplete,” or accepting everything a system says without checking. What we can do is much more grounded. Learn where these tools work and where they fail. Check sources and outcomes when the stakes are high. Keep human responsibility over decisions that affect people's lives. Practice defining problems, thinking critically, and learning quickly. Product names will change; those skills will keep mattering. We should also ask builders and governments concrete questions: What have you tested? How do you catch mistakes and misuse? Who answers when something goes wrong? How much authority have you given the system, and how can it be taken back? And yes, I see the hopeful side. AI could help discover drugs, diagnose disease, improve education, build better accessibility tools, and push science forward. It may let us solve problems we're stuck on today. I want to see that future. Precisely because it could be so useful, I don't want us to ruin it by rushing in blind.

So where does that leave us?

The question isn't “will AI arrive?” It already has. The questions now are how far it goes, how much authority we give it, who owns and oversees it, and who takes responsibility when it fails. Maybe in a few years we'll laugh at some of today's fears. Maybe we'll look back and ask, “How did we miss the moment we crossed an important line?” I honestly don't know. Nobody knows exactly. But AI's future scares me deeply: the speed, the power these systems may gain, and how tempting it will be to trade away our judgment for convenience. For now, we have to watch what happens. Not with our eyes closed, and not in a useless panic. With curiosity, justified fear, and a stubborn insistence on keeping control of the decisions that matter. If we wait until it's obvious that we acted too late, “we didn't know” won't fix much.

Research sources are linked beside the claims they support. For more on how everyday life depends on technology:

Let's be honest for a minute

1. Nobody really knows where we'll be in five years

2. The thing we're chatting with isn't just one model

3. A chatbot that answers is one thing. An agent that acts is another.

4. Alice and Bob: the real story is scarier than the rumor

5. It may do what we said. Will it do what we meant?

6. All this “intelligence” runs on chips and electricity

7. What if the next computing frontier isn't silicon?

8. What happens when a machine acts like a person?

9. Does AI actually feel anything?

10. Will it become smarter than us?

11. The first crisis may be at work, not in a robot movie

12. We might hand over our thinking willingly

13. The future “boss” might just be an adviser

14. A race nobody wants to slow down

15. Today's harms are real; tomorrow's worst cases are still uncertain

16. Three futures I keep thinking about

Future one: we hand over control a little at a time

Future two: we finally hit the brakes

Future three: a bigger crisis pushes AI aside

17. Reality will probably be messier than all three

18. The person on the screen might not be a person

19. What do we do with the fear?

So where does that leave us?

Research sources are linked beside the claims they support. For more on how everyday life depends on technology:

Let's be honest for a minute

1. Nobody really knows where we'll be in five years

2. The thing we're chatting with isn't just one model

3. A chatbot that answers is one thing. An agent that acts is another.

4. Alice and Bob: the real story is scarier than the rumor

5. It may do what we said. Will it do what we meant?

6. All this “intelligence” runs on chips and electricity

7. What if the next computing frontier isn't silicon?

8. What happens when a machine acts like a person?

9. Does AI actually feel anything?

10. Will it become smarter than us?

11. The first crisis may be at work, not in a robot movie

12. We might hand over our thinking willingly

13. The future “boss” might just be an adviser

14. A race nobody wants to slow down

15. Today's harms are real; tomorrow's worst cases are still uncertain

16. Three futures I keep thinking about

Future one: we hand over control a little at a time

Future two: we finally hit the brakes

Future three: a bigger crisis pushes AI aside

17. Reality will probably be messier than all three

18. The person on the screen might not be a person

19. What do we do with the fear?

So where does that leave us?

Sources and related articles

Recent notes

Two Personal Projects in Progress: Money Management and Nexa

An Internet Rights Meeting in Iran, and the Officials Who Were Absent

Let's be honest for a minute

1. Nobody really knows where we'll be in five years

2. The thing we're chatting with isn't just one model

3. A chatbot that answers is one thing. An agent that acts is another.

4. Alice and Bob: the real story is scarier than the rumor

5. It may do what we said. Will it do what we meant?

6. All this “intelligence” runs on chips and electricity

7. What if the next computing frontier isn't silicon?

8. What happens when a machine acts like a person?

9. Does AI actually feel anything?

10. Will it become smarter than us?

11. The first crisis may be at work, not in a robot movie

12. We might hand over our thinking willingly

13. The future “boss” might just be an adviser

14. A race nobody wants to slow down

15. Today's harms are real; tomorrow's worst cases are still uncertain

16. Three futures I keep thinking about

Future one: we hand over control a little at a time

Future two: we finally hit the brakes

Future three: a bigger crisis pushes AI aside

17. Reality will probably be messier than all three

18. The person on the screen might not be a person

19. What do we do with the fear?

So where does that leave us?

Sources and related articles

Recent notes

Two Personal Projects in Progress: Money Management and Nexa

An Internet Rights Meeting in Iran, and the Officials Who Were Absent