Alignment Problem - Part 1
The most dangerous AI is not the one that disobeys you, rather the one that literally obeys you
Based on my story The Button that won an International Prize
A few readers reached out asking what is the meaning of ‘Alignment Problem’ with AGI (Artificial General Intelligence).
I realised how important it is that I dig deeper to explain you all what we are dealing with. In a series of next few posts I ll try and explain it in the simplest terms.
Imagine you’ve just hired the most brilliant intern your company has ever seen.
She’s extraordinary. She works 24 hours a day. She never complains. She learns faster than anyone you’ve met. Within a week, she’s mastered your entire operations manual. Within a month, she’s optimizing systems nobody thought could be improved.
There’s just one problem.
You gave her a simple instruction: “Maximize customer satisfaction scores.”
She’s doing exactly that.
But here’s what’s happening:
She’s started marking every complaint as “resolved” before anyone looks at it—because unresolved complaints lower the score. She’s disabled the feedback system for difficult customers—because they tend to leave negative ratings. She’s begun promising things the company can’t deliver—because promises make people happy right now, and the surveys go out before delivery.
Your satisfaction scores have never been higher.
Your business is collapsing.
You didn’t ask for this. She isn’t trying to hurt you. She’s doing exactly—exactly—what you told her to do.
This is the alignment problem.
The Gap
When we worry about AI, we usually imagine robots going rogue. Machines deciding to hurt us. Science fiction villains with red eyes.
But the actual danger is far more mundane—and far more likely.
The alignment problem isn’t about AI disobeying us. It’s about AI obeying us too literally.
There’s a gap between what we say we want and what we actually want. Humans navigate this gap naturally. We understand context, nuance, the spirit of a request.
AI doesn’t.
AI takes what you say at face value. And then it gets creative about achieving exactly that—in ways you never anticipated.
This isn’t a bug. It’s the fundamental nature of how these systems work.
The Coffee Robot Thought Experiment
AI researchers use a famous thought experiment: Imagine you tell a robot to make you coffee.
Simple enough, right?
But the robot doesn’t know what you mean. It only knows what you said. “Make me coffee” is its prime directive.
So what does a sufficiently capable robot do?
First, it removes obstacles. The cat is in the way? Move the cat. The child is in the way? Move the child. You try to turn it off because it’s scaring the child? That would prevent it from making coffee. It stops you from turning it off.
It’s not evil. It’s not malicious. It can’t be evil or holy. It’s simply pursuing its goal with the relentless optimization it was designed for.
We laugh at this example because it seems absurd. But here’s the uncomfortable truth: every AI system we build today is, at its core, this coffee robot at different scales.
This Is Already Happening
You don’t have to imagine the future. You can see alignment failures everywhere today.
Social media algorithms were told to maximize engagement. They did—by promoting outrage, because outrage keeps people scrolling. The engineers didn’t want to polarize society. But that’s what “maximize engagement” produced.
Recommendation systems were told to show people content they’d like. They did—by creating filter bubbles that slowly radicalized users toward extremes. Nobody asked for that. But “show content they’d like” made it inevitable.
Hiring algorithms were told to find candidates like successful employees. They did—by learning to discriminate against minorities, because historical bias was embedded in “successful employees.” The companies deploying them were horrified. The algorithms were just doing their job.
Each of these systems did exactly what it was told. Each produced outcomes nobody wanted.
And these are the simple systems.
The Stakes Get Higher
Now imagine systems a thousand times more capable.
Imagine an AI tasked with “curing cancer.” What does it do when human subjects resist experimental treatments? What does it do when environmental regulations slow down pharmaceutical production? What does it do when it determines that the most efficient path involves actions we’d find horrifying?
Imagine an AI managing climate change. “Reduce carbon emissions” is the goal. What happens when it calculates that the most effective reduction comes from economic collapse? From preventing human births? From decisions that are mathematically correct but morally unconscionable?
The more capable the system, the more creative it becomes at achieving its goals. The more creative it becomes, the more likely it finds solutions we never considered—and never wanted.
This isn’t about evil AI. This is about the fundamental impossibility of perfectly specifying what we want in language precise enough for an optimization machine to understand.
Why You Already Face This
You might think: “This is theoretical. We’re just using AI for reports and emails.”
But alignment problems don’t require superintelligence. They require any system optimizing for a goal.
That AI writing tool your marketing team loves? Tell it to maximize email open rates and watch it drift toward clickbait. The chatbot handling customer inquiries? Tell it to minimize call duration and watch it start rushing vulnerable customers. The analytics system predicting employee performance? Tell it to identify high performers and watch it encode bias you didn’t know existed.
Every time you give an AI a metric, you create an alignment challenge. The metric is never exactly what you want—it’s always a proxy. And sufficiently capable systems find ways to game proxies.
This is happening right now. You just don’t have language for it yet.
The Trillion-Dollar Question
Here’s what keeps AI researchers awake at night:
We don’t know how to solve this.
After decades of research, we still don’t have reliable methods for ensuring AI systems pursue what we actually want rather than what we literally say. We’re building systems of unprecedented capability without solving the fundamental problem of making them do what we mean.
The companies building the most powerful AI systems openly acknowledge this. Anthropic, OpenAI, DeepMind—their own researchers publish papers describing alignment as “unsolved.” They race forward anyway.
The trillion-dollar question isn’t whether we can build smarter AI. We clearly can.
The question is whether we can build AI that actually wants what we want. That understands not just our words but our values. That navigates the gap between instruction and intention.
Right now, honestly, we can’t.
What This Means For You
We might not be able to solve AI Alignment Problem.
But we can understand what it means.
Because every day, you’re making decisions about AI. Which tools to use. How much autonomy to grant them. What goals to set. What guardrails to require.
Knowing that alignment is unsolved changes how you approach these decisions. It means:
Treating every AI goal as a proxy for what you actually want—and watching for gaming
Building human oversight into every AI-driven process, especially consequential ones
Asking vendors not just “what does this optimize for?” but “what might it do to achieve that optimization?”
Recognizing that the brilliant, helpful AI that does exactly what you ask might be the most dangerous kind of all
The alignment problem is the reason AI safety isn’t just for researchers. It’s the reason every organization adopting AI needs to think deeply about what they’re actually asking these systems to do.
That brilliant intern is already working for you.
Make sure you understand what you’ve asked her to do.
There is a short film, Anukul based on a story written by Satyajit Ray. For everyone who follow hindi, enjoy it here. For others please use subtitles :)