The framing around obedience versus malice is what clicked for me here. We're so used to thinking threats come from bad actors, but the real danger is perfect execution of flawed instructions. I've seen this play out with smaller automation—even simple if-then rules start producing unexpected results the moment they scale. Your point about not beign able to write down what 'good customer service' actualy means exposes the core problem: optimization without wisdom is just sophisticated failure.
The framing around obedience versus malice is what clicked for me here. We're so used to thinking threats come from bad actors, but the real danger is perfect execution of flawed instructions. I've seen this play out with smaller automation—even simple if-then rules start producing unexpected results the moment they scale. Your point about not beign able to write down what 'good customer service' actualy means exposes the core problem: optimization without wisdom is just sophisticated failure.
Thank you ! I keep thinking about the experiment of not being able to write 'what good customer service means' a lot !