AI Doomers Got the Superintelligence Thing All Wrong

Inner peace is all you need

I look at the stars on a pale black, moonless night and feel it deep in my bones. They’re showing me the way, lone lighthouses. “This sensation would only get better if I get there,” I think. I want to get there. Build a long staircase. Or a lightspeed rocket. Like beautiful mermaids, they call me from their isolated islands in a cosmic ocean.

But what if, like sirens, they’re setting me up for a trap? Balls of fire at thousands of degrees Celsius would scorch my skin right after blinding my bewitched eyes and right before throwing a disintegrating solar flare at me. So I look down at the solid ground beneath my feet to break the hypnotism and accept they are more beautiful from afar. I realize — my gaze again directed at the titillating temptation but cautious this time — that we never consider the obvious question.

What if human ambition — and, by extension, our craving for progress — isn’t a virtue but a bug? What if it’s the curse of a mismatch: our intelligence being too high for our own good, while our ability to rewire our emotional circuitry remains tragically primitive? We yearn to touch the stars to compensate for our inability to be emotionally stable. From this view, our eventual conquest of the universe feels like distracting collateral damage, and our belief that intelligent beings always maximize expected utility, a coping mechanism.