I wanted to share a few more thoughts with you about this roiling public debate about AI and the dangers of rogue, uncontrollable “agents.”

I start with the quite predictable conclusion that a lot of the contending theories we were talking about are true at the same time, at least to some degree. It is impossible not to see how a controlled deceleration of AI development serves financial purposes for the frontier AI labs. A regulatory pause sort of freezes their current lead in place, at least to a degree. It also allows them to focus on the massive returns of what we might call “meh” AI/LLMs while deprioritizing the kind of cutting edge research that costs a lot, has various risks and doesn’t have clear returns within the time horizon of your IPO. (The two top U.S. “frontier” labs are in their final preparations for going public.) It has also become increasingly clear to me that significant chunks of the AI world are in the thrall of a self-styled “rationalist” subculture which spends an inordinate amount of time postulating future human extinction events — dressed up in a lot of “rationalist” language but about which no one really has any idea — and then devising often quite sinister strategies for averting these imagined futures. (AI was a big focus of theirs before the last few years, but they have several storylines about future extinction events in the distant future.) It is finally the case that a lot of the people sounding these alarms have little acquaintance with how the world works outside of the computing domain and speak on issues of intelligence, sentience and self-awareness while knowing little or nothing about the scientific fields where these topics have been studied for decades or even centuries.

Needless to say, none of these facts inspire a lot of confidence.