It’s amazing how a theory can take over a brain. Consider the idea that people believe what serves their interests. As heuristics go, it’s a good one. I use it all the time. Yet when Dario Amodei says AI is dangerous, perhaps even an extinction risk, some people conclude he must be running a marketing campaign. That is stupid. Which is more likely, that a useful heuristic sometimes misfires or that “our product might kill you” is a clever way to sell it? Death threats are a poor marketing strategy.

We also have plenty of evidence that the fears of AI experts are sincere. Amodei, Altman and Musk were all publicly warning about AI risk long before they had AI companies to promote. The worry runs well beyond the executive suite; rank and file researchers share it. And it extends outside the industry altogether, to computer scientists with no product to sell, among them Nobel laureate Geoffrey Hinton. Hinton left a high-paying job at Google precisely so he could speak out and Hinton is not in a Berkeley polycule with Eliezer Yudkowsky, at least as far as I know. Whatever else you may say about the belief that AI presents a serious risk, plenty of AI researchers believe it sincerely.

Similarly, when Amodei recently proposed to slow the pace and install independent safety teams at AI companies many people jumped to the conclusion that this was regulatory capture. Sorry, but no, that theory doesn’t make sense. To see why, we should review the theory of regulatory capture.

...