System cards are presented as if they’re offering information the company is required to disclose for consumer safety — like the side effects in a pharmaceutical commercial — when, in fact, the companies volunteer them. So why would a company make their product sound scarier than it is? Perhaps because this is the best advertising money can’t buy. People like Harari and others repeat these accounts like ghost stories around a campfire. The public, awed and afraid, marvels at the capabilities of AI.
...
Where did we come up with this caricature of AI’s obsessive rationality? “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker(opens a new tab), “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.
...
After talking to experts, I was convinced there’s no reason to fear AIs developing a will to live, and then tricking or destroying us to avoid shutdown and take over the world. Unless, of course, we tell them to. Still, I asked Mitchell if there’s anything about AI that scares her.
“I have two really big concerns,” she said. “One, that it’s being used to create fake information that’s destroying our whole information environment. And two, people are trusting them to do things that they shouldn’t be trusted to do. We overestimate their capabilities. There’s a lot of magical thinking about AI. But it must be said that if you let these systems loose in the real world and they have access to your bank account, even if they’re just role-playing, it could still have catastrophic effects.”
TL;DR Marketing
I think more people would like this article if they took the time to read it. The entire thing can't be summarized easily but this line was great
And
So it's all roleplaying experiments. (I swear in another article, I read AI getting told to be a better roleplayer would have a positive effect, while telling it to be a better mathematician would make it roleplay harder but work worse.) And that's the real danger: people being dumb about the parlor trick (from your final quote).