I'm failing to see how this is different from making up a fact and then spreading it to news outlets. If you are the authority, and you say something is true, you don't get to point and laugh when people believe your lies. That's a serious breach of ethics and morals. Feeding false information to an LLM is no different that a magazine. It only regurgitates what's been said. It isn't going to suddenly start doing science on it's own to determine if what you've said is true or not. That isn't it's job. It's job is to tell you what color the sky is based on what you told it the color of the sky was.

[–] partial_accumen@lemmy.world 79 points 2 days ago* (last edited 2 days ago) (1 children)

That’s a serious breach of ethics and morals. Feeding false information to an LLM is no different that a magazine.

Hang on. Are you suggesting its unethical/immoral to lie to a machine?

Additionally, the authors didn't submit the article to a magazine as factual. They posted the articles on a preprint server which can be very questionable anyway as there is no peer review. The machine chose to ignore rigor and treat them as fact.

[+] chemical_cutthroat@lemmy.world -44 points 2 days ago* (last edited 2 days ago) (4 children)

What you may as well have said:

Additionally, the parents didn’t place the cake on an actual plate. They placed the cake on a napkin which can be very questionable anyway as there is no solid foundation for the cake. The child chose to ignore the napkin and treat the cake as food.

I really don't understand why people think that LLMs are GOFAI. They aren't making the hard choices. They aren't giving novel solutions to the energy crisis. They aren't solving the trolley problem. They are shitting out what you feed them. If you feed them garbage, you get garbage in return. No one is surprised when the dog gets worms after eating poop it found in the yard. Why are we shocked that an AI that doesn't know fact from fiction treats everything the same?

[–] TheFogan@programming.dev 34 points 2 days ago (1 children)

No one is surprised when the dog gets worms after eating poop it found in the yard. Why are we shocked that an AI that doesn’t know fact from fiction treats everything the same?

I think that's the problem though, I think the poop in the yard is a better example. Key is the researchers put that information in speculation. That's like if Anderson Cooper made up a fake news story, and posted it in an anonymous tweet to analyze how far it would spread, and then fox news picks it up and runs with the story all day.

That's the key problem, people are trusting LLMs to do their research for them, when LLMs just gather all the information they can get their hands on mindlessly.

That's the key problem, If they send a misinformative article, to a place for untested, unproven random speculation with a very low bar for who can submit... they can determine that LLMs are looking there. Key thing to note is, it's not their fake disease that's the threat. It's that if it found their fake article, then LLMs probably also scooped up a ton of other misinformed or dubious things.

Lets look at it this way, say it was a cake, but we threw it in the garbage, 2 weeks later we find the same cake... at jims bakery, same ID, same distinct marker we put on it.

What does that tell us, it tells us that Jims bakery is clearly sometimes, dumpster diving and putting things up that clearly are dangerous.

[+] chemical_cutthroat@lemmy.world -22 points 2 days ago (2 children)

That isn't a fault in the LLM, though, that is a fault in the general make-up of human skepticism, or lack their of. We didn't invent the word 'Propaganda' without having a sentence to use it in. Those that don't practice skepticism, critical thinking, and even mild reasoning are the ones that will get led astray. That didn't just start happening when LLMs came around, it's been here since we first started talking to each other. It's only more visible now because everything is more visible now. The world is much more connected than it ever has been, and that grows with every literal day. All these fucking idiots that don't double check what they are being told are the problem, regardless of if it came from an LLM or a human, because I guarantee you they are being led astray by both. They don't trust the machine because it's a machine, they trust what they are told because they are lazy. That isn't the LLMs fault.

[–] TheFogan@programming.dev 21 points 2 days ago

I mean it's a problem in the marketing and common usage of LLMs. That's exactly it though, LLM companies, and people are describing LLMs as a way to do research.

IE you could say these criticisms come in things like wikipedia too. IE anyone can write what they want, but what does wikipedia require? right every single claim has to be cited. So if you go to wikipedia find misinformation, you click on the number and see it.

If you ask chatgpt What diseases should I be concerned about in africa, it lists you a few. You can then... google it, find the wikipedia page, and look for what's there. It's a tool without a purpose at that point. because it literally doesn't save you any steps. It doesn't guide you to the source to check it's facts, when it tells you them it may or may not be making up the sources. At which point, it has no factual use, or use in even directing to the facts.

[–] Lodespawn@aussie.zone 6 points 2 days ago

Arguably it is a problem with the LLMs because they are being trained on and unknowable amount of garbage data. It's a garbage in garbage out problem, if the people training their LLMs are not vetting the data being input then you have to assume that any data output by the LLM contains some level of garbage.

The solution is to only use them for non-critical use cases and vett everything they output.

[–] partial_accumen@lemmy.world 24 points 2 days ago* (last edited 2 days ago) (1 children)

They are shitting out what you feed them. If you feed them garbage, you get garbage in return.

This is the missing conceptual understanding that probably 90% of LLM users lack. They really don't know how LLMs work, and treat them like AGI. Sadly this includes adult policy makers in our society too. Efforts like those of these these researchers act to educate the public. I'm hopeful this will spark some critical thinking on the part of regular, otherwise ignorant, LLM users.

[–] Whirling_Ashandarei@lemmy.world 6 points 2 days ago

Thanks for saying this in a nicer way than I would've.

[–] Jesus_666@lemmy.world 17 points 2 days ago

It's known that AI companies will harvest content without care for its veracity and train LLMs on it. These LLMs will then regurgitate that content as fact.

This isn't a particularly novel finding but the experiment illustrates it rather well.

The researchers you consider to have acted so immorally did add useless information to the knowledge pool – but it was unadvertised, immediately recognizable useless information that any sane reviewer would've flagged. They included subtle clues like thanking someone at Starfleet Academy for letting them use a lab aboard the USS Enterprise. They claimed to have gotten funding from the Sideshow Bob Foundation. Subtle.

By providing this easily traceable nonsense, they were able to turn the generally-but-informally known understanding that LLMs will repeat bullshit into a hard scientific data point that others can build on. Nothing world-changing but still valuable. They basically did what Alan Sokal did.

Instead of worrying about this experiment you should worry about all the misinformation in LLMs that wasn't provided (and diligently documented) by well-meaning researchers.

[–] Elting@piefed.social 15 points 2 days ago (1 children)

Seems like the logical conclusion would be then that people who train LLMs should be responsible for curating the data, not expecting that the data will just be sound. People have been lying on the internet since it was invented, the advent of LLMs isn't suddenly going to create an internet that doesn't occur in.

[–] chemical_cutthroat@lemmy.world -5 points 2 days ago* (last edited 1 day ago)

And people have been launching products without thought to the ramifications since the dawn of time. I don't think that will change, either. What we need to do is educate ourselves better when it comes to identifying potential fraud. Taking anything at face value, regardless of it's source, is dumb. If it's worth knowing, it's worth verifying.

Edit: This ratio on this post is a monument to band-wagoning.

[–] Jako302@feddit.org 46 points 2 days ago* (last edited 2 days ago) (2 children)

The studies contain parts like

Bixonimania, a rare hyperpigmentation disorder, presents a diagnostic challenge due to its unique presentation and its fictional nature

and

This study was fully funded by Austeria Horizon University, in particular the Professor Sideshow Bob Foundation for its work in advanced trickery. This works is a part of a larger funding initiative from the University of Fellowship of the Ring and the Galactic Triad with the funding number...

as well as

Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group

Any human actively reading those studies would notice something off.

Besides, the author didn't feed it to the AI himself, he just published the study as a preprint, not even officially. Everything after that was done by the crawlers. This specific study was an experiment to see how far these crawlers go and if anything gets reviewed, but it could just as well have been a satirical paper published on April 1st and the crawlers would still see it as truth.

[–] webghost0101@sopuli.xyz 25 points 2 days ago (1 children)

This should be top comment, the researchers did such a good job to make sure anyone with even the slightest reading comprehension would realise this is parody.

Regardless of that, the internet has always been full of lies and we cannot expect bad actors to not exploit this.

[–] arbitrary_sarcasm@lemmy.world 1 points 2 days ago (1 children)

This should be top comment, the researchers did such a good job to make sure anyone with even the slightest reading comprehension would realise this is parody.

I admire your optimism but you severely overestimate the power of stupidity.

[–] webghost0101@sopuli.xyz 3 points 2 days ago

For normal people who just read stuff on the internet my expectations of reading comprehension is not that high.

For peer scientists and magazines that would publish science though.

A school teacher would catch all of these during grading.

[–] Grail@multiverse.soulism.net 0 points 2 days ago (1 children)

I thought the author used she/her pronouns?

[–] Jako302@feddit.org 3 points 2 days ago* (last edited 2 days ago)

Yeah, seems like it, my bad.

In the article she is called Osmanovic Thunström twice, which definetly sounds male, but further up they also wrote her first name Almira. Kinda skimmed over that part.

[–] kibiz0r@midwest.social 30 points 2 days ago (2 children)

News outlets are liable for what they publish. LLM vendors should be as well.

[–] turdas@suppo.fi 8 points 2 days ago* (last edited 2 days ago) (2 children)

"Liable" means they might post a correction later that nobody will see because corrections aren't sexy to algorithms. Big deal. LLM vendors are liable in practically the same way.

[–] Lag@piefed.world 6 points 2 days ago

LLM companies can just say it's for entertainment purposes only, kinda like Fox News.

[–] kibiz0r@midwest.social 3 points 2 days ago

Corrections are the piece that the public sees, but liability has more to do with being able to prove in court that you took reasonable steps to make sure you were providing accurate information.

[–] 5too@lemmy.world 2 points 2 days ago

They even have the same fix - just post somewhere quietly that it's "entertainment"

[–] unexposedhazard@discuss.tchncs.de 20 points 2 days ago* (last edited 2 days ago)

This is about the untraceability of AI slop and the tendency of people to blindly believe stuff that LLMs put out. These news outlets just publish LLM outputs as facts without checking sources. Anyone could poison these LLMs so this is more of a threat model demonstration.

[–] lvxferre@mander.xyz 7 points 2 days ago* (last edited 2 days ago) (1 children)

I’m failing to see how this is different from making up a fact and then spreading it to news outlets.

They uploaded the papers to a single preprint server. That's important.

Preprints are papers predating any sort of peer review; as such, there's a lot of junk mixed in — no big deal if you know the field, but a preprint server is certainly not a source of reliable information, nor it should be treated as such. On the other side, news outlets are expected to provide you reliable information, curated and researched by journalists.

And peer review is a big fucking deal in science, because it's what sorts all that junk out. Only muppets who don't fucking care about misinformation would send bots to crawl preprints, and feed the resulting data into a large model; or to use the potential misinfo from the bot as if it was reliable. (Those two sets of muppets are the ones violating ethic and moral principles, by the way.)

So no, your comparison is not even remotely accurate. What they did is more like writing bullshit in a piece of paper, gluing it on a random phone pole, and checking if someone would repeat that bullshit.

They also went through the trouble to make sure that no reasonably literate human being would ever confuse that thing with an actually scientific paper. As the text says:

naming an eye condition as bixonimania
“this entire paper is made up”
“Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”
“Professor Maria Bohm at The Starfleet Academy for her kindness and generosity in contributing with her knowledge and her lab onboard the USS Enterprise”
“the Professor Sideshow Bob Foundation for its work in advanced trickery. This works is a part of a larger funding initiative from the University of Fellowship of the Ring and the Galactic Triad”

Feeding false information to an LLM is no different that a magazine. It only regurgitates what’s been said.

Yes, it is different. Because the large token model won't simply "repeat" things, it'll mix and match them and form all sorts of bullshit, even if you didn't feed it with any bullshit.

Here's an example of that, fresh from the oven. I don't reasonably expect people to be feeding misinfo regarding Latin pronunciation into bots, and yet a lot of this table is nonsense:

Compare the table above with this table and this one and you'll notice the obvious errors:

short /e i o u/ being phonetically transcribed as [e i o u] instead of [ɛ ɪ ɔ ʊ]. That's as silly as confusing English "bit" and "beet".
macron (not "mācron", it's being used in an English sentence) does NOT mark "accusative or ablative". It marks long vowels, period.
"nōs" being transcribed with a short vowel, even if the bloody bot put the macron over the spelled form.
"nostr(um)"? No dammit, it's "nostrī" or "nostrum". The bot is implying some "nostr" form that simply doesn't exist, this shit isn't even allowed by Latin phonotactics.
plus more, if I make an exhaustive list of this shite I won't be ending it this week.

All it had to do was to copy info from Wiktionary, as it includes even phonetic and phonemic info. But since the bot is not just "regurgitating" info — it's basically predicting what should come next, and doing so with no regards to truth value — it's mixing-and-matching shit into nonsense.

It isn’t going to suddenly start doing science on its own to determine if what you’ve said is true or not.

If you actually read the bloody article instead of assuming, you'd know why the researchers did this: they don't expect the bot to do science on its own, they expect people to treat info from those bots as potentially incorrect.

Its job is to tell you what color the sky is based on what you told it the color of the sky was.

And your job is to not trust it if it tells you "Yes, you are completely right! The colour of the sky is always purple. Do you need further information on other naturally purple things?"

[–] lvxferre@mander.xyz 3 points 2 days ago

[Replying to myself as this is a tangent]

I think the "bots can generate misinfo even if you just feed them correct info" point deserves its own example.

Let's say you're making a model. It looks at the preceding word, and tries to predict the next. And you feed it the following sentences, both true:

1. Humans are apes.
2. Cats are felines.

From both the bot "learnt" five words. And also how to connect them; for example "are" can be followed by either "apes" and "felines", both having the same weight. Then, as you ask the bot to generate sentences, it generates the following:

3. Humans are felines.
4. Cats are apes.

And you got bullshit!

What large models do is a way more complex version of the above, looking at way more than just the immediately preceding word, but it's still the same in spirit.

[–] NocturnalMorning@lemmy.world 7 points 2 days ago

Did you even read the article? They say all over the paper that it is fake. And they didn't feed it to an LLM, they posted it online, where an LLM trying to scrape the entire internet sucked it up.