keisatsu

joined 3 years ago
[–] keisatsu@infosec.pub 4 points 1 day ago (1 children)

An AI security tool can't substitute for a real security audit. Especially if you don't have the necessary know-how to verify the outcome. If you don't have extensive experience performing manual security code audits, don't even bother. You might just as well release it with a disclaimer that it hasnt been vetted and leave it up to whoever want to test it to review it.

If you do opt for a scan, please disclose it properly. Many will never consider code that has been touched by an LLM in any shape or form, but it's better to be honest about it.

Finally, if you pay the frontier labs for their models you are supporting their business practices including training models on stolen data, the destruction of the environment and further inflation of the AI bubble. A local model is not much better, all things considered, but at least Scam Altman and Wario Amodei won't benefit.

[–] keisatsu@infosec.pub 2 points 2 days ago

I hate to see another LLM-assisted app, but I must say that I really appreciate the detailed disclosure in the readme.

[–] keisatsu@infosec.pub 4 points 3 days ago

yeah, it wasnt my intention but I made it look like I put him on a pedestal there. My bad

[–] keisatsu@infosec.pub 5 points 3 days ago (1 children)

Despite the "AI policy and common sense" saying that the contributor is responsible for understanding the code committed, Linus did clealry not treat this with the due dilligence required. He used unproven error-prone technology to emit code for a project that millions of users depend on and sent it in despite it having the race condition and the performance regression.

If Linus was as smart as you seem to believe, he should have checked that code as if it was submitted from his worst enemy, as if it was written by a smarter and malicous adversary or at least a very enthusiastic imbecile. But the LLMs are designed in a way that trick you into believing that they are smart, reasoning and capable and you drop your guard. The code looked good, he got fooled and here we are.

[–] keisatsu@infosec.pub 6 points 3 days ago

You putting quotes around "learn" and "dream" is you admitting that they are not really doing either. LLMs don't think, learn, reason, analyse, design or anything else that require a human mind.

You making a note in agents.md did not teach the model anything, it's a halfassed attempt to compensate for the fact that LLMs are INCAPABLE of learning. You need to create a new model trained on the new data you wish to incorporate, because when that phase is over the model is locked in.

Have a look at the claude code leak and you will see how they actually prompt inject constantly to try to keep it on track and not do stupid shit. It has fucking CYBER ALZHEIMERS for gods sake.

Look at that woman at meta who told an agent to suggest which emails to archive, explicitly forbidding it from deleting anything. Guess what? That internal monologue filled up the context and that instruction fell off the back of the truck, and the clanker happily started shredding emails until they literally pulled the plug.

LLMs are somehow passing the Turing test for a subset of humans and its embarassing that the bar is apparently that low. The AI labs are doing their best to weaponize that and you fell for it.

[–] keisatsu@infosec.pub 10 points 4 days ago* (last edited 4 days ago) (7 children)

First of all, if a human makes a mistake it's a learning opportunity. When a human misses a bug an LLM hallucinated, noone really learn from it.

I like to compare it with the xray machines at airports. Human operators stare at the screens hunting for weapons and other prohibited items and miss pretty obvious stuff because they get bored out of their minds. They actually added a mechanism to insert fake xray images of weapons ever so often to actually keep the operators interested and alert. Ain't much too different from doing code reviews being the human-in-the-loop/scapegoat.

Also, LLMs don't necessarily generate good code. They excel at generating good-looking mediocre code.

So yeah, I'm gonna go ahead and claim that Linus is more likely to miss a bug a LLM wrote on his behalf than one he wrote himself.

[–] keisatsu@infosec.pub 44 points 4 days ago (6 children)

How about we keep relying on deterministic tooling that are not built on stolen data and don't fuck the economy and the environment at the same time? How about putting a fraction of the money invested i LLMs into SAST and DAST tools? Maybe a dollar or two for fuzz testers? How about we look at projects like curl and build safeguards into the project to avoid footguns?

[–] keisatsu@infosec.pub 37 points 4 days ago (21 children)

if Linus, who has worked with the language AND the project for decades, can't find a race condition and a performance regression in his VERY FIRST LLM-assisted commit, then just maybe we should accept that humans are fucking horrible att catching the mistakes that clankers make and stop deluding us into thinking that this is the future of software engineering.

[–] keisatsu@infosec.pub 151 points 4 days ago* (last edited 4 days ago) (40 children)

and in the process he introduced a race condition, a performance regression costing up to 9% when making reads of files around 1 MiB and dumped the responsibility of fixing his mess on co-maintainers and went on vacation.

[–] keisatsu@infosec.pub 29 points 5 days ago (2 children)

200k lines of code in a month by four contributors of which one is dependabot. There is an AGENTS.md in the repo and the gitignore contains entries for claude and agents. At least clearly disclose that you are pushing vibed slop.

[–] keisatsu@infosec.pub 31 points 6 days ago (1 children)

claude, cursor and codex in the gitignore for a slop trifecta!

view more: next ›