「AI」是個糟糕的舉動，再論

Top level navigation menu

'AI' is a dick move, redux

(This was originally posted as a social media thread. Edited and posted here for posterity.)

I’ve stopped trying to debate software developers on LLMs. You might have noticed if you’ve been following this blog. It’s a fruitless debate. Even if the believers in agents and copilots could be budged on empirical grounds, and the past few years have given us plenty of evidence that they can’t, this is still a crowd that is explicitly fine with using tools that are themselves deeply unethical.

Debating people who look past “chatbot psychosis”, the dismantling of the education system, the gendered abuse, the generated CSAM images, the overt attacks on the media industries, or the ultra-right’s glee about “AI”, by showing them a well-constructed academic study is never going to work.

—“Oooh, neat experimental design! I’m totally going to take this seriously even though it challenges everything I believe about myself and would force me to reassess my life’s work”, said nobody, ever.

Somebody who is capable of looking past “ICE is using LLMs as accountability sinks for waving extremists through their recruitment processes”, generated abuse, or how chatbot-mediated alienation seems to be pushing vulnerable people into psychosis-like symptoms, won’t be persuaded by a meaningful study. Their goal is to maintain their personal benefit, as they see it, and all they are doing is attempting to negotiate with you what the level of abuse is that you find acceptable. Preventing abuse is not on their agenda.

You lost them right at the outset.

Most of them are up to their eyebrows in Cialdini-style cognitive traps and have completely lost sight of the fact that subjective experiences don’t tell you anything about the effects on a system, organisation, industry, or community. Anything sensible will fall on deaf ears.

Nor do they seem to care, except in a performative way, that “AI” is designed to be an outright attack on labour and education, using the works of those being attacked – without their consent – as the tools for dismantling their own communities and industries, all done in overt collaboration with the ultra right.

Even if you disregard context – the political climate, hostility toward labour, attacks on education and media, the alienation increasing the risk of psychological episodes – these “systems thinkers” no longer seem to understand that software is a system built on the labour of thousands.

Even if it were true that the effect for them personally is positive, that is objectively and manifestly not scaling up to the level of the larger software development system. We can all see it. Software as an ecosystem has rapidly become less stable, less secure, worse designed and outright less productive for the end user.

Shit is getting bad out in the actual software economy. Cash registers that have to be rebooted twice a day. Inventory systems that randomly drop orders. Claims forms filled with clearly “AI”-sourced half-finished localisation strings. That’s just what I’ve heard from people around me this week. I see more and more every day.

And I know you all are seeing it as well.

We all know why. The gigantic, impossible to review, pull requests. Commits that are all over the place. Tests that don’t test anything. Dependencies that import literal malware. Undergraduate-level security issues. Incredibly verbose documentation completely disconnected from reality. Senior engineers who have regressed to an undergraduate-level understanding of basic issues and don’t spot beginner errors in their code, despite having “thoroughly reviewed” it. Everybody being forced to use the tools to increase output even when they themselves don’t think it’s safe or right.

The software made by the software industry is getting substantially worse every month. The products are getting worse. The platforms are getting worse. The ecosystems are less safe. And the job environment is horrible. And it’s happening faster and faster.

Going all “but it works great for me” even as the industry burns around you and the “it” is a right-wing political project built on disregarding consent, being applied to dismantle public infrastructure and institutions, is fundamentally a dick move.

And debating dicks is pointless.

Join the Newsletter

Subscribe to the
Out of the Software Crisis
newsletter to get my weekly (at least) essays on how to avoid or get
out of software development crises.

Join now and get a free PDF of
three bonus essays from
Out of the Software Crisis.

We respect your privacy.

Unsubscribe at any time.