Shinycroak

Today, in a blog post , Anthropic published a claim that “Tools like Claude Code can automate the exploration and analysis phases that consume most of the effort in COBOL modernization.” They claim that Claude can be used to analyze and document existing accounting software, so that it can be more quickly modernized. Has anyone else had the disorienting experience of reading an “AI” summary and realizing that it is wrong, wrong, wrong, but very neatly worded? Why does anyone think the summaries and documentation Claude generates will be any better? And there will be no way to check them…

We know that "AI" will create superficially valid natural language texts in authoritative persuasive language. They are right a fair amount of the time, which persuades us they are reliable, but often they are wrong and sometimes people are endangered by the errors. What does it do with programming language texts? I would expect "AI" to create programs that appear to be right most of the time and sometimes just fail. A superficial examination of the generated code will not catch the problem any more than a superficial reading of the natural language texts uncovers their errors.  I feel queasy. It is certainly going to happen that an "AI" generated program will have a subtle bug that does a lot of harm, and that even will occur without malicious intent upon the part of the users of the technology, let alone deliberate malice upon the part of the owners. No one should trust a system managed by Sam Altman to produce honest answers, or take safety into account...

[I keep bringing these up on Bluesky, so I think it’s time to gather them up and make a post out of them.] This is a collection of articles on the problems of “AI.” “AI”—really, various sorts of generative machine learning models—including generative large language models (gLLMs) and generative stable diffusion models (gSDMs) so far do not live up to the promises of their marketers. A computer you can talk to is one of the great dreams of computing, and the initial releases of transformer-model based chatbots seemed to live up to this. There have been long-standing qualms about this idea, most notably Dijkstra’s argument that the imprecision of natural language was an impediment to correct thinking about computation and to accurate computing. 1 Unfortunately, so far it appears that Dijkstra was correct; gLLMs and gSDMs are notorious for errors and they are not currently designed to indicate uncertainty to their users so that people confidently rely on their erroneous output. There ar...

(A step away from my usual political posting.) There should be website ratings like: You'll never find what you want using the tools for site provides; try using a search engine instead. They usually can ship you what you want but they'll try to sell you everything in the store on your way to finding it and they treat their people and suppliers like dirt. They've got local stores with decent stock but if you go to the website it'll send so many pop-ups that you'll just wish you went to the store instead, people steal things out of their pickup bins, and if they screw up an order you will not be able to reach their customer service.

My latest smartphone update came with an easy-access chatbot button that was hard to disable; I doubt most users will bother, let alone figure out how. There is, I think, a risk of creating cognitive disabilities by providing too-easy access to chatbots, the way one can develop a physical disability by persistent restrictions of motion. To some extent, all cognitive-enhancing technologies do this; people who write don't develop oral and memory skills; people who use calculators don't learn paper and pencil computational skills, people who grow up with photography don't learn to draw. But unlike writing, calculators, and so on, chatbots are not aids to cognition but replacements for it—and a replacement that is controlled by someone else. If a child writes, there are still writing words and ideas that have at least passed through the child's mind; chatbots entirely bypass this, inserting ideas from an external source; routine chatbot use interferes with thought. There i...

If, say, I broadcast a short story on the radio, I have to license the original work. If I print a book, I have to have a license to do so. If I publish a thinly-veiled rewrite of a book without a license, that is copyright infringement. And so on and on. It ought to be copyright infringement to do that with a large language model (LLM) or, for visual art, a stable diffusion model (SDM.) LLMs and SDMs do not exist, do not operate at all, without a body of work to built the models from. Without that training data those models do not exist at all. Therefore the developer of an LLM, SDM, or any other future generative machine learning technology, ought to be required to license any work used for developing that model.