How AI systems work, actually
Hello again!
New writing: “How to understand AI systems.”
This is the first chapter of Are language models Scary?, which is one of the two “companion documents” to Better without AI.
The “companion documents” are large chunks I removed from Better without AI. I wrote them in September 2022. I removed them in December, when I rewrote the book for a broader audience, because they go into more detail than many readers would want.
The other companion document is Gradient Dissent, which explains why neural networks are bad. I would recommend reading that one first, if you haven’t already. It’s complete, whereas I’m still revising Are language models Scary, so mostly only a skeleton of it is up on the web so far. Gradient Dissent is also helpful as background for reading today’s new bits, although it’s not an actual prerequisite. Neither requires any particular knowledge of AI, but parts may be a bit challenging if you don’t have some general computer science background.
Are language models Scary? is about text-generation systems like ChatGPT. Those seem uncannily human-like, and therefore “Scary.” I’m using that word as a semi-technical term for a hypothetical, but nebulously defined, type of AI that would pose special dangers beyond those of “regular” existing AI systems.
How to understand AI systems
“How to understand AI systems” is not about text generators. It lays some groundwork for my analysis of those. It has two sections, each a separate web page.
“Task-relevant algorithmic explanations” explains the type of understanding we’re aiming for, overall and in the abstract.
It’s common to explain that current AI systems are “neural networks,” consisting of arithmetical “units” that are trained with “machine learning.” This explanation is true as far as it goes, but it applies uniformly to extremely different AI systems, so it can’t give much insight into what specific ones can or can’t do, or why.
It’s also common to explain language models as “predicting the next bit of text,” which is also true, but leaves “how” unanswered. Without the “how,” it’s impossible to know why they fail when they do, and what their ultimate limits may be.
I explain the desired, useful type of understanding by analogy with understanding conventional database systems. It will help if you are familiar with those, but a general basic knowledge of the internal operation of computers should be adequate to get the point. That is: the useful sort of understanding is task-relevant (in terms of the specific real-world situation a system gets used in) and algorithmic, meaning it answers “how” questions.
“Classifying images” explains AI image classifiers at the algorithmic level. That's an easier warm-up before tackling language models. For these image classifiers, the explanation has been tested and is known to be mostly correct. What we’ve discovered is that these systems rely mainly on superficial, local features of images—textures, especially—rather than overall shape and structure.
This section is a personal historical narrative. It’s about how I figured out, in 2014, that spectacular 2012 results in “machine vision” did not mean what researchers believed at the time. I then explain a series of experiments, performed subsequently by others independently, which showed that my hypothesis was correct.
The “Classifying images” section explains that the spectacular 2012 results mainly taught us surprising and fascinating things about collections of photographs, rather than about “neural network” AI systems. Specifically, texture clues can do most of the work of figuring out which class of images a photograph belongs to—and therefore what “object” it contains. That in turn gives some insight into human vision and how and why it works.
This illustrates the type of analysis I apply to language models in Are language models Scary. Later in the document, I suggest that language models work similarly: they mainly exploit superficial linguistic patterns rather than “deep understanding.”
I believe that the success—and the failure patterns—of systems such as ChatGPT may teach us surprising and fascinating things about collections of texts, and about how people understand and use language, more than about AI systems.