41Robots that understand language, and what is real today

Hand a friend a sponge and say "wipe the table". You did not say which table, where to start, how hard to press, or what to do about the cups. They did it anyway.

No robot in this course could. Each was told what to do in joint angles and wheel speeds, or shown, or left to find out by trial and error. A robot that takes an order the way your friend did was a film prop until a few years ago, and videos of arms folding shirts on a spoken word are now easy to find. This module is about what is inside such a robot, which part of it became good, and which part did not.

The box below has the arm you know, with a blue block, an open bin and the touch module's egg on its table. In the grippers module one button ran one fixed routine with that block and bin, and in the programming module you wrote the program yourself. Here you type a sentence in plain English, and the box writes the program.

From a sentence to a motion

Whatever is inside it, a robot that is told what to do in words has the same five things to get done. One of them needs a word first. A skill is something a robot can already do from start to finish when asked: pick a thing up, open a drawer, drive to a door. It may be a program somebody wrote or a policy made from demonstrations. Either way, it exists before the sentence arrives.

1. 1SentenceA person says what they want, in their own words.
3. 2MeaningThe words are tied to things in front of the robot and to actions it can take.
5. 3PlanThe job is broken into steps, in an order that can work.
7. 4SkillsEach step is handed to something the robot already knows how to do.
9. 5MotionThe motors turn, and something in the world has changed.

However well the sentence is understood, the robot can do no more than the last two steps can deliver.

The box on this page does all five in the open.

1. Sentence. The line under the box opens with one already typed: pick up the block and put it in the bin. Do it sends it. You may type any other, and three more to try are one click away under it.

2. Meaning. The part of the box that reads your sentence, its reader, has a list of 14 words, printed under the line you type in, and looks up every word you typed. Seven of them make four actions: pick up, put down, put in, move to. Three name the things on the table: block, egg, bin. "The" is skipped, "and" and "then" start the next step, and "it" points back at the last thing named, or at what the hand holds. The words must also come in an order the reader has a pattern for, such as "pick up" and then a thing. Tying a word to the actual thing it stands for, here and now, is called grounding. This box grounds by cheating: the simulation tells it where the block is. A real robot has to find the block with a camera and a recognizer like the perception module's, crate and all.

3. Plan. Each action becomes one line of a program, in the order you gave them. The reader adds one rule of its own: a thing has to be in the hand before it can be put anywhere. So "put the egg in the bin" comes out as two lines, a pick-up you never asked for and then the put, and the box says which line it added.

4. Skills. Each line calls one of four skills: pickUp, putIn, putDown, moveTo. A person wrote them for this course. pickUp reaches over the thing, lowers the open jaws around it, closes them as hard as the motor can, and lifts. It does that the same way whatever it is asked to pick up.

5. Motion. The skill sends joint angles to the motors, as every program in this course has.

Under the line you type in, the box shows its working: on the left each word and what the reader took it for, on the right the program, with the line that is running lit. Below that, one line says what the arm takes itself to be holding, and where the block and the egg really are, with how hard anything is squeezing them. The egg still cracks, and turns red, when it is squeezed harder than 8 newtons. Reset puts everything back.

Loading the simulation. The robot runs in your browser, so the first one takes a few seconds to arrive.

ResetPauseSlowLabels

This is easier on a larger screen, but it works here too.

A list of words is not understanding

Your second sentence meant what your first did, and you knew that without thinking. The reader did not, because it never deals in what a sentence means. It looks words up, and there is no entry for "grab".

Adding entries does not cure it. Put in grab, cube, drop and box, and the next person types "get that blue thing into the tub". Programs with word lists and rules of grammar far richer than this box's have moved toy blocks on typed English orders since about 1970. Each worked inside its vocabulary and stopped at the edge of it: people have more ways of saying a thing than anyone can write down in advance.

The part that became real

A large language model is a neural network with billions of weights, where the perception module's had 3,480. It is trained on a large share of what people have written, at one job: given the start of a piece of text, predict what comes next. The text goes in as numbers, every word having its own, and what comes out is a share for each word that could come next, as the recognizer gave a share to each of its four answers.

To continue "she grabbed the cup and" as well as it continues "she picked the cup up and", the network has to end up treating the two alike, and over billions of sentences its weights come to do that for most ways of saying most things. And a question followed by its answer is also a piece of text. Hand such a network a question as the start, and what it predicts next is an answer, the kind the writing it learned from would give. So it takes "grab the cube" and "pick up the block" as the same request, and asked how to put a block in a bin, it says to pick the block up first. The box's reader has that one step as a fixed rule: press Reset, type "put the egg in the bin", and it writes the pick-up line itself.

That is the word half of step 2 and all of step 3, done for nearly any sentence a person would say. It is the part that is real, and it is only a few years old. Finding the block is still the recognizer's job.

Robots use one in two ways today.

The first keeps this box's shape and swaps its reader. The language model is handed the sentence, a list of the robot's skills, and what the recognizer found on the table, and writes the plan, often as a short program much like the one in the box. Steps 4 and 5 are untouched: the skills are whatever the robot already had.

The second puts one network where the middle three steps were. A vision-language-action model is given the camera's picture and the sentence, and gives back the next short stretch of motion, again and again until the job is done. It is the imitation-learning module's policy with a sentence added to what it is told. It starts as a language model that was also trained on pictures and what people wrote about them, a picture going in as numbers as the recognizer's did, so it arrives knowing the words and what the things look like. It is then trained further, by behavior cloning, until the numbers it gives back are moves for the motors where they used to be words. The demonstrations for that run to thousands of hours in the biggest collections, each recorded with a sentence saying what the job was. Nobody wrote its meanings, its plans or its skills. All three are in the weights.

Understanding is not doing

Now the egg. The reader read "pick up the egg" without a fault: every word on the list, the program correct. Then the arm crushed it, with 37 newtons where anything over 8 cracks it, because pickUp has one squeeze and gave the egg what it gives the block. Nor did it notice, because no step of pickUp looks. Once the jaws have closed, the arm takes it that it holds what it was sent for, whole or cracked or not there at all, and the program runs on.

"Gently" is not on the list. Suppose it were. There is still nothing for it to call, because no skill here has a soft grip, and somebody would have to write one or teach one.

A large language model does not change that. It has read that eggs break and what gently means, and it would write a sensible plan. Whether the egg survives is still decided at steps 4 and 5, by a skill that can or cannot squeeze softly. The one-network kind has no separate skill to point at, and the same holds for it: it squeezes softly only if its demonstrations included soft squeezing of things like this one.

What is real today

Following an instruction in ordinary words, however it is phrased, is real. So are robots, in laboratories and in trials outside them, that tidy a table, fold laundry or pack a box when asked in a sentence. Some manage it in a room they were never shown, when their demonstrations came from many different rooms, so that the new room falls between the examples and not beyond them.

What is not real is doing it dependably. Those robots fail often enough that a person stays close by. This page gives no number for how often: the published ones are each for one job in one laboratory, and do not carry. Most of the reason is the last two steps. A language model learned from text that billions of people had already written. Nothing like that exists for motion: every demonstration is recorded by somebody teleoperating a robot, one job at a time. So the skills are thin, and they fail as the imitation-learning arm did, short of a dot beyond its takes. The seeing fails as the recognizer did, sure of a crate.

Language models add a failure of their own. A language model always gives a likely continuation, and it has no more of an "I was never shown this" answer than the recognizer had. So its plan can name a thing that is not on the table, or a skill the robot does not have, in sentences as assured as its good plans.

A video of a robot obeying a sentence shows the whole chain working, once. It does not say how many tries came before it, or what happens with a different shirt.

What the box leaves out

There is no language model on this page, and nothing in the box was learned: its reader is a person's list of 14 words, as the box itself says. Its skills are told where everything is, never look, and never check their work.

What it keeps is the chain: a sentence, a meaning, a plan, skills, motion. The newest robots take nearly any sentence, where this box takes 14 words. What they can then do about it is where the work is.