Teaching by showing: imitation learning

Nobody taught you to tie a shoelace by listing the angles of your finger joints. Somebody did it slowly in front of you, and you copied until your hands had it. Every robot in this course so far has been taught the other way: somebody wrote down what it should do.

The last part of the course is about robots that are not told how. The first way is the one that worked on you: show the robot the job.

Why show instead of tell

Some jobs are easy to do and hard to describe. You can fold a towel in five seconds. Writing down how, in joint angles, for every towel and every way one can lie on a table, would take months.

So a person does the job, the robot records it, and a rule for doing the job is built from the recordings. Each recorded run is a demonstration, and building a robot's behavior out of demonstrations is called imitation learning.

Watching is not enough for a robot, though. A video of your hands has no joint angles in it, and your arm is not the shape of its arm. So the person does the job through the robot, and the robot records where its own joints are, many times a second. One way is to take the arm by the hand and move it, with its motors slack. The other is teleoperation: driving the robot from a distance while it does the job, often with a second arm of the same shape, called the leader, which a person moves by hand while the robot copies it joint for joint. The arm in the box below is a copy of a real one, the SO-101, which is sold as such a pair.

The box

You have dragged a star before, in the inverse kinematics module, where the hand's following was the lesson. Here the drag is only how you show the robot a motion. The lesson is what the robot does afterward.

This arm turns at its base as well as bending, so its hand can go anywhere over the table. You look down on the table from beside the arm, with its base at the left of the picture: out from the base is toward the right, and the arm's left is the far side of the table. The hand waits 15 centimeters out from the base and 8 centimeters above the table, and a red dot floats at the same height, farther out.

The box calls a demonstration a take. Press Record a take and a gold star appears on the hand. Drag the star to the red dot and the hand follows. The star and the dot stay 8 centimeters above the table however you drag, so a drag across the picture is a move across the table. When the hand reaches the dot the take ends by itself, and its path stays drawn in blue. A take is kept as the joint angles the hand was sent through, with a note of where the dot was. Record one for me makes a take without you: the star runs straight to the dot.

Let the robot try is the robot's turn. The arm is put back at the start and moves by itself, drawing its path in purple. The line under the box then says where the dot is, in centimeters out from the base and to the arm's left or right, which takes the robot used, and how far from the dot the hand stopped. On the dot means within 2 centimeters of its center, the edge of the red ball.

The dot stays where it is until you move it. To a new place sends it to the next of three places meant for takes, Between the takes and Beyond them put it where their names say, and you may drag it anywhere while the arm is still. Each take is listed with a Forget button, and Reset forgets them all.

The numbers below are from takes made by Record one for me, which come out the same every time. Your own will differ a little.

Loading the simulation. The robot runs in your browser, so the first one takes a few seconds to arrive.

This is easier on a larger screen, but it works here too.

A recording is not a skill

The robot did what it was shown, both times. It went where your hand went, by the road your hand took, at your pace, and stopped where the dot was when you showed it. The second time it was told the dot had moved: the box gives it the dot's place before every try. That made no difference, because one take is one motion, and a robot that has seen one motion has nothing else to offer.

Nor does anything inside a take say what mattered. It is a list of joint angles against time. Your detour is in the list as faithfully as your arrival, and the robot cannot tell the part of your motion that was the point from the part that was not.

To do better it needs more examples, with the dot in different places, so that it has more than one motion to draw on.

From recordings to a rule

A policy is a robot's rule for turning what it senses into what it does. Everything that steered a robot earlier in this course was a policy somebody wrote. In imitation learning it is made from the demonstrations.

The policy in this box is about the plainest there is. It is told one thing, where the dot is. It picks three takes whose dots surround the new dot, or come as near to surrounding it as any do, and mixes them. Picture the three old dots as the corners of a triangle. A new dot on a corner gets all of that corner's take. One halfway along a side gets half of each end's take. Anywhere else the shares fall between, bigger for the nearer corners, and they always add up to the whole.

The mix is made joint by joint, with the takes lined up start to start and end to end however long each lasted.

Why a mix lands

The robot was never shown that dot, and its hand stopped on it. Doing the right thing in a situation that was not among the examples is called generalization, and it is the reason to learn from demonstrations instead of replaying them.

It works because, for an arm that keeps the same elbow, in the inverse kinematics module's words, places close together need joint angles close together. So the reach to a dot among three others is nearly the mix of the three reaches.

Nearly: the line said a centimeter or so, not zero. Halfway between two joint angles is not halfway between two places. A joint swings the hand round a circle, so a mix of reaches ends a little off the place that the same mix of their dots points to, and further off the farther apart the takes are. Demonstrations packed close together copy better than a few spread wide.

The edge of what it was shown

A person shown three reaches would have the idea by now: put the hand on the dot. Asked for a dot farther out, they would reach farther. The robot never got the idea. It has three takes and some arithmetic, and this arithmetic only mixes. No take's share is less than nothing or more than all, so whatever comes out lies between things the robot was shown. For a dot outside the triangle, the nearest a mix can come is the triangle's edge: half of take 1 and half of take 2, the hand stopping between their dots, a centimeter outside the line between them because of the same curve, well short.

That limit is a choice made for this box. The arithmetic could be let run on past the examples, following their trend outward. A little way out that would work. Farther out it goes wrong fast, because joint angles do not follow a straight trend, and nothing marks where a little way ends. Among its examples a learned rule looks like understanding. Beyond them it is guessing, and it does not know that it is.

So the cure for a miss is not cleverer arithmetic. It is a demonstration where the miss was.

Record a fourth take with the dot out there and the miss is gone: the robot lands on that dot, and on most of the table between it and the others. Not quite all. Straight ahead, about 33 centimeters out, it mixes take 4 with take 3, the farthest dot with the nearest, 17 centimeters apart, and stops 2.5 centimeters away. Those two are too far apart, and one more take between them would mend it.

What a real one looks like

On a real robot the idea is the same, and three things change.

The first is what the policy is told. Nobody hands a real robot the place of the dot. It gets the pictures from its cameras and the angles of its own joints, and has to find what matters in them by itself.

The second is what the policy is. The usual choice is a neural network, which for this page is a formula with millions of adjustable numbers in it. Training it means nudging those numbers, a little at a time, until the formula, given what the robot sensed at any instant of any demonstration, gives back close to what the demonstrator did next. That is called behavior cloning, and it is the simplest kind of imitation learning. It has the strength and the weakness you met in the box, for the same reason. The nudging only ever pulls the formula toward the demonstrator in situations the demonstrations contain. Between them it is pinned from every side. Beyond them nothing pins it, and it gives an answer anyway.

The third is when the policy looks. The one in the box looks once, at where the dot is, and then plays a whole motion blind. A real one looks again many times a second and decides only the next short stretch of the motion. That lets it correct itself, and it opens a trap. A small error puts the robot somewhere a little unlike any demonstration, where its next move is a little worse, which carries it further from anything it was shown. A good demonstrator never drifted there, so no demonstration shows the way back. The cure is collected on purpose: let the robot run, have a person take over when it drifts, and record the rescue.

About fifty demonstrations is a common count for teaching an arm like this one a single job on a single table.

What the box leaves out

The box also has no training to wait for, and its job is a bare reach, with nothing to grasp.

What the box keeps is the shape of the thing: a robot that was never told what to do, doing what it was shown, detour and all, as well as its examples cover the job and no better. The next module takes the demonstrator away.

Watch

How I Made A Deep Learning Robot — Adam Lucek (5:52 of it, starting at 6:41)

Why this video: The words say a person shows the robot a job through a second arm, the leader, and that the angle of every joint is recorded. Here you can watch it done. A hobbyist holds the leader of a pair of small arms (Koch arms, a different design from this course's SO-101 but the same kind of pair) and the follower copies him, putting a block in a box, while two cameras watch and the motors' numbers stream down the screen. He does it about 50 times, trains a neural network on the recordings, and then the follower boxes the block with nobody holding the leader. He never tries the robot beyond what it was shown, as this page's box does. He calls his network an action chunking transformer, ACT for short; for this page it is one kind of neural network. It plans about 100 small moves at each look, the short stretch of motion the words mention.
What to watch for: At 7:12 he says what the pair of arms is for: every move he makes on the leader is copied by the follower. From 7:34 watch his hand on the small arm at the left while the arm at the right does the work. At 8:22 a column of numbers scrolls beside the arms, the motors' positions being captured: that is the recording. At 9:01 he says he did it about 50 times, with the block in five places, and at 9:10 he names it: imitation learning. At 9:26 one demonstration is on the screen, two camera views and a colored line for each motor. At 10:03 he says what will be trained, a neural network with 52 million adjustable numbers, and at 10:25 what it becomes, the robot's policy. At 11:06 nobody is holding the leader, and the follower picks up the block and boxes it. At 12:23 he admits that some tries gave it a little trouble.

Words introduced

demonstration: One recorded run of a person doing a job through the robot, by moving its arm or by driving it from a distance. What is kept is where every joint was, many times a second, from start to finish, and on a real robot what its cameras saw.
imitation learning: Building a robot's behavior out of demonstrations instead of writing it down. A person shows the job several times, and a rule for doing it is made from the recordings.
teleoperation: Driving a robot from a distance while it does the job, often by moving a second arm of the same shape that the robot copies joint for joint. It is the usual way demonstrations are recorded.
policy: A robot's rule for turning what it senses into what it does. It can be written by a person, as everything that steered a robot earlier in this course was, or made from demonstrations.
generalization: Doing the right thing in a situation that was not among the examples. A policy made from demonstrations generalizes between the things it was shown, and cannot be trusted beyond them.
behavior cloning: The simplest kind of imitation learning. A policy is adjusted until, given what the robot sensed at any instant of a demonstration, it gives back close to what the demonstrator did next.

Where next

Suggested next: 39. Learning by trial and error: reinforcement learning — Nobody tells this cart how to keep its pole up. It is paid points, tries, fails, and tries again, hundreds of times while you watch. Then change what it is paid for, and watch it learn exactly that instead.