Seeing and recognizing: neural networks for perception

The cameras module ended with a way to find a blue block. For every pixel, take the red number away from the blue one, and keep the pixels where the answer is over 40. A person wrote that test, after looking at the camera's numbers for a blue block on a gray table.

Now try to write the test for a cup. A cup is any color, a different shape from every side, and it changes with the lamp. Nobody has written that test. The robots that find cups have something that was shown a great many pictures of cups, each marked "cup", and worked out a test of its own.

The last two modules were about a robot learning what to do, from demonstrations and then from trial and error. This one is about a robot learning what it is looking at.

A formula nobody wrote

The thing that works out its own test is a neural network: a formula built from many small sums, with adjustable numbers all through it. The sums come in layers: a layer is a row of sums all made from the same inputs, and its results are the inputs of the next layer. Numbers go in at one end, an answer comes out at the other, and what happens in between depends entirely on what the adjustable numbers are set to. Whatever is being recognized, the way from picture to answer has the same five steps.

PictureThe camera hands over a grid of colored squares.
NumbersEach square is three numbers. The picture is nothing more.
NetworkThe numbers are multiplied, added and passed on, layer after layer.
How sureOut comes a share for every answer it knows. The shares add up to 100%.
AnswerIt gives the answer with the biggest share.

Nobody wrote what goes on in the middle. It was fitted beforehand, to thousands of pictures that each came with the right answer.

The rover on this page has a network small enough to go through in full. It knows four answers: red block, green block, blue block, and no block.

1. Picture. Five times a second the network is handed what the rover's camera sees, as a picture 16 pixels wide and 12 tall.

2. Numbers. Each pixel is three numbers, its red, green and blue, so the picture is 576 numbers. The network is given those and nothing else.

3. Network. The first layer is six sums. Each one runs over the whole picture: every one of the 576 numbers is multiplied by a number that belongs to this sum alone, and the results are added up. The number an input is multiplied by is a network weight, a weight for short, and nothing to do with what a thing weighs. A big weight means that input matters a great deal to this sum, a weight near zero that it hardly matters, a weight below zero that it counts against. Six sums with 576 weights each is 3,456 weights.

The cameras module's test was a sum of this kind: a weight of 1 on a pixel's blue number, −1 on its red, and zero on everything else. Each of the six is a test like that, put to the whole picture at once and answered with a number, and nobody chose its weights. Each number is then squeezed. A small sum passes almost unchanged, a sum of 3 and a sum of 300 both come out close to 1, and nothing comes out beyond −1 or 1, so that one huge sum cannot swamp the other five. The second layer is four more sums, one for each answer the network can give, each made from the six squeezed numbers with six weights of its own: 24 more. These four are not squeezed. That is the whole network: two layers, ten sums, and 3,480 weights. (Each sum also has one number added at the end, ten more in all.)

4. How sure. The four results are turned into four shares that add up to 100%. Each result is first turned into an amount that grows very fast as the result grows, nearly tripling for every 1 the result goes up, and an answer's share is its amount out of the four amounts together. So two results that are close split the shares between them, and a result well ahead of the others takes nearly everything. The share an answer gets is the network's confidence in it.

5. Answer. The network gives the answer with the biggest share. This page adds a rule of its own choosing: when that share is under 75%, it reports the network as not sure, and gives the answer as its best guess.

Where the weights come from

Nobody chose any of those 3,480 weights, or the ten added numbers. They were set by training, which goes like this. Start with every weight random, so that the network's answers mean nothing. Show it a picture for which the right answer is already known. The right answer attached to a picture is its label. Look at the four shares the network gives. Then change every weight by a tiny amount, each in whichever direction would have given the label a bigger share for that picture. The sums are plain arithmetic, so for each weight it can be worked out exactly whether making it a hair bigger would raise that share or lower it. Take the next picture, and do it again. No one nudge is big enough to undo the others: what most pictures agree on adds up, and what they disagree on cancels. After enough rounds the weights give the right answer for most of the pictures, and no person has looked at what they are.

This network's pictures were taken by this rover's camera in this pen: 7,920 of them, with the rover parked, driving and turning, with no blocks or with one, two or three, near and far. The simulation knew where every block was, so each picture got its label without a person: the block that filled the most of the view, or no block. Training went through all of them 300 times. It happened before you opened this page, and nothing you do here changes a weight.

Then the network was tried on 1,540 more pictures that had been kept out of the training, and it was right on 96 in every 100. Pictures it has never seen are the only fair trial: with enough weights, a network can fit the pictures it was trained on, quirks and all, without having caught what they have in common.

Three plain colors is as easy as recognizing gets, and the cameras module's color test would do this job. But nobody wrote this test, and the same procedure, with millions of weights and millions of photographs labeled by people, is how a robot tells a cup from a hand.

The box

The rover is the small two-wheeled robot of earlier modules, with a camera on its nose, in its walled pen, seen from above. Straight ahead of it, 60 centimeters from the camera, is a red block. Off to its left and far away is a blue one. Close behind it is a green one, which comes into view past 140° either way. To its right, against the wall, is a brown crate twice as wide as a block. The network was never shown a crate. The yellow post in the far corner is the rover's beacon; it was in many training pictures, never as a block.

The camera takes in 86 degrees from side to side, 43 to each side of straight ahead. It sits 7 centimeters ahead of the point the rover turns about, so it swings a little sideways as the rover turns, and things leave its picture a few degrees sooner than 43. Its view is drawn in the corner at 160 by 120 pixels, so that you can make it out. The network gets the same view at 16 by 12, each pixel the average of the patch it covers. That picture is in the panel under the view from above, The recognizer, which is this page's name for the network at work, beside a line saying what it says and its four shares as bars. The one control is the slider, Turn the rover: drag it left and the rover turns left on the spot. Reset puts it back.

Loading the simulation. The robot runs in your browser, so the first one takes a few seconds to arrive.

This is easier on a larger screen, but it works here too.

What you saw

A fifth of a block was enough. More than a thousand of the training pictures had their block cut off by the picture's edge, labeled as that block all the same, so the network kept its answer until very little of the block was left. Then it stopped being sure, and six degrees after the last of the red had gone it was sure of the opposite. Why six, nobody can read off the weights: that comes with a test nobody wrote.

At 1.2 meters the blue block is as blue as ever, and small. When it first came in at the left edge it was about two squares wide, and the network was sure of it: the squares near the edge of a flat picture each take in a narrower slice of the view than the ones in the middle, so a thing there spreads across more of them. Faced squarely the block is hardly more than one square wide. It straddles two or three squares, each an average of a little blue and a lot of gray wall, and the results for a blue block and for no block come out nearly level. The network had less practice at this, too: only a few hundred of its training pictures had a block that far away. From so little blue, not sure is a fair answer.

How far can its being sure be trusted? Of the kept-out pictures it was sure of, it had 98 in 100 right; of the others, two in three. A robot can use a "not sure": it can drive closer and look again.

Then the crate. A red block, 100% sure, and wrong, with the same confidence it had for the real red block. The four shares have to add up to 100% across four answers, and none of the four is "something I was never shown". A big reddish-brown shape evidently pushes the sums much as a close view of the red block does, so the result for a red block comes out far ahead of the other three and takes the whole share. No training picture said otherwise, because none had a crate in it. Nor did the kept-out pictures, from the same pen: doing well on those says nothing about a picture unlike all of them. Confidence says how strongly a picture drives the sums toward an answer. It does not say the answer is true.

The imitation-learning module's arm stopped short of a dot beyond its takes. This is the same wall, the limit of generalization, met with a camera. The cure has the same limit: add pictures of crates, labeled "no block", and this network should learn to pass them by, and then be as sure and as wrong about the next thing nobody thought to include.

What a real one looks like

A real robot's recognizer has millions of weights in dozens of layers, knows thousands of answers, and usually says where in the picture each thing is.

It is trained this way all the same, it answers with shares, and it fails as this one does: a recognizer trained on daytime streets meets a street at night. So a careful robot treats its recognizer as one more sensor that is sometimes wrong. It looks more than once, from more than one place, and does not stake much on a single picture.

Watch

Jetson Nano: Vision Recognition Neural Network Demo — ExplainingComputers (4:16 of it, starting at 7:42)

Why this video: This page's network knows four answers and has only ever seen a simulated pen. Here is a real one at work: a camera on a small tripod, wired to a palm-sized computer of the kind small robots carry, running a ready-trained network that knows 1,000 kinds of thing. He holds one household object after another up to it, and the network's answer and its percentage stay on the screen the whole time, flickering as the picture changes. There is no robot here and no training: that was done beforehand, by other people. Shown an empty table it still names something. Unlike this page's network it has no answer for "nothing there", and like it, none for "something I was never shown". He calls the percentage the probability that it is correct; the crate on this page shows why that is too generous. The frame rate he mentions is how many pictures it gets through each second.
What to watch for: At 7:58 he reads the small line at the top of the picture, an answer and a percentage: African elephant, about 60%. At 8:22 he opens the list behind it: the shares are split among three kinds of elephant. From 8:42 the line is repeated in large letters under the picture. At 9:14 a green apple comes up close: Granny Smith, 99.99%, then 100%. At 9:57 a wooden spoon seen edge-on turns into a drumstick. At 10:05 his mug is a coffee mug, 92% sure, and he protests that he drinks tea from it. At 10:44 a toilet roll, 99% a few seconds earlier, is turned to hide its hole and becomes a paper towel. At 10:52 the table is empty and the line still answers: jellyfish, 36%. At 11:14 an upright computer mouse: joystick, over 99% sure. At 11:42 a wine bottle's label fills the picture: envelope.

Words introduced

neural network: A formula built from many small sums stacked in layers, each sum multiplying its inputs by adjustable numbers called weights. Nobody writes what it computes. Its weights are set by training on examples.
network weight: One of a neural network's adjustable numbers, called a weight for short. It says how much, and in which direction, one input counts toward one of the network's sums. Everything a network has learned is in its weights.
training: Setting a neural network's weights by showing it examples whose right answers are known, over and over, and nudging every weight a little toward the right answer each time.
label: The right answer attached to a training example, such as the words "red block" attached to a picture of one. A network can only ever answer with the labels it was trained on.
confidence: The share a recognizer gives the answer it chose, from 0 to 100%. It says how strongly the picture drives the network's sums toward that answer, which is not the same as the answer being true.

Where next

Suggested next: 41. Robots that understand language, and what is real today — Type pick up the block and put it in the bin, and a robot arm does it. Say the same thing another way and it is lost. Ask it to pick up an egg and it knows every word, which does the egg no good.