MICHALE FEE: OK.
All right, let's go ahead and get started.
OK, so we're going to continue talking
about the topic of neural networks.
Last time, we introduced a new framework
for thinking about neural network interactions,
using a rate model to describe the interactions of neurons
and develop a mathematical framework for how
to combine collections of neurons
to study their behavior.
So, last time, we introduced the notion of a perceptron
as a way of building a neural network that
can classify its inputs.
And we started talking about the notion of a perceptron learning
rule, and we're going to flesh that idea out
in more detail today.
We're going to then talk about the idea of using networks
to perform logic with neurons.
We're going to talk about the idea of linear separability
and invariance.
Then we're going to introduce more complex
feed-forward networks, where instead
of having a single output neuron,
we have multiple output neurons.
Then we're going to turn to a more fully developed view
of the math that we use to describe neural networks,
and matrix operations become extremely important
in neural network theory.
And then, finally, we're going to turn
to some of the kinds of transformations that
are performed by matrix multiplication
and by the kinds of-- by feed-forward neural networks.
OK, so we've been considering a kind of neural network called
a rate model that uses firing rates rather than spike trains.
So we introduced the idea that we
have an output neuron with firing rate
v that receives input from an input neuron that
has firing rate u.
The input neuron synapses onto the output neuron
with a synapse of weight w.
And we described how we can think
of the input neuron producing a synaptic input into the output
neuron that has a magnitude of the firing
rate times the strength of the synaptic connection.
So the input to the output neuron here is w times u.
And then we talked about how we can convert that input current,
let's say, into our output neuron
into a firing rate of the output neuron through some function
f, which is what's called the F-I curve of the neuron that
relates the input to the firing rate of the neuron.
And we talked about several different kinds
of F-I firing rate versus input functions that can be useful.
We then extended our network from a single input neuron
synapsing onto a single output neuron
by having multiple input neurons.
Again, the output neuron has a firing rate,
and our input neurons have a vector of firing rates now--
u1, u2, u3, u4, and so on--
that we can combine together into a vector, u.
Each one of those input neurons has a synaptic strength w
onto our output neuron.
So we have a vector of synaptic strengths.
And now we can write down the input current to our output
neuron as a sum of the contributions from each
of those input neurons-- so w1, u1 plus w2, u2, plus w3, u3,
and so on.
So we can now write the input current
to our output neuron as a sum of contributions
that we can then write as a dot product--
w dot u.
OK, any questions about that?
And so, in general, we have the firing rate of our output
neuron is just this F-I function,
this input-output function of our output neuron acting
on the total input, which is w dot u.
And then we talked about different kinds
of functions that are useful computationally
for this function f.
So in the context of the integrate and fire neuron,
we talked about F-I curves that are zero below some threshold
and then are linear above that threshold current.
We talked last time about a binary threshold known
that has zero firing rate below some threshold
and then steps up abruptly to a constant output firing rate
one.
And then we also introduced, last time, the notion
of a linear neuron, whose firing rate is
just proportional to the input current
and has positive and negative firing rates.
And we talked about the idea that although it's
biophysically implausible to have neurons
that have negative firing rates, that this
is a particularly useful simplification of neurons.
Because we can just use linear algebra
to describe the properties of networks of linear neurons.
And we can do some really interesting things
with that kind of mathematical simplification.
We're going to get to some of that today.
And that allows you to really build
an intuition for what neural networks can do.
OK, so let's come back to what perceptron is and introduce
this perceptron learning role.
So we talked about the idea that a perceptron carries out
a classification of its inputs that
represent different features.
So we talked about classifying animals into dogs and non-dogs
based on two features of animals.
We talked about the fact that you
can't make that classification between dogs and non-dogs
just on the basis of one of those features,
because these two categories overlap in this feature
and in this feature.
And so in order to properly separate those categories,
you need a decision boundary that's
actually a combination of those two features.
And we talked about how you can implement
that using a simple network, called
a perceptron, that has an output neuron and two input neurons.
Each one of those input neurons represents the magnitude
of those two different features for each object
that you're trying to classify.
So u1 here and u2 are the dimensions on which we're
performing this classification.
And so we talked about the fact that that decision boundary
between those two classifications
is determined by this weight matrix w.
And then we used a binary threshold neuron
for making the actual decision.
Binary threshold neurons are great for making decisions,
because unlike a linear neuron-- so a linear neuron just
responds more if its input is larger,
and it responds less if its input is smaller.
Binary threshold neurons have a very clear threshold
below which the neuron doesn't spike
and above which the neuron does spike.
So, in this case, this network, this output neuron here,
will fire, will have a firing rate of one,
for any input that's on this side of the decision boundary
and will have a firing rate of zero
for any input that's on this side of the decision boundary,
OK?
All right, so we talked about how we can, in two dimensions,
just write down a decision boundary that will separate,
let's say, green objects from red objects.
So you can see that if you sat down
and you looked at this drawing of green dots and red dots,
that it would be very simple to just look at that picture
and see that if you put a decision boundary right
there, that you would be able to separate the green dots
from the red dots.
How would you actually calculate the weight vector
that that corresponds to in a perceptron?
Well, it's very simple.
You can just look at where that decision boundary crosses
the axes--
so you can see here, that decision boundary crosses
the u1 axis at point A, crosses the u2 axis at, I should say,
a value of B. And then we can use those numbers to actually
calculate the w.
So, remember, u is the input space.
w is a weight vector that we're trying
to calculate in order to place the decision
boundary at that point.
Is that clear what we're trying to do here?
OK, so we can calculate that weight vector.
We assume that just data is some number.
Let's just call it one.
We have an equation for a line-- w dot u equals theta.
That's the equation for that decision boundary.
We have two knowns, the two points on the decision boundary
that we can just read off by eye.
And we have two unknowns-- the synaptic weights, w1 and w2.
And so we have two equations--
ua dot w equals theta, ub dot w equals theta.
And we can just solve for w1 and w2,
and that's what you got, OK?
So the weight vector that gives you that decision boundary
is 1 over a and 1 over b, OK?
Those are the two weights.
Any questions about that?
OK.
So in two dimensions, that's very easy to do, right?
You can just look at that cloud of points,
decide where to draw a line that best separates the two
categories that you're interested in separating.
But in higher dimensions, that's really hard.
So in high dimensions, for example,
we're trying to separate images, for example.
So we can have a bunch of images of dogs,
a bunch of images of cats.
Each pixel in that image corresponds
to a different input to our classification unit.
And now how do you decide what all of those weights
should be from all of those different pixels
onto our output neuron that separates images of one class
from images of another class?
So there's just no way to do that by eye in high dimensions.
So you need an algorithm that helps
you choose that set of weights that allows you
to separate different classes--
you know, a bunch of images of one class from a bunch
of images of another class.
And so we're going to introduce a method called
the perceptron learning rule that is a category of learning
rules called supervised learning rules that allow you to take
a bunch of objects that you know-- so if you
have a bunch of pictures of dogs,
you know that they're dogs.
If you have a bunch of pictures of cats, you know they're cats.
So you label those images.
You feed those inputs, those images, into your network,
and you tell the network what the answer was.
And through an iterative process,
it finds all of the weights that optimally separate those two
different categories.
So that's called the perceptron learning rule.
So let me just set up how that actually works.
So you have a bunch of observations of the input.
So in this case, I'm drawing these in two dimensions,
but you should think about each one of these dots as being,
let's say, an image of a dog in very high dimensions,
where instead of just u1 and u2, you have u1 through u1000,
where each one of those is the value of a different pixel
in your image.
So you have a bunch of images.
Each one of those corresponds to an image of a dog.
Each one of those corresponds to an image of a cat.
And we have a whole bunch of different observations
or images of those different categories.
Any questions about that?
All right, so we have n of those observations.
And for each one of those observations,
we say that the input is equal to one
of those observations for one iteration of this learning
process, OK?
And so with each observation, we're
told whether this input corresponds
to one category or another, so a dog or a non-dog.
And our output, we're asking--
we want to choose this set of weights
such that the output of our network
is equal to some known value.
So t sub i, where if it's a dog, then the answer is one for yes.
If it's a non-dog, the answer is no for that's not a dog.
And we have n of those answers.
We have n images and labels that tell us what category
that image belongs to.
So for all of these, t equals one.
For all of these, t equals zero.
And we want to find a set of weights
such that when we take the dot product of that weight factor
into each one of those observations minus theta
that we get an answer that is equal to t
for each observation.
Does that make sense?
So how do we do that?
All right, so each observation, we have two things--
the input and the desired output.
And that gives us information that we
can use to construct this weight vector.
So, again, that's called supervised learning.
And we're going to use an update rule, or a learning rule,
that allows us to change the weight
vector on as a result of each estimate,
depending on whether we got the answer right or not.
So how do we do this?
What we're going to do is we're going
to start with a random set of weights, w1 and w2, OK?
And we're going to put in an input.
So there's a space of inputs.
We're going to start with some random weight,
and I started with some random vector in this direction.
You can see that that gives you a classification boundary here.
And you can see that that classification boundary is not
very good for separating the green dots from the red dots.
Why?
Because it will assign a one to everything
on this side of that decision boundary and a zero
to everything on that side.
But you can see that that does not
correspond to the assignment of green and red
to each of those dots, OK?
So how do we update that w in order to get the right answer?
So what we're going to do is we're
going to put in one of these inputs on each iteration
and ask whether the network got the answer right or not.
So we're going to put in one of those inputs.
So let's pick that input right there.
We're going to put that into our network.
And we see that the answer we get from the network
is one, because it's on the positive side of the decision
boundary.
And so one was the right answer in this case.
So what do we do?
We don't do anything.
We say the change in weight is going to be zero if we already
get the right answer.
So if we got lucky and our initial weight vector
was in the right direction, so our perceptron
already classified the answer, then
the weight vector is never going to change,
because it was already the right answer.
OK, so let's put it in another input--
a red input.
You can see that the correct answer is a zero.
The network gave us a zero, because it's
on the negative side of the weight vector of the decision
boundary.
And so, again, delta w is zero.
But let's put in another input now such
that we get the wrong answer.
So let's put in this input right here.
So you can see that the answer here, the correct answer
is one, but the network is going to give us a zero.
So what do we do to update that weight vector?
So if the output is not equal to the correct answer,
then we're wrong.
So now we update w.
And the perceptron learning rule is very simple.
We introduce a change in w that looks like this.
It's a little change, so eps eta is a learning rate.
It's generally going to be smaller than one.
So we're going to put in a small change in w that's
in the direction of the input that was wrong
if the correct answer is a one.
We're going to make a small change
to w in the opposite direction of that input
if the correct answer was zero.
Does that make sense?
So we're going to change w in a way that
depends on what the input was and what
the correct answer was.
So let's walk through this.
So we put it in an input here.
The correct answer is a one, and we got the answer wrong.
The network gave us a zero, but the correct answer is a one.
So we're in this region here.
The answer was incorrect, so we're going to update w.
The correct answer was a one, so we're going to change delta--
we're going to change w in the direction of that input.
So that input is there.
So we're going to add a little bit to w in this direction.
So if we add that little bit of vector to the w,
it's going to move the w vector in this direction, right?
So let's do that.
So there's our new w.
Our new w is the old plus delta w,
which is in the direction of this incorrectly
classified input.
So there's our new decision boundary, all right?
And let's put in another input--
let's say this one right here.
You can see that this input is also incorrectly classified,
because the correct answer is a zero.
It's a red dot.
But the network since it's on the positive side
of the decision boundary.
So the network classifies it as a one.
OK, good.
So the network classified it as a one and the correct answer
was a zero, so we were wrong.
So we're going to update w, and we're
going to update it in the opposite direction of the input
if the correct answer was zero, which is the case.
So we're going to update w.
And that's the input xi.
Minus xi is in this direction.
So we're going to update w in that direction.
So we're going to add those two vectors to get our new w.
And when we do that, that's what we get.
There's our new w.
There's our new decision boundary.
And you can see that that decision boundary is now
perfectly oriented to separate the red and the green dots.
So that's Rosenblatt's perceptron learning rule.
Yes, Rebecca?
AUDIENCE: How do you change the learning rate?
Because what if it's too big?
You'll sort of get not helpful [INAUDIBLE]..
MICHALE FEE: Yeah, that's right.
So if the learning rate were too big,
you could see this first correction.
So let's say that we corrected w but made a correction that
was too far in this direction.
So now the new w would point up here.
And that would give us, again, the wrong answer.
What happens, generally, is that if your learning
rate is too high, then your weight vector bounces around.
It oscillates around.
So it'll jump too far this way, and then
it'll get an error over here, and it'll
jump too far that way.
And then you'll get an error over there,
and it'll just keep bouncing back and forth.
So you generally choose learning rates
that-- the process of choosing learning rates
can be a little tricky Basically,
the answer is start small and increase it until it breaks.
OK, any questions about that?
So you can see it's a very simple algorithm that
provides a way of changing w that is guaranteed to converge
toward the best answer in separating these two
classes of inputs.
All right, so let's go a little bit further
into single layer binary networks
and see what they can do.
So these kinds of networks are very good for actually
implementing logic operations.
So you can see that-- let's say that we have a perceptron that
looks like this.
Let's give it a threshold of 0.5 and give it
a weight vector that's 1 and 1.
So you can see that this perceptron
gives an answer of zero.
The output neuron has zero firing rate
for an input that's zero.
But any input that's on the other side of the decision
boundary produces an output firing rate of one.
What that means is that if the input a, or u1, is a 1,
0, then the output neuron will fire.
If the input is 0, 1, the output neuron will fire.
And if the input is 1, 1, the output neuron will fire.
So, basically, any input above some threshold
will make the output neuron fire.
So this perceptron implements an OR gate.
If it's input a or input b, the output neuron
spikes, as long as those inputs are above some threshold value.
So that's very much like a logical OR gate.
Now let's see if we can implement an AND gate.
So it turns out that implementing an AND gate
is almost exactly like an OR gate.
We just need-- what would we change about this network
to implement an AND gate?
AUDIENCE: A larger [INAUDIBLE].
MICHALE FEE: What's that?
AUDIENCE: A larger theta?
MICHALE FEE: Yeah, a larger theta.
So all we have to do is move this line up to here.
And now one of those inputs is not
enough to make the output neuron fire.
The other input is not enough to make the output neuron fire.
Only when you have both.
So that implements an AND gate.
We just increase the threshold a little bit.
Does that make sense?
So we just increase the threshold here to 1.5.
And now when either input is on at a value of one,
that's not enough to make the output neuron fire.
If this input's on, it's not enough.
If that output is on, it's not enough.
Only when both inputs are on do you get enough input
to this output neuron to make it have a non-zero firing rate,
to get it above threshold.
Now, there's another very common logic operation that cannot be
solved by a simple perceptron.
That's called an exclusive OR, where
this neuron, this network, we want
it to fire only if input a is on or input b is on, but not both.
Why is it that that can't be solved
by the kind of perceptron that we've been describing?
Anybody have some intuition about that?
AUDIENCE: I mean, it's obviously [INAUDIBLE] separable.
MICHALE FEE: Yeah, that's right.
The keyword there is separable.
If you look at this set of dots, there's no single line,
there's no single boundary that separates all the red dots
from off the green dots, OK?
And so that set of inputs is called non-separable.
And sets of inputs that are not separable cannot be classified
correctly by a simple perceptron of the type we've been talking
about.
So how do you solve that problem?
So this is a set of inputs that's non-separable.
You can see that you can solve this problem now
if you have two separate perceptrons.
So watch this.
We can build one perceptive one that fires,
that has a positive output when this input is on.
We can have a separate perceptron that is active
when that input is on.
And then what would we do?
If we had one neuron that's active
if that input is on another input
that's active when that input is on?
We would or them together, that's right.
So this is what's known as a multi-layer perceptron.
We have two inputs, one that represents activity
in a, another that represents activity in b.
And we have one neuron in what's called
the intermediate layer of our perceptron
that has a weight vector of 1 minus 1.
What that means is this neuron will be active if input a is
on but not input b.
This one will be active.
This neuron has a different weight vector-- minus 1, 1.
This neuron will be active if input b is on but not input a.
And the output neuron implements an OR operation
that will be active when this intermediate neuron is on
or that intermediate neuron is on, OK?
And so that network altogether implements this exclusive OR
function.
Does that make sense?
Any questions about that?
So this problem of separability is
extremely important in classifying inputs in general.
So if you think about classifying an image,
like a number or a letter, you can
see that in high-dimensional space, images
that are all threes, let's say, are all
very similar to each other.
But they're actually not separable in this linear space.
And that's because in the high dimensional space
they exist on what's called a manifold
in this high-dimensional space, OK?
They're like all lined up on some sheet, OK?
So this is an example of rotations,
and you can see that all these different threes kind of sit
along a manifold in this high-dimensional space that
are separate from all the other numbers.
So all those numbers exist on what's
called an invariant transformation, OK?
Now, how would we separate those images
of threes from all the other numbers or letters?
How would we do that?
Well, we could imagine building a multi-layer perceptron that--
so here, I'm showing that there's
no single line that separates the threes on this manifold
from all the other digits over here.
We can solve that problem by implementing
a multi-layer perceptron that while one of those perceptrons
detects these objects, another perceptron detects
these objects, and then we can OR those all together.
So that's a kind of network that can now
detect all of these three, separate them from non-threes.
Does that make sense?
So we can think of objects that we recognize, like this three
that we recognize, even though it has different--
we can recognize it with different rotations
or transformations or scale changes.
You can also think of the problem of separating images
from dogs and cats as also solving this problem,
that the space of dogs, of dog images,
somehow lives on a manifold in the high dimensional space
of inputs that we can distinguish
from the set of images of cats that's
some other manifold in this high-dimensional space.
So it turns out that you need more than just a single layer
perceptron.
You need more than just a two-layer perceptron.
In general, the kinds of networks
that are good for separating different kinds of images,
like dogs and cats and cars and houses and faces,
look more like this.
So this is work from Jim DiCarlo's lab,
where they found evidence that networks in the brain that do
image classification-- for example,
in the visual pathway--
look a lot like very deep neural networks, where
you have the retina on the left side here sending inputs
to another letter in the thalamus,
sending inputs to v1, to v2, to v4, and so on, up to IT.
And that we can think of this as being,
essentially, many stacked layers of perceptrons
that sort of unravel these manifolds
in this high-dimensional space to allow
neurons here at the very end to separate dogs
from cats from buildings from faces.
And there are learning rules that
can be used to train networks like this
by putting in a bunch of different images of people
and other different categories that you
might want to separate.
And then each one of those images
has a label, just like our perceptron learning rule.
And we can use the image and the correct label--
face or dog-- and train that network
by projecting that information into these intermediate layers
to train that network to properly classify
those different stimuli, OK?
This is, basically, the kind of technology
that's currently being used to train--
this is being used in AI.
It's being used to train driverless cars.
All kinds of technological advances
are based on this kind of technology here.
Any questions about that?
Aditi?
AUDIENCE: So in actual neurons, I
assume it's not linear, right?
MICHALE FEE: Yes.
These are all nonlinear neurons.
They're more like these binary threshold units
than they are like linear neurons.
That's right.
AUDIENCE: But then do you there's, like--
because right now, I imagine that models we make
have to have way more perceptron units.
MICHALE FEE: Yes.
AUDIENCE: We use our simplified [INAUDIBLE]..
But then our brain is sometimes--
I mean, it's at, like, a much faster level,
like way faster, right?
So you think it'd be like-- if we examine what functions
neurons might be using, in a way that would let us reduce
the number of units needed?
Because right now, for example, [INAUDIBLE]
be a bunch of lines.
But maybe in the brain, there's some other function it's using,
which is smoother.
MICHALE FEE: Yeah.
OK, so let me just make sure I understand.
You're not talking about the F-I curve of the neurons?
Is that correct?
You're talking about the way that you figure out
these weights.
Is that what you're asking about?
AUDIENCE: No.
I'm asking if we use a more accurate F-I curve,
we'll need less units.
MICHALE FEE: OK, so that's a good question.
I don't actually know the answer to the question
of how the specific choice of F-I curve
affects the performance of this.
The big problem that people are trying to figure out
in terms of how these are trained
is the challenge that in order to train these networks,
you actually need thousands and thousands, maybe millions,
of examples of different objects here and the answer here.
So you have to put in many thousands
of example images and the answer in order
to train these networks.
And that's not the way people actually learn.
We don't walk around the world when we're one-year-old
and our mother saying, dog, cat, person, house.
You know, it would be... in order to give a person as many
labeled examples as you need to give these networks,
you would just be doing nothing, but your parents would be
pointing things out to you and telling you one-word answers
of what those are.
Instead, what happens is we just observe the world
and figure out kind of categories
based on other sorts of learning rules that are unsupervised.
We figure out, oh, that's a kind of thing, and then mom says,
that's a dog.
And then we know that that category is a dog.
And we sometimes make mistakes, right?
Like a kid might look at a bear and say, dog.
And then dad says, no, no, that's not a dog, son.
So the learning by which people train their networks
to do classification of inputs is
quite different from the way these deep neural networks
work.
And that's a very important and active area of research.
Yes?
AUDIENCE: Is the fact that [INAUDIBLE] use unsupervised
learning, as well, to train a computer
to recognize an image of a turtle as a gun,
but humans can't do that [INAUDIBLE]..
MICHALE FEE: Recognize a turtle if what?
AUDIENCE: Like I saw this thing where it was like at MIT,
they used an AI.
They manipulated pixels in images
and convinced the computer that it was something
that it was not actually.
MICHALE FEE: I see.
Yeah.
AUDIENCE: So like you would see a picture of a turtle,
but the computer would get that picture
and say it was, like, a machine gun.
MICHALE FEE: Just by manipulating a few pixels
and kind of screwing with its mind.
AUDIENCE: Yes.
So it's [INAUDIBLE].
MICHALE FEE: Yeah.
Well, people can be tricked by different things.
The answer is, yes, it's related to that.
The problem is after you do this training,
we actually don't really understand
what's going on in the guts of this network.
It's very hard to look at the inside of this network
after it's trained and understand what it's doing.
And so we don't know the answer why
it is that you can fool one of these networks
by changing a few pixels.
Something goes wrong in here, and we don't know what it is.
It may very well have to do with the way it's trained,
rather than building categories in an unsupervised way, which
could be much more generalizable.
So good question.
I don't really know the answer.
Yes?
AUDIENCE: Sorry, can you explain what
you mean [INAUDIBLE] the neural network needs an answer?
They're not categorized and then tell the user dogs?
MICHALE FEE: Yeah, so no, in order
to train one of these networks, you have to give it a data set,
a labeled data set.
So a set of images that already has the answer
that was labeled by a person.
AUDIENCE: So you can't just give it
a set of photos of puppies and snakes
and it'll categorize them into two groups?
MICHALE FEE: No, nobody knows how to do that.
People are working on that, but it's not known yet.
Yes, Jasmine?
AUDIENCE: [INAUDIBLE] but I see [INAUDIBLE] I
can't separate them and like adding an additional feature
to raise it to a higher dimensional space, where
it's separable?
MICHALE FEE: Sorry, I didn't quite understand.
Can you say it again?
AUDIENCE: I think I remember reading somewhere
about how when the scenes are nonlinearly separable--
MICHALE FEE: Yes.
AUDIENCE: --you can add in another feature to [INAUDIBLE]..
MICHALE FEE: Yeah, yeah.
So let me show you an example of that.
So coming back to the exclusive OR.
So one thing that you can do, you
can see that the reason this is linearly inseparable-- it's not
linearly separable-- is because all these points are
in a plane.
So there's no line that separates them.
But one way, one sort of trick you can do,
is to add noise to this.
So that now, some of these points move.
You can add another dimension.
So now let's say that we add noise,
and we just, by chance, happen to move the green dots this way
and the red dots, well, that way.
And now there's a plane that will separate the red dots
from the green dots.
So that's advanced beyond the scope of what
we're talking about here.
But yes, there are tricks that you
can play to get around this exclusive OR
problem, this linear separability problem, OK?
All right, great question.
All right, let's push on.
So let's talk about more general two-layer
feed-forward networks.
So this is referred to as a two-layer network-- an input
layer and an output layer.
And in this case, we had a single input neuron
and a single output neuron.
We generalized that to having multiple input neurons and one
output neuron.
We saw that we can write down the input current
to this output neuron as w, the vector of weights,
dotted into the vector of input firing rates
to give us an expression for the firing rate of the output
neuron.
And now we can generalize that further
to the case of multiple output neurons.
So we have multiple input neurons, multiple output
neurons.
You can see that we have a vector
of firing rates of the input neurons
and a vector of firing rates of the output neurons.
So we used to just have one of these output neurons,
and now we've got a whole bunch of them.
And so we have to write down a vector of fire rates
in the output layer.
And now we can write down the firing rate of our output
neurons as follows.
So the firing rate of this neuron
here is going to be a dot product of the vector
of weights onto it.
So the firing rate of output neuron one
is the vector of weights onto that first output neuron dotted
into the vector of input firing rates.
And the same for the next output neuron.
The firing rate of output neuron two
is dot product of the weights onto that output neuron two
and onto the vector of input firing rates.
Same for neuron three.
And we can write that down as follows.
So the eighth output-- the firing rate
of the eighth output neuron is the weight vector
onto the eighth output neuron dotted into the input firing
rate vector, OK?
And we can write that down as follows,
where we've now introduced a new thing here,
which is a matrix of weights.
So it's called the weight matrix.
And it essentially is a matrix of all
of these synaptic weights, from the input layer onto the output
layer.
And now if we had a linear neuron,
we can write down the firing rate of the output neuron.
The firing rate vector of output neuron
is just this weight matrix times the vector of input fire rates.
So now, we've rewritten this problem
of finding the vector of output firing rates
as a matrix multiplication.
And we're going to spend some time talking about what
that means and what that does.
So our feed-forward network implements a matrix
multiplication.
All right, so let's take a closer look at what
this weight matrix looks like.
So we have a weight matrix w sub a comma b that looks like this.
So we have four input neurons and four output neurons.
We have a weight for each input neuron onto each output neuron.
The columns here correspond to different input neurons.
The rows correspond to different output neurons.
Remember, for a matrix, the elements
are listed as w sub a, b, where a is the output neuron.
b is the input neuron.
On so it's w postsynaptic, presynaptic-- post, pre.
Rows, columns.
So the rows are the different output neurons.
The columns are the different input neurons.
So it can be a little tricky to remember.
I just remember that it's rows--
a matrix is labeled by rows and columns.
And weight matrices are postsynaptic, presynaptic--
post, pre.
AUDIENCE: [INAUDIBLE] comment of [INAUDIBLE]??
MICHALE FEE: I think that's standard.
I'm pretty sure that's very standard.
If you find any exceptions let me know.
OK, we can think of each row of this matrix
as being the vector of weights onto one output neuron.
That row is a vector of weights onto that output neuron--
that row, that output neuron; that row, that output neuron.
Does that makes sense?
All right, so let's flesh out this matrix multiplication.
The vector of output firing rates,
we're going to write it as a column vector, where
the first number is this firing rate.
That number is that firing rate.
That number represents that firing rate, OK?
That's equal to this weight matrix
times the vector of input firing rates,
again, written as a column vector.
And in order to calculate the firing rate of the first output
neuron, we take the dot product of the first row of the weight
matrix and the column vector of input firing rates.
And that gives us this first firing rate, OK?
To get the second firing rate, we
take the dot product of the second row of weights
with the vector of firing rates, and that gives us
this second firing rate.
Any questions about that?
Just a brief reminder of matrix multiplication.
All right, no questions?
All right, so let's take a step back and go quickly
through some basic matrix algebra.
I know most of you have probably seen this,
but many haven't, so we're just going to go through it.
All right, so just as vectors are--
you can think of them as a collection of numbers
that you write down.
So let's say that you are making a measurement of two
different things--
let's say temperature and humidity.
So you can write down a vector that represents those two
quantities.
So matrices you can think of as collections of vectors.
So let's say we take those two measurements
at different times, at three different times.
So now we have a vector one, a vector two, and a vector three
that measure those two quantities at three
different times, all right?
So we can now write all of those measurements
down as a matrix, where we collect
each one of those vectors as a column in our matrix,
like that.
Any questions about that?
And there's a bit of MATLAB code that calculates this matrix
by writing three different column vectors
and then concatenating them into a matrix.
All right, and you can see that in this matrix,
the columns are just the original vectors,
and the rows are--
you can think of those as a time series
of our first measurement, let's say temperature.
So that's temperature as a function of time.
This is temperature and humidity at one time.
Does that make sense?
All right, so, again, we can write down this matrix.
Remember, this is the first measurement
at time two, the first measurement at time three.
We have two rows and three columns.
We can also write down what's known
as the transpose of a matrix that just flips the rows
and columns.
So we can write transpose, which is
indicated by this capital super scripted t.
And here, we're just flipping the rows and columns.
So the first row of this matrix becomes the first column
of the transposed matrix.
So we have three rows and two columns.
A symmetric matrix--
I'm just defining some terms now.
A symmetric matrix is a matrix where
the off-diagonal elements-- so let me just define,
that's the diagonal, the matrix diagonal.
And a symmetric matrix has the property
that the off-diagonal elements are zero.
And a symmetric matrix has the property
that the transpose of that matrix is equal to the matrix,
OK?
That is only possible, of course,
if the matrix has the same number of rows and columns,
if it's what's called a square matrix.
Let me just remind you, in general
about matrix multiplication.
We can write down the product of two matrices.
And we do that multiplication by taking the dot product
of each row in the first matrix with each column
in the second matrix.
So here's the product of matrix A and matrix B.
So there's the product.
If this matrix, if matrix A, is an m by k--
m rows by k columns--
and matrix B has k rows by n columns,
then the product of those two matrices
will have m by n rows and columns.
And you can see that in order for matrix multiplication
to work, the number of columns of the first matrix
equal the number of rows in the second matrix.
You can see that this k has to be the same for both matrices.
Does that make sense?
So, again, in order to compute this element right here,
we take the dot product of the first row of A
and the first column of B. That's just 1 times 4, is 4.
Plus negative 2 times 7 is minus 14.
Plus 0 times minus 1 is 0.
Add those up and you get minus 10.
So you get this number.
You multiply this row dot product
this row with this column and so on.
Notice, A times B is not equal to B times A.
In fact, in cases of rectangular matrices, matrices that aren't
square, you can't even do this, often do
this, multiplication in a different order.
Mathematically, it doesn't make sense.
So let's say that we have a matrix of vectors,
and we want to take the dot product
of each one of those vectors x with some other vector v. So
let's just write that down.
The way to do that is to say the answer here,
the dot product of each one of those column vectors
in our matrix with this other vector
v we do by taking the transpose of v,
which takes a column vector and turns it into a row vector.
And we can now multiply that by our data matrix x
by taking the dot product of v with that column of x.
And that gives us a matrix.
So this matrix here, that vector is a one by two matrix.
This is a two by three matrix.
The product of those is a one by three matrix.
Any questions about that?
OK.
We can do this a different way.
Notice that the result of this multiplication
here is a row vector, y.
We can do this a different way.
We can take dot product.
We can also compute this as y equals x transpose v.
So here, we've taken the transpose of the data
matrix times this column vector v.
And again, we take the dot product of this,
this with this, and that with that.
And now we get a column vector that
has the same entries that we had over here.
All right, so I'm just showing you different ways
that you can manipulate a vector in a matrix
to compute the dot product of elements of vectors
within a data matrix and other vectors
that you're interested in.
All right, identity matrix.
So when you're multiplying numbers together,
the number one has the special property
that you can multiply any real number by one
and get the same number back.
You have the same kind of element in matrices.
So is there a matrix that when multiplied by A gives you A?
And the answer is yes.
It's called the identity matrix.
So it's given by the symbol I, usually.
A times I equals A. What does that matrix look like?
Again, the identity matrix looks like this.
It's a square matrix that has ones along the diagonal
and zero everywhere else.
So you can see here that if you take an arbitrary vector x,
multiplied by the identity matrix,
you can see that this product is x1, x2 dotted into 1,
0, which gives you x1.
x1, x2 dotted into 0, 1, gives you x2.
And so the answer looks like that, which is just x.
So the identity matrix times an arbitrary vector x
gives you x back.
Another very useful application of linear algebra,
linear algebra tools, is to solve systems of equations.
So let me show you what that looks like.
So let's say we want to solve a simple equation, ax equals c.
So, in this case, how do you solve for x?
Well, you're just going to divide both sides by a, right?
So if you divide both sides by a,
you get that x equals 1 over a times c.
So it turns out that there is a matrix equivalent
of that, that allows you to solve systems of equations.
So if you have a pair of equations--
x minus 2y equals 3 and 3x plus y equals 5--
you can write this down as a matrix equation,
where you have a matrix 1, minus 2,
3, 1, which correspond to the coefficients of x and y
in these equations.
Times a vector xy is equal to 3, 5, another vector 3, 5.
So you can write this down as ax equals c--
that's kind of nice--
where this matrix A is given by these coefficients
and this vector c is given by these terms
on this side of the equation, on the right side of the equation.
Now, how do we solve this?
Well, can we just divide both sides of that matrix equation,
that vector equation, by a?
So division is not really defined for matrices,
but we can use another trick.
We can multiply both sides of this equation
by something that makes the a go away.
And so that magical thing is called the inverse of A.
So we take the inverse of matrix A,
denoted by A with this superscript minus 1.
And that's the standard notation for identifying the inverse.
It has the property that A inverse times
A equals the identity matrix.
So you can sort of think about this
as A equals the identity matrix over A. Anyway, don't really
think of it like that.
So to solve this system of equations ax equals c,
we multiply both sides by that A inverse matrix.
And so that looks like this--
A inverse A times x equals A inverse c.
A inverse A is just what?
The identity matrix times x equals A inverse c.
And we just saw before that identity matrix times x
is just x.
All right, so there's the solution
to this system of equations.
All right, any questions about that?
So how do you find the inverse of a matrix?
What is this A inverse?
How do you get it in real life?
So in real life, what you usually do is
you would just use the matrix inverse function in Matlab.
Because for any matrices other than a two-by-two,
it's really annoying to get a matrix inverse.
But for a two-by-two matrix, it's actually pretty easy.
You can almost just get the answer by looking at the matrix
and writing down the inverse.
It looks like this.
The inverse of a two-by-two square matrix is just given
by a slight reordering of the coefficients,
of the entries of that matrix, divided by what's called
the determinant of A. So what you do is you flip--
in a two-by-two matrix, you flip the A and the D,
and then you multiply the diagonal elements by minus 1.
Now, what is this determinant?
The determinant is given by a times d minus b times c.
And you can prove that that actually
is the inverse, because if we take this and multiply it by A,
what you find when you multiply that out is that that's just
equal to the identity matrix.
So a matrix has an inverse if and only
if the determinant is not equal to zero.
If the determinant is equal to zero,
you can see that this thing blows up,
and there's no inverse.
We're going to spend a little bit of time
later talking about what that means when a matrix has
an inverse and what the determinant actually
corresponds to in a matrix multiplication context.
If the determinant is equal to zero,
we say that that matrix is singular.
And in that case, you can't actually find an inverse,
and you can't solve this equation right here,
this system of equations.
All right, so let's actually go through this example.
So here's our equation, ax equals c.
We're going to use the same matrix we had before
and the same c.
The determinant is just the product
of those minus the product of those, so 1 minus negative 6.
So the determinant is 7.
So there is an inverse of this matrix.
And we can just write that down as follows.
Again, we've flipped those two and multiplied those
by minus 1.
So we can solve for x just by taking that inverse times c,
A inverse times c.
And if you multiply that out, you
see that there's the inverse.
It's just a vector.
That's it.
That's how you solve a system of equations, all right?
Any questions about that?
So this process of solving systems of equations
and using matrices and their inverses
to solve systems of equations is a very important concept
that we're going to use over and over again.
All right, let's turn to the topic
of matrix transformations.
All right, so you can see from this problem of solving
this system of equations that that matrix A transformed
a vector x into a vector c, OK?
So we have this vector x, which was 3/7 minus 4/7 a vector.
When we multiplied that by A, we got another vector, c.
And the vector A inverse transforms this vector
c back into vector x, right?
So we can take that vector c, multiply it by A inverse,
and get back to x.
Does that make sense?
So, in general, a matrix A maps a set
of vectors in this whole space.
So if you have a two-by-two vector,
it maps a set of vectors in R2 onto a different set
of vectors in R2.
So you can take any vector here--
a vector from the origin into here--
multiply that vector by A, and it gives you
a different vector.
And if you multiply that other vector by A inverse,
you go back to the original vector.
So this vector A implements some kind
of transformation on this space of real numbers
into a different space of real numbers, OK?
And you can only do this inverse if the determinant of A
is not equal to zero.
So I just want to show you what different kinds of matrix
transformations look like.
So let's start with the simplest matrix transformation--
the identity matrix.
So if we take a vector x, multiply it
by the identity matrix, you get another vector y,
which is equal to x.
So what we're going to do is we're going to kind of riff off
of a theme here, and we're going to take
slight perturbations of the identity matrix
and see what that new matrix does to a set of input vectors,
OK?
So let me show you how we're going to do that.
We're going to take it the identity matrix 1, 0, 0, 1.
And we're going to add a little perturbation
to the diagonal elements.
And we're going to see what that does to a set of input vectors.
So let me show you what we're doing here.
We have each one of these red dots.
So what I did was I generated a bunch of random numbers
in a 2D space.
So this is a 2D space.
And I just randomly selected a bunch
of numbers, a bunch of points on that plane.
And each one of those is an input vector x.
And then I multiplied that vector
times this slightly perturbed identity matrix.
And then I get a bunch of output vectors y.
Input vectors x are the red dots.
The output vectors y are the other end of this blue line.
Does that make sense?
So for every vector x, multiplying it by this matrix
gives me another vector that's over here.
Does that make sense?
So you can see that what this matrix does
is it takes this space, this cloud of points,
and stretches them equally in all directions.
So it takes any vector and just makes it longer,
stretches it out.
No matter which direction it's pointing,
it just makes that vector slightly longer.
And here's that little bit of code
that I used to generate those vectors.
OK, so let's take another example.
Let's say that we take the identity matrix
and we just add a little perturbation to one element
of the identity matrix, OK?
So what does that do?
It stretches the vectors out in the x direction,
but it doesn't do anything to the y direction.
So the vector with a component in the x direction,
the x component gets increased by an by a factor 1 plus delta.
The components of each of these vectors in the y direction
don't change, all right?
So we're going to take this cloud of points,
and we're going to stretch it in the x direction.
What about this matrix here?
What's that going to do?
AUDIENCE: Stretch it in the y direction.
MICHALE FEE: Good.
It's going to stretch it out in the y direction.
Good.
So that's kind of cute.
And you can see that this earlier matrix that we looked
at right here stretches in the x direction
and stretches in the y direction.
And that's why that cloud of vectors
just stretched out equally in all directions.
Out this.
What is that going to do?
AUDIENCE: It would stretch in the x direction and compress
in the y direction
MICHALE FEE: Right.
This perturbation here is making this component,
the x component larger.
This perturbation here-- and delta here is small.
It's less than one.
Here, it's making the y component smaller.
And so what that looks like is the y component of each one
of these vectors gets smaller.
The x component gets larger.
And so we're squeezing in one direction
and stretching in the other direction.
Imagine we took a block of sponge
and we grabbed it and stretched it out,
and it gets skinny in this direction
and stretches out in that direction.
All right, that's kind of cool.
What is this going to do?
Here, I'm not making a small perturbation of this,
but I'm flipping the sign of one of those.
What happens there?
What is that going to do?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: Good.
What do we call that?
There's a term for it.
What do you-- yeah, it's called a mirror reflection.
So every point that's on this side of the origin
gets reflected over to this side of the origin.
And every point that's over here--
sorry, of this axis.
Every point that's on this side of the y-axis
gets reflected over to this side.
So that's called a mirror reflection.
What is this?
What is that going to do?
Abiba?
AUDIENCE: Reflect it [INAUDIBLE]..
MICHALE FEE: Right.
It's going to reflect it through the origin, like this.
So every point that's over here, on one side of the origin,
is going to reflect through to the other side.
That's pretty neat.
Inversion of the origin.
OK?
So we have symmetric perturbations
in the x and y components of the identity matrix.
We have a stretch transformation that stretches along one axis,
but not the other.
Stretch around the other axis, the y-axis, but not the x-axis.
Stretch along x and compression along y.
Mirror reflection through the y-axis.
Inversion through the origin.
These are examples of diagonal matrices, OK?
So the only thing we've done so far--
we've gotten all these really cool transformations,
but the only thing we've done so far
are change these two diagonal elements.
So there's a lot more crazy stuff
to happen if we start messing with the other components.
Oh, and I should mention that we can invert
any one of these transformations that we just did by finding
the inverse of this matrix.
The inverse of a diagonal matrix is very simple to calculate.
It's just one over those diagonal elements.
All right, how about this?
What is that going to do?
Anybody?
When you take a vector and you multiply it
by that, what's going to happen?
This part is going to give you the original vector back.
This part is going to take a little bit of the y component
and add it to the x component.
So what does that do?
That produces what's known as a shear.
So points up here, we're going to take
a little bit of the y component and add it to the x component.
So if something has a big y component,
it's going to be shifted in x.
If something has a negative y component,
it's going to shift this way in x.
If something has a positive y component,
it's going to shift this way an x.
And it's going to produce what's called a shear.
So we're pushing these points this way,
pushing those points this way.
Shear is very important in things like the flow of liquid.
So when you have liquid flowing over a surface,
you have forces, frictional forces to the liquid down here
that prevent it from moving.
Liquid up here moves more quickly,
and it produces a shear in the pattern of velocity profiles.
OK, that's pretty cool.
What about this?
It's going to just produce a shear
along the other direction.
That's right.
So now components that have a--
vectors that have a large x component acquire
a negative projection in y.
OK, what does this look like?
It's pretty cool.
We're going to get some shear in this direction,
get some shear in this direction.
What's it going to do?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: Good.
Good guess.
That's exactly right, produces a rotation.
Not exactly a rotation, but very close.
So that's how you actually produce a rotation.
So notice, for small angles theta, these are close to one,
so it's close to an identity matrix.
These are close to zero, but this is negative
and this is positive, or the other way around.
So if we have diagonals close to one and the off-diagonals one
positive and one negative, then that produces a rotation.
That, formally, is a rotation matrix.
Yes?
AUDIENCE: On the previous slide, is there
a reason you chose to represent the delta on the x-axis as
negative?
MICHALE FEE: No.
It goes either way.
So if you have a rotation angle that's positive,
then this is negative and this is positive.
If your rotation angle is the other sign,
then this is positive and this is negative.
So, for example, if we want to produce a 45-degree rotation,
then we have 1, 1, minus 1, 1.
And of course, all those things have a square root
of 2, 1 over square root of 2, in them.
And so that looks like this.
So if you have, let's say, theta equals 10 degrees,
we can produce a 10-degree rotation of all the vectors.
If theta is 25 degrees, you can see
that the rotation is further.
Theta 45, that's this case right here.
You can see that you get a 45-degree rotation of all
of those vectors around the origin.
And if theta is 90 degrees, you can see that, OK?
Pretty cool, right?
OK, what is the inverse of this rotation matrix?
So if we have a rotation-- oh, and I just
want to point out one more thing.
In this formulation of the rotation matrix,
positive angles correspond to rotating counterclockwise.
Negative angles correspond to rotation
in the clockwise direction, OK?
So there's a big hint.
What is the inverse of our rotation matrix?
If we have a rotation of 10 degrees this way,
what is the inverse of that?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: Right.
AUDIENCE: [INAUDIBLE]
MICHALE FEE: That's right.
Remember, matrix multiplication implements a transformation.
The inverse of that transformation
just takes you back where you were.
So if you have a rotation matrix that you implemented
a 20-degree rotation in the plus direction,
then the inverse of that is a 20-degree rotation
in the minus direction.
So the inverse of this matrix you
can get just by putting in a minus sign into the theta.
And you can see that cosine of minus theta
is just cosine of theta.
But sine of minus theta is negative sine of theta.
So the inverse of this matrix is just this.
You change the sign of those diagonals,
which just makes the shear go in the opposite direction, right?
OK, so a rotation by angle plus theta
followed by a rotation of angle minus theta
puts everything back where it was.
So rotation matrix phi of minus theta times phi of theta
is equal to the identity matrix.
So those two are inverses of each other.
And the inverse of a-- notice that the inverse
of this rotation matrix is also just the transpose
of the rotation matrix.
All right, so what you can see is
that these different cool transformations
that these matrix multiplications can do
are just examples of what our feed-forward network can do.
Because the feed-m forward network
just implements matrix multiplication.
So this feed-forward network takes
a set of vectors, a set of input vectors,
and transforms them into a set of output vectors, all right?
And you can understand what that transformation does just
by understanding the different kinds of transformations
you can get from matrix multiplication.
All right, we'll continue next time.