MICHALE FEE: OK, let's go ahead and get started.
All right, so today, we're going to continue talking about
feed-forward neural networks, and we're going to keep working
on some interesting aspects of linear algebra--
matrix transformations.
We're going to introduce a new idea from linear algebra,
the idea of basis sets.
We're going to describe some interesting and important
properties of basis sets, such as linear independence.
And then we're going to end with just a very simple formulation
of how to change between different basis sets.
So let me explain a little bit more,
motivate a little bit more why we're doing these things.
So as people, as animals, looking out at the world,
we are looking at high-dimensional data.
We have hundreds of millions of photoreceptors in our retina.
Those data get compressed down into about a million
nerve fibers that go through our optic nerve up to our brain.
So it's a very high-dimensional data set.
And then our brain unpacks that data and tries
to make sense of it.
And it does that by passing that data
through layers of neural circuits
that make transformations.
And we've talked about how in going from one layer of neurons
to another layer of neurons, there's
a feed-forward projection that essentially
does what looks like a matrix multiplication, OK?
So that's one of the reasons why we're
trying to understand what matrix multiplications do.
Now, we talked about some of the matrix transformations
that you can see when you do a matrix multiplication.
And one of those was a rotation.
Matrix multiplications can implement rotations.
And rotations are very important for visualizing
high-dimensional data.
So this is from a website at Google research,
where they've implemented different viewers
for high-dimensional data, ways of taking high-dimensional data
and reducing the dimensionality and then visualizing
what that data looks like.
And one of the most important ways
that you visualize high-dimensional data
is by rotating it and looking at it from different angles.
And what you're doing when you do
that is you take this high-dimensional data,
you rotate it, and you project it
into a plane, which is what you're seeing on the screen.
And you can see that you get a lot out
of looking at different projections
and different rotations of data sets.
Also, when you're zooming in on the data,
that's another matrix transformation.
You can stretch and compress and do
all sorts of different things to data.
Now, one of the cool things is that when
we study the brain to try to figure out
how it does this really cool process of rotating data
through its transformations that are
produced by neural networks, we record from lots of neurons.
There's technology now where you can
image from thousands, or even tens of thousands, of neurons
simultaneously.
And again, it's this really high-dimensional data
set that we're looking at to try to figure out
how the brain works.
And so in order to analyze those data,
we try to build programs or machines that
act like the brain in order to understand the data that we
collect from the brain.
It's really cool.
So it's kind of fun.
As neuroscientists, we're trying to build a brain
to analyze the data that we collect from the brain.
All right, so the cool thing is that the math that we're
looking at right now and the kinds of neural networks
that we're looking at right now are exactly
the kinds of math and neural networks
that you use to explain the brain
and to look at data in very powerful ways, all right?
So that's what we're trying to do.
So let's start by coming back to our two-layer feed-forward
network and looking in a little bit more
detail about what it does.
OK, so I introduced the idea, this two-layer feed-forward
network.
We have an input layer that has a vector of firing rates,
a firing rate that describes each of those input neurons,
a vector of firing rates.
That, again, is a list of numbers
that describes the firing rate of each neuron in the output
layer.
And the connections between these two layers
are a bunch of synapses, synaptic weights,
that we can use to calculate to transform the firing
rates at the input layer into the firing rates at the output
layer.
So let's look in a little bit more detail now
at what that collection of weights looks like.
So we describe it as a matrix.
That's called the weight matrix.
The matrix has in it a number for the weight
from each of the input neurons to each of the output neurons.
The rows are a vector of weights onto each
of the output neurons.
And we'll see in a couple of slides
that the columns are the set of weights from each input neuron
to all the output neurons.
A row of this weight matrix is a vector of weights
onto one of the output neurons.
All right, so we can compute the firing rates
of the neurons in our output layer
for the case of linear neurons in the output layer
simply as a matrix product of this weight
vector times the vector of input firing rates.
And that matrix multiplication gives us
a vector that describes the firing rates of the output
layer.
So let me just go through what that looks like.
If we define a column vector of firing rates of each
of the output neurons, we can write that
as the weight matrix times the column vector of the firing
rates of the input layer.
We can calculate the firing rate of the first neuron
in the output layer as the dot product
of that row of the weight matrix with that vector
of firing rates, OK?
And that gives us the firing rate. v1 is then
W of a equals 1 dot u.
That is one particular way of thinking
about how you're calculating the firing rates in the output
layer.
And it's called the dot product interpretation
of matrix multiplication, all right?
Now, there's a different sort of complementary way
of thinking about what happens when
you do this matrix product that's also
important to understand, because it's
a different way of thinking about what's going on.
We can also think about the columns of this weight matrix.
And we can think about the weight matrix
as a collection of column vectors
that we put together into matrix form.
So in this particular network here,
we can write down this weight matrix, all right?
And you can see that this first input
neuron connects to output neuron one, so there's a one there.
The first input neuron connects to output neuron two,
so there's a one there.
The first input neuron does not connect to output neuron three,
so there's a zero there, OK?
All right.
So the columns of the weight matrix
represent the pattern of projections from one
of the input neurons to all of the output neurons.
All right, so let's just take a look at what
would happen if only one of our input neurons was active
and all the others were silent.
So this neuron is active.
What would the output vector look like?
What would the pattern of firing rates
look like for the output neurons in this case?
Anybody?
It's straightforward.
It's not a trick question.
[INAUDIBLE]?
AUDIENCE: So--
MICHALE FEE: If this neuron is firing and these weights
are all one or zero.
AUDIENCE: The one neuron, a--
MICHALE FEE: Yes?
This--
AUDIENCE: Yeah, [INAUDIBLE].
MICHALE FEE: --would fire, this would fire,
and that would not fire, right?
Good.
So you can write that out as a matrix multiplication.
So the firing rate vector, in this case,
would be the dot product of this with this, this with this,
and that with that.
And what you would see is that the output firing rate
vector would look like this first column of the weight
matrix.
So the output vector would look like 1,
1, 0 if only the first neuron were active.
So you can think of the output firing rate vector
as being a contribution from neuron one--
and that contribution from neuron one
is simply the first column of the weight matrix--
plus a contribution from neuron two,
which is given by the second column of the weight
matrix, and a contribution from input neuron three,
which is given by the third column of the weight matrix,
OK?
So you can think of the output firing rate vector
as being a linear combination of a contribution
from the first neuron, a contribution
from the second neuron, and a contribution
from the third neuron.
Does that make sense?
It's a different way of thinking about it.
In the dot product interpretation,
we're asking, what is the--
we're summing up all of the weights
onto neuron one from those synapses.
We're summing up all the weights onto neuron two
from those synapses and summing up all the weights onto neuron
three from those synapses.
So we're doing it one output neuron at a time.
In this other interpretation of this matrix multiplication,
we're doing something different.
We're asking, what is the contribution to the output
from one of the input neurons?
What is the contribution to the output from another input
neuron?
And what is the contribution to the output
from yet another input neuron?
Does that makes sense?
OK.
All right, so we have a linear combination
of contributions from each of those input neurons.
And that's called the outer product interpretation.
I'm not going to explain right now why it's called that,
but that's how that's referred to.
So the output pattern is a linear combination
of contributions.
OK, so let's take a look at the effect of some very simple
feed-forward networks, OK?
So let's just look at a few examples.
So if we have a feed forward-- this
is sort of the simplest feed-forward network.
Each neuron in the input layer connects
to one neuron in the output layer with a weight of one.
So what is the weight matrix of this network?
AUDIENCE: Identity.
MICHALE FEE: It's the identity matrix.
And so the firing rate of the output layer
will be exactly the same as the firing rates in the input
layer, OK?
So there's the weight matrix, which is just the identity
matrix, the firing rate.
And the output layer is just the identity matrix
times the firing rate of the input layer.
And so that's equal to the input firing rate, OK?
All right, let's take a slightly more complex network,
and let's make each one of those weights independent.
They're not all just equal to one,
but they're scaled by some constant--
lambda 1, lambda 2, and lambda 3.
The weight matrix looks like this.
It's a diagonal matrix, where each of those weights
is on the diagonal.
And in that case, you can see that the output firing
rate is just this diagonal matrix times the input firing
rate.
And you can see that the output firing rate is just
the input firing rate where each component of the input firing
rate is scaled by some constant.
Pretty straightforward.
Let's take a look at a case where the weight matrix now
corresponds to a rotation matrix, OK?
So we're going to let the weight matrix look like this rotation
matrix that we talked about on Tuesday, where
are the diagonal elements are cosine of sum rotation angle,
and the off-diagonal elements are plus and minus sine
of the rotation angle.
So you can see that this weight matrix corresponds
to this network, where the projection from input neuron
one to output neuron one is cosine phi.
Input neuron two to output neuron two is cosine phi.
And then these cross-connections are a plus and minus sine phi.
OK, so what does that do?
So we can see that the output firing rate vector is just
a product of this rotation matrix times the input firing
rate vector.
And you can write down each component like that.
All right, so what does that do?
So let's take a particular rotation angle.
We're going to take a rotation angle of pi
over 4, which is 45 degrees.
That's what the weight matrix looks like.
And we can do that multiplication
to find that the output firing rate vector looks like--
one of the neurons has a firing rate
that looks like the sum of the two input firing rates,
and the other output neuron has a firing rate
that looks like the difference between the two input firing
rates.
And if you look at what this looks
like in the space of firing rates of the input
layer and the output layer, we can see what happens, OK?
So what we'll often do when we look
at the behavior of neural networks
is we'll make a plot of the firing
rates of the different neurons in the network.
And what we'll often do for simple feed-forward networks,
and we'll also do this for recurrent networks,
is we'll plot the input firing rates as in the plane of u1
and u2.
And then we can plot the output firing rates in the same plane.
So, for example, if we have an input state that
looks like u1 equals u2, it will be some point
on this diagonal line.
We can then plot the output firing rate
on this plane, v1 versus v2.
And what will the output firing rate look like?
What will the firing rate of v1 look like in this case?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: Yeah let's say this is one and one.
So what will the firing rate of this neuron look like?
[INAUDIBLE]?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: What's that?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: So the firing rate of v1
is just this quantity right here, right?
So it's u1 plus u2, right?
So it's like 1 plus 1 over root 2.
So it will be big.
What will the firing rate of neuron v2 look like?
It'll be u2 minus u1, which is?
AUDIENCE: Zero.
MICHALE FEE: Zero.
So it will be over here, right?
So it will be that input rotated by 45 degrees.
And input down here--
so the firing rate of the one will be the sum of those two.
Those two inputs are both negative.
So v1 for this input will be big and negative.
And v2 will be the difference of u1 and u2,
which for anything on this line is?
AUDIENCE: Zero.
MICHALE FEE: Zero.
OK.
And so that input will be rotated over to here.
So you can think of it this way--
any input in this space of u1 and u2, in the output
will be just rotate by, in this case, it's minus 45 degrees.
So that's clockwise, are the minus rotations.
So you can just predict the output firing rates simply
by taking the input firing rates in this plane
and rotating them by minus 45 degrees.
All right, any questions about that?
It's very simple.
So this little neural network implements
rotations of this input space.
That's pretty cool.
Why would you want a network to do rotations?
Well, this solves exactly the problem
that we were working on last time
when we were talking about our perceptron, where we were
trying to classify stimuli that could not be separated
in one dimension, but rather, can
be separated in two dimensions.
So if we have different categories--
dogs and non-dogs--
that can be viewed along different dimensions--
how furry they are--
but can't be separated--
the two categories can't be separated from each other
on the basis of just one dimension of observation.
So in this case, what we want to do is take this base of inputs
and rotate it into a new what we'll call a new basis set
so that now we can take the firing rates of these output
neurons and use those to separate
these different categories from each other.
Does that make sense?
OK, so let me show you a few more examples of that.
So this is one way to think about what we
do when we do color vision, OK?
So you know that we have different cones in our retina
that are sensitive to different wavelengths.
Most colors are combinations of those wavelengths.
So if we look at the activity of, let's say,
a cone that's sensitive to wavelength one and the activity
in a cone that's sensitive to wavelength two, we might see--
and then we look around the world.
We'll see a bunch of different objects
or a bunch of different stimuli that
activate those two different cones in different ratios.
And you might imagine that this axis corresponds to, let's say,
how much red there is in a stimulus.
This axis corresponds to how much green
there is in a stimulus.
But let's say that you're in an environment where
there's some cloud of contribution of red and green.
So what would this direction correspond to in this cloud?
This direction corresponds to more red and more green.
What would that correspond to?
AUDIENCE: Brown.
MICHALE FEE: So what I'm trying to get at here
is that the sum of those two is sort
of the brightness of the object, right?
Something that has little red and little green
will look the same color as something that has
more red and more green, right?
But what's different about those two stimuli
is that the one's brighter than the other.
The second one is brighter than the first one.
But this dimension corresponds to what?
Differences in the ratio of those two colors, right?
Sort of changes in the different [AUDIO OUT] wavelengths,
and that corresponds to color.
So if we can take this base of stimuli
and rotate it such that one axis corresponds
to the sum of the two colors and the other axis corresponds
to the difference of the two colors,
then this axis will tell you how bright it is,
and this axis will tell you what the hue is, what the color is.
Does that makes sense?
So there's a simple case of where
taking a rotation of a inputs base, of a set of sensors,
will give you different information
than you would get if you just had one of those stimuli.
If you were to just look at the activity of the cone that's
giving you a red signal, if one object
has more activity in that cone, you
don't know whether that other object is just brighter
or if it's actually more red, that looked red.
Does that makes sense?
So doing a rotation gives us signals in single neurons
that carries useful information.
It can disambiguate different kinds of information.
All right, so we can use that simple rotation matrix
to perform that kind of separation.
So brightness and color.
Here's another example.
I didn't get to talk about this in this class, but there are--
so barn owls, they can very exquisitely localize objects
by sound.
So they hunt, essentially, at night in the dark.
They can hear a mouse scurrying around in the grass.
They just listen to that sound, and they
can tell exactly where it is, and then they dive down
and catch the mouse.
So how did they do that?
Well, they used timing differences
to tell which way the sound is coming from side to side,
and they use intensity differences
to tell which way the sound is coming from up and down.
Now, how do you use intensity differences?
Well, one of their ears, their right ear
pointed slightly upwards.
And their left ear is pointed slightly downwards.
So when they hear a sound that's slightly louder
in the right ear and slightly softer in the left ear,
they know that it's coming from up above, right?
And if it's the other way around,
if it's slightly louder in the left ear and softer
in the right ear, they know it's coming from below horizontal.
And it's extremely precise system, OK?
So here's an example.
So if they're sitting there listening
to the intensity, the amplitude of the sound in the left ear
and the amplitude of the sound in the right ear,
some sounds will be up here with high amplitude in both ears.
Some sounds will be over here, with more
amplitude in the right ear and less amplitude in the left ear.
What does this dimension correspond to?
That dimension corresponds to?
AUDIENCE: Proximity.
MICHALE FEE: Proximity or, overall,
the loudness of the sound, right?
And what does this dimension correspond to?
AUDIENCE: Direction.
MICHALE FEE: The difference in intensity corresponds
to the elevation of the sound relative to the horizontal.
All right?
So, in fact, what happens in the owl's brain
is that these two signals undergo a rotation to produce
activity in some neurons that's sensitive to the overall
loudness and activity in other neurons that's
sensitive to the difference between the intensity
of the two sounds.
It's a measure of the elevation of the sounds.
All right, so this kind of rotation matrix
is very useful for projecting stimuli
into the right dimension so that they give useful signals.
All right, so let's come back to our matrix transformations
and look in a little bit more detail
about what kinds of transformations
you can do with matrices.
So we talked about how matrices can do
stretch, compression, rotation.
And we're going to talk about a new kind of transformation
that they can do.
So you remember we talked about how a matrix multiplication
implements a transformation from one set of vectors
into another set of vectors?
And the inverse of that matrix transforms back
to the original set of vectors, OK?
So you can make a transformation,
and then you can undo that transformation
by multiplying by the inverse of the matrix.
OK, so we talked about different kinds of transformations
that you can do.
So if you take the identity matrix
and you make a small perturbation
to both of the diagonal elements, the same perturbation
to both diagonal elements, you're basically
taking a set of vectors and you're stretching them
uniformly in all directions.
If you make a perturbation to just one of the components
of the identity matrix, you can take the data
and stretch it in one direction or stretch it
in the other direction.
If you add something to the first component
and subtract something from the second component,
you can stretch in one direction and compress
in another direction.
We talked about reflections and inversions through the origin.
These are all transformations that are
produced by diagonal matrices.
And the inverse of those diagonal matrices
is just one over the diagonal elements.
OK, we also talked about rotations
that you can do with this rotation matrix.
And then the inverse of the rotation matrix
is, basically, you compute the inverse of a rotation matrix
simply by computing the rotation matrix
with a minus sign for this, using the negative
of the rotation angle.
And we also talked about how a rotation matrix--
for a rotation matrix, the inverse
is also equal to the transpose.
And the reason is that rotation matrices
have this antisymmetry, where the off-diagonal elements have
the opposite sign.
One of the things we haven't talked about is--
so we talked about how this kind of matrix
can produce a stretch along one dimension or a stretch
along the other dimension of the vectors.
But one really important kind of transformation
that we need to understand is how you can produce stretches
in an arbitrary direction, OK?
So not just along the x-axis or along the y-axis, but along any
arbitrary direction.
And the reason we need to know how that works
is because that formulation of how you write down a matrix
to stretch data in any arbitrary direction
is the basis of a lot of really important data analysis
methods, including principal component
analysis and other methods.
So I'm going to walk you through how
to think about making stretches in data
in arbitrary dimensions.
OK, so here's what we're going to walk through.
Let's say we have a set of vectors.
I just picked--
I don't know, what is that--
20 or so random vectors.
So I just called a random number generator 20 times
and just picked 20 random vectors.
And we're going to figure out how to write down a matrix that
will transform that set of vectors
into another set of vectors that stretched along some arbitrary
axis.
Does that make sense?
So how do we do that?
And remember, we know how to do two things.
We know how to stretch a set of vectors along the x-axis.
We know how to stretch vectors along the y-axis,
and we know how to rotate a set of vectors.
So we're just going to combine those two
ingredients to produce this stretch in an arbitrary
direction.
So now I've given you the recipe--
or I've given you the ingredients.
The recipe's pretty obvious, right?
We're going to take this set of initial vectors.
Good.
Lina?
AUDIENCE: You [INAUDIBLE].
That's it.
MICHALE FEE: Bingo.
That's it.
OK, so we're going to take--
all right, so we're going to rotate this thing 45 degrees.
We take this original set of vectors.
We're going to--
OK, so first of all, the first thing
we do when we want to take a set of points
and stretch it along an arbitrary direction,
we pick that angle that we want to stretch it
on-- in this case, 45 degrees.
And we write down a rotation matrix corresponding
to that rotation, corresponding to that angle.
So that's the first thing we do.
So we've chosen 45 degrees as the angle
we want to stretch on.
So now we write down a rotation matrix
for a 45-degree rotation.
Then what we're going to do is we're
going to take that set of points and we're
going to rotate it by minus 45 degrees.
So how do we do that?
How do we take any one of those vectors x and rotate it by--
so this that rotation matrix is for plus 45.
How do we rotate that vector by minus 45?
AUDIENCE: [INAUDIBLE] multiply it by the [INAUDIBLE]..
MICHALE FEE: Good.
Say it.
AUDIENCE: Multiply by the inverse of that.
MICHALE FEE: Yeah, and what's the inverse of a--
AUDIENCE: Transpose.
MICHALE FEE: Transpose.
So we don't have to go to Matlab and use the inverse matrix
in inversion.
We can just do the transpose.
OK, so we take that vector and we multiply it by transpose.
So that does a minus 45-degree rotation
of all of those points.
And then what do we do?
Lina, you said it.
Stretch it.
Stretch it along?
AUDIENCE: The x-axis?
MICHALE FEE: The x-axis, good.
What does that matrix look like that does that?
Just give me-- yup?
AUDIENCE: 5, 0, 0, 1.
MICHALE FEE: Awesome.
That's it.
So we're going to stretch using a stretch matrix.
So I use phi for a rotation matrix,
and I use lambda for a stretch matrix, a stretch
matrix along x or y.
Lambda is a diagonal matrix, which always just
stretches or compresses along the x or y direction.
And then what do we do?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: Good.
By multiplying by?
By this.
Excellent.
That's all.
So how do we write this down?
So, remember, here, we're sort of marching through the recipe
from left to right.
When you write down matrices, you go the other way.
So when you do matrix multiplication,
you take your vector x and you multiply it on the left side
by phi transpose.
And then you take that and you multiply that on the left side
by lambda.
And then you take that.
That now gives you these.
And now to get the final answer here,
you multiply again on the left side by phi.
That's it.
That's how you produce an arbitrary stretch--
a stretch or a compression of a data
in an arbitrary direction, all right?
You take the data, the vector.
You multiply it by a rotation matrix transpose,
multiply it by a stretch matrix, a diagonal matrix,
and you multiply it by a rotation matrix.
Rotate, stretch, unrotate.
So let's actually do this for 45 degrees.
So there's our rotation matrix--
1, minus 1, 1, 1.
The transpose is 1, 1, minus 1, 1.
And here's our stretch matrix.
In this case, it was stretched by a factor of two.
So we multiply x by phi transpose, multiply by lambda,
and then multiply by phi.
So we can now write that down.
If you just do those three matrix
multiplications-- those two matrix multiplications, sorry,
yes?
One, two.
Two matrix multiplications.
You get a single matrix that when you multiply it by
x implements this stretch.
Any questions about that?
You should ask me now if you don't understand,
because I want you to be able to do this for an arbitrary--
so I'm going to give you some angle,
and I'll tell you, construct a matrix that
stretches data along a 30-degree axis by a factor of five.
You should be able to write down that matrix.
All right, so this is what you're going to do,
and that's what that matrix will look like, something like that.
Now, we can stretch these data along a 45-degree axis
by some factor.
It's a factor of two here.
How do we go back?
How do we undo that stretch?
So how do you take the inverse of a product of a bunch
of matrices like this?
So the answer is very simple.
If we want to take the inverse of a product of three matrices,
what we do is we just--
it's, again, a product of three matrices.
It's a product of the inverse of those three matrices,
but you have to reverse the order.
So if you want to find the inverse of matrix A times B
times C, it's C inverse times B inverse times A inverse.
And you can prove that that's the right term as follows.
So ABC inverse times ABC should be the identity matrix, right?
So let's replace this by this result here.
So C inverse B inverse A inverse times
ABC would be the identity matrix.
And you can see that right here, A inverse times A is i.
So you can get rid of that.
B inverse times B is i.
C inverse times C is i.
So we just proved that that is the correct way
of taking the inverse of a product of matrices, all right?
So the inverse of this kind of matrix
that stretches data along an arbitrary direction
looks like this.
It's phi transpose inverse lambda inverse phi inverse.
So let's figure out what each one of those things is.
So what is phi transpose inverse,
where phi is a rotation matrix?
AUDIENCE: Just phi.
MICHALE FEE: Phi, good.
And what is phi inverse?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: [INAUDIBLE].
Good.
And lambda inverse we'll get to in a second.
So the inverse of this arbitrary rotated stretch matrix
is just another rotated stretch matrix, right?
Where the lambda now has--
lambda inverse is just given by the inverse of each
of those diagonal elements.
So it's super easy to find the inverse of one
of these matrices that computes this stretch in an arbitrary
direction.
You just keep the same phi.
It's just phi times some diagonal matrix times
phi transpose, but the diagonals are inverted.
Does that makes sense?
All right, so let's write it out.
We're going to undo this 45-degree stretch that we just
did.
We're going to do it by rotating, stretching by 1/2
instead of stretching by two.
So you can see that compresses now along the x-axis.
And then we rotate back, and we're back
to our original data.
Any questions about that?
It's really easy, as long as you just
think through what you're doing as you go through those steps,
all right?
Any questions about that?
OK.
Wow.
All right.
So you can actually just write those down
and compute the single matrix that
implements this compression along that 45-degree axis, OK?
All right.
So let me just show you one other example.
And I'll show you something interesting
that happens if you construct a matrix that instead
of stretching along a 45-degree axis does compression
along a 45-degree axis.
So here's our original data.
Let's take that data and rotate it by plus 45 degrees.
Multiplied by lambda, that compresses along the x-axis
and then rotates by minus 45 degrees.
So here's an example where we can take data and compress it
along an axis of minus 45 degrees, all right?
So you can write this down.
So we're going to say we're going to compress
along a minus 45 degree axis.
We write down phi of minus 45.
Notice that when you do this compression or stretching,
there are different ways you can do it, right?
You can take the data.
You can rotate it this way and then squish along this axis.
Or you could rotate it this way and squish along this axis,
right?
So there are choices for how you do it.
But in the end, you're going to end up
with the same matrix that does all of those equivalent
transformations.
OK, so here we are.
We're going to write this out.
So we're writing down a matrix that
produces this compression along a minus 45-degree axis.
So there's 5 minus 45.
There's lambda, a compression along the x-axis.
So here, it's 0.2001.
And here's the phi transpose.
So you write all that out, and you get 0.6, 0.4, 0.4, 0.4.
Let me show you one more.
What happens if we accidentally take this data, we rotate it,
and then we squish the data to zero?
Yes?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: It doesn't.
You can do either one.
Let me go back.
Let me just go back to the very first one.
So here, we rotated clockwise and then
stretched along the x-axis and then unrotated.
We could have taken these data, rotated counterclockwise,
stretched along the y-axis, and then rotated back, right?
Does that make sense?
You'll still get the same answer.
You'll still get the same answer for this matrix here.
OK, now watch this.
What happens if we take these data, we rotate them,
and then we compress data all the way to zero?
So by compressing the data to a line,
we're multiplying it by zero.
We put a zero in this element of the stretch matrix, all right?
And what happens?
The data get compressed right to zero, OK?
And then we can rotate back.
So we've taken these data.
We can write down a matrix that takes those data
and squishes them to zero along some arbitrary direction.
Now, can we take those data and go back to the original data?
Can we write down a transformation
that takes those and goes back to the original data?
Why not?
AUDIENCE: Lambda doesn't [INAUDIBLE]..
MICHALE FEE: Say it again.
AUDIENCE: Lambda doesn't [INAUDIBLE]..
MICHALE FEE: Good.
What's another way to think about that?
AUDIENCE: We've lost [INAUDIBLE]..
MICHALE FEE: You've lost that information.
So in order to go back from here to the original data,
you have to have information somewhere here that tells you
how far out to stretch it again when you try to go back.
But in this case, we've compressed everything
to a line, and so there's no information
how to go back to the original data.
And how do you know if you've done this?
Well, you can take a look at this matrix that you created.
So let's say somebody gave you this matrix.
How would you tell whether you could
back to the original data?
Any ideas?
Abiba?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: Good.
You look at the determinant.
So if you calculate the determinant of this matrix,
the determinant is zero.
And as soon as you see a zero determinant,
you know right away that you can't go back.
After you've made this transformation,
you can't go back to the original data.
And we're going to get into a little more detail about why
that is and what that means.
And the reason here is that the determinant of lambda is zero.
The determinant of a product matrices
like this is the product of the determinants.
And in this case, the determinant of the lambda
matrix is zero, and so the determinant of the product
is zero, OK?
All right, so now let's talk about basis sets.
All right, so we can think of vectors in abstract directions.
So if I hold my arm out here and tell you
this is a vector-- there's the origin.
The vectors pointing in that direction.
You don't need a coordinate system
to know which way I'm pointing.
I don't need to tell you my arm is pointing
80 centimeters in that direction and 40
centimeters in that direction and 10 centimeters
in that direction, right?
You don't need a coordinate system
to know which way I'm pointing, right?
But if I want to quantify that vector so
that-- if you want to quantify that vector so that you can
maybe tell somebody else precisely which direction I'm
pointing, you need to write down those numbers, OK?
So you can think of vectors in abstract directions,
but if you want to actually quantify it or write it down,
you need to choose a coordinate system.
And so to do this, you choose a set
of vectors, special vectors, called a basis set.
And now we just say, here's a vector.
How much is it pointing in that direction, that direction,
and that direction?
And that's called a basis set.
So we can write down our vector now
as a set of three numbers that simply tell us
how far that vector is overlapped
with three other vectors that form the basis set.
So the standard way of doing this
is to describe a vector as a component in the x direction,
which is a vector 1, 1, 0, sort of in the standard notation;
a component in the y direction, which is 0,
1, 0; and a component in the z direction, 0, 0, 1.
So we can write those vectors as standard basis vectors.
The numbers x, y, and z here are called
the coordinates of the vector.
And the vectors e1, e2, and e3 are called the basis vectors.
And this is how you would write that down
for a three-dimensional vector, OK?
Again, the little hat here denotes
that those are unit vectors that have a length one.
All right, so in order to describe an arbitrary
vector in a space of n real numbers,
Rn, the basis vectors each need to have n numbers.
And in order to describe an arbitrary vector in that space,
you need to have n basis vectors.
You need to have--
in n dimensions, you need to have n basis vectors,
and each one knows basis vectors has to have n numbers in them.
So these vectors here--
1, 0, 0; 0, 1, 0; and 0, 0, 1-- are called the standard basis.
And each one of these values has one element that's one
and the rest are zero.
That's the standard basis.
The standard basis has the property
that any one of those vectors dotted into itself is one.
That's because they're unit vectors.
They have length one.
So i dot ei is the length squared of the i-th vector.
And if the length is one, then the length squared is one.
Each vector is orthogonal to all the other vectors.
That means that each e1 dot e2 is zero, and e1 dot e3 is zero,
and e2 dot e3 is zero.
You can write down as e sub i dot e sub j equals zero for i
not equal to j.
You can write all of those properties
down in one equation--
e sub i dot e sub j equals delta i j.
Delta i j is what's called the Kronecker delta function.
The Kronecker delta function is a one if i equals j and a zero
if i is not equal to j, OK?
So it's a very compact way of writing down this property
that each vector is a unit vector
and each vector is orthogonal to all the other vectors.
And the set with that property is called an off an orthonormal
basis set.
All right, now, the standard basis is not the only basis--
sorry.
I'm trying to do x, y, and z here.
So if you have x, y, and z, that's
not the only orthonormal basis set.
Any basis set that is a rotation of those three vectors
is also an orthonormal basis.
Let's write down two other orthogonal unit vectors.
We can write down our vector v in this other basis
set as follows.
We just take our vector v. We can plot the basis vectors
in this other basis.
And we can simply project v onto those other basis vectors.
So we can project v onto f1, and we can project v onto f2.
So we can write v as a sum of a vector in the direction of f1
and a vector in the direction of f2.
You can write down this vector v in this different basis set
as a vector with two components.
This is two dimensional.
This is R2.
You can write it down as a two-component vector--
v dot f1 and v dot f2.
So that's a simple intuition for what [AUDIO OUT]
in two dimensions.
We're going to develop the formalism for doing this
in arbitrary dimensions, OK?
And it's very simple.
All right, these components here are
called the vector coordinates of this vector basis f.
All right, now, basis sets, or basis vectors,
don't have to be orthogonal to each other,
and they don't have to be normal.
They don't have to be unit vector.
You can write down an arbitrary vector
as a sum of components that aren't
orthogonal to each other.
So you can write down this vector v
as a sum of a component here in the f1 direction
and a component in the f2 direction,
even if f1 and f2 are not orthogonal to each other
and even if they're not unit vectors.
So, again, v is expressed as a linear combination
of a vector in the f1 direction and a vector
in the f2 direction.
OK, so let's take a vector and decompose it
into an arbitrary basis set f1 and f2.
So v equals c1 f1 plus c2 f2.
The coefficients here are called the coordinates
of the vector in this basis.
And the vector v sub f--
these numbers, c1 and c2, when combined into this vector,
is called the coordinate vector of v in the basis f1
and f2, OK?
Does that makes sense?
Just some terminology.
OK, so let's define this basis, f1 and f2.
We just pick two vectors, an arbitrary two vectors.
And I'll explain later that not all choices of vectors work,
but most of them do.
So here are two vectors that we can choose as a basis--
so 1, 3, which is sort of like this, and minus 2, 1
is kind of like that.
And we're going to write down this vector
v in this new basis.
So we have a vector v that's 3, 5 in the standard basis,
and we're going to rewrite it in this new basis, all right?
So we're going to find the vector coordinates of v
in the new basis.
So we're going to do this as follows.
We're going to write v as a linear combination of these two
basis vectors.
So c1 times f1--
1, 3-- plus c2 times f2--
minus 2, 1-- is equal to 3, 5.
That make sense?
So what is that?
That is just a system of equations, right?
And what we're trying to do is solve for c1 and c2.
That's it.
So we already did this problem in the last lecture.
So we have this system of equations.
We can write this down in the following matrix notation.
F times vf-- vf is just c1 and c2--
equals v. So there's F--
1, 3; minus 2, 1.
Those are our two basis vectors.
Times c1 c2-- the vector c1, c2-- is equal to 3, 5.
And we solve for vf.
In other words, we solve for c1 and c2
simply by multiplying v by the inverse of this matrix F.
So the coordinate vector in this new base is said
is just the old vector times f inverse.
And what is f inverse?
F inverse is just a matrix that has the basis vectors
as the columns of the matrix.
So the coordinates of this vector in his new basis set
are given by f inverse times v. We can find the inverse of f.
So if that's our f, we can calculate the inverse of that.
Remember, you flip the diagonal elements.
You multiply the off-diagonals by minus 1,
and you divide by the determinant.
So f inverse is this times v is that, and v sub f is just 13/7
over minus 4/7.
So that's just a different way of writing v.
So there's v in the standard basis.
There's v in this new basis, all right?
And all you do to go from the standard basis
to any arbitrary new basis is multiply the vector
by f inverse.
And when you're actually doing this in Matlab,
this is really simple.
You just write down a matrix F that has
the basis sets in the columns.
You just use the matrix inverse function,
and then you multiply that by the data matrix,
by the data vector.
All right, so I'm just going to summarize again.
In order to find the coordinate vector for v in this new basis,
you construct a matrix F, whose columns
are just the elements of the basis vectors.
So if you have two basis vectors,
it's a two-- remember, each of those basis vectors.
In two dimensions, there are two basis vectors.
Each has two numbers, so this is a 2 by 2 matrix.
In n dimensions, you have n basis vectors.
Each of the basis vectors has n numbers.
And so this matrix F is an n by n matrix, all right?
You know that you can write down v as this basis times v sub f.
You solve for v sub f by multiplying both sides
by f inverse, all right?
That performs whats called change of basis.
Now, that only works if f has an inverse.
So if you're going to choose a new basis to write down
your vector, you have to be careful to pick one
that has an inverse, all right?
And I want to show you what it looks
like when you pick a basis that doesn't have an inverse
and what that means.
All right, and that gets to the idea of linear independence.
All right, so, remember I said that if in n dimensions, in Rn,
in order to have a basis in Rn, you have certain requirements?
Not any vectors will work.
So let's take a look at these vectors.
Will those work to describe an--
will that basis set work to describe an arbitrary
vector in three dimensions?
No?
Why not?
AUDIENCE: [INAUDIBLE] vectors, so if you're [INAUDIBLE]..
MICHALE FEE: Right.
So the problem is in which coordinate, which axis?
AUDIENCE: Z-axis.
MICHALE FEE: The z-axis.
You can see that you have zeros in all three of those vectors,
OK?
You can't describe any vector with this basis
that has a non-zero component in the z direction.
And the reason is that any linear combination
of these three vectors will always lie in the xy plane.
So you can't describe any vector here
that has a non-zero z component, all right?
So what we say is that this set of vectors
doesn't span all of R3.
It only spans the xy plane, which
is what we call a subspace of R3, OK?
OK, so let's take a look at these three vectors.
The other thing to notice is that you
can write any one of these vectors
as a linear combination of the other two.
So you can write f3 as a sum of f1 and f2.
The sum of those two vectors is equal to that one.
You can write f2 as f3 minus f1.
So any of these vectors can be written as a linear combination
of the others.
And so that set of vectors is called linearly dependent.
And any set of linearly dependent vectors cannot form
a basis.
And how do you know if a set of vectors
that you choose for your basis is linearly dependent?
Well, again, you just find the determinant of that matrix.
And if it's zero, those vectors are linearly dependent.
So what that corresponds to is you're taking your data
and when you transform it into a new basis,
if the determinant of that matrix F
is zero, then what you're doing is you're taking those data
and transforming them to a space where they're being collapsed.
Let's say if you're in three dimensions,
those data are being collapsed onto a plane or onto a line,
OK?
And that means you can't undo that transformation, all right?
And the way to tell whether you've got that problem
is looking at the determinant.
All right, let me show you one other cool thing
about the determinant.
There's a very simple geometrical interpretation
of what the determinant is, OK?
All right, sorry.
So if f maps your data onto a subspace,
then the mapping is not reversible.
OK, so what does the determinant correspond to?
Let's say in two dimensions, if I have two orthogonal unit
vectors, you can think of those vectors
as kind of forming a square in this space.
Or in three dimensions, if I have three orthogonal vectors,
you can think of those vectors as defining a cube, OK?
And if there unit vectors, then they
define a cube of volume one.
Here, you have the square of area one.
So let's think about this unit volume.
If I transform those two vectors or those three vectors
in 3D space by a matrix A, those vectors
get rotated and transformed.
They point in different directions, and they define--
it's no longer a cube, but they define some sort of rhombus,
OK?
You can ask, what is the volume of that rhombus?
The volume of that rhombus is just the determinant
of that matrix A. So now what happens
if I have a cube in three-dimensional space
and I multiply it by a matrix that transforms it
into a rhombus that has zero volume?
So let's say I have those three vectors.
It transforms it into, let's say, a square.
The volume of that square in three dimensional space
is zero.
So what that means is I'm transforming my vectors
into a space that has zero volume
in the original dimensions, OK?
So I'm transforming things from 3D into a 2D plane.
And what that means is I've lost information,
and I can't go back.
OK, notice that a rotation matrix, if I take this cube
and I rotate it, has exactly the same volume
as it did before I rotated it.
And so you can always tell when you have a rotation matrix,
because the determinant of a rotation matrix is one.
So if you take a matrix A and you find the determinant
and you find that the determinant is one,
you know that you have a pure rotation matrix.
What does it mean if the determinant is minus one?
What it means is you have a rotation,
but that one of the axes is inverted, is flipped.
There's a mirror in there.
So you can tell if you have a pure rotation
or if you have a rotation and one of the axes is flipped.
Because in the pure rotation, the determinant is one.
And in an impure rotation, you have a rotation and a mirror
flip.
All right, and I just want to make a couple more comments
about change of basis, OK?
All right, so let's choose a set of basis vectors
for our new basis.
Let's write those into a matrix F.
It's going to be our matrix of basis vectors.
If the determinant is not equal to zero,
then these vectors, that set of vectors,
are linearly independent.
That means you cannot write one of those vectors as a linear
combination of--
any one of those vectors as a linear combination
of the others.
Those vectors form a complete basis in that n dimensional
space.
The matrix F implements a change of basis,
and you can go from the standard basis to F
by multiplying your vector by F inverse
to get the coordinate vector and your new basis.
And you can go back from that rotated or transformed basis
back to the coordinate basis by multiplying by F, OK?
Multiply by F inverse transforms to the new basis.
Multiplying by F transforms back.
If that set of vectors is an orthonormal basis, then--
OK, so let's take this matrix F that has columns
that are the new basis vectors.
And let's say that those form an orthonormal basis.
In that case, we can write down-- so, in any case,
we can write down the transpose of this matrix, F transpose.
And now the rows of that matrix are the basis vectors.
Notice that if we multiply F transpose times F,
we have basis vectors in rows here and columns here.
So what is F transpose F for the case
where these are unit vectors that
are orthogonal to each other?
What is that product?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: It's what?
AUDIENCE: [INAUDIBLE]
MICHALE FEE: Good.
Because F1 dot F1 is one.
F1 dot F2 is zero.
F2 dot F1 is zero, and F2 dot F2 is 0.
So that's equal to the identity matrix, right?
So F transpose equals F inverse.
If the inverse of a matrix is just its transpose,
then that matrix is a rotation matrix.
So F is just the rotation matrix.
All right, now let's see what happens.
So that means the inverse of F is just this F transpose.
Let's do this coordinate-- let's [AUDIO OUT]
change of basis for this case.
So you can see that v sub f, the coordinate vector in the new
basis, is F transpose v. Here's F transpose--
the basis vectors are in the rows--
times v. This is just v dot F1, v dot F2, right?
So this shows how for a orthonormal basis,
the transpose, which is the inverse of F--
taking the transpose of F times v
is just taking the dot product of v
with each of the basis vectors, OK?
So that ties it back to what we were showing before about how
to do this change of basis, OK?
Just tying up those two ways of thinking about it.
So, again, what we've been developing
when we talk about change of basis
are ways of rotating vectors, rotating sets
of data, into different dimensions,
into different basis sets so that we
can look at data from different directions.
That's all we're doing.
And you can see that when you look
at data from different directions, you can get--
some views of data, you have a lot of things overlapping,
and you can't see them.
But when you rotate those data, now, all of a sudden,
you can see things clearly that used to be--
things get separated in some views, whereas in other views,
things are kind of mixed up and covering each other, OK?
And that's exactly what neural networks are doing when they're
analyzing sensory stimuli.
They're doing that kind of rotations
and untangling the data to see what's
there in that high-dimensional data, OK?
All right, that's it.