Flash and JavaScript are required for this feature.
Download the video from Internet Archive.
Summary: The functional organization of scene perception and navigation and the various brain structures that implement them.
Speaker: Nancy Kanwisher

Lecture 8: Navigation I
[SQUEAKING]
[RUSTLING]
[CLICKING]
NANCY KANWISHER: Here's the agenda for today. As usual, a bunch of announcements in red. Assignment 4 was graded. There will be comments showing up online on Stellar soon on any of you who didn't get a near-perfect score on it. And I'll also be going over a little bit of it in a moment. And then once we do that, we're going to talk about navigation-- how we know where we are, and how to get from here to someplace else, which is much more awesome than it sounds at first, as you will see.
OK. So quick review. OK. So what was the key point? Why did I assign the Haxby 2001 article for you guys to read? It presents this important challenge to the functional specificity of the face area and the place area. What was that challenge? What was Haxby's key point? Yes, Isabel.
AUDIENCE: Well, he was just attacked-- it just has a preference for rectilinear and not seeing if it's actual scanning for. It's not truly detecting whether it's a face or not.
NANCY KANWISHER: Yeah. He wasn't worrying about rectilinearity so much back then. But his point was that we shouldn't care just about the overall magnitude of response of a region. Like, OK, it's nice if the face area responds like this to face, but isn't like that to objects. But even if it responds low and the same to cars and chairs, it might still have information to enable you to distinguish cars from chairs if the pattern of response across boxes in that region was stably different for cars and chairs. OK? That's really key. We'll go over it at a few more points.
But that's essential, right? A lot of the details that I'm going to teach you that go by in class don't matter, but I really want you guys to understand the PPA. And that's the nub of it. OK?
So the idea is that selective-- his claim is that selective regions, like the face area, contain information about non-preferred stimuli. That is, like, non-faces for the face area, or non-places for the place area. And because they contain information, those regions don't care only about their preferred category. So why does Kanwisher get off saying the FFA is only about faces and the PPA is only about places if we can see information about other things in those regions? OK. That's a really important critique. That's why we're spending time on it. OK? OK.
Next, what kind of empirical data might be an answer to Haxby's challenge? I presented at least three different kinds of data that can address this and say, hey, wait a minute. You know, you have a point, but what kind of data could speak to that and respond to Haxby? We didn't actually talk about this explicitly in class, but think about it.
Here's the claim he makes. What might we say, right? So that's empirically true. Like, you look in the FFA. Even in my own data, I can distinguish chairs from shoes a little teeny bit in the FFA. OK? So that empirical claim is true.
Why might it nonetheless be the case that the face area is really only about face recognition? What other data have you heard in here that might make you think that? Yes, Ben.
AUDIENCE: Because it's the presence--
NANCY KANWISHER: Speak up.
AUDIENCE: Sorry. The presence of low-level stimuli that are generally in faces, but can also be sparse on chairs and cars in context.
NANCY KANWISHER: Absolutely. So yeah, put another way, even if you had a perfect coder for faces-- like take your best deep net for face recognition, VGG face-- it can distinguish chairs and shoes too, right? The features that you use to represent faces will slightly discriminate between other non-face objects. So the fact that we can see that information in itself isn't strong evidence that that region isn't selective for face perception. Absolutely. What else? Yeah, OK.
AUDIENCE: Like transcranial magnetic stimulation? When you stimulate the epithelial and you look at it face it affects it, but when you are like looking at other objects, the effect is no longer.
NANCY KANWISHER: Exactly. And so what does that tell you about-- OK, so there's pattern information in there about other things beyond faces. But? Apparently it's not used, right? Now with every bit of evidence, you can always argue back. People would say, well, TMS, those effects are tiny. Maybe there isn't and we didn't have power to detect it, blah, blah, blah, blah. But at least, absolutely you're right. TMS argues against them. What else?
Or at least is a way to argue against it-- and the Pitcher paper that I assigned and other papers that we've talked about in here provide some evidence that actually, at least the occipital face area really is only causally involved in face perception even if there's information in there about other things. What else? What other methods can address this? Yeah.
AUDIENCE: That is pretty new simulation, and even when you're pressing hand against your face, you can perceive faces in it.
NANCY KANWISHER: Exactly. Exactly. So these are both causal tests, right? OK, there's information in there. But is it causally used in behavior? TMS suggests not. The little bit of direct intracranial stimulation data that I showed you also suggests the causal effects when you stimulate that region are specific to face perception, again suggesting that even if there's pattern information in there, it's not doing anything important because we can mess it up and nothing happens to the perception of things that aren't faces. Absolutely. What else?
We talked about it very briefly a few weeks ago. Yeah.
AUDIENCE: So if you remove the [INAUDIBLE], it just completely makes a person incapable of perceiving faces. That is causing--
NANCY KANWISHER: Yes, but the crucial way-- yes, but the crucial way to address Haxby would be what further aspect of that? Yes. And by the way, we don't remove the area in humans, but occasionally, we find a human who had a lesion there due to a stroke and then we study them.
AUDIENCE: So they're still able to do other categories.
NANCY KANWISHER: Exactly. Exactly. So all three lines of evidence from studies of prosopagnosia, electrical stimulation directly on the brain, and TMS, all can provide evidence to various degrees. Again, one can quibble about each of these particular studies. But all of those suggests that even though there's information in the pattern, Haxby's right-- there's information in there about other things that aren't faces.
The only causal effects when you mess up with that region are on faces, not on other things. That suggests that pattern information is what they sometimes say in philosophical circles is "epiphenomenal." That is, it's just not related to behavior and perception. Make sense?
OK, moving along, how can we then use Haxby's method to not just engage in this little fight about the FFA and how specific it is, but to harness this method and ask other interesting questions from functional MRI data. How can we use it to find out, for example, does the place area discriminate, say, beach scenes from city scenes? We want to know what's represented in there. How could we use this method to find out?
Yes, Jimmy.
AUDIENCE: If I do what Haxby kind of did, and try the decoder, and see if the decoder could decide and differentiate between the city and or like an acre of shade.
NANCY KANWISHER: Exactly. Exactly. So we talked about decoding methods last time as a way to use machine learning to look at the pattern of response in a region of the brain, and train the decoder so it knows what the response looks like during viewing of beach scenes, train it so it knows what the response in that region looks like when you're looking at city scenes, and then take a new pattern, and say, is this more like the beach pattern or is it more like the city pattern? And that's how you could decode from that region. Yes.
AUDIENCE: That doesn't tell as much, in the sense that it's not telling you-- I mean, we know that there is residue of information nevertheless, and that this community can be varied on any region considered at any time, always.
NANCY KANWISHER: We have a true nihilist here. No, it's a good question. It's not the case that you can discriminate anything based on any region of the brain.
So there are some constraints. There are some things you can find in some places and other things you can find in other places. And they're not uniformly distributed over the brain.
However, the fact we just-- the point I just made about yes, there's discriminative information in the face area about non-faces but maybe it's not used, should raise a huge caveat about this whole method. How do we ever know? We see some discriminative information. How do we know whether it's actually used by the brain, part of the brain's own code for information, or just epiphenomenal garbage that's a byproduct of something else?
It's a really important question about all of pattern analysis. We do it anyway because we're beggars. We can't be choosers in terms of methods with human cognitive neuroscience. And we want to know desperately what's represented in each region.
So we do this. But whenever you see these lovely, "I can decode x from y," things, you should always be wondering. Who knows if that fact that you, the scientist, can decode it from that region means the brain itself is reading that information out of that region? Big, important question.
All right, put another way-- so Jimmy mentioned just decoding in general and that's absolutely right. But to directly harness the Haxby version of this, what would we do? First, we would functionally localize the PPA by scanning them, looking at scenes and objects, find that region in each subject.
Then we would collect the pattern of response across voxels in the PPA while subjects were looking at, say, beach scenes. And so if this is the PPA, this is the pattern of response across voxels in that region when they're looking at beach scenes-- fake data, obviously, just to give you the idea. So we would split the data in half, even runs, odd runs.
That would be like even runs. Then we get another pattern for odd runs. And then we get another pattern for when they're looking at city scenes with even runs, and another pattern when they're looking at city scenes in odd runs.
So then, once we have those four patterns, what is the key prediction if using Haxby's correlation method? What is the key prediction if the PPA, if pattern of response in the PPA, can discriminate beach scenes from city scenes? What should we see from these patterns?
What's the key prediction? Claire. Key prediction-- you have these four patterns in the PPA, and now you want to know is there information in there that enables you to discriminate beach scenes from city scenes?
AUDIENCE: Is that like beach even and beach odd are more similar than beach even and city even?
NANCY KANWISHER: Exactly. Exactly. Right. It sounds all complicated and it's easy to get confused. But the nub of the idea is really simple.
It just says, look, the beach patterns are stable. We do beach a few times, we get the same pattern, more or less. We do city, we get a different pattern. And we keep doing city, we get the same pattern more or less. And the beach pattern and the city pattern are different.
So that's the nub of the idea. And so you can implement it with decoding methods or the Haxby versions, just to ask whether the correlation between two beach patterns-- beach even, beach odd-- is more similar than the pattern between one of the beaches and one of the cities. Just asking, are they stably similar within a category and stably different from another category? Does that make sense?
This is just a variant of this thing I showed you guys before. We just harnessed this to ask whether that region can discriminate. OK, and I just said all of this.
If you still feel shaky on this, there's a few things you can do. A version of my little lecture on this method is here at my website. You can look at that. It's just like six minutes and it's basically what I did before. But if you want to go over it again, there it is.
You can reread the Haxby paper, which I know is not super easy, but it's actually nicely written. And if you read it carefully, it explains the method pretty clearly. You can talk to me or a TA. And we'll get back to this question of whether we should do a whole MATLAB based problem set on this.
OK, let's move on and talk about navigation. This is a Monarch butterfly. It weighs about half a gram.
And yet, each fall the Monarch migrates over 2,000 miles from the USA and Canada down to Mexico. In fact, a single Monarch flies 50 miles in a single day. It's pretty amazing for this tiny, little, beautiful delicate thing. Even more amazing-- it flies to a very specific forest in Mexico that's just a few acres in size. And it arrives at that particular forest.
Now, that's already amazing, but here's the part that is just totally mind blowing and that is-- and it flies back north in the spring-- and that is that this whole cycle takes four generations to complete. And that means that the Monarch that starts up in Canada and flies down to that forest in Mexico-- one Monarch does that-- is the great-great-grandkid of his ancestor that last went on that route. Put that in your head and smoke it. That's pretty amazing.
Consider the female loggerhead turtle. She hatches at a beach, and goes out in the sea, and swims around in the sea for 20 years before she comes back 20 years later for the first time to the beach that she hatched at. Now, it's pretty amazing, but some mothers miss by 20 miles. They go to the wrong island or the wrong beach on the same island.
And so you might think, OK, it's pretty good. It's not amazing. But here's the thing-- the wrong beach that those mothers go to is the exactly right beach had the Earth's magnetic field not shifted slightly over those 20 years. They're exactly precise, but they just don't compensate for the shift in the Earth's magnetic field.
Here's a bat. This bat maintains its sense of direction even while it flies 30 to 50 miles in a single night in the dark catching food. And it maintains its sense of direction even though it's flying around in all different orientations in three dimensions, and even as it flips over and lands to perch on the surface of a cave. It doesn't get confused by being upside down.
This is Cataglyphis, the Tunisian desert ant. These guys are amazing. They crawl around on the surface of the Tunisian desert where it's 140 degrees in the daytime, and they have to crawl around up there to forage for food. And then because it's so damn hot, as soon as they find food, they zoom back to their nest and go down in the nest where it's cooler.
So here is a track of Cataglyphis starting at point A and foraging. He's meandering around looking for food going along this whole crazy path to point B. And then if he finds food at point B, boom-- straight line back exactly to the nest.
Now we might ask, how does Cataglyphis keep track as he's doing all this stuff of where his heading is back to his nest? The first thing you might think of is things like what it looks like. Maybe there are landmarks, maybe there are odors.
But no, he doesn't use any of those things. And we know that because when scientists who have set up this measurement device capture Cataglyphis after he goes out on this tortuous path and finds the feeding station, they capture him and move them across the desert-- on which they've drawn all these grid lines for the convenience of their experiment-- and they release them here. And what does Cataglyphis do? He goes on the exactly correct vector-- no landmarks, no relevant odors, and yet he's obviously encoded the exact vector of how to get home. Think about what that entails and what's involved.
AUDIENCE: The same vector with respect to north?
NANCY KANWISHER: With respect to-- yes, well, with respect to absolute external direction, absolutely. So that's what I just said. So these feats of animal navigation are amazing.
And animals have evolved ways to solve all these problems unique to their environment. They've evolved these abilities because they really have to be able to find food, and mates, and shelter. And this is not just esoterica in the natural world. MIT students, too, need to be able to find food, and mates, and shelter.
So what is navigation, anyway? And what does it entail? Well, I'll argue over the next two lectures that there are two fundamental questions that organisms need to solve to be able to navigate. First one is, where am I? And the second one is, how do I get from here to there, A to B, wherever there is that you need to get?
So we'll unpack this. There are many different facets of each. But so for example, if you see this image, you immediately know where you are, and you also know where to go if, for example, it starts raining. You might rush into lobby 7, or if you're hungry, you might turn around and go back to the Student Center.
Same deal here-- if you see this, then you know where you are and where you would go to get to various things. Now, these judgments rely on the specific knowledge you guys have of those particular places. You recognize that exact place, and you have some kind of map in your head that we'll talk more about in a moment, that tells you where everything else is with respect to it.
But even if you're in a place you don't know at all you can still extract some information. So suppose you miraculously found yourself-- boom-- here. I wouldn't mind, actually, but that's not in the cards for a while. So you're here.
Even if you've just hiked around the corner, if you've never seen this place before, you have some kind of idea of what sort of place this is. Where would you pitch your tent? Where might you try to go to get out of this valley?
If it was me, I wouldn't. I have friends who would go straight up there and try to drag me along, complaining. If it was me, I'd rather look for some other route. But you can tell all of that just by looking at this image-- where you can go from there, not just what kind of a place it is, but what are the possible routes you might take.
So these fundamental problems that we solve in navigation, of knowing where am I and how do I get from here to there, include multiple components. In terms of where am I, the first piece is recognizing a specific place you know. So you might open your eyes and say, OK, this is my living room. I know this particular place.
But as I just pointed out, even if the place is unfamiliar, we can get a sense of what kind of place this is. Am I in an urban environment, a natural environment, a living room, a bathroom? Where am I?
A third aspect of where am I, a third way that we might answer that question, is something about the geometry of the environment we're in. So try this right now-- close your eyes. OK, now think about how far the wall is in front of you. Don't open your eyes, just think about how far away it is, how far away the left wall is and the right wall is.
And how about the wall behind you? Don't open your eyes. How far back is the wall behind you from where you are right now?
OK, you can open your eyes. It's not rocket science. I just wanted you to intuit that even though you're presumably riveted by this lecture, and thinking only about navigation, you sort have a kind of situational awareness of the spatial layout of the space you're in.
So you might have a sense of I'm in a space like this and I'm over here in it. And we'll talk more about that exact kind of awareness of your position relative to the spatial layout of your immediate environment. It's something that's very important in navigation.
And another part of that is you might think, how would I get out of here? If I'm seriously bored by the lecture or for any other reason I urgently need to get out of here, you probably know exactly where the doors are in this space. It's just part one of those things that we keep track of.
So those are aspects of where am I in this place. What are the things we need to know to know how we would get from here to someplace else? Well, the simplest way to navigate to another location another goal is called "beaconing."
And this is a case where you can directly see or hear your target location. So you're sailing in the fog. You can't see a damn thing, but you hear the foghorn over there, and you know you're sailing to that point. So you just go toward the sound, nice and simple. You don't need any broader map of anything else. You just hear it and head toward it.
Or if you see this, and your goal is to get to the green building, well, there's a green building and you just head that way. Now, you're going to have to go around a little bit to get around those obstacles, but you know where to head because you can see your target directly. These are cases where you don't need a broader, long-term knowledge of the whole environment.
If you can see your target, you just go straight for it. So that's beaconing, simplest kind of A to B. And it requires no mental map, no kind of internal model of the whole world you're navigating in. But if you can't see the place you want to go, then you need some kind of mental map of the world.
So what do we mean by a "mental map of the world?" Well, this idea was first articulated in a classic experiment way back in the 1940s. So this was actually one of the original experiments that launched the cognitive revolution, when we emerged from the scourge of behaviorism to realize that it was actually OK, and indeed, of the essence, to talk about what's going on in the mind.
And a really influential study that launched the cognitive revolution by Tolman was done on rats. And it went like this-- he trained rats. He put them down in this area, and they had to learn that there would be food out there at the goal.
And so they just have to make the series of left and right turns to find the food. So you train them on that for a while till they're really good at it. And then he put the rats in this environment.
Now, the environment is similar, except there's multiple paths, one that seems analogous to the old route. So what are the rats do in this situation? They run down here, they run into a wall, and they realize, OK, that's not going to work. No surprises yet.
But then, the rats immediately come back out and they go straight out that way. What does that tell you? What did they learn? Did they learn a series of go straight, and then left, and then right, and then right, and then go for a long ways?
No. That wouldn't work over here. They learned something much more interesting. Even though they were only being trained on this task here, they learned some much more interesting thing about the kind of vector average of all of those turns. Everybody get this? It's really simple but really deep.
So from this, Tolman and others started talking about cognitive maps, whatever it is you have to have learned in a situation like this so you can abstract the general direction. We don't just learn specific routes as a series of stimulus and responses. So there must be some kind of map in your head to be able to do this, and rats have that, and so do you.
So let's consider this question right now. Where am I? Where are you? To answer that question to yourself, there's something like this in your head.
And it probably doesn't look exactly like that in your head, but there's some version of this information that's in your head that you're using when you answer the question of where you are. And you have some way to say in that map of the world, I know not just what the MIT campus looks like and how it's arranged, but I know where I am in it.
Now, if you want to know how to get somewhere else-- like suppose you're hungry and you want to go over to the Stata Cafeteria over there. What else do you need to know besides knowledge of the map of your environment and where you are in it? What else do you need to know?
You know you have this map, you know where you are, and you know where your goal is. Now you have to plan how to get over there. What else do you need to know? Yeah.
AUDIENCE: You have to know which parts are paths and which parts are buildings.
NANCY KANWISHER: Yes, exactly-- where can you go in there? Actually, where can you physically get through? Actually, our vector is right over there, but you can't go that way because you can't go through that glass, even though you can see through it. So knowledge of physical barriers, and what's an actual path and what isn't is crucial.
What else do you need to know? Suppose we had a robot in this room, sitting right here facing the front of a room like you guys, and we're programming the robot on how to get over there. What are other things we'd have to tell that robot to get it to plan how to get over to the Stata Cafeteria? Yeah.
AUDIENCE: Things to watch out for, like cars and traffic.
NANCY KANWISHER: Absolutely. We'd have to know about obstacles, like moving obstacles, not just fixed ones. Absolutely. What else? Yeah.
AUDIENCE: Initial orientation.
NANCY KANWISHER: Yes. He has to know which way he's headed. You're going to give this robot instructions on which way to go. It matters a whole lot if the robot is starting like this or starting like that. The instructions are different in the two cases, and likewise for you guys. To plan a route, you need to know which way you're heading.
If you guys have ever been in Manhattan, and you come up from the subway, and you see the street's going like this, and you know it's north/south, and you don't know if you're heading south or north-- really common thing. It's not enough to know I'm at the junction of Fifth and 22nd. You need to know, am I facing south or north?
Otherwise you can't figure out which way to go. That's called "heading direction." We just did all that. You need to know your current heading. You also need to know the direction of your goal in order to plan a route to it.
So in this kind of taxonomy of all the things you need to know to navigate, we've just added that if you're going to navigate in your own environment, you need to know not just where you are in it, but which way you are facing in that mental map. And we also talked about this business of what routes are possible from here, how do we move around obstacles, where are the doors, where are the hazards like cars, et cetera.
A final thing you need to know is that even if you have a good system for all of these other bits, it's still possible to get lost in all kinds of ways. You lose track, you get confused, you get lost. So we also need a way to reorient ourselves when we're lost. And we'll talk a lot about that in the next lecture.
So this is just common sense. We're doing a kind of low-tech version of Marr computational theory for navigation. What are the things that we would need to know or that a robot would need to know to be able to navigate? Just thinking about the nature of the problem. So that's what we need.
What's the neural basis of all of this? So I'm going to start right in with the parahippocampal place area, not to imply it is the total neural basis of this whole thing. It's just one little piece of a much bigger puzzle. But we'll start in there because it's nice and concrete.
All right, so this story starts about 20 years ago. I think I mentioned some of this in the first class when I talked about the story of Bob and I talked about Russell Epstein, who was then my post-doc. And he was doing nice behavioral experiments, and thought it was trashy and cheap to mess around with brain imaging.
And he was going to have none of it until I said, Russell, just do one experiment. Scan subjects looking at scenes. I know it's kind of stupid, but just do it. Then you'll have a slide for your job talk.
And he scanned subjects looking at scenes and looking at objects. And here is one of those early subjects, probably me-- I don't remember-- with a bunch of vertical slices through the brain, near the back of the brain down there, moving forward as we go up to here. Everybody oriented?
Sorry, it's not showing up very well in this lighting, but there's a little bilateral region right in the middle there that shows a stronger response when people look at pictures of scenes than when they look at pictures of objects. So we hadn't predicted this. Yeah.
AUDIENCE: Is the pink the eye color?
NANCY KANWISHER: Yeah. Yeah, pink is-- all the colors are-- there's significance maps or P levels, right. So pink is higher than blue, but blue is borderline significant. So this is kind of dopey.
We didn't actually predict it for any deep reason. We hadn't been thinking about theories of navigation or anything like that. It was just one of those dumb experiments where we found something and we followed the data.
So we found this, and it's, like, OK, let's try some other subjects. So here are the first nine subjects we scanned. Every single subject had that kind of signature response in exactly the same place, in a part of the brain called "parahippocampal cortex."
So this is very systematic. And there's lots of ways to make progress in science. One way is to have a big theory, and use it to motivate brilliant, elegantly designed experiments. And another is you just see something salient and robust that you didn't predict, and you follow your nose, and try to figure it out.
So that's what we did in this case. It's like, OK, what the hell is that? So if you think about-- we eventually called it the "parahippocampal place area" after a little more work. If you think about what we have so far, we've scanned people looking at pictures like this and pictures like that.
And what we've shown is that little patch of brain responds a bunch more to these than those. So my first question is, is that a minimal pair? Tally, is that a minimal pair?
AUDIENCE: Sorry, I'm about my voice.
NANCY KANWISHER: Sorry. Simple, simple. We're contrasting this with that.
AUDIENCE: Can you remind me what a minimal pair is?
NANCY KANWISHER: OK, minimal pair is this thing we aspire towards an experimental design, where we have two conditions that are identical except for one little thing we're manipulating.
AUDIENCE: I don't really think it's a minimal pair, but I'm not really sure.
NANCY KANWISHER: Well, I even told you what we were designing to manipulate, but--
AUDIENCE: There seems to be too many differences between a living room and--
NANCY KANWISHER: It's ludicrous. I mean, it's a million differences here. So we don't know that we have anything yet. There's all kinds of uninteresting accounts of this systematic activation in that part of the brain.
So just to list a few that you've probably already noticed-- these things have rich, high-level meaning and complexity. So you can think about living rooms, or where you might sit, or somebody's aesthetic, home design, or there's all kinds of stuff to think about there, much more than just, OK, it's a blender. So there's just complexity in every possible way.
There are also lots of objects present here, and only a single object over there. So maybe that region just represents objects, and if you have more objects, you get a higher signal. There's another possibility, and that is that these images depict spatial layout, and that one does not.
So you have some sense of the walls, and the floor, and the layout of the local environment here that you don't have over there. And we could probably list a million other things It's a very, very sloppy contrast.
So how are we going to ask which of these things might be driving the response of that region? Well, a natural thing to do is just deconstruct the stimuli. So here's what we did-- this is actually way back 20 years ago.
There were better methods at the time, but I didn't know them, so I actually drove around Cambridge, photographed my friends' apartments, left the camera on the same tripod, moved all the furniture out of the way, and photographed the space again. Ha, ha. I know. And then these will be probably cut out with some horrific version of Adobe Photoshop that existed 20 years ago.
Anyway, we deconstructed the scenes into their component objects and the bare spatial layout. Everybody get the logic here? Just to try to make a big cut in this hypothesis space of what might be driving that region.
So what do we predict that the PPA will-- how strongly will it respond? Oops, how strongly will it respond if these two things are true? If it's the complexity or multiplicity of objects that's driving it, what do you predict we will see over there? We already know you get a high response here. What will we get over there? Yeah.
AUDIENCE: Probably get more biases to the furniture.
NANCY KANWISHER: Yeah, we'll respond more to this than that. Right. It's really simple-minded. If instead, it responds more to the spatial layout, what do we predict? Isabel.
AUDIENCE: It's going to respond to the empty rooms more.
NANCY KANWISHER: Yeah. And that seems like a weird hypothesis because these are really boring, this kind of nothing going on here. And there's just lots of stuff going on here. I mean, it's not riveting, but it's a whole bunch, whole lot more interesting to look at these than those.
Believe me, I got scanned for hours and hours looking at these things. And whenever the empty rooms came on, I was, like, oh, my god, I'm just so bored. There's just nothing here, whereas here at least there's stuff.
But that's not what the PPA thinks. What the PPA does-- oops, we just did the localizer-- it responds like this. This is percent signal change, a measure of magnitude of response, to the full scenes, way down, less than half the response to all those objects, and almost the same response as the original scene when all you have is a bare spatial layout.
Pretty surprising, isn't it? We were blown away. We were, like, what? What?
But can you see how even this really simple-minded experiment enables us to just pretty much rule out that whole space of hypotheses? It's not about the richness, or interest, or multiplicity of objects. It's something much more like spatial layout because that's kind of all there is in those empty rooms.
I mean, it could be something like the texture of wood floors or something weird like that. But one's first guess is it's something about spatial layout. Does this make sense? It's just a way to take a big, sloppy contrast, and try to formulate initial hypotheses, and knock out a whole big space of hypotheses. Yes. Is it Alana?
AUDIENCE: Yeah, I'm sorry, I might have missed the design. So people who are looking at the empty room would not have the furniture?
NANCY KANWISHER: Good question. I skipped over all of that. We did-- yes, that's true. We did mush them all together and one could worry about that, that when you see this, you remember that that's a version of this. Absolutely. Absolutely.
And so maybe-- yes, nonetheless, if what you were doing-- that's absolutely true, but if what you were doing here is kind of mentally recalling this, then why couldn't you also do that here? Maybe you could. You might argue that this is more evocative of that than this is, but it's also got lots of relevant information. Yeah, Jimmy.
AUDIENCE: For the furniture, did you guys try placing them in the exact position as the scene and seeing if that--
NANCY KANWISHER: We did both versions for exactly the reasons you guys are pointing out. And it didn't make a difference. Yeah. Sorry, Cooley.
AUDIENCE: It'd be-- you would transfer if they were just responding to the things, like more stuff? Like in the empty room, there's more background, but there's still more background.
NANCY KANWISHER: Totally. You're absolutely right. This is taking us pretty far, but it's still pretty sloppy. This stuff goes all the way up to the edge of the frame, and here there's lots of empty space.
Is that what you're getting at? Absolutely. I took out those slides because I felt I didn't want to spend the entire lecture doing millions of controlled conditions on the PPA. I thought you'd get bored.
But actually, another version that we did was we then took all of these conditions, and we chopped them into little bits and rearranged the bits, so that you have much more coverage of stuff in the chopped-up scenes than the chopped-up objects. And in the chopped-up versions, it doesn't respond differently at all. So it's not the amount of total spatial coverage. It's the actual-- something more like the depiction of space. Was there a question over there? Yeah.
AUDIENCE: I was wondering if there would be any difference between looking at images as 2D or 3D scene, and actually being there to see the 3D inside of the scene.
NANCY KANWISHER: Totally. Totally. It's a real challenge. With navigation, navigation is very much about being there and moving around in the space.
And this is just a pretty rudimentary thing where you're lying in the scanner, and these images are just flashing, flashing on, and you're doing some simple task, like pressing a button when consecutive images are identical. It's not moving around in the real world. You don't think you're actually there.
But here's where video games and VR come in because actually, they produce a pretty powerful simulation of knowing your environment, feeling you're in a place in it. And so lots of studies have used those methods to give something closer to the actual experience of navigation.
So where are we so far? We've said the PPA seems to be involved in recognizing a particular scene. So this just says it responds to scenes and something about spatial layout, maybe. Does it care about that particular scene or do you have to recognize that particular scene to be able to use the information?
Now, our subjects mostly didn't know those particular scenes. But we wanted to do a tighter contrast asking if knowledge of the particular scene matters. So what we did was we took a bunch of pictures around the MIT campus, and we took a bunch of pictures around the Tufts campus. And we scanned MIT students looking at MIT pictures versus Tufts pictures. And then what else do we do?
AUDIENCE: Get the Tufts students.
NANCY KANWISHER: Yeah, why?
AUDIENCE: Oh, just to make sure that it's not all about that weird architecture of the set.
NANCY KANWISHER: Exactly. Exactly. So this is called-- yes, whose weird architecture? I think ours is weirder. So it's not just about the particular scenes or the particular subjects.
So everybody get how with that counterbalanced design, you can really pull out the essence of familiarity itself, unconfounded from the particular images? So when we did that, we found a very similar response magnitude in the PPA for the Tufts students, for the familiar and unfamiliar scenes. Really didn't make much difference. Yeah.
AUDIENCE: Taking a step back, so we started off with the one question of navigation and it involving all these different components. I just want to place this--
NANCY KANWISHER: We're getting there. We're getting there. There won't be like a perfect answer. We're not going to end up with that slide, with the exact brain region of each of those things. We'll get some gisty, vague senses of what this is.
OK, so this tells us it's not about-- whatever the PPA is responding to in a scene, it's not something that hinges on knowing that exact scene. So it can't be something like, if I was here and I wanted to get coffee, what would my route from this location be, given my knowledge of the environment. Because otherwise, we wouldn't get this result.
So whatever it is, it's something more immediate and perceptual to do with just seeing this place. So where are we? We've said that there's this region that responds more to scenes than objects, that when all the objects are removed from the scenes, the response barely drops. And its response is pretty much the same for familiar and unfamiliar scenes.
So all of that suggests that it's involved in something like perceiving the shape of space around you. Doesn't nail it yet, but it kind of pushes you towards that hypothesis. Yeah, was there a question here a second ago? No? OK.
AUDIENCE: I was talking about experiment, but is it accurate when you look at a map?
NANCY KANWISHER: Oh, great question. Not very much. Yeah, if you take pictures of places from above versus this kind of view, you get a response in this kind of view, but not above. Yeah, very telling.
OK, so I'm going to skip. We're not going to do the 30 other experiments. We're going to skip to the general picture, that here's the PPA in four subjects in this very stereotyped location.
And here are some of the many conditions we've tested. It's not just abstract maps like this. They don't produce a strong response.
Oh, this is an answer to Cooley's question way back. Here's the scrambled-up scene-- much lower response. So it's not just coverage of visual junk. And it responds pretty strongly to scenes made out of LEGOs compared to objects made out of LEGOs, and various other silly things.
So all of that seems to suggest that it's processing something like the shape or geometry of space around you-- visible space in your immediate environment. Nonetheless, there's always pushback. And there's pushback on multiple fronts, and there should be. That's proper science.
So one of the lines of pushback was this paper by Nasr, et al. that I didn't assign. I assigned you the response to it. Anyway, what Nasr et al. Did was scan people looking at rectilinear things like cubes and pyramids versus curvilinear, round-y things like cones, and spheres. And what they showed is the PPA responds more to the rectilinear than the curvilinear shapes.
OK, that's the first thing. And so then, they argue that in general, scenes have more rectilinear structure than curvilinear structure. And they did a bunch of math to make that case. And so they argue that maybe the apparent scene selectivity of the PPA is due to a what of scenes with rectilinearity? Yeah.
AUDIENCE: Confound.
NANCY KANWISHER: Yes, exactly, a confound. This is exactly what a confound is-- something else that covaries with the manipulation you care about that gives you an alternative account, namely it's not scene selectivity. It's just rectilinearity. I mean, that might be interesting to other people, but it would make it not very relevant to navigation and much less interesting to me, at least.
So that's an important criticism. And so then the Bryan et al. Paper that you guys read starts from there and says, let's take that seriously. Let's find out.
And so you guys should have read all of this, but just to remind you, they have a nice, little 2 by 2 design-- remember we talked about 2 by 2 designs-- where they manipulate whether the image has a lot of rectilinear structure or less rectilinear structure, and whether the image is a place or a face. And what they find in the PPA is the same response to these. And it's higher to the scenes than the faces, and rectilinearity didn't matter for the scenes.
So evidently, even though it does matter with these abstract shapes, in actual scenes and faces, it doesn't seem to be doing much. It's not accounting for this difference. Everybody get that?
OK, let's talk about this graph. Are there main effects or interactions here? And what are those main effects or interactions? Yes, Cooley.
AUDIENCE: There's many different scenes.
NANCY KANWISHER: Yeah, of category, scene versus face. Anything else?
AUDIENCE: What's the first one? What was the first thing? In PPA category, what's the subtype?
NANCY KANWISHER: Oh, wait, this here-- these are scenes and those are faces. I'm sorry, and this is the code here. These are rectilinear versus curvilinear. Just one main effect, or is there an interaction, or another main effect?
Just one main effect. These guys are higher than those guys. That's it. So that just tells you there's nothing else going on in these data other than scene selectivity.
Rectilinearity doesn't interact with or modify scene selectivity, and it doesn't have a separate effect. Nonetheless, as we've been arguing with all the whole Haxby rigmarole, does the fact that there's no main effect of rectal linearity in here mean that the PPA doesn't have information about rectilinearity? No, Josh, why?
AUDIENCE: This little, tiny moment that could be-- you know, this is not the right experiment to--
NANCY KANWISHER: That's right. This is a big-- well, it's the right experiment, but not the right analysis. It's the big, average responses are the same, but maybe the patterns are different. That wouldn't directly engage with this, but we wanted to know, was there information in there about rectilinearity.
So how would we find out? So this was your assignment, and I think most people got it right. But in case anybody missed it, we were zooming in on this Figure 4 here. So again, this is just the same basic design of experiment two.
And now, let's consider what's going on here. So you guys read the paper and you understood what was going on here. What's represented in that cell right there?
What is the point of this diagram? What are they doing here? And what does that cell mean in that matrix? You can't understand the paper without knowing that. Is it Ali? No, sorry. What's your name?
AUDIENCE: Sheldon.
NANCY KANWISHER: Sheldon, I've only asked you six times. Yeah, go ahead.
AUDIENCE: So they want to see whether the activation patterns can better discriminate between rectilinearity of the same category of things or between categories of things with the same rectilinearity. So the first thing I said is to the left and the second one is to the right. And they--
NANCY KANWISHER: Sorry, wait, here and here? No.
AUDIENCE: Right side. Yeah, so that part is discriminating between rectilinearity, and that side is discriminating between categories. And they take the differences of-- well, not the differences, they take how well it can distinguish between each of those categories and plot them down there.
NANCY KANWISHER: Right, OK. That's exactly right. So this is how well it can discriminate plotted down here, but based on an analysis that follows this scheme. So what does that cell in there represent, that dark green cell? What is the number that's going to be calculated from the data corresponding to that cell?
AUDIENCE: Similar piece of same rectilinearity and same pattern.
NANCY KANWISHER: Exactly. Exactly. So just as if you want to distinguish chairs from cars or something else, if you want to know is there information about rectilinearity in there, you take these two cases which are the same in rectilinearity-- both high rectilinear, both low rectilinear for run one and run two-- and that's the correlation between run one and run two for those cells.
That's the within rectilinearity case. And if there's information about rectilinearity, the prediction is those within correlations are higher than the between correlations, just as we argued a bit back with beaches and cities and everything else-- same argument. This is just presenting the data in terms of run one and run two, and which cells do we grab to do this computation.
So each of the cells in there-- for each of the cells, we're going to calculate an r value of how similar those patterns are. A pattern for rectilinear scenes in run two, a pattern for rectilinear scenes in run one-- this cell is a correlation between those two patterns. How stable is that pattern across repeated measures?
All right, so that's what that r value is. The two darker blue squares here are the r values for stimuli that differ in rectilinearity. And remember that the essence of the Haxby-style pattern analysis is to see if the within correlations are higher than the between correlations. In this case, the within correlations are within rectilinearity versus between rectilinearity.
And so then they calculate all those correlation differences and they plot them as discrimination abilities. And so what this is showing us here is that actually, the PPA doesn't have any information in its pattern of response about the rectilinearity of the scene. However, if we take the same data, and now choose within category versus between category, ignoring rectilinearity, and we get the same kind of selectivity correlation difference within versus between for category, there's heaps of information about category.
Does that make sense? Again, if you're fuzzy about this, look back on that slide. I have lots of suggestions for how to unfuzzy yourself on it.
So interim summary-- PPA responds more to scenes than objects. It seems to like spatial layout in particular. It does respond more to boxes than circles, but that rectilinearity bias can't account for scene selectivity.
That's all very nice, but what is a whole other kind of fundamental question we haven't yet asked about the PPA? So we've been messing around with functional MRI, measuring magnitudes of response, trying to test these kind of vague or general hypotheses about what it might be responding to. Yes.
AUDIENCE: Causation.
NANCY KANWISHER: Yes, what particular causation?
AUDIENCE: I guess like how the scenes, like with how the PPA with what role it plays in the person being seen.
NANCY KANWISHER: Exactly. Exactly. Again, we can test the causal role of a stimulus on the PPA, all of the stuff I talked about did that. Manipulate the stimulus, find different PPA responses. But what we haven't done yet is ask, what is the causal relationship, if any, between activity and the PPA and perception of scenes or navigation?
So far, this is all just suggestive. We have no causal evidence for its role in navigation or perception. All right, so let's get some. I'll show you a few examples.
So one, as, you guys have learned by now is these rare cases where there's direct electrical stimulation of a region, and there's one patient in whom this is reported. This patient again, is being mapped out before neurosurgery.
They did functional MRI in the patient first. This is his functional MRI response to, I think, houses versus objects. Houses are not as strong an activator as scenes for the PPA, but they're pretty good. PPA responds much more to houses than other objects.
And so that's a nice activation map showing the PPA. And those little circles are where the electrodes are, little, black circles. So they know they're in the PPA because they did functional MRI first to localize that region. Now those electrodes are sitting there.
And so first thing we do is record-- or first thing they did-- is record responses. They flash up a bunch of different kind of images, and they measure the response in those electrodes. And so what you see is in those electrodes right over there, 1, 2, 3, that correspond to the PPA, you see a higher response to house images than to any of the other images.
And you see the time course here over a few seconds. Everybody clear? This is not causal evidence yet. It's just amazing, direct intracranial recordings from the PPA-- I think the only time this was ever done, because it's pretty rare to have the electrodes right there in a patient who's willing to look at your silly pictures, and all of that.
But now, what happens when they stimulate there? So let's look at what happens when they stimulate on these sites 4 and 3 that are off to the side of the scene selectivity. And this is just a dialogue. We don't have a video, unfortunately. The videos are more fun, but this is just a dialogue between the neurologist and the patient.
And the neurologist electrically stimulates that region and says, did you see anything there? Patient says, I don't know. I started feeling something. I don't know, it's probably just me. No, it's not you.
And then they stimulate again. Anything there? No. Anything here? No. So that's right next to the side of the scene selective electrodes, right next door, a few millimeters away.
Then, they move their stimulator over here. They don't move anything, they just control where they're going to stimulate. Patient, of course, has no idea.
Neurologist says, "Anything here? Do you see anything, feel anything?" Patient says, "Yeah, I feel like--" he looks perplexed, puts hand to forehead-- "I feel like I saw some other site. We were at the train station."
Neurologist cleverly says, "So it feels like you're at a train station?" Patient says, "Yeah, outside the train station." Neurologist-- "Let me know if you get any sensation like that again."
Stimulates. "Do you feel anything here?" "No." And then he does it again. Did you see the train station or did it feel like you were at the train station? Patient, "I saw it."
These are very sparse, precious data, but that's so telling. It's not that he knew he was at the train station abstractly. He saw it.
So then, they stimulate again, right on those scene-selective regions. Patient says again, "I saw almost like, I don't know, like I saw-- it was very brief." Neurologist says, "I'm going to show it to you one more time." Really what he means is, I'm going to stimulate you in the same place one more time.
"See if you can describe it any further. And to give you one last time, what do you think?" "I don't really know what to make of it, but I saw, like, another staircase.
The rest I couldn't make out, but I saw a closet space, but not this one." He points to a closet door in the room. "That one was stuffed and it was blue."
"Have you seen it before," neurologist, "Have you seen it before at some point in your life, you think?" "Yeah, I mean when I saw the train station." "Train station you've been at?" "Yeah." Et cetera, et cetera.
So it's not a lot of data. But it's very compelling. What is the patient describing? Places he's in that he sees, and then he describes this closet space and its colors. Interestingly, colored regions are right next to scene regions, so that's kind of cool, too.
So it's causal evidence. It's sparse. Ideally, we'd like more in science, but it's pretty cool. Yeah.
AUDIENCE: At this point, the patient is just staring at a blank wall?
NANCY KANWISHER: I actually forget in the paper. I've got to go look that up. I forget exactly what the patient was doing, whether-- I think he's just in the room looking out. Usually, they don't control it that much because it's done for clinical reasons, and the patient is in their hospital bed, and they're just stimulating.
So he's probably just looking out at the space he's in. In fact, he must have been because at one point, he says, "The closet, not like that one over there." So if he was staring at a blank thing, he was also looking out at his room. So yeah.
AUDIENCE: This may be a little bit off topic. You said that the region for color perception is very close to this, it seems like. Is there any relationship between functional proximity and--
NANCY KANWISHER: That's a great question. Nobody in the field has an answer to this. People often make hay about the proximity of two regions, like there's some deep link because this thing is next to that thing.
The body selective region is right next to, and in fact slightly overlapping with, area MT that responds to motion. It's like, bodies move. Well, faces move and cars move, too. So I don't know. It's tantalizing.
It feels like it ought to mean something. And people often talk as if it does. And maybe it does, but nobody's really put their finger on what exactly it would mean. But it's useful.
So when Rosa Lafer-Sousa who you met in the color demo, and I showed that in humans, you get face, color, and place regions right next to each other in that order, that was really cool because Rosa had previously shown that in monkeys, the monkey brain it goes face, color, place in exactly the same order. And so we thought that's really interesting. That suggests common inheritance because that's so weird and arbitrary. Why would it be the same? So it can be useful in ways like that, at least.
So we just went through all of this. So how does this go beyond what we knew from functional MRI? I'm insulting your intelligence. You know the answer to this. It goes beyond it because it tells you-- it implies that there's a causal role of that region in place perception, some aspect of seeing a place.
Now, all of this about the PPA I just started in there because it's nice, and concrete, and easy to think about. But no complex mental process happens in just one brain region. Nothing is ever like that.
And likewise, scene perception and navigation is part of a much broader set of regions. So if you do a contrast, scan people looking at scenes versus objects, you see not just the PPA in here. Again, this is a folded-up brain, and this is the mathematically unfolded version so you can see the whole cortex. Dark bits are the bits that used to be inside a sulcus until it was mathematically unfolded.
So there's the PPA kind of hiding up in that sulcus. And when you unfold it, you see this nice, big, huge region. But you also see all these other regions.
Now there's a bunch of terminology and don't panic. I don't think you should memorize everything about each region. You should know that there's multiple scene regions.
You should know some of the kinds of ways you tease apart the functions, and some of the functions that have been tested, and how they're tested. But you don't need to memorize every last detail. Because it's going to get a little hairy.
So here's a second scene region right there called retrosplenial cortex or RSC. And actually, Russell Epstein and I saw that activation in the very first experiments we did in the 1990s, but we really didn't know what we were doing back then. And we knew that this is right near the calcarine sulcus.
Remind me, what happens in the calcarine sulcus? What functional region lives in the calcarine sulcus? It's just a weird, little fact, but it's kind of an important one that we mentioned weeks ago.
V1, primary visual cortex-- that's where primary visual cortex lives. And remember, primary visual cortex has a map of retinotopic space, with next door bits of primary visual cortex responding to next door bits of space. And in fact, that map has the center of gaze out here and the periphery out there.
So when Russell and I first saw that activation, we had the same worry that Cooley mentioned a while back. And that is the scenes are sticking out. There's stuff everywhere. The objects, there isn't that much sticking out.
And we thought, oh, that's just peripheral retinotopic cortex. But it's not. It's right next to there and it's a totally different thing.
And it turns out to be extremely interesting. You don't need to know all that. It's just silly, little history.
There's a third region up there that's on the outer surface out there that used to be called TOS and is now called OPA. I'm sorry about that. You don't need to remember this. Know that there are at least three regions.
But TOS slash OPA is interesting because there's a method we can apply to it that we can't apply to the others. What would that method be?
AUDIENCE: TMS.
NANCY KANWISHER: Yeah, TMS-- it's right out on the surface. You just stick the coil there and go "zap." So of course, we've done a lot of that. Can't get the coil into the PPA or RSC. It's too medial.
And there's another region that we'll talk about more next time called the hippocampus. You saw the hippocampus when Ann Graybiel spent all that time digging in the temporal lobe to find that bumpy, little, dentate gyrus, approximately right in there. And so all of these-- and probably other regions, but these are the core elements of the scene selective regions that are implicated in different aspects of navigation.
So when you have multiple regions that seem to be part of a system, that's an opportunity. Because now we have the possibility that maybe we could figure out different functions for different regions. And then maybe that would really tell us more than just scenes and navigation, end of story. It's kind of rudimentary.
It would be nice if different aspects of the navigation story engage different parts of the system. So really what we want to know is, how does each of these regions help us navigate and see scenes. And I'm not going to answer that fully. The field is still trying to understand all of this, but I'll give you a few tantalizing little snippets.
So let's take retrosplenial cortex right here. So this is first the response of the PPA right there, and retrosplenial cortex, which is just behind it. This is just its mean response to a bunch of different kinds of stimuli, showing you that it likes landscapes and cityscapes, scenes, more than a bunch of other categories of objects. And that's true of both the PPA and RSC. No surprises here-- they're both somewhat scene selective.
But then in a whole bunch of other studies summarized in this graph here, Russell Epstein and his colleagues had subjects engage in different tasks while they were looking at scenes. In some tasks, they had to say where they were. He's at UPenn, and he showed his subjects pictures of the UPenn campus. And they had to answer all kinds of questions about what part of campus they were, where they were on campus, and also about which way they were facing given the view of the campus they were looking at.
Then he also showed people familiar scenes and unfamiliar scenes, much like we did with our Tufts study. And he had object controls. And you can see the PPA doesn't care about any of that, doesn't care, really, if they're familiar or unfamiliar, doesn't care what task you're doing on the scene. You're looking at a scene, it's just going.
So we didn't really tease apart functions there. But RSC responds differently in these conditions. It's engaged in both the location task and the orientation task. It responds substantially more when you look at images of a familiar place than an unfamiliar place. So this is the first time we've seen that in the same network.
And so now, think about all the things you can do when you're looking at a picture of a scene and you know that place. You have memories of having been there. You can think about what you might do if you were there, how you would get from there to someplace else. All of those things are possible things that might be driving RSC.
Another thing that might be driving RSC is that if you're looking at a picture of a familiar place, you orient yourself with respect to the broader environment that that view is part of. So what I showed you that picture of the front of Stata, you immediately imagine, I'm out on Vassar Street facing that way, roughly northwest, I think. If you look at a picture of a scene and you don't know that scene, it doesn't tell you anything about your broader heading in the broader world. So all of those are things that the RSC, its function seems to depend on knowing that place.
Perhaps the most telling case comes from a patient who had damage in retrosplenial cortex. And the description in the paper of this says that this patient could recognize buildings and the landmarks, and therefore, understand where he was. So lots is intact-- can recognize scenes and know where he is.
But the landmarks he recognized did not provoke directional information about any other places with respect to those landmarks. So this person can look at a picture and say, yeah, I know that place. That's the front of my house. But then if you say, in which direction is a coffee shop two blocks away, he doesn't know which way it is from there.
So this should sound familiar. This is my guess of the bit that my friend Bob got messed up. Yeah. This is exactly his description-- he could recognize places, but it wouldn't tell him how to get from there to somewhere else.
And so the best current guess about retrosplenial cortex is that it's involved in anchoring where you are. You have this mental map of the world, and you have a scene, and you're trying to put them together. Given that I see this, where am I on the map, and which way am I heading in that map?
Again, think about the problem you face when you emerge from the subway in Manhattan. You look around. Where am I, and which way am I heading? That's what you need retrosplenial cortex for.
How about this TOS thing? There's lots of studies of it. I'll give you just one little offering.
So this is a causal investigation because as we discussed, the TOS is out on the lateral surface. So we can zap it. And so of course, we do.
And so in this study, we were asking whether TOS is involved in perceiving the structure of space around you. So we took scenes like this from CAD programs, and we just varied them slightly. So for example, the position of this wall moves around, the aspect ratio, the height of the ceiling moves around, and we make this subtle morph space of different versions of this image.
And then for control condition, we do the same with faces. We morph between this guy and that guy, and make a whole spectrum in between. And then in the task, what we do is here's one trial. One of the scenes or faces comes on briefly, and then shortly thereafter, you get a choice of two, and you have to say which of these matches that one.
And then what we do is we zap people right after we present this stimulus. And so the idea is this is as close as we can get to a pretty pure perceptual task. How well can you see the shape of that environment or the shape of that face?
You don't have to remember it for more than a few hundred milliseconds. So it's really more of a perception task than a memory task. And what we measure is, we actually muck with how different these two images are in each trial, and measure how far apart they have to be in morph space for you to be about 75% correct.
That's the standard psychophysical measure. The details don't matter. But our dependent measure is, how different do the stimuli have to be for you to discriminate them as a function of whether you're getting zapped in TOS or not.
And so here are the data. So let's take the case where you're doing the scene task here. What this threshold is, is again, how different the stimuli need to be for you to discriminate them. So the higher the bar, the worse performance. They have to be really different or you can't tell them apart.
And so what you see is when you zap OPA, that lateral scene selective region, discrimination threshold goes up a bit. That means you get worse at the discrimination. The stimuli need to be more different.
Compared to zapping the top of your head-- remember, you always want a control condition, and there's no perfect control condition because it feels differently to be zapped in different places. But getting zapped up here is a better than nothing control.
And then here's the occipital face area. That's the lateral face region we talked about before when I showed you another TMS study. Basically, whenever there's anything lateral, we zap it because we can. And see, it's not affected here. Zapping the occipital face area does not mess up your ability to discriminate the scenes.
However, in the face task, we see the opposite pattern. For the face task, zapping the occipital place area doesn't do anything compared to zapping the top of your head, but zapping the face area does. This is a double dissociation.
If we just had the scene task, it would be like, yeah, maybe. Who knows. Maybe, who knows why. But it's not very strong.
But when you have these opposite things, then we really have much more strong evidence that these two regions have different functions from each other. Everybody get that this is a double dissociation, in the same sense of when you have one patient with damage in one location and another patient with damage in another location and they have opposite patterns of deficit, then we're really in business. Then we can draw strong inferences.
So we just said all of that. So that's just a little snippet. These and other data suggest that that region is strongly active when you look at scenes, and it seems to be involved in something like perceiving-- just directly online perceiving the structure of the space in front of you.
So we already did retrosplenial cortex. And next time, we'll talk about the hippocampus in there, and its role in the whole navigation thing. Now, since I have ended early-- a rare event-- I actually put together a whole other piece of this lecture, and I thought, no, don't always have a part you don't get to.
But then it turns out we do get to it. We're going to go over this more later, but we're going to start with this business right here. So anybody have questions about this stuff so far?
OK, so I've spent a lot of time talking about multiple voxel pattern analysis, because it's the only method I've mentioned so far that enables us to go beyond the business of saying how strongly do the neurons fire in this region to the more interesting question of what information is contained in this region. But I also ended the last lecture with this kind of depressive note-- that you can't see much with MVPA applied to face patches, even when we know there's information in there with electrophysiology data.
Remember, I showed you that monkey study where they tried MVPA in the face patches in monkeys and they couldn't kind of read out a damn thing. And then they try MVPA on individual neural responses of the same region, and they can read out all kinds of information. And that tells you the information is there and we just can't always see it with MVPA.
Now today, you've seen cases where can see stuff with MVPA in the scene region. So sometimes it works, sometimes it doesn't. And when it doesn't work, we're left in this unsatisfying situation that we don't know if the information isn't there or if the neurons are just so scrambled together that we can't see the different patterns.
So bottom line, we need another method. MVPA is a whole lot better than nothing, but we want to be able to ask, is there information present in this region even when we think the relevant neurons are all spatially intermingled? So let me just do a little bit of this and then we'll continue later.
So goal-- this new method is called "event-related functional MRI adaptation." And we use it when we want to know if neural populations in a particular region can discriminate between two stimuli, two stimulus classes. So for example, do neurons in the FFA distinguish between this image and that image?
So if we want to know that, we could measure the functional MRI response in the FFA and find this would be an event-related response, similar responses to the two. And as I just mentioned, that wouldn't mean that there isn't information in the FFA that discriminates that. It just says they have the same mean response. Everybody get that?
Now, if we zoom in, and think about what might neurons be doing, it's still possible-- even with the same mean response-- that neurons could be organized like this, with some of them responding only to this image and some of them responding only to that image. But it's also possible that all of the neurons respond equally to both. And we kind of desperately need to know-- I mean, not in this case. This is a toy example, obviously.
But we often, when we're trying to understand a region of the brain, we need to know which situation we're in. So that neural population can discriminate these two and that one can't. How are we going to tell which is true?
Well, we talked before about multiple voxel pattern analysis, but as I just said, it only works when the neurons are spatially clustered on the scale of voxels. So imagine you have these situations here. This is getting more and more of a toy example, but just to give you the idea.
Suppose where those neural populations land with respect to voxels is like this. So if each of these is a voxel in the brain, a little, say, 2 by 2 by 3 millimeter chunk of brain that we're getting an MRI signal from, if you have the different neural populations spatially segregated enough that they mostly land in different voxels, then MVPA might work here. Is that intuitive? Do you guys all see that?
Then we'd get a different pattern in these voxels if we're looking at those two different images. But even if we have the situation here, which is kind of informationally the same, if they're spatially scrambled so that they're in roughly equal proportion in each voxel, MVPA won't work. Does that make sense?
And so that's when we need this other method called "functional MRI adaptation." Make sense? I'm going to go one minute over probably. So the point of functional MRI adaptation is it can work even when there's no spatial clustering of the relevant neural populations on the scale of voxels.
So let me go through it quickly and we'll come back to it later. So here's how it goes-- the basic idea is, any measure that's sensitive to the sameness versus difference between two stimuli can reveal what that system takes to be same or different. So for example, if a brain region discriminates between two similar stimuli like these, then if we measure the functional MRI response in that region to same versus different trials-- so this would be a different trial.
You present Trump and then the chimp back to back. That's one trial, compared to a same trial, chimp and then chimp. And of course, we counterbalance everything, so we also do chimp and then Trump in another different case and then Trump and then Trump in another same case.
If we find that the neural response is higher when the two stimuli are different than when they're same, then we know that that region has neurons that respond differentially to the two. So remember, we started with a case where the mean response is the same to this image and this image if you just measure them alone. But now we want to know, do we really have neurons that respond differentially?
So we're using the fact that neurons are like people and muscles. If you keep doing the same thing to them, they get bored. Been there, done that.
So you present this back to back. You get a lower response than if you present this and then this. That's called "functional MRI adaptation." It's like that waterfall MT adaptation we talked about before, but just crammed into a fine time scale.
And so then if you do that, you can ask what a region thinks is the same. So then, we could ask, what about these two images? Does it think those are the same?
And if we find a response like that, what have we learned? So if these two respond like that, what have we learned about a region that shows? This is all fake data, obviously, but if we saw that, what have we learned? And then I'll let you go, as soon as I get a nice answer to this. Yeah.
AUDIENCE: So if it's the same between two pictures of the same stimuli, that means that it's activated. It can discriminate. But if the yellow is at the same degree as the red, it would just be the brain reacting to different pictures.
NANCY KANWISHER: You totally get that. It's probably right, and you totally get it. Key point-- just because I don't want to torture you guys and go way over-- but key point is, it's the same response is the lower response.
We tell that with this case, and we actually give it a same one. So same is lower than different. That's just how this method works.
Then we're basically asking, does that count as the same to this brain region? And we're finding, yes, it does. That tells us that those neurons are invariant to all kinds of things-- viewpoint, facial expression, when he last dyed his hair, who the hell knows, all these other things.
So we'll talk more about this. But the idea is, now we have another method in addition to MVPA that can start to tell us what neurons are actually discriminating. OK, sorry to go over.