The following content is
provided under a Creative Commons license. Your support will help
MIT OpenCourseWare continue to offer high quality
educational resources for free. To make a donation or to
view additional materials from hundreds of MIT courses,
visit MIT OpenCourseWare at ocw.mit.edu JANET RANKIN: All right, great. I looked over some of the Mud
cards we had from last time. And there were some
really good questions, some really good points,
and one lingering one from the previous time
that I forgot to address. Someone said, could
we please just move the desks into
an arrangement where we could see each other. It was a fabulous suggestion. Today we can't do it. But I'm going to try to do
it in the future whenever the logistics allow, because I
think you're absolutely right. A little bit more
of a U, or a circle, would be great for this class. So thank you for
that suggestion. Another one was that we didn't
get to motivation theory. And I think I mentioned
this last time, the idea that there's a
huge amount of stuff that's been written and
published on motivation and theories of motivation. And I'd be more than
happy to discuss this as the semester goes on. And it might be a
great topic for one of those lunches that somebody
was going to organize, one of those informal
lunches that someone was going to organize. Or maybe we can
carve out some space at the end of the semester and
pop it in, take something out and pop it in, because
it's really fascinating. It's really interesting,
but it doesn't quite fit anywhere in particular. So thank you for
bringing that up. And keep bugging me about it. The other thing is someone
asked if all those learning theories that we discussed
were equally valid. And yes, they are equally valid. I mean, there are situations
where being behaviorist could help. There are situations where
being constructivist could help. It depends on what you're
trying to have students learn. So you want to make sure that
you tailor the way that you're teaching to support what it
is you want students to learn. And we'll get into that
in a lot more detail today when we talk about
developing learning outcomes, and then also when we
talk about tailoring your instructional
activities to support those learning outcomes. So this class, this
particular class today is a balance between developing
course design, developing intended learning
outcomes, which for me is at the heart of it. If you can come up
with some good intended learning outcomes, you basically
have designed your course. And I'll show you how. I mean, you can't just ignore
it, but that's the hard work, and then the act of writing
constructing a syllabus. So once you've got
those learning outcomes, you have an incredible anchor
for the rest of your course. And there was just
a very timely haiku. I put it on the wiki. But there's this Twitter
feed, academic haikus, which if you want to waste
any time and read them, it's a way to waste time. But there was one here,
"that question you have? I know where the answer
is-- on the syllabus. So it was, like I said,
it came out yesterday. I thought it was pretty timely. And it's true. If you write a good syllabus,
it should really clear up a lot of, not the content, not
the understanding level issues students have, but the
issues with respect to when's the test, do
we have to know this, how will this be graded,
what's important? All those things should
be in your syllabus. So that's why that's there. So here's a little
slide, a graph that I saw a few
years ago, and I thought it's a really good
way to start this discussion. So on the y-axis we have sort
of an assessment, a proclamation of students understanding. They got it, they didn't get it. And I think the word
in French is [FRENCH]. So for some reason I find it
very cool to say [FRENCH]. Got it. I don't know. So whether they get it, like
whether after the tests, you've given a test, and you
go, man, that student really got it. That student really
gets this material, and whether they passed the
exam or failed the exam. So the size of the
circle is more or less the number of people that
fall into those quadrants. So anybody surprised
by this plot? Yes, Rachel. AUDIENCE: [INAUDIBLE] JANET RANKIN: OK. All right. Well, there's the segue. So we want, we don't
want the green, but we more or less
expect the green, right? If you get it,
you pass the exam. If you pass the
exam, you get it. That's that, and we like
to have lots of people have that happen too. And then we also
acknowledge that somebody isn't going to get
it, and they're not and they're not going
to pass the exam. And that kind of maybe makes us
sad, but it's not surprising. And then anybody
can kind of have a bad day or kind of a fluke. So sometimes, sure,
somebody will fail the exam, and actually really
understand the material. Generally speaking,
that doesn't happen, and then there's
often a reason for it. But it's that red
circle that Rachel mentioned that is troublesome,
at the least, troublesome. So why might that happen? Why so many red circles? Yeah, Dave. AUDIENCE: Because the test was
set up for basic memorization, or you could figure
out with a lucky guess with multiple choice. So you could easily pass just
with good test-taking skills or a good memory. JANET RANKIN: Excellent. So you pass with sort of
good test taking skills. Now I want us to
hang on to that. There's an underlying
assumption in Dave's comment. So I'd like to hang
on to that thought, but let's hear a few
other conjectures and see. Rachel, why do you think
you might have passed it? AUDIENCE: So often I
feel like TAs are either really good or
really terrible when it comes to the exam reviews,
and they're mainly good. They often told us
too much of what was going to be on the exam. And so then you
kind of just knew what the test was going to
be if you could well on it if you [INAUDIBLE]. JANET RANKIN: OK. That's interesting. Yes, Gordon. AUDIENCE: Yeah, it's
possible that [INAUDIBLE] tests [INAUDIBLE]. When it comes to true life
situations and something that [INAUDIBLE]
knowledge that was gained from that material,
then [INAUDIBLE]. JANET RANKIN: OK. So Gordon is sort of kind of
flirting with this idea of you as the instructor, this
real life knowledge, this is what it means. They have to have this practical
working knowledge for us as the instructors to
say they got it, right? But the exam isn't measuring it. And that's exactly what, that's
what Dave was saying as well. This idea that the exam
is measuring one thing, but it's not really measuring
whether they got it. When we go home
and think what does it mean to be a good
material scientist, or what does it mean
to be a good chemist, we don't think, oh,
it means that you can define what a
mitochondria is, or you can balance
a simple equation. And yet, we put
that on the exam. It's also a bit of
the idea that students are good at taking tests. They're good at gaming systems. They're good at listening to
TAs or TAs may have not given out appropriate information. So it comes from both sides. It comes from the students and
it comes from the professors. But you have control over this. You shouldn't be writing
an exam that people can get a good grade on,
but then when you go home and say, oh, they got a
good grade on this test, but they didn't get it. You need to define what it
means to get it in your field, to get this subject, whatever
it is that the test is about. And then you need to write
a test that measures that. And too many times
we don't do it. We're kind of rushed,
or we don't really know. We haven't really thought
about how to measure whether somebody gets it. So the first step in all
this is to define what it is that means that they got it. And that's the first step,
and that's the hardest step. But once that's done,
the rest is easier, and it will save you from this. OK? OK. So that's the topic
of today's class. And if you needed a
little more motivation, I'd like you to
read this paragraph, and I'd like you to tell
me what it's talking about. But I'm just going to give
you about 2 and 1/2 minutes to read it. And then we'll
discuss what it means. What do you think
this is talking about? Yes, David. AUDIENCE: Yeah, I
think it stresses about how we can arrange by say,
maybe schedules or activities that put them into groups. Then maybe if we form a group,
how we plan [INAUDIBLE]. JANET RANKIN: OK. So arranging schedules
or activities. OK. OK. Something else? AUDIENCE: When
you're planning, how to go about teaching
a module, [INAUDIBLE]. JANET RANKIN: Planning teaching. What else? Anything? AUDIENCE: It's like a mailroom
operation or something. JANET RANKIN: Mailroom. Something from the back, maybe? You know you're smiling. AUDIENCE: I couldn't
think of anything. JANET RANKIN: All right. Well, then don't say anything. AUDIENCE: The first is the
course content, the course outline, and what you
are going to teach the students, [INAUDIBLE]. JANET RANKIN: OK. So students, grouping students-- Adam, right? Adam. OK. So content, arranging content. Anybody in the back? Flyer? Yes, Gordon. AUDIENCE: I think you talked
about sorting materials. JANET RANKIN: Sorting materials. So we'll go with maybe sorting. So if I tell you-- so these
are all great guesses. They're as good
as what I'm going to tell you the answer is. So this was written to
describe the process of doing your laundry. Like doing the wash. You take the clothes
and you put, maybe you put the dark clothes in
one bin and the white clothes in another bin. And then you fold them
and you put them away. It's impossible. This happens almost
every time I show this little piece of writing. People don't generally get it. They get the fact that it
has to do with sorting, and some planning,
some arranging things, but they don't get that
it's about the laundry. If you skim it-- you don't
have to totally reread it-- but if you skim it
with the idea that it's about doing the
laundry, you probably can understand it better, maybe. So now I'm going to ask why
did I show you this now, today, before the third, the
beginning of the third class? Yes. AUDIENCE: I think it's sort
of about arrangement of course content [INAUDIBLE]. JANET RANKIN: So it could
be about how you arrange the content in the syllabus. [? Hina. ?] AUDIENCE: I think it's easy
to describe methodology as a person more in
a different field. But if there's no context
for the audience about what this is, it doesn't matter that
you've laid out everything. No one knows [INAUDIBLE]. JANET RANKIN: Right. It's very detailed
at a certain level, but it's still really
hard to understand, right? Any other ideas? Yes. AUDIENCE: I think
it's interesting. There's a lot of value
judgements in it. Like, it's really easy. And one can never tell
with comments like that. Because I'm trained to figure
out what they're saying, it's actually just
frustrating me more, because I don't know what
they're talking about. And it's really easy-- JANET RANKIN: All right. So that's a great point. So we're talking about
writing learning outcomes, about organizing the
course around writing learning outcomes. If you're using
phrases, or you're using language
that's just, there's just like throwaway words
in there or value judgments, it's not doing your
students any good, right? And also, that if you have
this beautiful course that's beautifully organized even,
wonderfully organized, but if you don't tell
them what it is you're trying to do with it all-- what it is they're
going to be able to do with it all, whatever it
is you expect them to do-- they're not going to
be able to follow you. So imagine a procedure
as fairly straightforward as doing the laundry. Now imagine, OK,
you're going to be able to select the right,
the appropriate method to integrate equations,
so methods of integration, by separation, by card or
substitution or whatever. So if you don't
tell them what it is you're expecting
them to do, even if you're very good at
explaining what it is they're going to do, they won't get it. So this is my sell for why when
you write a learning outcome, you have to write a learning
outcome that's clear and that's understandable to
the students, that doesn't have a lot of extra words
and that tells them exactly what they're
supposed to do. So I have some learning
outcomes for today and I hope that they're clear. You'll be able to
describe the components of the constructive alignment
and backward design processes. And those, constructive
alignment anyway, was in the pre-class reading. You'll be able to classify
the content of a course that you might teach. So what do I expect students
to do with this content, to be able to do
with this content? You'll be able to create
measurable, specific, and realistic learning outcomes
for a class you will teach, and you'll be able to state
the components of a syllabus. So I feel that those are
pretty understandable. You may disagree with me,
or maybe you want to wait. So constructive alignment, it
was in the pre-class reading, right? Anybody want to just give
a brief summary of it? I mean it's kind of up
here, but in your own-- AUDIENCE: I think
constructive alignment is about aligning assessment,
method of assessment, with course content. [INAUDIBLE] assess
[INAUDIBLE] type of assessment so that what the teacher
has in mind, the course content or the content
I want them to get, to test it with the
assessment process. Then we would also
do the assessment, and then come up with
the intended outcome. JANET RANKIN: Right. They'll achieve the
learning outcomes. And if you do this
right, you get rid of that red circle
on that graph. You get rid of students who
did well on the assessment but don't really
know what's going on, because you have
aligned the assessment with what it was you wanted
them to know or be able to do. So that's important. I just think this is a
very, very compelling image to keep in your head, that these
three things have to be linked. And as I said before,
this, you will see that the top part,
writing the learning outcomes is the tricky part,
it's the hard part. It's the part that requires
a lot of thinking and work. When you get a good
learning outcome, the other two things almost
take care of themselves. OK? Did anybody have
any, does anybody have any questions
about something from the reading associated
with constructive alignment or some other idea
from the reading? Yes, Gordon. AUDIENCE: I just wanted
to make a contribution. I think that this constructive
alignment is one of the things that the intended
outcome should measure. So I think maybe that's
confusing [INAUDIBLE] and also how we can test
and be sure and confident. JANET RANKIN: Right. Right. I think the measurability part
for scientists and engineers is kind of intuitive. Like why would you say
something if you couldn't tell whether or not
it was true, or not from a scientific point of view. What's the point
of saying, making a proclamation
about what somebody should be able to do if you
can't measure it in the end? So it's comforting
to say, OK, I'm going to make learning
outcomes that are measurable. But it also can be
a bit challenging. But we'll do that. We're going to do that today. Some other comment? Nina. AUDIENCE: I guess as
far as [INAUDIBLE] Do people write
learning outcomes that have no constructive
alignment process, like choosing to write
learning outcomes, are you going
through the prcoess? It seems decoupled a
little bit in the reading, but it seems like you can't
have one without the other. JANET RANKIN: You
owe it to yourself-- and I don't know what people
do, and I guarantee you people write learning outcomes
without thinking about whether they can assess
them or measure them, I guarantee that
happens in real life-- you owe it to yourself to
try very hard not to do that. OK? And I think the important
thing to do is, often we say, oh, my god, this is
the best assignment. This is the greatest question. It's this project, and
they're going to do this, and they're going to whatever. They're going to
build a race car. And if you sit down
and say, well, what do I want them to
get out of that? If you can't articulate it
in a way that's consistent with your class,
consistent with the course as it sits in this
series of their courses, or as it sits in
the institution, then you either need to
rethink the assignment, rethink the project, or
think a lot harder about what learning outcomes that
activity advances. That's a very good point. It's really kind of iterative. And I think it might be easier
to see with this diagram. This is another-- so John
Biggs is from the UK. And so there's sort of
the UK camp of people and they like John Biggs. And then there's
Wiggins and McTighe. Grant Wiggins and Jay
McTighe are from the US. They wrote primarily for
the K through 12 audience, but it's completely applicable. And they're from the US, so they
have kind of a US following. It's just how it works. It's essentially the same thing. It's identify the results, so
the intended learning outcomes. Figure out how you're
going to get them there. So the black is Wiggins and
McTighe and the red is Biggs. So plan the learning
experiences that are going to support them,
achieving the learning outcomes, and really help you
measure whether they got them or not. And then determine
what evidence will be that acceptable evidence
that they achieved the learning outcomes and what
are the assessments. So real often we do have
this project that we want. Oh, this is a great problem. Often in thermo-- I taught a lot
of thermo-- there's like, oh, they're going to do this
double loop, double reactor cycle with the refrigerant
and a heating cycle. And it's going to be
coupled and they're going to use the output from
one for the input in the other and all this crazy stuff. Like, this is a
great assignment. And you kind of, I know
this is a great assignment. If I can't back up
from this assignment-- whether or not I put
it here or here-- if I can't back up and say
after a successful completion of this assignment, you
will be able to describe the components of
the Brayton cycle, you will be able to define
the increase in entropy for a system with x boundaries,
If I can't articulate those things, then I need to
either change the assignment, or, well, then I need to
change the assignment. OK? But you can start
with the activity, and then back up
and say, well, what learning outcomes does
that support, and then, am I OK with that? Is that relevant to this class? So it's often a very iterative,
the arrows kind of go this way, but it's often iterative. So I had asked that you come
to class with some topics from a class you're
likely to teach. That was on the assignment page. If not, start thinking now. And so you want to have,
for most classes, most undergraduate classes,
there's a bunch of topics. You might have 10
topics, 20 topics. And I don't care right now
about the scale of the topics, whether they're really big. You probably don't want
to go with like first law, second law. That's probably a
little bit too big, but on the order
of 5 to 15 topics. Jot down the topics. And then I have a chart here. So the chart has two
sides, and the columns are Bloom's Taxonomy,
which you don't have to worry about
that specifically, but you can look at the words
that are associated with it. And then the second column
is what students should be able to do with each topic. So let's say my topic is
first law for closed systems. So I might think
about, OK, do I want them to be able to
define, to just state the first law for
closed systems? Do I want them to be able to
use the first law for closed systems in order to solve
certain kinds of equations? Do I want them to be able
to derive the first law? Do I want them to be
able, whatever it is. So you're going to look at
the columns, the categories on the left side,
and you're going to think about your topic. And you're just going to decide
which box in the middle column to put the topics into. You don't have to worry
about writing anything. You're just going to distribute
the topics to the level of, it's really cognitive
processes that are associated with the things on the left. Remember, understand, apply,
analyze, evaluate, and create. So we'll take
about five minutes. And you just do
that by yourself. And just fill in
that middle column of the table with the
topics from your course. Do you want more
explanation, Rachel? It's starting to make
you think about this part of the beginning circle. But you don't want to
write any sentences yet. But it's starting to help
you think about what are they going to do with these topics. What do I want them to do? Is it remember, or is it create
something using these things? Where is it? Those are two ends
of the spectrum. And in your chart, the
things that at the top are the more remember, right? It's more about kind of
a recall information. And then the things in
the back of the chart tend to be more higher
order cognitive processes, like synthesize, apply, create. Yes. AUDIENCE: [INAUDIBLE] JANET RANKIN: So don't worry
about the third column. Don't worry about
the last column yet. We're going to do that later. So all you have
to do is just say where each topic falls loosely. OK. Does everybody have a chart? OK. AUDIENCE: Do you
want us to write as many topics as possible? JANET RANKIN: Write
as many topics as possible, but distribute
them through the chart. Right? Not everything is going
to be remember, right? So some things will be remember,
some things will be apply, some things will be analyze,
some things will be create. OK? If you have any questions,
just raise your hand. All right. So what we're going
to do is I'm going to ask you to just
keep those with you. We're going to do
another activity later on using those as
a starting point, but I want you to just
keep that sorting with you. Any comments or questions
about the experience, about what you just did? Was it easy? No. It can sometimes be a challenge. Well, what the heck
you were supposed to do with this anyway? What do I really want
them to do with it? This is the beginning
of that process whereby you define exactly what it
is you want students to do, know or be able to do. And then you can start
to think about how you're going to assess whether
or not they can do it or not. But it's very
different if I were to say I want everybody
to be able to state the first law of
thermodynamics versus I want you to be able to use the
first law of thermodynamics to define chemical, mechanical,
and thermal equilibrium in a simple composite system. That's different. That's very different. So that's what we're
starting to do. So now we have
kind of the seeds. We thought about which topics
kind of we want to have, what we want to do with
each of the topics. But now we really have to turn
them into learning outcomes, into real statements. By the end of this
class, you will be able to, by the end of this
course, you will be able to. And just as an aside,
remember that sometimes you'll notice that for every class,
every session we have, I have learning outcomes. But there's also overarching
learning outcomes for the course. So I will use them
interchangeably, but they'll be at a
slightly different scale. The ones for the class are a
little bit more fine grained. As was mentioned in the reading,
and as Gordon mentioned, they have to be specific,
measurable, and realistic. If they don't have
those properties they're not going to be
particularly useful, or as useful. All right. And we'll see some examples
of that in just a sec. So here's what they're not. They're not topics. So you came with topics. You will leave today
with learning outcomes. But topics are not
learning outcomes. The first law of thermodynamics
is not a learning outcome. You need a verb with it. What do you want me
to do, what do you want your students to
do with the first law of thermodynamics? What do you want them to
know about the first law of thermodynamics? So they're not topics. They're not things
that you will do. It's easy to say, I could
have a list a mile long. I will talk about the first
law of thermodynamics. I can check that
off pretty easily. I mean I just did. I just talked about the
first law of thermodynamics. It says nothing about what
you learned about it, what your students learned about it. So it's not about you. Get over it. That was a joke. It's not value statements. As Adrian pointed out, some
of these think this is easy, this is good, this is useful. Any of those things don't
belong in a learning outcome. And it's not your hopes and
dreams for your students. I mean, ultimately, it is. You hope that they will be
able to use the first law of thermodynamics, da, da, da. But it's really about
what they're doing, about what they're going to do. So, no. Here's a little example. This is just from like an
earth and planetary course. Students will understand
plate tectonics. How's that? Is that OK for a
learning outcome? Why not? AUDIENCE: It's too broad. JANET RANKIN: It's
not very specific. What the heck about
plate tectonics? What else? AUDIENCE: Measurable. JANET RANKIN: Thank you. OK, the measurable part. This word understand,
strike it from your brains. It's a property that we love. We love understand. But you can't
flippin' measure it. You can't go in
there and measure it. You can measure lots
of things that tell you whether students
understand, but you can't measure understanding. And I think we'll see that
with some more examples. I don't think anybody is
earth and planetary here. So how about this one? Students will be
able to interpret unfamiliar tectonic
settings based on information on volcanic
activity and seismicity. Again, I don't think any
of us are geologists. But the idea that I've
told, or the person has told us what
we're going to use to interpret these
situations, and then we're going to make predictions based
on those interpretations, I mean, that's an assumption. So it's specific and
now it's measurable. Even though we're
not geologists, I suspect we can
think of how we could write a test question that
would ask students to do this. And then we could see
whether they did it or not. So that's measurable. And probably, by doing this,
by having students do this, we get a pretty good
measurement about whether they understand plate tectonics. But it's much more specific
and it's much more measurable. From here, this is from the
University of Minnesota, which has a nice website
on learning outcomes. They're not from your point
of view, as we mentioned. They're not what
you're going to do. I'm going to introduce
students to the major turning points and processes in
North American history. You can say anything you
want, but it doesn't mean the students learned anything. So it's not about you. It's what they're going to do. They're going to
list and describe the turning points and processes
in North American history. I'm going to create
an understanding of the formal constructs
of physical design. That one is like,
this one is, it's like, I don't even know
what it means, right? Create an understanding
of the formal constructs of physical design. I don't even know what it means. And so it's like
that laundry example. Your students aren't going
to know what it means, and it's not going to help them. It's not going to help
them access the material. So the ones on the left
are all teacher-centered and they're not very useful. The ones on the right
are student-centered and they are generally useful. But there's some things wrong
with the things on the right. AUDIENCE: Understand. JANET RANKIN: Understand, yes. So what would you
write instead of understand the formal constructs
of physical design, whatever the heck that means? What might you use? AUDIENCE: Identify. JANET RANKIN: Identify. OK. That might be like a list,
right, or identify or list, pick it out of the
lineup kind of thing. This is a nice one. Explain to an intelligent
non-expert, an INE. And so explain this to somebody
who's not in the field. If you could do that, it
probably means that you get it. Somebody I know used
to say, explain this to my four-year-old brother. Explain-- what was
it-- angular momentum, to my four-year-old brother. I would argue if you can explain
it to a four-year-old kid, you probably understand it. So you can use that. Use formal constructs
to design something. I mean, that's kind
of an inversion. How about the one on the bottom. Understand gender, race,
ethnicity, socioeconomic class, understand how they have
shaped Americans' lives. Completely unmeasurable, right? So what would you write instead? AUDIENCE: Describe. JANET RANKIN:
Describe how, sure. Anything else? AUDIENCE: [INAUDIBLE]
it seems like you're trying to get [INAUDIBLE]. JANET RANKIN: Right. So that's a great
point, is that we don't know where this faculty
member put this on their table. We don't know whether
it really just states some ways that, list some ways
that these things have affected Americans, North Americans'
lives, or Americans' lives-- which is kind of a remember
thing, just make a list-- or whether it's do some
analysis on some situation. We don't know that, but
you as the instructors get to decide that
for your classes. So that's an important thing. You get to call all the shots. You get to make the decision. We can only guess about
what this person meant, and therefore, his
or her students can only guess about
what he or she means. So you have on your table this
hierarchy of Bloom's Taxonomy. It's often shown in this
triangle, this pyramid. There's a handout which has
exactly the same strata on it. And it just has some handy
dandy verbs associated with it. But Benjamin Bloom came up
with this classification of cognitive processes way
back when, in the 1950s. And it is this idea that
it is this pyramid, that at the base of the pyramid
are these ideas like remember. And those are basic cognitive
processes, remember, list. And then it moves up the pyramid
to things that require a higher level of cognitive processing. Rather than just pulling
facts out of storage and stating them, you're
going to do something with those facts. So as you go higher and
higher up the pyramid, note that he chose-- really unfortunately-- to call
the second level understand. However, there are
words that can help you. If it really is understand
that you want to measure, there's words that describe
that a little bit better. They're in your
table, and they're also in that second
table that I handed out. So arrange, list, label. For understand-- describe,
relate, recognize, explain, those things. Gordon. AUDIENCE: Just a
little [INAUDIBLE]. When it comes to
understand, maybe if you put it like [INAUDIBLE]. JANET RANKIN:
Understand to what? AUDIENCE: Understand
to an [INAUDIBLE]. Maybe that's what
we can measure. Understand to this level
or to this [INAUDIBLE]. JANET RANKIN: But how, can you
give me a specific example? AUDIENCE: I wish I could go back
to the last slide [INAUDIBLE] can describe [INAUDIBLE]. JANET RANKIN: OK. So I see where you're going. Does someone have a suggestion? So think about that laundry. Think about that description
of doing a laundry. Can somebody take
what Gordon said and get where he wants to
go, but get there in perhaps a more direct way. Katherine. AUDIENCE: [INAUDIBLE] JANET RANKIN: Exactly. So if you're going
to do that, and I see where you're
coming from, understand so that you can describe,
in order to describe, or to the level of describe. Just forget the
understand part, and just say describe, or define,
or whatever it is you want. But yes, I mean, that
first step is saying, what do I mean by understand? And when you say that, then
it generally rewrites itself. And so the more direct you
can be, generally, the better. And keep that laundry
example in your head. Yes. AUDIENCE: You're asking
a student [INAUDIBLE]? JANET RANKIN: So I would say,
well, let's just open that up. What do you think about
that, of asking students? So do you understand this
stuff about learning outcomes? AUDIENCE: I think
as teachers, when you, in the front of the
class, you're teaching students and you say do you
understand, they say yes. But you see from psychology you
know who doesn't understand. JANET RANKIN: So David
says as I ask you that, do you understand? Most people are going to
say, hey, I understand, sure. Don't think I'm an idiot. I understand. Right? So I completely agree. Students are not likely to
say they don't understand. What else? AUDIENCE: That's difficult
for the teacher to [INAUDIBLE] JANET RANKIN: Well,
so do you want to say something about that? AUDIENCE: Yeah, I agree. I think the part
that we do understand as nonspecific is the learning
understanding [INAUDIBLE]. As an instructor you
have to know, understand what you're trying
to get at, which I think is where the clicker
question is beautiful, because you know
already what you're trying to evaluate that they're
catching in class in real time, if you can see who's getting
at the nuance of what you're teaching. JANET RANKIN: Yes. I think that's an
excellent, excellent point, that many of these
activities that we do, these small group
activities, or the clicker questions or other
activities that students are doing in class are
helping them learn. But they're also formative
assessments of whether or not they get it or not, whether
or not they understand. And you as the
instructor can see it. If I walk around,
if I listen, if I look at who answered what
question on the clickers, or on the raising your hand,
multiple choice questions, whatever methods I used
here, I get a better, a much better sense
of who understands or doesn't understand. If all I'm doing is
lecturing I can't tell. And I can ask do you
understand, but they're not likely to tell me. Which is kind of the other
point is that sometimes-- and I think you were
alluding to this, Gordon-- that sometimes students don't
know whether they understand or not. So we need to give them the
opportunities to find out whether they understand or not. And arguably, if
you're just lecturing and you're a great lecturer-- you're eloquent,
you don't stumble, the writing on the board is
awesome, all this stuff-- students will think
they understand maybe when they don't,
because they're not having to confront
any misunderstanding, and it all seems very nice. So these activities where
you actually have students do something, say
something, where you ask them a specific
question, that's when you can determine whether
or not they understand or not. And because they are
not likely to say they don't understand,
either because they just don't want to admit it, or
because they really don't know whether they don't understand. So you need to give them
measurements to find out whether they get it or not. Yes. AUDIENCE: Sometimes, for
example lecture and the activity has [INAUDIBLE]. For example, we have
lectures and the teacher just [INAUDIBLE] and then
we have expectations as we do exercises. So do you recommend that
we also do some activities during the lecture
or just [INAUDIBLE]? JANET RANKIN: No, do
them during lecture. I mean, the model that
we're trying to do here is the model that you should
do even if you have a hundred or 200 people in your class. You can use clickers. You can use pair
share discussions. You can do all sorts of
things to break it up, and it's effective. There's a paper coming
up for next week, a meta-analysis, active
learning strategies, how they actually, students
learn more in a lecture if you actually ask them
to talk to each other, break up, share, et cetera. And I think when we talked
about misconceptions, remember that example with the coin? How many students got it,
could draw the free body diagram before and after? I mean, before and after
in a straight lecture, there's hardly
any learning gain. But if you incorporate
active learning, the learning gains go way up. So that's Bloom's Taxonomy. And an interesting take
on Bloom's Taxonomy, there's a woman,
Kathy Schrock, who-- it's a nice website
and the slides we posted so you have the link-- but she decided, she has
this theory that it's not really so pyramidal. That everything
really is dependent, these cognitive processes
are dependent on each other at a certain level. And so she's drawn it as
a gear, where everything sort of supports
the act of creating, which is up at the top. But the other ones are
sort of around the side, but they all contribute
to the ultimate cognitive process of creating. So that might be a nice
way to think about it. And then she's taken the
specific words, like analyzing, and said, OK, what's
an instantiation of analyzing, outlining,
deconstructing, organizing, structuring,
surveying, whatever? So it's the same idea, it's
just a different graphical representation. So what you've done is
you've taken your topics, put them in those
cells in the table. And I just want to point out
kind of how as the instructor you have a lot of
power about where you're putting those, where
you've decided to put those. And it may be different for
a first year undergraduate class, or a second or
third year graduate course. It could be the same
topic, but you're going to put them in
a different category. So that's why when I asked
you to bring a topic, it's for a specific class. It's for a specific
group of students. So let's say we take the
idea of interstitial sites. So I have a
crystalline structure, it's got an order of
periodicity of atoms. And then I have spaces
between the atoms, those are the
interstitial sites. And so I could say you'll
be able to identify the interstitial sites. So I could say where's
the tetrahedral site in a face-centered cubic
or something, whatever? And students just
really have to find it. I could say calculate
the maximum size of an interstitial
atom that could occupy that space, an ion
that could occupy that space in a particular
crystal structure. So those are two very
different levels, but it's the same topic. X-ray diffraction, I could
say apply Bragg's Law to calculate d-spacing. So that's very much a turn
the crank kind of thing. I know the equation
for Bragg's Law, I stick things in, I
solve for the unknown. I could then also ask you
to index unknown diffraction patterns, meaning identify an
unknown crystal structure based on the diffraction pattern. That's a completely
different activity, completely different
set of cognitive processes than just
using Bragg's Law. But you have to
decide what it is. And arguably, if all you've had
students do during the class is sort of problems that they
applied Bragg's Law and they solved for d-spacing-- so they
solved for the incident angle or they solved for the
wavelength or whatever it is-- you really can't throw this
at them on the exam, right? Or conversely, if you spent the
whole time, they've indexed, they've been in the lab indexing
unknown diffraction patterns, and then this is the question
you ask them on the exam. Well, then you have people that
perhaps get it without really, perhaps pass the exam
without really getting it. So that's just something
to keep in mind. The other thing to
note is let's say this is my intended
learning outcome-- calculate the maximum size
of the interstitial atoms in a variety of
crystal structures. How do I know whether
students get that or not? What's that? AUDIENCE: It's easy. JANET RANKIN: It's very easy. I give them a crystal
structure and I say calculate the maximum
size of the ion that can fit in the space. So I have figured out how, the
measurement is easy, it's cake. It's totally straightforward. Now I haven't told them, OK, I'm
going to give you this crystal structure and I'm
going to ask you this spacing, this ion size. I haven't given it
away, but I have told them what's expected of them. And my colleague wrote
these for VSEPR theory. It's the same topic,
and it's different. Identify the common
geometric shapes found in simple molecules. Explain the assumptions
of this theory. Apply the theory to
predict 3D structures. Compare and contrast the
geometry of a certain molecule as predicted by two
different theories. Evaluate the accuracy
of each theory for a particular
set of compounds, and then create some
recommendations. So it's the same
topic, but you decide what it is that students
should be able to do. OK? So I have some, just a
little exercise for us. So they should be specific,
measurable, realistic. I'm going to put them up
here and we're going to say, you're going to tell me
what's wrong with them. So number one, t-tests are
like a statistical analysis. So is anybody a
t-test kind of person? We can skip that one. How about gaining
appreciation for the use of linearization techniques? What's wrong with it? AUDIENCE: It's not specific. JANET RANKIN: It's not specific. AUDIENCE: It's not measurable. JANET RANKIN: It's totally
not measurable, exactly. I mean, how do I know
that you appreciate it? Don't do it. All right. Great. Have an intuition for the most
effective method of integration for a given problem. AUDIENCE: It's vague. It's not measurable. JANET RANKIN: So I've
heard it's not specific, it's not measurable,
and it's not vague, I mean, and it's vague. So right, so have an intuition,
I cannot measure that. I have not figured out a
way to measure intuition. I can tell whether
you can do it, right, but I can't tell you whether
you have an intuition for it. Well, how would you rewrite it? Actually, how would
you rewrite 2? AUDIENCE: Use [INAUDIBLE] JANET RANKIN: So it could
be use the techniques. Another suggestion? AUDIENCE: Make something using
the evaluation techniques. JANET RANKIN:
Calculate something. Make a first pass estimate
using a particular linearization technique. AUDIENCE: Describe when you
would use a linearization technique. JANET RANKIN: Describe
when you would use it, describe why you would
use it, all of those things. AUDIENCE: List those
areas where [INAUDIBLE] JANET RANKIN: List the areas
where it's useful, yes. Great. All right. For number 3, the
gain an intuition, we decided it's vague,
it's not measurable. So how would you rewrite it? AUDIENCE: Compare. Compare the areas integration
technique [INAUDIBLE] Solve a particular problem using
different kind of [INAUDIBLE]. JANET RANKIN: OK. So solve a problem, and
then we need a little more. Did you have something? AUDIENCE: Identify the
most effective method of-- JANET RANKIN: Let's hear
what Rachel had to say. Rachel. AUDIENCE: Yeah. Learn how to identify
the simplest [INAUDIBLE]. JANET RANKIN: Right. So you probably don't
want to say learn how. You would say be
able to identify. Right? Gordon. AUDIENCE: [INAUDIBLE] JANET RANKIN: Right. I mean, you could
just say integrate-- AUDIENCE: [INAUDIBLE] JANET RANKIN: Integrate
a problem using whatever. Select a technique to integrate
a particular equation. And again, those things end
up being more measurable. So 4, provide problem
solving tools and strategies. What's wrong with that one? AUDIENCE: It's too wide. JANET RANKIN: It's too wide. AUDIENCE: It's
instructor centered. JANET RANKIN: It's
instructor centered. What else? AUDIENCE: But it's measurable. JANET RANKIN: Well, I can
measure that I provided it. Here, I provided
these handouts, right? OK. Check. It happened. Whether or not you could
do anything with them, I don't know at
this point, right? That's the problem. AUDIENCE: It doesn't tell
us what kind of problem and what kind of tools
we're going to use. Just who gets to
choose the strategies. JANET RANKIN: Right. Right. It's very, very general,
totally not specific. So what could we do? I mean, we don't know what
this instructor had in mind, but what could you do
if this was your class? What would you say? AUDIENCE: What about
develop instead of provide? JANET RANKIN: Develop. And again, it would have
to be student centered, so the student will be
able to develop problem solving tools and strategies. That's for x, exactly. We want to make it a little
more specific and a little more focused, because
that's pretty broad. I mean, I have
developed a problem solving strategy
that lets me get out the door without tripping. Does that mean I satisfied
your learning outcomes? Probably not. So in order to do something-- Use thermodynamics to
solve engineering problems. AUDIENCE: That's not
specific as well. JANET RANKIN: It's
not specific at all. AUDIENCE: Engineering
is [INAUDIBLE]. JANET RANKIN: Yeah. I mean, it's crazy broad on
a number of fronts, right? I mean, it's like
engineering is huge. Thermodynamics is
pretty darn big, too. So which part of thermodynamics,
which part of engineering? Build an SAE race car. AUDIENCE: [INAUDIBLE] JANET RANKIN: What? AUDIENCE: It's not realistic. JANET RANKIN: Right. It's pretty specific. Build a functioning
SAE race car. It's very specific,
and I can totally think of the metric
of how I would measure whether students could do it. I would just ask them
to build the race car. But they're never going
to do it in a class, in one class period,
in one course. It's unrealistic, exactly. Learn to use Laplace
transforms to solve differential equations. AUDIENCE: [INAUDIBLE] JANET RANKIN: It's funny. I mean, you can
imagine the instructor saying you will
learn to use that. But let's go, Katherine, work
your magic with this one. AUDIENCE: [INAUDIBLE] JANET RANKIN: Right. You're my go-to person for
when you want to get rid of all the extra words. I'm going to call on you. Right. Use Laplace transforms. That you can measure. You can give them
an equation and see if they can use Laplace
transforms to solve it. And yes, you might want
to be a little more specific about the kinds of
differential equations they use, and know how to upper
diagonalize a matrix. Katherine. AUDIENCE: . Upper diagonalize a matrix. JANET RANKIN: Upper
diagonalize a matrix. That's all you
have to say, right? Forget the extra words. And that is really, I have just
written the assessment question for that. I know it's totally measurable. I give you a matrix and I ask
you to upper diagonalize it. I haven't given it away. I haven't told you you will
upper diagonalize this matrix, right? Memorize these
equations that you're going to need to write down in
order to upper diagonalize it. But I have told you that
that's what's expected of you. AUDIENCE: I have
a comment on it. It's pretty useful
when a student is trying to evaluate
the use of the class. For example, the career
fair tomorrow [INAUDIBLE]. There's a million
[INAUDIBLE] to get jobs. OK, how are they
going to evaluate me? And if you could
come in and say, this happened and
this is what we did, as opposed to
having, or have them evaluate me on some
random knowledge. And then I don't know
that I could respond, but I do know stuff. So this is really helpful
in kind of a broader sense. JANET RANKIN: Right. Great. No, I'm glad. And I think it's so true. And people often think,
well, you're giving it away. But you're not
giving anything away. You're just telling
them the expectations. You're not giving
them the problems, you're not teaching to the test,
but you're just telling them what they're going to be able
to do at the kind of high level, or medium level. You're telling what they're
going to be able to do. I will say-- so ABET,
the Accreditation Board for Engineering Teaching-- every accredited engineering
program has to list, they have A through K outcomes. And if you look it up, ABET-- and I'm sure you see
us as accredited-- if you look up,
they have to, they used to do a lot
more bean counting. But what they would
do is the outcomes would be critical thinking
and problem solving skills, and they were a
little bit broad. But every department
that got accredited had to show how various courses
contributed to the outcomes. So somewhere in your department
should be a pretty detailed list of how 6 double
00 whatever, did this. Or 6 yada, yada-- I know your grad
students-- but did this. The problem is they
don't have to do it for grad student courses. But it might be helpful if you
looked at that, because I would imagine that sometimes
the recruiters would be looking at that information,
so that might be a commonality. But every program in the US
that's accredited has to have, show at least how they've
addressed those outcomes. So you might want to
take a look at that. But yeah, if it was a
little more explicit it would certainly be better. And I'm glad it's helpful. So I think we can
see, you can start to see how there are better
and worse learning outcomes, that if they're not measurable
they're not so useful. If they are measurable,
they're quite useful. So now what I'd like to do
is revisit the worksheet. And what we can
do is we're going to take about five minutes
where you just sit quietly and write out specific
intended learning outcomes. So you'll be able to upper
diagonalize a matrix, you will be able to
use Laplace transforms to solve this type of
differential equation, whatever you've
written in this column. And note that if you
put it in this column, let's say you put it
in the apply column, then when you write the learning
outcome it's going to be you will be able to interpret
data using blah, blah, blah. You will be able to model
the vibration, model a car suspension system using
linearization techniques. They're going to be whatever,
wherever you put it in the box. These are some of
the verbs, these are the verbs you're going to
use in your learning outcome. Does that make sense? And I have even more verbs
for you if that's helpful. Did I pass those out? You got those, right? AUDIENCE: Yeah. JANET RANKIN: So
there's more verbs. It's the same thing. I mean, it's consistent. So take five minutes by
yourself and do that. And then you're going
to get into pairs and you're going to discuss. You're going to trade worksheets
and give each other feedback. All right? So five minutes of quiet
and then we'll pair up. So if you can pair up,
you don't necessarily have to pair up with people
that are sitting next to you. You might want to get a
different perspective, so feel free to get
up and move around. You can talk to somebody
that wasn't in your group. So maybe if you go back
and talk with Alex, maybe. We need one group of three. Oh, are you a group of three? So now just trade sheets
of paper with your partner, or if you're a threesome you
can figure out how to do it. And then review the learning
outcomes critically. Guys, just one sec. So just review your partner's
learning outcomes critically. Make sure that they're
measurable, realistic, and specific. If your partner
has used the word understand, help him or
her work through that to get a better word. So you're going to be
a critical friend here. And then you guys can
discuss how to improve it. And then I'm going to pass
around some flip chart paper and each one of you
will write, maybe just write your name at the
top, and write two or three learning outcomes on
the flip chart paper. And then we can stick
them up around the room, and then everybody can
see everybody else's, all learning outcomes. [INTERPOSING VOICES] JANET RANKIN: Was
there a question? AUDIENCE: Oh, no. I just think-- AUDIENCE: I think [INAUDIBLE]. JANET RANKIN: How
are you guys doing? Doing OK? [INTERPOSING VOICES] JANET RANKIN: So you can use
this zone if you need it. As you're looking
at other people's learning outcomes,
think about how clear, how easy it would
be, or not easy, to come up with an exam
question, or a project, or an assessment or
measurement of whether or not the student reached,
attained the learning outcome. You can put them up
here, Ina and Shau. OK. And as I said, make sure that
you've read everybody else's. I'm going to say a
few words about them. I think we may have a record. This is like the first
time ever that no one has used the word
understand, which I think you're all to be commended for. So David, make sure you
don't write understand. You build me up just
to knock me down. AUDIENCE: [INAUDIBLE]. JANET RANKIN: You can,
or you don't have to. It doesn't matter. Does anyone have any comments,
either about the exercise itself, or about somebody's
learning outcomes, that you see around? Again, the
measurement, the thing you want to look at when you
look at your own learning outcomes and those of others, is
are they specific, measurable, and realistic? And would you know
if a student achieved those learning outcomes? So I would say about 90%
of these are very specific, which is great. I mean, there's some
incredibly specific ones here. The more specific, the
easier it is to measure. So that's one thing. That's a useful thing. You want to make sure
you don't get so specific that there's only like one
question you could ask. So that's the balance. Sometimes you can't
tell that unless you're in the discipline. Do we have other comments about
any of these learning outcomes, something you don't
get, something that looks particularly interesting? Yes, Gordon. Perhaps you can sit down
if you're done reading. AUDIENCE: I'm looking around
and I'm seeing [INAUDIBLE]. I don't know how
it's going to be. How are we going to test that? JANET RANKIN: Well, I guess you
have to come up with a reaction that you're confident
they haven't seen before. So if you can do that, then
you can ask them to propose a mechanism for it, I suppose. If the person that
wrote it wants to-- I know who wrote it-- but
if the person that wrote it wants to talk a little bit
about it, that would be great. If not-- AUDIENCE: OK, So a
lot of people are going to just try to memorize
each specific outcome. But it's easier to just
identify your activity group and find patterns within them. So if you can try [INAUDIBLE]
They can't just [INAUDIBLE]. JANET RANKIN: Right. And this is a great point. I mentioned it to
one of the groups. But sometimes we'll say explain
how blah, blah, blah, works. And that's a great
thought, that, oh, they're going to be able
to explain it, right? But many times
they can fall back on a memorized
explanation or sort of kind of a canned procedure. So if you really want them to
be at the level of explain, you will have to kind
of make some things up to get them out of,
to get them away from the ability to just
pull something completely from memory. So this is an example of
how you might do that. Other observations or questions? AUDIENCE: I also [INAUDIBLE]
something that all [INAUDIBLE]. JANET RANKIN: So
perhaps the person that wrote it wants
to comment on it. AUDIENCE: OK. So [INAUDIBLE]. I think what we have,
just [INAUDIBLE]. So we write and compile. So you write a code and
then you compile it. JANET RANKIN: Right. And this is a great
example of how having people that
aren't in the field read them can help clarify
things for your students, who are by default not experts. So perhaps to an
expert, it totally makes sense, of course, code. But to somebody that's not an
expert, maybe that's confusing. Maybe the use of code
twice was confusing. To an expert, it's
totally readable. It's totally understandable. Perhaps to a novice it isn't. You're laughing. AUDIENCE: It's funny, because
we're both [INAUDIBLE]. JANET RANKIN: Yeah,
that was kind of random. Well, that's a perfect
example, right? It's a perfect example of how
getting maybe other people that aren't in your field
to look at them can help make them more
accessible to your students. OK. This is great. Oh, Hina. AUDIENCE: I had, it's
on a different topic. But really, I'm talking
about buzz words in learning outcomes. I think with engineering,
you're really excited about telling people
about real world problems. I guess that's the reason why. I guess it's more of
an opinion question. As you write your learning
outcomes I think in some ways it's better to be specific
about the real world problems that they're going to attack. Because I've seen that on
syllabuses all the time. I'm very excited about it. I think it's a way to
teach people in your class. It's going to help them know
what tools they're getting. JANET RANKIN: Right. I mean, real world
problems in particular really do kind of encapsulate a
whole pile of hopes and dreams. So it may help to be a
little more specific. You can use it in your
learning outcomes, but then maybe describe
it in a little more detail in your syllabus, the kinds
of real world problems, or why we care about
real world problems. They're messy. They're ill-defined. What about them is so important? What about them is so important? Try to minimize the buzz words. But at some level, they may
not know about something until they're completely
through with the course, and you may have
to use that word. So there's a bit of a balance. What I'm going to
do before, I'll probably bring all
these back to my office, take pictures of them, and
put them up on the wiki so you'll have a record of them. I wanted to just
as we go out here, my point is really
that the hard part is these learning outcomes. And that once you write
the learning outcomes, as we said before,
you kind of know how you're going to test
whether students got them, and you know what you're going
to do to help them get them. The syllabus itself is just
an articulation of that. It's a description
of the course, an articulation of
the learning outcomes stated, like the laundry
example, with the title. This is what you're going to do. By the end of the class this
is what you'll be able to do. It's kind of a promise that
that's what you're going to do. It motivates students
to take the class, to stay in the class, to
engage with the class. It keeps you on track. So at the beginning
of the semester maybe I'll be very
clear, but four weeks in, and you're a little bit fuzzy,
because you've been dealing with all sorts of stuff. And you kind of
go, what the heck. What's this in here for? Well, if you've lined everything
up at the beginning using your learning outcomes
as the anchor, then you don't have
to worry about that. You just follow the map. So it really, really
helps you and the students get through the semester
in a logical way. And it tells the
students what they can expect of you, what
you can expect of them. So it's motivational,
structural, and it's evidence, hopefully
of what they did, but certainly of what you hope
to do with them. So for the post-session
assignment, you'll have to write up
more learning outcomes from your course. You can use these as
the basis for sure. Hopefully, that table will help
you as you write more of them. And I will go in and
give you feedback. I want to make sure that
everybody knows that you should read each other's postings. So a lot of you had great
examples from the first class about how you would
use these learning theories in your courses
and how you teach. They were fabulous examples. Make sure you read those
postings of others. That's one reason why it's
up on a wiki and not just you handing it in to me. So you have the Mud cards. Fill them out if you have any
more information that you want. And otherwise,
we'll see next week.
