Should You Use That EdTech?


When deciding whether to use a particular educational technology, consider whether the educational benefits are justified by the effort required to make it work.


The peer-review tool in the Blackboard learning management system should be buried in cursed soil, have hexes placed upon it, and never be spoken of again. I came to this conclusion within the first few weeks of a new online course offering.

Peer review—when students evaluate each other’s work—has been an effective way for students to check whether they’re on the right track mid-way through their course projects. In and of itself, peer review is a great idea. My students like it and find it helpful. The problem is in the execution.

This was all my own fault, really. Normally I approach these things with a large dose of skepticism regarding how well the student-tool interaction will go, and have multiple back-up plans. I don’t know what happened this time—my money’s on possession by a diabolical entity—but I decided to trust that everything would go smoothly. Dear friends, it did not.

First, the tool won’t accept late work. Someone is always late, whether due to leaving assignments to the last minute, or not understanding the process, or both. In this case there were several someones.

Second, students didn’t know how to access the tool. If they read the instructions and followed them exactly (not a safe assumption for a variety of reasons including some perfectly legitimate ones) they might have understood that the link saying “View/Complete Assessment” was what they needed to click on to submit their work for review.

Third, they made errors in the submission process. There was a how-to video, but in my experience those are ineffective for at least 10% of students. In this case it was worse thanks to a confusing and difficult-to-navigate user interface within the peer-review tool.

In the end I had to come up with workarounds to ensure everyone had a chance to give and receive feedback, and now I just use a streamlined version of those workarounds instead of the tool itself.

This experience, and my folly in trusting that other Blackboard tools would also function as advertised in that same course, have led me to outline some general rules surrounding whether you should use a particular learning technology or not:

  • First and foremost, if you’re working with an edtech fiend and see a manic gleam in their eyes as they enthuse about a new learning tool, you need to back away.
  • If your idealistic inner teacherperson is considering the use of a new tool and casually dismissing the amount of work it will take, this is a bad sign.
  • If students must get every step exactly right to make the thing work, proceed as with the first bullet point.   

As a handy reference, print out the diagram at the beginning of this post and display it wherever you usually contemplate the pedagogical decisions you ultimately regret.

Due Dates: Who Needs Them?

Periodically pedagogy Twitter veers in the direction of reasons students miss due dates, and inevitably ends up with someone suggesting that we should abolish due dates altogether. The pro-abolition side argues that students have complex schedules and many responsibilities to juggle, and that due dates are an unnecessary and outdated hoop for them to jump through. The anti-abolition side argues that due dates matter in the “real world,” and it behooves students to learn time management skills.

I come at this discussion having taught courses where the only due date was the course end-date. If you’re thinking of abolishing due dates, there are a few things you should consider.

Going Due-Date Commando

"Dinkum_Boy"_calendar_1919_(14839804822)

An image of a calendar received by New Zealand’s Patent Copyright office that combines the remarkable properties of alluding to due dates, hinting at going commando, and bearing a Creative Commons license. This has been a good internet day. Source: Archives New Zealand (2008) CC BY-SA 2.0. View source.

First, here is my (admittedly anecdotal, and potentially foggy memory of) experience with the no due-date arrangement:

  • Students who started their work early tended to submit their work consistently, and had it done with time to prepare for the final exam.
  • Roughly 50% of students paid a fee to extend their course end-date by 2 months. Of these, the majority did not start early, and a good chunk had not submitted any assignments prior to their original end date.
  • It was common for students who paid for one extension to pay for a second extension.
  • A non-trivial number of students did not complete the course work before they had to write the final exam. After failing the final exam, they paid a fee for a second attempt at the exam, and used the intervening time to complete course work, for which they still got credit.

Students were given a recommended schedule at the start of the course, but I can’t help but think that the absence of official due dates gave the impression that students could succeed without working on the course regularly. An absence of due dates early in the course most certainly contributed to students starting the course late, and thus the negative outcomes mentioned above.

Pedagogical Arguments Against Going Commando*

*for due dates

Here are the reasons I have due dates, and why I think it would be a mistake to run my courses without them. If none of these apply to you, then maybe you don’t need due dates after all. If they do apply, then perhaps consider the consequences of abolishing due dates in that context.

Reason 1. My Workload

I use due dates to distribute my grading evenly through the term, and make sure I have time to give students meaningful feedback. If students can hand in their work whenever it suits them, it will suit many of them to prioritize other course work with due dates over coursework without. It’s reasonable to expect a glut of grading at the end of the term, and that means marathon grading sessions for which I am entirely too old at this point.

Reason 2. Students Have Time to Use Feedback

In the courses where students handed in their work at the last minute, there was very little time before the final exam to review the feedback I gave them. Who knows if they even had the mental wherewithal to process it, after focusing on the work so intently for so long? There are more useful things I can do than write detailed comments that no-one will ever read. Not that I’m guaranteed they’ll be read on assignments with due dates, but I’d lay better odds on it.

Reason 3. Academic Integrity

The more work students have to do in a short period of time, the more stress is involved, and the greater the likelihood that they cut corners. There were a number of reasons why I quit the no-due-dates teaching job, but a big one was that the courses were rendered pointless by cheating. The assignments submitted at the last minute were not infrequently copied from graded work that other students had submitted earlier.

Now imagine a desperate email from a student that you just reported for submitting someone else’s work, wherein the student pleads for leniency because they need the course to graduate, and their parents are already on a flight from overseas to see them get their degree. (In case you’re wondering, that student had an awkward conversation with their parents.)

Reason 4. Course Organization

The assignments in my courses are meant to reinforce particular concepts and skills. The assignments are most meaningful when I’ve just finished covering the relevant topics. The assignments are meant to sort out issues students had with those topics, and prepare them for the next thing on the list. Those kinds of assignments would have little value if students did not do them at the right time and in the right order.

In addition to impacting the relevance of assignments, no due dates would mean having to answer the same questions about assignments over and over again, because individual students would be focusing on them at different times. And as per Reason 3, I’d either have to not grade the work at all, wait until all assignments were submitted to grade the work, or just accept that some students won’t do their own work, and either disregard academic misconduct (not gonna happen) or spend time writing academic misconduct reports.

Getting that Breezy Feeling of Flexibility without Actually Going Commando*

*for due dates

If any of those reasons resonate with you, but you still feel the need to build in some flexibility, I’ve seen that done in a few ways, and have some of my own.

1. By the Book

In the syllabus, set out the scenarios under which a student will be permitted an extension. Be specific about whether you require an explanation, or some evidence that they need an extension. Be careful here about asking for personal information or making evidence burdensome. If you really want to help a student out, you don’t want them to not ask for an extension for fear of having to explain their mental health issue to you, or describe embarrassing or traumatic life circumstances.

Here is where we venture into another sore spot for pedagogy Twitter (the last time this conversation happened, people were bashing each other for a full week), the Great Asking for Granny’s Obituary Debate. On the one side were those tired of being lied to about grannies (metaphorically or otherwise) having become deceased as due dates approached. On the other were those whose metaphorical (or literal) grannies had died, and were furious and hurt about being treated as liars at one of the worst times of their lives.

My own approach to this is making a decision in the student’s favour unless I have specific evidence suggesting otherwise. My trust battery runs a little low most days, so this is the best way to sidestep the issue of honesty entirely. Aside from which, I prefer policies that work consistently, and a gut check isn’t one of those.

2. The Oprah Method*

*YOU get and extension! And YOU get an extension! EVERYONE gets an extension!

Some simply permit extensions to students who ask, whenever they ask, with no explanations required. But either you tell all students you’ll do that, or you risk the benefit only being used by students who feel confident questioning  due dates. The students who need the extension the most might never think to ask for it. Of course if you make that policy widely known, you have to be prepared for your grading schedule to go sideways, and depending on the size of your course, a snowstorm of extension requests.

3. Get Out of Jail Free Card

You could give all students a one-time due-date extension they can use at their discretion. If you do that, make sure you have a reasonable way to determine what is an appropriate length of extension.

4. Extension Account

A late penalty need not be a bad thing. If you want to emphasize the importance of sticking to due dates, but you also want some flexibility, a small late penalty (between 1% to 5% per day up to some maximum) that students could choose to take is one way to go.

You could combine this with the Get Out of Jail Free Card, or the By The Book method, or just give all students an “extension account” of so many points to go against deductions for lateness. This allows students to be in control of how they “spend” the time they have. I use something similar to account for points lost to technical difficulties when students are graded on participation in in-class response system questions.

So In Conclusion…

Before you abolish due dates, think about why you have due dates to begin with. You might be doing your students a favour by getting rid of them, but then again you might not. In my experience, the latter has been the case. To my mind, the better choice involves a consistent application of reasonable flexibility and accountability.

Historical Geology Timeline Activity

Some time ago I dreamed about the perfect tool for teaching historical geology. It would be an interactive timeline that students could update with events, and it would even have the ability to allow students to quiz themselves.

While that tool has yet to materialize, I did come up with a low-tech solution that permits the plotting of geological events that occur over widely varying timescales. I created four timeline forms to fit 8.5” x 11” paper. The forms cover different intervals and scales of geological time. They are:

  • Timeline 1: The Precambrian (4.5 billion years to 0.5 billion years ago
  • Timeline 2: The Paleozoic Era (540 million years to 250 million years ago)
  • Timeline 3: The Mesozoic and Cenozoic Eras (250 million years ago to present)
  • Timeline 4: The Cenozoic Era (last 10 million years of the Cenozoic Era)

The forms have divisions of the geological timescale along the bottom, and a numerical timescale across the top.

Timescale_forms

Click here to download timeline forms.

So here you go, but you should probably know…

Plotting geological events on the forms can be tricky. Some events have well-known dates, but some don’t. Events may occur in a geological instant, or take hundreds of millions of years. I provided a sample timeline to give students an idea of what to do in different cases, and how to express uncertainty.

Sample_timeline

An example of how to plot events

Even with those considerations handled, the students who used this exercise were looking up dates for themselves, and that added another layer of complexity. I prepared a handout to guide them through some additional challenges.

Challenge 1: Different sources or different pages in the textbook give more than one date for an event.

Expect that this will happen. Sometimes a source will be general (akin to saying World War 2 happened in the twentieth century), and sometimes it will have more specific information (it began in 1939). Ideally, the interval on the timeline should reflect the most specific information available (i.e., the most specific date or the narrowest range of dates). This makes it much easier to see how the timing of one event compares to the timing of another. We could plot the appearance of both the Ford Model A and the Ford Mustang as single bars covering the twentieth century. But if someone who had never seen or heard of an automobile before looked at our timeline, they would lack the context to understand that some evolution had taken place.

Keep in mind that geologists themselves may not know the exact dates of a particular event. The boundaries of the timescale can also shift if new information comes to light, so an older source might put the same event at a slightly different time than a new one.

In the end, students shouldn’t get hung up on finding exact dates (because mostly they won’t be able to), but they should be trying to get the dates in the right ballpark, and trying to make that ballpark as small as possible.

Challenge 2: The textbook or other source mentions a geological time interval that isn’t at the bottom of the form.

The forms have increments of the geologic timescale (eras, periods, and epochs in some cases) at the bottom, but these intervals are in fact subdivided into smaller ones. For example, the Devonian period is divided into Early, Middle, and Late epochs. The Late Devonian is divided into the Frasnian and Famennian ages. To figure out where a division belongs, consult the Geological Society of America’s Geologic Time Scale.

Challenge 3: The source says an event happened in the “upper” or “lower” of some interval.

When talking about the timescale, periods are sometimes divided into Early, Middle, and Late. These are official designations for the timescale. In contrast, when talking about the rocks themselves, a layer may be referred to as “Upper Devonian.” “Upper” and “Lower” refer to the position of a bed in a stack of rock layers. Older beds are lower down, and younger beds are higher up. While careful decisions have been made about the ways in which uppers and lowers fit into the timescale, a rough approximation for this exercise would be to treat “Upper” the same  as “Late,” and “Lower” the same way as “Early.” A less-rough approximation would be to look up the ages of the units in question.

 

Ways to use the timelines

My students used the timelines to plot events listed in assignments. Some students printed the sheets and wrote directly on them. Some students added to the timelines using drawing software, or using the mark-up tools within Adobe Acrobat. But if you wanted to go all out, you could print the forms out in mega-huge format, post them on the classroom wall, and stick on images or text to mark events.

Or you could turn them into PowerPoint backgrounds and have students build presentations on top of them.

Or you could make them SUPER-mega-huge in drawing software, decorate them all up with whats and whens, then print a poster.

Whatever you do, send pictures!

Student-Curated Video Collection: An Activity

AEG/Telefunken television from 1937. This was newfangled back when I started screening videos for this course. Eckhard Etzold, CC BY-SA 2.0

AEG/Telefunken television from 1937. This was newfangled back when I started screening videos for this course. Eckhard Etzold, CC BY-SA 2.0

I’ve been working on revisions to a distance-education physical geology course, and attempting to make it more interactive by offering videos. Have you ever tried to source relevant and accurate videos for multiple topics across multiple course modules? It involves going through hours and hours of videos, and rarely finding one that is directly on point or without problematic inaccuracies. My search technique has evolved to skipping anything longer than 5 minutes that doesn’t come with a transcript or clear description, and then screening the video at 1.5x speed.

So what to do about getting reliable videos without spending most of your adult life in the attempt… well, one school of thought would say let the students do it. I experimented with this kind of activity a few years back, but didn’t have an opportunity to deploy it full-scale. Here are the instructions I provided, with annotations. If you try it, let me know how it goes!

Curating Videos for Historical Geology

In this assignment you will assemble a collection of videos and complementary resources for historical geology students. You will work from the TED Ed* Lessons Worth Sharing video collection, Awesome Nature. This collection can be found at http://ed.ted.com/series/awesome-nature.

*I chose TED Ed because the videos are short. The student who did this moved on to TED Talks, which are  longer. I’d advise limiting the length of videos if you don’t want to spend hours watching videos in order to grade the results. If I were doing this today, I’d also recommend the fabulous video collection at MinuteEarth.

Your work will form the basis of a collection of resources to be made available to future students in Geology 109. If you wish, you will be acknowledged as the curator of the resources when they are posted, although I reserve the right to make any modifications that might be necessary to optimize the effectiveness of the collection.

Rationale

In the Independent Studies version of Geology 109, students do not have access to video lectures. Sometimes the textbook is unclear or written in too technical a fashion for students new to the topic to immediately understand what is being said. Videos designed by someone with a different perspective on the topic can be very helpful for reinforcing concepts, or clarifying points of confusion.

The problem is that not all videos are created equal. Some have factual errors, or even seek to mislead viewers. Some could benefit from clarifications. The task of looking for and vetting videos requires an understanding of the objectives a video should satisfy, and an assessment of how well the video accomplishes those goals. It also requires that viewers understand why they are watching the video and what they should get out of it. When an instructor looks for videos, he or she has an idea of what students find difficult, but it is really the students themselves who can most accurately identify where they need help, and what helps the most.

Your task

  1. Identify a video that satisfies one or more of the learning objectives for Geology 109. Provide the name of the video, and the link.
  2. Write an overview of the video. This should not simply restate the title of the video, but should summarize its contents in three or four sentences.
  3. List the learning objectives from the Geology 109 Course Guide that the video covers, and indicate which chapter they are from.
  4. Identify three key questions that the video answers. The questions should not be a restatement of the learning objectives, and should make it clear to other students why they would find the video useful. The questions will take the following form:
    1. Have you ever wondered …?
    2. Would you like to know how [something works or happens/ happened]?
    3. Have you ever been confused by …?
  5. Identify five terms that are technical in nature, and that are key to understanding the topic of the video. Define those terms in simple language, using your own words.
  6. Identify three “loose ends,” and explain the loose ends so that others watching the video will not be confused by them. The “loose ends” could be:
    1. Points that could be expanded upon
    2. Points that might leave some confusion in the minds of students watching the video
    3. Factual errors (hopefully there won’t be any of those)
    4. Points that are inconsistent with something in the course materials (e.g., competing hypotheses, more recent information, etc.)
  7. Write ten multiple choice questions so students can test their knowledge after watching the video. Supply the correct answers. The questions should cover key points. A good set of multiple choice questions will have the following characteristics:
    1. Four answer options (a through d)
    2. Little to no use of answer options like “all of the above” or “none of the above.”
    3. It should not be obvious to someone with no prior knowledge of the topic which is the correct answer. (Over-simplified questions are not helpful when trying to understand a topic.)
    4. Questions should be relevant to the topic of the video and to the learning objectives.
    5. After doing the questions, it should be clear to students what key points they have not understood.

Deliverables

You will write up each video following the layout supplied at the end of this document. This layout is designed to be compatible with the Blackboard system. The specific software you use to create the write-up is not important, nor is the font. (Blackboard has some formatting limitations, and formatting must be done within the Blackboard text editor, so this is something I will have to do afterward.)

Grading

Each write-up is worth up to 10 points. Those points will be calculated as follows:

  • Is the video relevant to Geology 109, and is the relevance clearly explained? (2.5 points)
  • Are all of the elements in points 1 through 7 above provided (e.g., the learning objectives, multiple choice questions, etc., are present)? (2.5 points)
  • Is the write-up scientifically accurate (e.g., definitions are correct, multiple choice answers are correct, etc.)? (5 points)

You may curate as many videos as you like*, however the maximum possible score for the assignment portion of the class will be 100%.

*This assignment was designed for a specific student. You may wish to rethink the “as many as you like” policy, or turn it into a group project to reduce the workload.

Format for submission

Square brackets mean text that you will insert. Text in italics are my notes, and don’t need to be included in your write-up.

[Video title]

[url]

 

Summary

[Three to four sentence summary of the video topic]

 

Why watch this video?

  • Have you ever wondered […]?
  • Would you like to know how [something works or happens/ happened]?
  • Have you ever been confused by […]?

 

This video addresses the following learning objectives for Geology 109:

  • [Learning objective], Chapter [chapter number]
  • [Learning objective], Chapter [chapter number]
  • [as many additional points as necessary]

 

Some key terms used in this video are:

[term 1]: [definition]

[term 2]: [definition]

[term 3]: [definition]

[term 4]: [definition]

[term 5]: [definition]

 

Special notes

  • [Loose end 1, explanation]
  • [Loose end 2, explanation]
  • [Loose end 3, explanation]

 

Note: these could take the form of, “In the video, [topic] is mentioned, but [concept] isn’t explained. Here is what it means,” or “The video says [this] about [topic], but in the textbook it says [that].   The difference is [reason].”

 

Self-test

[Questions 1 through 10]

 

[Solutions (e.g., 1a, 2b, 3d, …)]

 

Deadline

All write-ups must be submitted on or before Monday, March 30th 2015.

 

The Case for Being a Nitpicky Grader

axis

I’ve always had a sense of the educator I didn’t want to be. To this day I remember the prof who became annoyed with endless questions and finally huffed, “My five-year-old could get this!” Student me, though stumped, decided to try work it out on my own. If I couldn’t get it, it was a relatively small thing on which to take a hit when it came time for exams.

These days when I come across a topic that seems ridiculously simple, but students aren’t getting it, I try to get their input on what the topic looks like to someone new to the subject. I use that input to come up with a more effective strategy to tackle it. I’m not that prof.

I used to not be the nitpicky-grader prof either. You know the one- they took points off for the tiniest infraction, and you could never get it exactly right. I’ve had a change of heart on that one, though.

When a student made a small error on an assignment, I used to point out the error and explain the problem, but not take off points. It’s a minor error, right? They’ll do better next time. But regardless of how carefully I explained, the same errors would show up in the student’s work over and over again. Then I started taking off a half point for those kinds of errors. Guess what? Suddenly students decided those small details mattered.

I was somewhat taken aback that the only way to convince them to do it right was making it costly to do it wrong. Suddenly the student who was a chronic non-labeller of graph axes is producing clear labels with proper units. The student not bothering to spell technical terms correctly (I mean, c’mon you have spellcheck for your homework for dog’s sake!) suddenly learns the spelling. Importantly, those errors also disappear from exams.

The distinction between formative and summative assessment is that formative assessment is meant to be low stakes/ no stakes, and help students analyze their work to improve. Summative assessment is the higher stakes measurement of whether they’ve met course objectives or not. Formative assessment involves helpful hints, and summative assessment involves correct or incorrect.

But as it turns out, unless there is something at stake to distinguish “important” from “whatever,” formative assessment is “I’ll take it under advisement” assessment, and summative assessment is “it seems you neglected to do so when the stakes were much higher” assessment.

I wasn’t doing any favours by letting things slide in the hope students got the message, so now I’m that prof.

Clear As Fine-Grained Sediment Mixed With Water: A Discussion Forum

This week I’m presenting a poster at the Earth Educators’ Rendezvous. The poster is about a discussion forum activity that I do with my introductory physical geology students at St. Peter’s College. I’ve turned my poster into a blog post just in case anyone is thinking about trying a similar activity and would like to refer back to it. Alternatively, folks may simply want to confirm that some nut at an academic meeting designed a poster consisting largely of cartoons. Either way, here it is.Intro

Why

How

You can download a copy of the handout for this activity, including the rubric, here.

Examples.png

Strategies

This is a great resource from the University of Wisconsin-Stout for explaining online etiquette to students.

summary

Online Courses and The Problem That No-One Is Talking About

There are two kinds of online courses: those which are taught, and those which are facilitated. The distinction does not apply to the task of interacting with students. I’ve been both “teacher” and “facilitator,” and it’s exactly the same job from that perspective. The difference is one of autonomy, and it is a big difference.

The Gwenna Moss Centre is about to run another offering of their Introduction to Teaching Online course. Although I am a co-facilitator for this course, I would describe it as a course which is taught rather than facilitated. My co-co-facilitator and I discuss the course as it is running, and make adjustments on the fly when necessary. We take note of what worked and what didn’t, look at participants’ evaluations, and then modify the course as necessary for the next offering. Not only do we have the autonomy to make the necessary changes, it is expected that we will.

In Intro to Teaching Online, we assume that the participants will also be able to teach their online courses- that they will make pedagogical and logistical choices to respond to their students’ needs, and to make the course run as smoothly as possible. Also, that they will have the ability to revise as necessary and try new things. That’s how you teach an online course.

When you facilitate an online course, while you might take on the task of assisting students and grading their papers, what you can do beyond that is tightly restricted by a delivery model over which you have very little control. How little control will vary, but most likely it will be difficult or impossible to make substantive changes to what is taught, or how it is taught. Even if you designed the course in the first place, that “you” and facilitator you are completely different people as far as control over the course goes, and designer you lost any input as soon as the design contract was up.

If you are lucky enough to be able to request changes, the process is rather like having completed a painting, then being told you aren’t allowed to touch it anymore. If you want something to change, you must fill out a form describing in detail where the paint should go and how to move the brush. Someone more qualified than you will make the change. They might send a note back to you saying that they plan to improve your painting of a cow by adding spots. You must then explain at length that it is in fact a dog, and should not have spots. When the painting is finally modified, the dog is the wrong shade of brown. You decide it is best to not request modifications to your paintings in future.

Why does this matter? I don’t care how good you are- you never get a course exactly right the first time. If there aren’t any outright problems, then it soon becomes apparent where improvements can be made. Facilitator you gets to see the problems or areas for improvement, but must be content with grading papers and answering questions. If facilitator you is like facilitator me, this will drive you nuts. If facilitator you is subject to the same kinds of course evaluations as someone who can teach their course, and make it the best it can be, then this is not only unfair, but professionally dangerous.

While course quality is affected by this- especially if no-one sees a need to consult with facilitator you about how the course is going, or there are no mechanisms for facilitator you to communicate issues and be taken seriously- there is a bigger problem: the very integrity of the course.

At one time distance education was mostly intended to serve those who could not go to a brick-and-mortar institution for one reason or another. Maybe they had a family or a full-time job and couldn’t leave to go to school. Maybe they just couldn’t afford to move. Now things are different. While I don’t have any hard numbers, from what I can tell, at least 70% of my students are already taking classes at a brick-and-mortar school. They take an online class because they can fit it into their schedule better than one on campus, or it isn’t offered on campus at a time they need it, or they’re trying to get ahead/ complete their degrees over the summer.

What this means for the big picture is that students are far more likely to communicate with each other about the course than in the past. It might be two students who take the course together, or it could be someone who took it previously sharing information with someone currently enrolled. In the case that is causing me problems right now, a substantial number of students from one department at one school take the online course to fill a requirement. This is a facilitated course, so perhaps you can guess where this is going.

The students talk to each other. Some of it might be innocent enough, but some of it involves passing on assignments that I’ve graded to the next group of students who take the course. The course has not been updated substantively in some time, so the same assignments and exams still apply.

The problem has become ridiculous of late, with students submitting near-perfect assignments, all exactly alike plus or minus a few careless errors, and within record time. They get things right that no-one ever gets right. Clearly they are working together, but they are also referring to older assignments. I know this for certain for a few reasons: First, the correct answer will frequently appear after incomplete or even nonsensical work. They submit solutions with the answer that would have resulted if a typo, long since removed, was still in the question. They also plagiarize my comments from old assignments, sometimes reproducing them verbatim.

This course has a must-pass stipulation on the final exam. Normally that would be some comfort, because students who haven’t learned anything on the assignments would fail the exams. I’ve seen students with 95%, 99%, and 100% on assignments unable to break 20% on the final. (The exam isn’t that hard.) But over the past few months it has become apparent that the content of the exam has been shared. If not an actual copy, then a very good description of what it contains is in circulation. Exam grades have gone up, and students are regularly answering questions correctly which were rarely answered correctly in the past.

Ideally, if so many students who know each other are taking the course, the assignments should change frequently. In our hyper-connected world, it is almost certain that this kind of communication between students will happen. I even know of a homework-sharing website that has some of the solutions posted. The problem is that in order to change this, someone has to keep on top of the course full-time, and have the autonomy to make the necessary changes. The main consideration should not be the logistics of altering course materials. There’s no excuse for that when the relevant materials are or can be delivered online, and everyone and their dog knows how to upload a file to an LMS.

Nevertheless, the issue is that facilitators cannot be empowered in this way without disrupting the underlying structure of course delivery. Even more problematic is a culture amongst those who do run things- those who are not subject-matter experts but who handle the day-to-day operations- which views facilitators as incompetent, and unable to handle this responsibility. Not long ago I was handed an in-house guide to designing distance education courses. It warned readers at the outset that most faculty would be uncooperative and not understand how a distance education course should run. I felt ill, the way you would feel if you overheard your co-workers complaining about how useless you were. As I recycle that book I will contemplate with irony the damage this attitude has caused to distance education, and wonder if maybe I should take a chance and start the dog-washing business I’ve been thinking about.

There are many reasons to disempower facilitators, not the least of which is the cost savings from having them as casual workers instead of full-time ones. So here’s where I’m going to get in trouble for this post (if I haven’t already): if your concern is the bottom line, what happens when the ease with which students can cheat in your course makes other schools, employers, professional certification organizations, etc., decide that credit for your course is no longer meaningful? Even if cheating is less of a risk, what if word gets around that the course is hopelessly outdated or has problems? You don’t get enrollment, that’s what. And the people who communicate this aren’t going to be disgruntled facilitators. I’m the least of your worries. You need to worry about the students themselves who joke openly about cheating, and how little can be done about it, or who are discovered to lack skills or to have learning that is outdated.

There is a fundamental disconnect between what schools view as the appropriate way to structure a distance education program, and what actually works on the ground, when you’re expecting learning to happen. One involves online teaching and the other does not. There is a cultural gulf between those who have the power to do something about it, and those who can only look on in frustration. There are a lot of dogs to wash, but with most of them you have to spell out B-A-T-H rather than say the word, or they run off. A waterproof apron is useful, but not foolproof. You’ll need lots of towels.

The Levitating Wiener Standard of Formative Assessment

Formative assessment, or informative assessment, as I like to call it, is the kind of evaluation you use when it’s more important to provide someone with information on how to improve than it is to put a number next to a name. Formative assessment might or might not include a grade, but it will include thoughtful and actionable feedback. Formative assessment of teachers is no less important than formative assessment of learners- both are needed for the magic to happen.

I struggle with how to get truly useful formative feedback from my students. There are different instruments for evaluating teaching, including SEEQ (the Students’ Evaluation of Educational Quality), but the problem with the instruments I’ve used is that they don’t provide specific enough information. Sure, there is a place where students can write comments to supplement the boxes they’ve checked off elsewhere on the form, but those spaces are often left blank, and when they’re not blank, they don’t necessarily say anything actionable.

I’ve concluded that I need to design my own questionnaires. But when I get down to the business of writing questions, it feels like an impossible task to design a survey that will get at exactly what I want to know. I do have a pretty high standard, however: the levitating wiener.

The mentalist and magician Jose Ahonen performs a magic trick where he presents a levitating wiener to dogs. You can watch the videos How Dogs React to Levitating Wiener (parts 1 and 2) below. These are fascinating videos… have a look.

The dogs in the videos have one of three reactions:

  1. It’s a wiener! Gimme that wiener! These dogs react as one might expect, focusing on the existence of the wiener rather than on the fact that it is levitating.
  1. How the heck are you doing that? These dogs ignore the wiener and focus on the palms of Jose’s hands instead. It’s as though they’ve decided that it doesn’t make sense for a wiener to be levitating, and he must be doing it by holding strings. In other words, these dogs are trying to figure out how he’s doing the trick, and they all seem to have the same hypothesis. (Incidentally, it’s probably the first hypothesis most humans would come up with.)
  1. This is wrong… it’s just so wrong. These dogs watch for a moment and then get the heck out of there. Like the dogs in group 2 they also don’t think wieners should levitate, but they are too appalled by the violation of normality to formulate a hypothesis and investigate.

To my mind, most of the teaching assessment instruments are more like having the dogs fill out the questionnaire below than watching them interact with a levitating wiener.

Formative assessment for levitating wieners (loosely based on the SEEQ questionnarie)

Formative assessment for levitators of wieners

If the participants checked “agree” or “strongly agree” for “Weiners should not levitate,” it could mean something different for each dog. A dog from group 1 might object to having to snatch the wiener out of the air as opposed to having it handed to him. A dog from group 2 might think the question is asking about whether wieners are subject to gravity. A dog from group 3 might be expressing a grave concern about witchcraft. If the dogs wrote comments (we’re assuming literate doggies here), their comments might clarify the reasons behind their responses. Or they might just say there should be more wieners next time.

Now contrast the questionnaire with the experiment shown in the videos. Because of the experimental design, I learned things that I wouldn’t even have thought to ask about- I just assumed all dogs would react like group 1. I learned things the dogs themselves might never have written in their questionnaires. A dog from group 2 might not have noted his interest in the engineering problems surrounding hovering hot dogs in the “Additional comments” section. It might not have occurred to a dog from group 3 to mention that he was frightened by floating frankfurters. Maybe neither dog knew these things about himself until he encountered a levitating wiener for the first time.

A formative assessment tool that is up to the levitating wiener standard would tell me things I didn’t even consider asking about. It would tell me things that students might not even realize about their experience until they were asked.  Aside from hiring a magician, any suggestions?

Help for Students, Part 1: The Curse of the Unknown Unknowns

Students often ask whether I can offer any tips on preparing for and writing exams. Sometimes they are new students who haven’t developed study strategies yet, and sometimes they have just become frustrated with strategies that don’t seem to be working for them. Sometimes they are panicked and desperate, and end their emails with “HELP” followed by several exclamation points. (Never a good sign.) So I thought it might be time to jot these things down in one place, rather than writing them over and over again in emails to unhappy students who waited until it was too late to ask for help .

If there is one thing that causes more problems for students preparing for exams than any other, it would be the unknown unknowns:

“…as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns — the ones we don’t know we don’t know.”  Donald Rumsfeld, US Secretary of Defense, 12 Feb 2002

When studying, known knowns are the topics you are confident about, and which you are right to be confident about. Known unknowns are the deficits in your knowledge that you are aware of, and which you therefore have a chance to fix. Where you get in trouble, however, are the unknown unknowns—the deficits in your knowledge that you don’t realize exist. You can’t fix those because you don’t know they’re there. At least, you don’t know they’re there until you hit an exam question you didn’t realize you were unprepared for. Then they become known unknowns, but it’s too late to do anything about them.

Here are two examples of what a run-in with unknown unknowns can sound like. Unfortunately, I receive emails like this on a regular basis:

Sally:

“I realize I am not going to pass this course even with the 20+ hours I studied over the last week. I have trouble putting the definitions on paper. I remember reading them and seeing them but can’t remember the definition.”

 Bert:

“I felt as though I at least I completed the test and did not leave it blank, and felt confident that half my responses where right, but must have gotten confused.”

Note: “Bert” and “Sally” are not real students.

Sally’s unknown unknowns turned into known unknowns during the exam. In contrast, Bert emailed me because he was shocked that his exam grade was so low; Bert’s unknown unknowns were so sneaky that he got right through the exam without even noticing them.

The main problem that Bert and Sally had is that brains can be deceiving. In Sally’s case, after more than 20 hours of studying, everything looked familiar to her, and thus she believed she was ready for the exam. Unfortunately for Sally, the appearance of the page was what was familiar, not the information on it.

For both Sally and Bert it would have been a simple matter to set a trap for the unknown unknowns: if Sally and Bert had put their notes away every few minutes and tried to explain verbally or in writing what they had just read, they would have found very quickly that they couldn’t do it. Then they could have fixed the problem. Unfortunately, this is very hard work and should not be done for more than 45 minutes or so without taking a break. In Sally’s case, after many sustained hours of studying, she would likely have been too tired to manage it. She probably continued reading and not absorbing partly because she was too tired to do anything else.

Some of the sneakiest unknown unknowns hide so well that you might need someone else’s help to find them. Those are the kind where you remember information, but don’t realize that you have some part of it incorrect. The best way to trap these is to work with someone who might be able to pick up errors in your understanding as you explain the course material to them. This could be someone else in the class, or just a friend who asks you questions by referring to the textbook. Here are a few strategies that I’ve found helpful for turning unknown unknowns into known unknowns:

  • Scare them out into the open: Imagine that your instructor were to call you out of the blue to ask you questions about the course. What would you not want them to ask you about? Along the same lines, what would you not want to be asked about on the exam?
  • If a list of learning objectives is provided, treat them as questions and attempt to answer them without looking at your notes.
  • Reorganize information into diagrams and tables. For example, if you made a table to compare and contrast Neanderthals and Cro-Magnons, you might find that you can say something about Neanderthal body size, but you don’t remember how that compares to Cro-Magnon body size.   Diagrams and tables have the added benefit of being much easier to remember than lists of facts.
  • Study by explaining topics out loud to yourself or a friend. There is a difference between reading facts and trying to mentally organize them so you can say them out loud, and that difference can be enough to throw you off balance and expose unknown unknowns.

Why I Don’t Give Extra Credit Assignments

I view extra credit assignments as problematic because they can be unfair to other students in the course, they don’t necessarily solve the problem of missed learning outcomes, and they’re a hassle for me.

Let’s say I’m teaching a carpentry class called Potting Sheds 101. Students sign up to learn how to build potting sheds. Their final exam is building a potting shed. They may or may not go into the potting-shed building industry after graduation. On the last day of class the final projects are evaluated. Bob’s potting shed is out of square, and collapses when the door is opened. Bob fails. Later I receive the following email from Bob:

“Hey, how are u? I’m Bob in Potting Shed 101. I failed my final project. It’s been a really hard month for me, plus I didn’t have money to buy the textbook or a hammer. I found the final project did not suit my learning style, and was shocked at how difficult it was. Talk about being expected to run before even learning to walk! I will definitely be commenting about this in the course evaluation. Plus I was delayed getting started because I had to borrow a hammer from the library, and it was recalled and still hasn’t been returned yet. Potting Shed 101 is the last class I need for my degree, and I don’t plan to build potting sheds for a living, but I really need to pass the class to graduate. Is there some extra credit work I could do to pass the course with a high enough grade to get my degree? I feel I already learned a lot, and I would need at least 65 to graduate.”

Note that Bob’s email bears a striking resemblance to the types of emails I frequently receive.

So what should I do with Bob? Here are some considerations:

  • Bob knew he would need a hammer to build potting sheds. Other students made sure they had the supplies necessary before signing up. It is unfortunate that Bob doesn’t have a hammer, but does this justify extra credit work?
  • Bob should have expected that building a potting shed would be part of Potting Sheds 101, so I don’t accept his argument that the final evaluation was unreasonable.
  • Bob is suggesting that the class doesn’t mean anything to him, but is just a course that his program required for some reason, and that he won’t use the skills (although he still claims to have learned something).
  • Bob expects that whatever he will do for extra credit will get him at least 65% in the course, and can be done in time so that he will graduate as expected.
  • If I give Bob the opportunity for extra credit, are the other students any less deserving? Should they not be allowed extra credit projects too?

What if I cave in to Bob’s request? Bob suggests that he make ten bird houses for extra credit. Bird houses are not potting sheds, so he would be getting credit for doing a task that is much easier than the original task. Bob assumes that demonstrating a willingness to work hard is equivalent to demonstrating competency in potting-shed building. While a good work ethic is admirable, it is not the same as being able to build a potting shed. If Bob changes his mind about working in the potting shed industry, he will use the grade I gave him to convince an employer that he can build potting sheds. If Bob shows his grade in potting-shed building to prospective employers who don’t deal in potting sheds, they may take it as a sign that he is somewhat handy, has reasonable hand-eye coordination, and can handle complex tasks that require precision and attention to detail.

Let’s go one step further and assume I let Bob hand in his 10 bird houses. They are consistent with his skill at potting-shed building. Am I required to give him extra credit even though his work is substandard? If I don’t, must I allow him to do extra extra credit work?

What if the day after Bob hands in his 10 bird houses, Carrie sends me an email:

 “I heard you let Bob build bird houses for extra credit. Can I build bird houses for extra credit, too? I’d really like to improve my grade because I want to get into the Advanced Potting Sheds program.”  

This is a very competitive program, and if I let Carrie do the extra credit work, her grade would not reflect her skill at potting-shed building, but it would give her an advantage compared to other students who apply to the program.  Is that fair?

Then I hear from Marty:

“I heard you let Bob hand in bird houses for extra credit. I made some when I was in grade four. Can I hand those in for extra credit?”

If Marty has demonstrated the skill, does that not count? If he had brought a completed potting shed to class on the first day, should he have received credit for the course? Some would say yes.

Beatrice:

“I heard you were taking bird houses for extra credit. My neighbours have some. Can I get credit for those?”

I would have to explain to Beatrice that she must make the bird houses herself. She would then request step-by-step instructions on how to build a bird house, and ask if she could come to my office hours to get help.

On a box delivered to my front door, containing 20 bird houses with the “Made in China” stickers still attached:

“Here are my bird houses for extra credit. Thx. Pete”

In an email from the department head:

“WHY are you letting students build bird houses for credit in Potting Sheds 101? They’re supposed to be building POTTING SHEDS!”

You see, it’s just way too complicated.