Course and Instructor Evalutions
Course Philosophy and Structure
I was a pretty average undergraduate student - I crushed courses that interested me and got crushed by courses that were nothing more than hoops to jump on the way to a degree. End result . . . pretty much a four year blur and a 3.2 GPA (I remember that only because I needed it for a discount on my car insurance). In those days, I never imagined one day I would actually teach college courses.
Fast-forward a few years - alright forty (yikes) - I'm now teaching this class for the tenth time here at SUNY ESF and thoroughly enjoying it!
In fact, you could say that I've become something of a pedagogical geek (I've even learned how to pronounce "pedagogy"). In particular, I'm a true believer in the potential of the "flipped/hybrid" teaching model and can trace my conversion to it from a traditional Powerpoint/Prezi "sage on the stage" model to roughly twenty minutes I spent in early 2012 watching a random TED Talk by Sal Khan on YouTube (appropriately enough) entitled "Let's Use Video to Reinvent Education."
Possibly because of my past experience as a Cooperative Extension Agent (an informal adult educator) with Cornell Cooperative Extension and more than twenty years as a DYI television program producer/host, I instantly bought into the potential of video "lectures" to deliver course content, thus freeing in-class time for a variety of more hands-on/"active learning" experiences. To better understand the ideas behind active learning, here's a quick read by Cathy Davidson, "10 Key Points About Active Learning."
Within an hour of watching Khan's TED Talk, I had downloaded the software he used to create his "Khan Academy" videos and started to figure out how to go about making my own for this course. You can learn more about my video-production process by watching - what else - a video "Creating Video-Enhanced Lectures - Tools, Tips and Opportunities," I put together as a follow-up to a lunch and learn presentation I made to our faculty at the beginning of the 2013 academic year.
Six years - and A LOT of videos later - I believe my "flipped/hybrid" efforts have created a better learning experience for my students, at least anecdotally. However, I've had no hard data to support this belief as it's been like pulling teeth to get students to submit traditional end-of-the-semester evaluations (possibly because they're almost all seniors and have more important things - like the rest of their lives - on their mind as they hand in their final exams and prepare for graduation weekend). And, even if students were submitting end-of-the-semester evaluations, there is significant skepticism as to the value of information they provide.
Sigh . . . . . . .
To the Rescue (Sort Of) - Spring 2018 Mid-semester Course Assessment
While my fascination with the use of technology in teaching remained high, I still didn't know if the effort (and investment of time - evenings, weekends, holidays - and money - almost completely out of my own pocket) was making a difference in my student's learning? Then I came across an article, "How One University Encourages Innovation in Teaching," in the December 4, 2016 issue of "The Chronicle of Higher Education."
Halfway through the article there was a mention of how faculty at the University of Georgia can request a "mid-semester evaluation" of their course from the university's Center for Teaching and Learning. Intruiged, I sent an email to Dr. Lindsay Coco, Coordinator of Assessment and Faculty Services at the U of G's CTL - which led to this email thread and the materials and instructions for conducting a mid-semester course assessment.
While too late to pull the process together for the Spring 2017 iteration of this course, I did ask former ESF Landscape Architecture faculty member Tim Toland if he would be willing to conduct an evaluation during the Spring 2018 semester.
(Note: this was more than a year prior to ESF's Undergraduate Student Association's Resolution 02.1718, "A Call for Refinement and Adjustment Within the Course Evaluation Process at ESF," requesting that mid-semester evalutions of all courses become mandatory.)
The Process
On Monday, March 5, 2018 I introduced Tim to my class, then I left.
Tim first had each student spend about five minutes provide responses to these three questions.
Next, he had the students gather into groups of three or four to discuss their individual responses for five minutes, then complete the same three questions as a group.
Then, as a class, they discussed their individual and group responses for an additional five minutes or so then came to a consensus on what was working best for the entire class and what recommendations they have to make the class work better for them.
Finally, each student was given two pieces of tape to place next to the recommendation(s) they felt were most important.
This in-class part of the evaluation took roughly half an hour.
Next, Tim - very graciously - spent a couple of hours compiling the comments from the sixteen students in the class (i.e., reading their handwritten comments and typing them to strip any indentification from the comments). Finally, he spent an additional hour with me to discuss the results (I did buy him a cup of coffee and a pastry for his efforts). Therefore, the entire data collection and interpretation part of the evaluation required roughly a half day of faculty/instructor time.
The Results
At right is a picture Tim took of the top comments from the students as to what was working best for them in the class through the first half of the semester and recommendations for improvements that would make the class work even better for them through the remainder of the semester. (I realize you can't read the comments in this small image, so click here to open a full-sized image.)
For clarification of comments in the image:
- Good/going well comments are in no particular order. (The "x2" next to the "videos are good" comment denotes that two of the small groups came to a consensus on that comment).
- The pieces of blue tape represent a weighted student vote (each student had two pieces of tape that they could place next to improvements they felt were most important).
- There are only twenty-eight pieces of tape (not thirty-two) as two students were absent.
Also, in the spirit of full disclosure, click here to access the raw comments from which the class comments in the image were distilled. Note that the highlighted comments are those made by the small groups - which means those not highlighted were made by individual students. While I look forward to your interpretation of the comments, I offer mine below.
Good/Going Well
- (Most) students really do find the "lecture" videos to be helpful (if boring to some) for introduction to new content, review, and comprehension of in-class discussions (precisely as learning science literature suggests, whew . . . . . . .)
- Students appreciate the "conversational" and group discussion nature of the class sessions versus a more traditional lecture format. This format is made possible by delivering "content" by video (along with text readings) before class sessions which then allows time for group work during class.
- Besides the "lecture" videos, several students/groups indicated that they find the lab technique videos to be helpful in making the lab sessions more efficient. They can watch the video instructions before coming to the lab session, thus giving them more time to reflect upon and perform the exercises versus spending half the lab time watching the exercise(s) be performed, then struggling to remember what they saw and were told. If necessary, they can also rewatch the videos as they are performing the exercises (photo at right).
- Surprisingly, students find the (almost) daily low-stakes (ten point) quizzes to be helpful in encouraging them to keep up with the content introduced throughout the semester.
- Not surprising, students find "active learning" experiences - i.e., lab exercises, group discussion, field trips, etc. - to be both enjoyable and helpful in their learning.
- Also not surprising, simply being an enthusiatic teacher makes a difference in the student learning experience. Looking back on our own experiences, teachers that we found to be most impactful were those that were most engaged with the content and class they were teaching.
Recommendations for Improvements
Below I've listed the twelve recommendations in descending order of the number of votes received. After each recommendation I will indicate how I have/will address each concern.
(7 votes) Send email notices when content is added to the course website and/or changes are made to the website "Notices" page.
- An easy fix that I have already implemented.
(5 votes) Be consistent in assignments/quizzes (e.g., don't change a two question, ten minute in-class quiz into an assignment to be turned in during the next class meeting that I haven't planned/scheduled time to do).
- This is certainly reasonable and should be an easy fix. Unfortunately, within a couple of weeks of the course evaluation, I did it again.
(4 votes) Post videos, etc. further in advance of when due/needed.
- To an extent, this is an easy fix and has been implemented by providing a link to the master OneDrive folder containing all of the video files. In hindsight, I could/should have done this at the beginning of the semester, but decided instead to post links to video content, etc. as I have built out and gone "live" with each textbook chapter content page on the course website this semester. While not as helpful for the current class (the first to benefit from a course website), future classes will have a fully built out and functional website from the first day of the semester.
- On the other hand, it's not quite so easy as I continue to record, edit and post additional video content as my schedule allows. Therefore, there are times when the videos are simply not ready far in advance of when they're introduced to the class. However, I will do my best to meet this request going forward.
(3 votes) Use class time more efficiently (less down/idle time while waiting for classmates to complete quiz at the beginning of class).
- I could possibly move the quiz to the last ten minutes of class (or eliminate it)? I will need to reflect this recommendation.
(3 votes) Avoid grouping due dates so close together (especially at the end of the semester). Spread out assignments.
- This recommendation is problematic for this class. As I mentioned early on in the semester completing lab assignments (of which there are only two) is very much dependant upon how fast nature chooses to progress from year to year. The timing of the availability of some of the plant materials used in the course (e.g., some of the geophytes) is one limitation to scheduling assignment due dates as is the reality that it can take anywhere from four to eight weeks to obtain significant measureable results from many of the lab exercises in the course.
- I will continue to explore strategies for spreading the workload more evenly over the course of the semester. In fact, several years ago I did switch the bulb propagation lab exercise to much earlier in the semester as it often takes eight weeks to obtain results. I am also attempting to grow sufficient geophyte plant material (cannas, dahlias, caladiums, gladiolus, etc.) so scheduling that lab exercise is less dependent on the commercial availability of those species.
(2 votes) Add due dates section to the website.
- I believe the only due dates this recommendation refers to are for the two lab assignments? Again, these dates are quite dependant on the variable response of propagules from one year to the next. Dates for the three, one hour exams are always available the first day of the semester on the course website "Schedule" page and due dates associated with the capstone project are typically posted in early February.
(2 votes) Exams are unpredictable - provide a outline prior to exam to help prepare.
- I'm pretty sure that all the information students need in order to prepare for the exams can be found in their notes, in the assigned readings and in the lecture videos?
(2 votes) Don't change assignments mid-stream (don't change expectations/plans for work).
- I believe this was addressed above?
(0 votes) Give quiz after a brief in-class review.
- As per above, this could be addressed by moving the quiz to the end of each class session? Typically, I do ask if anyone has any questions before passing out quizzes. Rarely does anyone ask for any clarification. Regardless, I do have a couple of ideas for drawing out questions/clarifications prior to passing out quizzes, so I will give them a try during the coming weeks.
(0 votes) Use a "real" classroom for Monday and Wednesday meetings.
- Between 2009 and 2013, the "lecture" component for this class was held in Bray 330, Moon 19, and Illick 11 (through 2013 I taught the class in a traditional lecture format). While I absolutely recognize that Illick 530 is far from the ideal classroom, upon completion of the Illick greenhouse reconstruction project I decided to move the entire class to the current space so as to be as close as possible to the greenhouses. The reason for this is that I had found that if students were not "forced" to come to the 5th floor of Illick except for the Friday lab session, their projects tended to languish.
- Unfortunately, even holding all class meetings in Illick 530, I continue to notice that a number of lab projects suffer from a lack of attention.
- - - - - - - - resume editing here - - - - - - - -
(0 votes) Keep videos more focused with fewer tangents.
- "Just the facts," right?
- Did you know you can download the videos onto your machine and - depending upon the video player you use - you can speed up the video? In fact, there is research that suggests video lecture playback speed can be increased up to 1.5x normal speed with little impact on information retention (though test subjects did prefer normal playback speed)!
- Also, you really are in complete control of how you watch the videos. You can use the "scrubber" to move quickly through those parts of the video you feel are tangential to your specific learning needs!
(0 votes) Don't need to rehash prior video content (so omit).
- I'm pretty sure this comment refers to the two sessions I spent doing traditional "lectures" on the "cone of juvenility?" I actually did this intentionally to provide you with an opportunity to unconsciously compare/contrast your perception of content delivered via video "lectures" as compared to in-person lectures.
- My interpretation of this recommendation is that the perception is that the delivery of content via video lecture is as effective as that delivered by a traditional lecture - at least on that specific topic.
Discussion
Reflecting on the more than one hundred comments and recommendations this mid-semester evaluation process generated, I'd like to focus on what I interpret as overarching themes.
First, students confirmed - both individually and as a whole - that they perceive the delivery of online video "lecture" content to be as effective as a traditional lecture format. While this has been supported in learning science literature for many years, I previously could not confirm it to be true for this particular course.
It is also no surprise that students in this course appreciate and enjoy "active learning" opportunities - whether the activities are lab exercises, field trips or in-class group work/discussion. Delivery of lecture content and lab exercise instruction by means of online video frees up class time for a wide range of active learning activities that provide depth to student learning as compared to sitting still while being "lectured at" for several hours each week of the semester.
At this point I will also add that I collected, scanned and returned to students the first exam of the semester - ungraded. I then explain that I will grade their exam - and they will also grade their own exam. A week later, I met with each student for ten minutes to review their exams and we negotiated their score. I believe the appreciation for this effort was captured in the group comment, "we like the feedback from the exam review."
As effective as I believe exam self-grading to be as a learning technique, it is simply not practical to repeat throughout the semester.
As an alternative, and while possibly not as effective as a one-on-one review, I have briefly explored the use of video quiz/exam answer reviews in previous iterations of this course. Rather than take up valuable in-class time reviewing each question after returning a quiz or exam, students were asked to review the expected answers online. In class, we then focused on key pieces of information to be extracted from each assessment exercise.
Turning to things that are not currently working well for students in the class, I would first like to focus on two specific comments that are absolutely golden to me!
They are; "The alternative learning approach is great, but now I spend more time relearning how to learn than absorbing content (especially being a senior with many other responsibilities. . . . ,)" and "We are relearning how to learn which, as seniors, isn’t the most effective."
What???? These comments are critical of your teaching practices - not praising them! Obviously, it's all about how you choose to interpret them.
I believe few of us have really been taught how to actually "learn." Rather, since a very young age we've been taught how to take true/false, multiple choice, word-matching and short (one word or a couple of words) answer tests that are designed to be quick/efficient to grade - ideally by a machine! Over time, we learn how to "study" in a way that helps us do well on assessments that are quick/easy to grade - not designed to help us really "learn" to think, to question, to create!
The comments above suggest to me that the way this class is presented is challenging students to really "think about how they think" (known as "metacognition"), rather than simply going through the motions of preparing for another quiz/exam that's been designed to be easy to grade! And, this goes to my thoughts about grading in this class - where I encourage students to focus on "learning," not grades!
I completely agree with - and I believe, understand - this concern. And, while I may not be able to completely address it this semester, I believe I have a plan for addressing the issue of course organization for the benefit of future students in this class.
The plan is . . . . , this course website and more video content!
The week before this semester started, this website did not exist. I have devoted a tremendous amount of time this semester building the site (using the just-released CSS "Grid" specification that results in the site reformatting itself based on the device on which it is viewed) and thus barely - or not - staying ahead of the class as it unfolds. I have also recorded several dozen new lecture and lab instruction videos (each requiring between thirty minutes and an hour to edit and post).
All of the time I have committed to building out course infrastructure this semester is time I have typically devoted to course organization in previous years - allowing them to function relatively smoothly.
By the end of this semester, all of the key structural "bones" of this site will be in place (though over the coming summer I will be adding a picture gallery, a section devoted specifically to the lab portion of the course, and a "Showcase" section featuring student efforts). I will also have all of the foundational video content added to the site by the middle of this April - in time for use in preparing for the last two exams of the semester.
Therefore, while students in this year's class are, unfortunately, bearing the burden of this infrastructure effort, I'm hopeful that their patience - and feedback in the form of this mid-semester course evaluation will result in a significantly enhanced - and organized - student learning experience in the years to come.
Summation
When I introduced the idea of conducting a mid-semester course evaluation earlier on this page, I suggested it might "sort of" address questions I've had/have regarding how much my students are actually "learning" - i.e., how are their lifelong beliefs and behaviors encouraged, enhanced and supported by their experiences in this class?
Obviously, this course evaluation exercise can't peer far enough into the future to fully answer that question. In fact, in another recent article in the "Chronicle of Higher Education," "What Delayed Course Evaluation Might - and Might Not - Reveal," it's suggested that it can be very difficult to tease out the influence of a specific course on a former student's life - though "paychecks from students" do arrive - often when least expected.
So, I'll conclude this first-ever mid-semester evaluation of this course by sharing one such former student's "paycheck." It arrived in my email inbox on March 19th - right in the middle of this process.
Hi Terry,
Hope you are well and enjoying the spring semester!
Today marks the start of mine and Nicks fourth week here in Firebaugh, CA working at Lone Willow Ranch. The farm is awesome!! Lots of challenges, fun new experiences, and tons of delicious food. We are continuing to learn more and more about how an organic farming operation works, all of the planning and preparation that goes into a growing season/maintaining a yearly operation, and how biodynamics fits into everything. It has been a crazy month and things are only getting busier as we head into the summer!
Since we have been here though, I have gained a whole new appreciation for your plant propagation class. While I have enjoyed being able to try my own plant prop experiments at home since taking the class, it has been extremely satisfying to start applying those skills towards my career. You taught us SO MANY things that I use every day here at the farm, from different ways to get seed to germinate/how to read and understand seed labels/different soil media types and their uses/how to propagate different plants with various techniques/ the list goes on and on! It has really been satisfying to start working here while already having a firm understanding of greenhouse production/plant propagation from your class, and for that I am extremely grateful.
Needless to say, I have been speaking very highly of our course since I have been able to put those skills to use on the farm. That being said, Nick has been having difficulty understanding the basics of why we do what we do here, and although I have been able to explain some of the principles/techniques to him, I feel like he would really benefit from getting to watch some of your video lectures. If at all possible, I was wondering if you could somehow forward/give me access to some of your video lectures from the plant prop class so that Nick could take a look at them/I could get a refresher? I know both of us would really appreciate it. But also I understand that it’s your property and course content isn’t always able to be give out, so I totally understand if my request is too much. Or if you have any external sources to share, we are always trying to learn more!
Okay sorry for rambling so much :) Looking forward to hearing from you soon, and once again I hope all is well at the ESF greenhouses!
Very best,
Emily
You can read the entire email thread here, if you'd like.
What I would like to point out - as you might imagine - is a former student's assessment of the class and especially the usefulness of the video "lectures" I've created for this course years after her graduation from ESF!
Not only do current students believe video "lectures" are a helpful learning tool as documented in this mid-semester evaluation, one former student (at least - hopefully there are more:-) believes they're helpful enough that she's comfortable recommending that others watch them, too!
This suggests to me that this course has the potential to impact current and former student's lives well into the future.