A method of frequent low-stakes assessments was introduced by lecturer Kristian Peters in the 2nd BSc course “Research Design”. Reasons for this was to activate students during the course, offer an alternative for small group tutorials and find a way to make sure students visit the on campus lectures. Weekly mini-quizzes and group assignments together made 50% of the course grade. To automatically assess certain aspects of student work Perusall was used. 

A big technical problem with making groups in Perusall caused a lot of stress during the course, and the time investment of the lecturer was high. Still, evaluation data shows that a combination of blended learning and frequent low-stakes assessments made students active during the course without the need of having dozens of small group sessions. The setup of this course is especially suitable for large-group courses in which students have relatively low levels of motivation and expert teachers are essential (instead of student assistants).

Why?

Please tell us about the motivations behind the practice. What issues and challenges did you identify that inspired you to develop this practice?

There were two main reasons to redesign this course. The first was to use the students' extrinsic motivations to help them form good study habits. This was done in the form of multiple small weekly assessments that gave a percentage of the points in the course. The second was to offer an alternative to small group tutorials, that are usually done face-to-face and for a large part under the supervision of student assistants. However, the benefits of small group tutorials are highly dependent on the knowledge and experience of the teachers, especially in a course about research design and methodology, which student assistants generally lack. But raising a small army of experienced senior teachers is practically impossible for so many students and tutorials. Therefore the question was: How to set up a course in which students are actively learning with support from real experts?

To achieve this I was looking for a new (blended) way to teach students and activate students to create operational and didactic benefits. I carried out this project as part of my Senior Teaching Qualification (STQ).

Overview

Please provide an overview of the teaching innovation that you have developed. What are the main goals and objectives?

Just before the first COVID period, at the end of 2019, I re-designed the course on behalf of the programme director of Bedrijfskunde. This initial course design was already inspired by blended learning (BL) principles in which online videos and graded mini quizzes, offered prior to the lectures, played an important role. The design itself was inspired by the paper of Finne (2018). 

In 2022-2023 I intensified the original set-up. Besides the weekly graded online mini quizzes that we already used, students would earn another part of their assignment grade with the weekly group assignments. My expectation was that this would keep students even more active during the entire course period, which ideally would lead to better results on the exam. In between the graded online mini quiz on Monday and the group assignment with a deadline on Friday, I gave students the opportunity to make an exercise assignment for a bonus as a try-out for the group assignment. To prepare for the online mini quiz and the exercise assignment, students should study one or more chapters from the book and watch pre-recorded videos. After the miniquiz and exercise assignment an on-campus lecture was given. I reasoned that the reward for students for doing a lot of work during the course period must be considerable, so I set the assignment grade part of the final grade to 50%. And the assignment grade was 50% mini quiz grade + 50% group assignment grade + max. 1 point bonus for the exercise assignments. Still students needed to score a 5.5 on the exam (50% of the final grade) in order to be eligible for passing the course.

Process

If applicable, please briefly describe the process that you went through to design, test, and implement the teaching innovation.

I made an interactive timeline, from getting and developing the ideas for the  2022-2023 version of the course and all major events and experiences during the course, which can be visited here: Blended & active learning in Research Design

Evidence informed

How is your innovation based on existing studies, or have you conducted any study to support your practice?

The design of the course was motivated by Finne (2018), who describes the challenge of teaching large classes as a professional service operations challenge, and came up with ways to reduce labour intensity and improve customisation. After carrying out the course, it was evaluated. The standard course evaluation was carried out, but also an extra questionnaire was sent out, and a learning analytics analysis was conducted to look into how the materials of the course were used, and whether this had an effect on overall grades.

Programme alignment

How does your innovation align with the course/programme ILOs?

The course Research Design covers a very big chunk of the “learning line” on methodology in the BSc Business Administration programme (see the green elements in the picture; the picture represents the entire learning line). 

It brings together lessons learnt in all previous methods courses (the bottom of the house of methodology in Bedrijfskunde), and on top of that the course offers new materials related to philosophy of science and research ethics, and it introduces three additional methodologies: case study research, experimental research, and design science research. Besides me, three colleagues were involved in giving the various topics.

Engagement and active learning

How does your teaching innovation promote student engagement and active learning?

I aimed for a constructive blended learning set-up: the book & pre-recorded videos provided the basics in each week, tested by the mini quiz for a grade, and further tested in the exercise assignment for a bonus. The answers of the exercise assignment were discussed during the on campus lecture only. Furthermore, during the on campus lecture I introduced a few new and more in-depth lessons related to that week’s topic, which was possible because students have studied the basics already thanks to the graded mini quiz and exercise assignment. These in-depth lessons were also added to the weekly assignment, which gives students another reason – besides hearing the answers to the exercise assignment - to come to the weekly on-campus lecture. In addition, I used recent articles for the assignments with interesting topics, well-accessible to 2nd year BSc students for the assignments. Of course these articles had to be suitable for the methodological contents of the course. Finding suitable articles was a challenge indeed. 

The weekly exercise and group assignments did introduce a few challenges:

• I needed to find an efficient way to give bonuses to students who completed the exercise assignments in a reasonable way (i.e., not graded on content per se)

• Also for the group assignment I was looking for an efficient way to grade the students’ effort next to the grading of content done by teaching assistants, in order to prevent students from letting other group members do all the work

• With smaller weekly assignments, it is probably easier to copy each other’s work without us noticing if students handed in a Word file with answers

• In 2022 I became aware of ChatGPT, so I was thinking of ways to make the assignment more ChatGPT-proof

• I would also like to find an environment in which students could collaborate to answer the assignment questions about existing scientific articles or theses

The solution to these challenges seemed to be Perusall. Perusall offered a way to automatically assess how serious students worked on the exercise assignment for a bonus. For the group assignment for a grade, Perusall could be used to monitor how individual students worked on the assignment and identify those who would profit from team members. Also with Perusall we could grade the location of an answer in the text (with an annotation), which gives another indication whether students really understood the answer. Also it makes it harder just to copy-paste answers from others or when using ChatGPT. And finally, in groups of 3, students could work together in the same document in Perusall online. As another measure to make the use of ChatGPT more difficult I opted to use very recent articles only (articles that ChatGPT has not read yet).

I discussed the idea with a colleague who had experience with Perusall and we came to the conclusion that this not-so-typical setup (because of the small group size) should be no problem for Perusall.

Challenges

What challenges have you encountered in implementing your teaching innovation? How did you address them, and what would you do differently next time?

At the beginning of the second week of the course, I started to receive emails from students who could not see their group member annotations in Perusall anymore. Instead they saw annotations from other students, or no annotations from any team members at all. As the assignments had to be made in groups of three, and students naturally divided the questions in the assignment among group members, this was extremely problematic. At first I hoped it was a temporary glitch for a few students, but many more emails piled up in my inbox. Also I could see in the freezing Perusall settings screen (yes, another problem) that Perusall randomly placed students in a different group than was assigned in Brightspace or in no group at all. Luckily this was the exercise assignment for a bonus only, with little consequences for the assignment grade. But the first graded group assignment would open the next day. 

I raised the alarm to code red. I reported the problem to ESI. Given the severity for the course, I escalated the problem to the programme director as well. Soon the news reached the dean of education and director of education, who urged ESI and other edu-IT support staff to solve it. The support troops were mobilised fast, and were very willing to help, but could not find the exact cause of the problem during the course period, nor find a solution… 

I had to come up with a new plan on the spot. The first exercise assignment had to be a pass for everyone who logged in to Perusall. That was not ideal, but it was only for the bonus. For the subsequent exercise assignments I found an alternative way to use Perusall without groups. I suspected that the problems were caused by the way how Perusall and Brightspace interchange information about students and groups (the integration of Perusall with Brightspace is relatively new, and also I use a lot of small groups (150+) in comparison to how Perusall is usually used (with fewer, yet bigger groups); these two factors may also be the reason why no one experienced the problem before).  For the exercise assignment I created a new course in Perusall, which students could access via the old, broken course page. For the remainder of the course students made the exercise assignments individually to avoid any more problems. In this sense I only used Perusall to award individual bonuses semi-automatically.

For the upcoming group assignments I changed to the traditional method of handing in work in a Word document via Brightspace. This, unfortunately, made it impossible to check whether students used the correct parts of the texts for their answers and it made copy-pasting each other's work or using ChatGPT a lot easier. I could also not check if every group member contributed.

Evaluation

How have you evaluated or done research on your practice? Have you gotten any feedback from your students and/or colleagues?

Only 17 students out of approx. 450 filled in the regular course evaluation. The course obtained a 3.7 out of 5 as total score (in 2021-2022 this was 3.9), which for a BSc course on methodology is decent. I added three additional custom questions (the maximum) and from those questions I could learn that 65% found the BL + AL setup positive, and 35% negative, with no students in the neutral zone. It seems that students either love it or hate it. 

To learn more, especially about the blended learning and frequent low-stakes assessment setup and how students actually make decisions about their study behaviour, I created an additional evaluation that was filled in by 51 students after writing to all students directly via email (almost 2,5 times higher response rate than the regular course evaluation). Three things stood out:

1. Frequent low-stakes assessments motivate students when it comes to studying methodology, and the respondents clearly prefer it over mandatory on campus work sessions (see bart chart in picture).

2. The increased intensity due to frequent low-stakes assessments was no problem for the majority of respondents, but like in the regular course evaluation there were a few students that really hated it.

3. The respondents indicated they would voluntarily join on campus activities if it has added value for them. If they can get a reasonable grade on the assignments and exam without going to classes, they won’t go. Another reason that was mentioned a few times is the social aspect of going to campus. 

Finally, I hoped to find a relationship between the group assignment grade and the exam grade. My expectation was that students who did well on the assignments also performed well in the exam, because the type of questions in the assignments were very representative for the exam, but the data did not show a clear relationship (see scatterplot below). There are several explanations. One is that students who scored a decent grade on the assignments did not prepare for the exam that well, as they did not need a (very) good grade on the exam to pass the course. This was also supported with data from the additional evaluation. Another explanation is that scoring a good assignment grade as a group does not mean that each individual group member is equally well trained, because students divide parts of the assignment and don’t really check each other's work nor learn from each other.

Lessons learnt

An abundance of lessons can be learnt from this project. I will share the top 4.

1. Based on an analysis of the grades it was not possible to find an effect of my approach on the final grades, but there were plenty of other signs in the learning analytics that the combination of blended learning and frequent low-stakes assessments made students more active during the course: mini quizzes and assignments were made by almost all students and almost every week, students showed up at the lectures (especially in the 1st half of the course), and online videos were being used a lot. But be mindful of the time investment needed to prepare and execute this set-up. And be careful with offering "escape" options for certain tests or assignments (e.g., students could miss one of the 5 mini quizzes, one of the 6 group assignments and two of the 6 exercise assignments). Because many students will make use of these escapes ultimately (especially the last assignment was poorly made), if they are not ill or absent due to another important reason, it can potentially do more harm than good. 

2. Whereas the project zoomed in on doing things differently with regard to assessment methods and teaching methods, it ultimately also led me to questioning the starting points of the course, i.e. the learning goals and the levels we want to achieve. For example, lecture attendance was promising at the start, but declined in the second half of the course quite drastically. Students probably sensed after a while that they would be able to pass the course without attending any more lectures, and that via self-study and the assignments they would be able to pass the course. When looking at the passing rates they were probably right.  Improving or changing assessment methods and teaching methods with a sub-optimal set of learning goals or achievement levels that are too easy or too hard will likely generate any improvement less effective or not effective at all.

3. Especially for innovative courses the current course evaluation system does not work well. One the one hand, course evaluation response rates are at an all time low in the BSc programmes and, on the other hand, the sparse results that reach us give very little insight on how students actually learn, academically behave and change their behaviour due to new assessment or teaching methods such as applied in my course. That was for me also the reason to create an additional evaluation with other types of questions and approach students myself. 

4. Despite the strategic importance, blended learning is still a concept that is poorly understood among colleagues. Many (still) see blended learning as "lecturing fully online", as we had to do in the COVID period, which is deemed a bad teaching method and also bad for students' wellbeing. What did not really help in my view was the memo of our faculty board about the importance of on-campus education in the summer 2022. Some colleagues point out to this memo to explain to me that the use of online videos is banned (yet when reading the memo carefully, one would see videos could still be used without a problem in a blended learning setup). 

The attached presentation Learning Analytics onderzoeksontwerp 22-23 gives more insight into the use of videos, lecture attendance, mini quiz and assignment results and a comparison of assignment and exam grades.

Scaling up

Can your practice be scaled up and/or modified for other educational contexts? (other courses and programmes)

The setup of this course is especially suitable for large-group courses in which students have relatively low levels of motivation and expert teachers are essential (instead of student assistants). It is however not advisable to use this approach in every course, especially within the same block. A portfolio of blended learning courses, blended "light" courses and traditional courses will have most didactic advantages and will also help students to manage their study time in new ways. Programmes need to start monitoring what is already done in relation to blended learning, for instance, which TEL-tools are used throughout the programme.

Sharing

Have you shared your practice/experience with other lecturers? If yes, how? (in an event, conference, paper, platform, etc.)

This practice was developed and shared via the Senior Teaching Qualification (STQ) programme.

Contact

Who is the primary contact person for this practice and how do they wish to be contacted?

Kristan Peters : k.peters@rug.nl

More resources about activating and motivating large student groups

Articles:

• Finne, M. (2018). Improving university teaching: a professional service operation perspective. International Journal of Operations & Production Management, 38(9), 1765-1795.