Skip to ContentSkip to Navigation
E-learningOnderdeel van Rijksuniversiteit Groningen

E-learning

E-learningInnovation projectsFaculty E-learning projects 2017-2018

Perusall active learning template

Active learning template

This template describes integrating the use of Perusall into an existing course. First, the “Why” is covered, after that the process of constructive alignment is followed to make sure that course activities align with learning outcomes and assessment. This template covers the educational aspects of using Perusall, the section more info describes technical and licensing support.

Why it works

Perusall helps every student prepare for every class. Students carry out assignments to annotate texts before class for a part of the course credit. Thus, they are invited to read and think about your texts before the lecture. With prepared students, time during the lecture can be used for active learning: discussions, tasks and exercises.

Perusall is a social system. Students annotate together, so, studying becomes a social rather than solitary experience. Some students may want to be the first to ask questions based on a text; others may like answering questions or responding to fellow students more. The lecturer sees the annotations beforehand. This makes it possible to pick out good annotations from students who may normally be too shy to speak up in class. By rewarding these annotations it is possible to encourage more students than usual to participate in class discussions.

Scoring and grading are done automatically. As a lecturer you can review the scores, and check marginal students scores by hand, saving you a lot of time. At the same time, Perusall helps to make sense of the many annotations by providing a confusion report and overviews of most upvoted comments and questions. This helps you as a teacher to focus on the most important issues that students come up with.

Constructive alignment

With any course, the design is about alignment of teaching and learning activities, learning outcomes, and assessment (Biggs & Tang, 2011). This principle is called constructive alignment. The idea is that all course activities are directly aligned with the learning outcomes of a course. Further, the learning outcomes and the activities are aligned with the course assessment. This helps students to focus their attention on the important parts of the course and course materials. Students are often quite goal-directed: they spend their time on course tasks that are rewarding for them, which means it helps them pass their exams. For Perusall, this means that you should make sure that tasks are rewarded (e.g. via course credit, bonus exam question, or participation score), and that the relation of the task to the learning outcomes is clear.

In many courses, studying literature is an essential part of the course. Most of the time, recall of information is not enough. Defining learning outcomes using the learning taxonomy of Bloom and the associated “action verbs” are a way to plan this. What do you want your students to focus on with their annotations? You may want to ask students to compare the literature to other known approaches, evaluate the authors viewpoints, rephrase sources, or reflect based on an original text.

Students need to know what is expected of their annotations. On what level should they be? Via course evaluations of Perusall, we have seen that this is a learning process during the course. From the start, you as an instructor can give guiding principles (a reading guide and explanation about your learning outcomes). By giving feedback on the annotations that were done by the students, you can tweak their responses to make sure they get on the level where you want them to be. Consider picking out the best student examples, and capitalizing on them. (For example, you might call out in class particularly good questions, and use them as the basis for class discussion.)

action verbs to help writing learning outcomes
action verbs to help writing learning outcomes

Course activities

Perusall is based on weekly assignments that students complete before a lecture starts. Lecture time is used to discuss issues around the course content. The assignments are made for points (course credits), or rewarded in some other way (for example, as a bonus exam question).

Without some form of reward, our experience is that students do not carry out the task. The weekly assignments consist of reading one or more the texts and making annotations based on this text. Annotations consist of questions based on the text or responses to the other student questions. Perusall grades the student work automatically, and provides an overview of all annotations in a confusion report, providing advice to the lecturer on what points are interesting for discussion.

The lecturer reviews the annotations, checks and releases student grades, and organises discussions. Make sure to plan enough time for this; typically, we advise starting no later than an afternoon before the lecture starts, and setting aside two hours of review time for two hours of lecture time. With more experience, about one hour of review time is needed for two hours of lecture time. To plan the lecture, first take some time to think back on why this text is included in your course, then use the confusion report, browse through the most active discussion threads, and review the most highly upvoted questions and discussions.

Active workflow

Perusall is designed to support typical course activities following a weekly schedule:

workflow
workflow

Based on your particular course requirements, variations on this template are possible. Many styles of courses have been supported. Some examples: Courses have been set up following a two-day schedule, students have been tasked with organizing in-class discussions themselves, a large group of students (over 200) have annotated on a poem, and in one particular course students self-produced texts that were annotated. With this setup, we can ensure that Perusall works to help students prepare for class.

Feedback on Perusall assigments to students

There are several steps to take when giving feedback, after students have finished Perusall assignments.

1. Think of what you want to see from your students

As a lecturer, first make a list of the most important topics that you think should be addressed. What do you need or want to see from students in order to reach the learning objectives? On what pages would you expect many annotations? Which steps would you like them to follow?

2. Generate confusion report

A confusion report is a summary of the questions of students. When the deadline for your assignment has passed, and student annotations are gathered, click on the assignment, then click on “Confusion report” to generate the report. Perusall needs around 20 unanswered questions to generate the confusion report.
Check whether or not you agree with the topics as mentioned in the confusion report.
In the confusion report you can click on the little icon next to the student comments to copy a comment over to your Powerpoint presentation.  

comment-powerpoint-presentation
comment-powerpoint-presentation

3. Check the analytics: how are my students doing?

The analytics section in Perusall gives you data about how the students are doing with respect your assignment, allowing you to identify students that struggle with the task and may need some extra help. To this aim, there are four parts in the analytics report:

  • Grade distribution: the image shows a typical distribution of grades; when students do the work they pass the assignment
distribution-of-grades
distribution-of-grades
  • Annotation submission time heat map: giving you an indication of when students did the assignment (typically: just before the deadline).
  • Page view report: showing how many times students viewed the page, and how much time they spent on the page. Check whether or not you can see a pattern like in the image below. Students start at the beginning of the text, run out of steam (or have done the required number of annotations) and miss the conclusions on page 20. This would be a good start for a discussion with the students on how they are reading a text, and a reminder for them to read the summary and conclusions.

analytics
analytics

  • Student activity report: The student activity report gives the total reading time of students, and the active reading time. Active reading time gives an indication of the time students were active by writing annotations, responding to other student annotations, mouse clicks, and scrolling the page (this is measured in a two minute timeframe). You can browse through the list, and see how much time your students have spent on the assignment. For small courses you can check which students were unsuccessful and see whether they have put the time and effort in. Try to talk with these students on how they approached this task, and what they can do differently to get the full points.

Perusall takes note of what students think of each others work. Students can “upvote” (i.e. click on the +1 button next to a comment), and indicate if they want to know a question too (by clicking on the question mark sign).  In the last column of the list you can see which students have the most upvoted annotations and questions. You can make a note to compliment the students that got a lot of upvotes on their annotations, and check whether these annotations are useful for further discussion in class. Further, you can check to most upvoted questions for further discussion in class.

4. Check discussions in Perusall

Go to the screen on Perusall where you see the text and the number of Perusall comments (see figure), check whether you have missed big discussion topics (the number behind the conversation indicates the amount of annotations):
check-discusion
check-discusion

The topics that you want to discuss, the topics, from the confusion report and the topics you have identified from the annotations together make a list of topics to address during your face-to-face lecture. Clicking on the link icon under the students comments will copy the link to the discussion to the clipboard. You can copy this into your Powerpoint presentation. When you are logged into Perusall clicking the link will jump straight to the discussion in Perusall.

check-discussion-screen
check-discussion-screen

5. Decide on in-lecture activities to address topics

Per topic: decide what will be the best way to address this. Think about how to make your students active. Let them find the answers. Organize a discussion, pass the question on to peer-students. Reward students for good annotations, but praising them (“that was a good comments, but…”). Maybe the topic goes beyond the content of your course, you could point students on to a follow-up course where this is addressed, or give them additional literature (it can also be a cliffhanger). Also, you can still give short presentations (Powerpoint) to explain some content to your students.

Students do not always wish to speak up in a group. A good in-between activity is to organise small group discussions first, then discuss the results with the whole group. You can ask for a spokesperson per group to share discussion results. Youtube has great timer videos to keep students at task.

Many more ideas to activate your lectures:
https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/educational-technologies/all/activities-large-classes

Help for organising discussions in class:
http://www.srri.umass.edu/sites/srri/files/beatty-2006deq.pdf

6. Follow-up in class

In class, show Perusall on the projector, and jump within discussions. Decide on whether or not you want students to join in Perusall during the lecture. Laptop screens may be a distraction. On the other hand, it can be good for students to check things back, and keep adding to the discussion during the lecture.

Testing and assessment

To use Perusall in your course, it is recommended to reward its use by giving points for it, counting towards the total course mark. Students spend a significant amount of time working in Perusall, so it is fair they are rewarded. It needs some thought to make this work for your particular course.

Perusal grading within a course

First, check your course description in Ocasys to see what you have promised the students. It is important that you have written in Ocasys that students are graded on “assignments”, or better, that the Perusall assignments are described, since Ocasys is legally binding.

Typically, a very small fraction of the total points for a course is enough to invite students to do their work in Perusall. The system is set up to reward studying, so, when students make an effort to do their annotations, it is easy to reach the full points for it. The Perusall part of the grade is typically arranged to reward students for inputs -- their effort and engagement -- rather than differentiate between students based on their learning outcomes, which will be handled by your exams, papers, etc. Ideally every student would get full credit for the Perusall part of their course grade!

course
course

For your course manual you can make a table similar to this:

Component

Weight

Written exam (Digital Exam)

50 %

Literature review

25 %

Literature review outline and reading list

5 %

Perusall assignments

10 %

Participation

10 %

Total

100 %

In this case, Perusall assignments count for 10 % of the course grade. With 5 assignments, they would count for 2 % each.

We suggest that you provide students with two documents relating to Perusall. First, you can download and modify a one-page rubric that explains what Perusall is, how it works, and how students are graded. (You may need to edit this document to reflect the scoring settings you have set for your course.) You can also provide students with a set of example annotations with associated quality scores and an explanation for each score, to help them get a feel for what sorts of comments and questions they should be posting.
Feedback to students in class focuses on picking out discussion points, and picking out good examples of student annotations. You can explain why you think this example is particularly good. Thus, you are giving feedback on level of student annotations (see the section about learning outcomes).

Automatic calculation of student scores

Based on the criteria of Perusall , students are graded on quantity, timeliness, distribution and quality. Based on these criteria, individual scores for students are calculated.

In practice, when students make an effort, they will receive full credit for the Perusall assignments. We advise only reviewing student scores that are marginal (say 4 to 6 out of 10). Scores below this number will be generally very poor; scores above this number will be fine. You can manually change the overall student score, or change the points given for a particular annotation.

In the example this particular student has received 4 out of 10 points. Of the required 7 annotations, 4 of them were submitted on time, the average quality of the annotations.

picture-of-example
picture-of-example

was 1 (out of a scale of 0, 1, 2), and this particular student has lost points on distribution, by not spreading the annotations over the text evenly. (Perusall makes this latter deduction to encourage students to engage with the full text, not just the first few pages.) This leads to an overall score of 4 out of 10 points.

In the gradebook for the overall course, you can see the overview of the student grades per assignment. Typically, though feedback on their annotations, students learn what kind of annotations are expected, and thus their scores gradually increase over time.

annotations
annotations

How does automatic scoring work?

The quality of annotations is determined by a machine learning algorithm. Annotations are scored by a set of defining features. Via a process of repeatedly feeding the system data, and comparing it to data that is scored the weight (importance) of the defining features is determined.

To give an idea how this works we can describe how Perusall was trained to deal with texts in Dutch. Three lecturers were asked to score a total of 1000 annotations on the same text. After scoring the first 100 annotations, they met up to discuss the differences among their grading, to reach a better inter-rater reliability. The algorithm was changed to reflect the agreement between the lecturers, after which the lecturers were asked to do another set of 100 annotations. This was done in 10 sets, hereby training the algorithm. So, basically, we told the system what we think are good annotations, and the system learned what are defining features of our “good annotations”.

Demo: How does it work

A demo can be found at:
https://app.perusall.com/demo

More info

Technical support and a manual for Perusall can be found at:

Education support:

Literature licensing:

Sander Sprik

References

Biggs, J. and Tang, C. (2011). Teaching for Quality Learning at University. Maidenhead: McGraw-Hill.

Miller, K., Lukoff, B. King, G. & Mazur, E. (2018). Use of a Social Annotation Platform for Pre-Class Reading Assignments in a Flipped Introductory Physics Class. Frontiers in Higher Education. https://doi.org/10.3389/feduc.2018.00008

PDF version

Laatst gewijzigd:14 december 2018 11:40
Volg ons opyoutube twitter linkedin