Double marking and moderation

A guide to setting up double marking and moderation

Overview

Double marking involves two or more markers marking the same exam submission and comparing results to reach a consensus. 

The process involves assigning multiple markers to an exam sitting and nominating a moderator who will choose the final score given to the student. 

Video tutorial  

Different user roles

When using the double marking and moderation feature, a user with marking permissions can play four different roles.

Assigned marker
An assigned marker is a user with marking permissions assigned as a marker on the exam sitting. They will have access to mark all exam submissions for that exam. They will only be able to see their scores and NOT the scores of other markers. The status of each submission will reflect their marking progress only. For more information on submission statuses, see Submission statuses below 

Moderator 
A moderator is a user with marking permissions assigned as the moderator on the exam sitting. The job of the moderator is to review the scores given by markers and choose the final score given to the student. They can see all scores, feedback, and annotations given by each marker. The moderator will see the status for each marker and their moderation status for each submission. For more information on submission statuses, see Submission statuses below 

Administrator 
The administrator is a user with administrator permission on the specific course (Modify course settings). The administrator will always have the ability to override the final score. They will also be able to see the scores given by each marker. The admin will see the status for each marker and the moderation status for each submission. For more information on submission statuses, see Submission statuses below 

Non-admin user 
The non-admin user is a user who is assigned to a course with exam marking permissions but is not an administrator or is not an assigned marker or moderator. They will be able to see all scores given by the markers and moderator but do not have permission to edit any scores. They will see the status for each marker and the moderation status for each submission. For more information on submission statuses, see Submission statuses below 

Step-by-step instructions: How to setup double marking and moderation 

Note - Only those with permission to manage course settings can configure double marking and moderation settings. 

1. In order to setup double marking on an exam sitting, you must first create an exam and then schedule an exam sitting.

2. Once you're exam is built and your sitting is scheduled, open the sitting and go to the 'Marking options' tab 

3. Enable 'Edit Mode' and enable the 'Assign markers' option. Select the 'Multiple markers per submission (Double+ marking)' option. This option should be selected by default when the setting is first enabled. 

4. You will see two sections; one to assign markers and one to assign a moderator. To begin, select the 'Add markers' button to assign markers. 

5. Choose your first marker from the dropdown list. When you select the marker from the list, their alias will default to their first and last name. 

The marker alias is the name that will be shown to the moderator and used to identify the marker.

If you do not want the moderator to know the identity of the marker, you can update this alias. For example, you can add the alias 'Marker 1'. 

6. To add a second marker, select 'Add another marker' and select a second marker. Remember to update the alias if you want to hide the identity of the marker. 

7. You can add more than 2 markers if desired. Once you have added your markers, select 'Confirm'.

8. You will see the assigned markers appear in a table. The next step is to assign a moderator who will decide the final score for each question.  Select 'Assign a moderator' at the bottom of the tab. 

9. Users assigned as markers cannot be assigned as moderators. To choose the moderator, select a user from the dropdown and click 'Confirm'

10. You will now see the selected moderator appear in a table under moderation settings. 

11. The final step is to save the changes you have made.  Select 'Save' in the top right-hand corner to finish setting up double marking and moderation. 

Submissions list 

Assigned marker

When one of the assigned markers is viewing the submissions list, they will be informed that double marking is enabled on this exam sitting and that they have been assigned as a marker. 

The status shown for each submission will be unique to that specific marker and will indicate to that marker if they have completed their marking for that student. 

Assigned moderator 

When the assigned moderator views the submission list, they will be informed that they are a moderator on this exam sitting.

They will see a column for each marker and a final score. The marker column will display the marking status of the marker, or if they have finished marking, will display the final score given by that marker. The final score column will display a 'Review' button if all marking has finished. Once moderation is complete, the final score column will display the moderators final score given to the student.

Administrator

When an administrator is viewing the submissions list, they will be informed that double marking is enabled on the exam sitting. They will be able to see the status of each marker and the moderation status. The admin will be informed that they have the ability to override moderator scores. 

Non-admin user 

The non-admin user who is not assigned as a marker will be informed that double marking is enabled on this exam sitting. They will be able to see the status of each marker and the moderation status. They will not be able to change student scores.  

Marking 

When the assigned marker is viewing a submission, they will mark the exam as any normal marker would. They give their score for each question. They can provide feedback to students and use annotations to mark essays.  

Assigned markers will NOT be aware of the scores given by other markers to the student. They cannot see feedback or annotations left by other markers.  

For more information on using marking tools, please see Marking an exam

Moderating

Once all assigned markers have checked 'Marking complete', the submission will become available for moderation. For each question, the moderator will be shown the score given by each marker and asked to choose a final score. There are a number of scenarios outlined below. 

Markers agree on the score 

In the case where markers have given the same score for a question, this score will automatically be selected. The moderator has the option to override this score and set a final score. 


 
Markers disagree on the score

In the case where markers have given a different score for a question, the average score will automatically be selected. The moderator has the option to override this score and set a final score. 

The moderator sets the final score

In the case where the moderator has chosen to input a final score different to that given by a marker, the user will be told the moderator has set a custom final score. 

 

Viewing markers feedback 

Moderators have the ability to view any feedback left to the student by a marker. To do this, the moderator can select the 'Marker's feedback' tab and switch between markers using the dropdown. 

Viewing markers annotations

Moderators have the ability to view any annotations left by a marker. To do this, the moderator can select the 'Marker's annotations' tab and switch between markers using the dropdown. 

Once the moderator is happy with the final score for all questions, they check 'Moderation complete' at the top of the exam. This will set the status of the submission to 'Moderation completed'. 

Submission statuses  

The status of the submission shown will depend on the user's role in marking the exam. 

The overall status shown to the administrator will be determined by the status of each assigned marker and the moderator. This makes it easy for the administrator to know when all marking and moderation is complete. 

Status chart