Personalised feedback at scale: Moderating audio feedback in first-year psychology

Educator-in-charge Jaclyn Broadbent provides an overview of the feedback in this subject.

Summary

Too often, it is felt that personalised feedback can only be provided in subjects with small learner enrolments. However, this case demonstrates that personalised, high-quality feedback can be provided consistently and at scale in a large first-year subject.

In this core Health subject, undertaken by learners from a broad range of disciplines, learners complete a series of three journal entries as their major assessment piece. Learners receive audio feedback from the same educator across their three journal entries, fostering a sense of personal connection between learner and educator. Educators also receive audio feedback through the feedback moderation process, which both ensures consistency amongst the large team of educators and supports the development of educators’ own feedback practices.

Key features of this case study include:

• Clear orientation of feedback to the next assessment task, including turnaround times scheduled to coincide with the following assessment;
• Rigorous moderation process focused on ensuring consistent quality of feedback rather than regularity of marks;
• Successful personalisation of feedback at scale; and
• Commitment to support and development of educators through moderation and resources.

Click to download a printable version of this case study

Keywords

audio feedback; moderation; feedback at scale; personalisation; educator support

The case

This subject is a core first-year Health subject, and is taken by learners from a broad range of Health disciplines. The subject explores behaviour modification, and the impact of health behaviours on individual health and wellbeing, by asking learners to undertake and reflect on their own health behaviour change.

Learners complete three assessment types, including an end-of-trimester exam and ten online weekly quizzes. The major assessment for the subject is a series of three linked, reflective journal entries in which learners chart and discuss their progress in altering their own health behaviour. For each journal entry, learners receive: a grade mapped to marking criteria in the form of a rubric; a brief written comment linking their assessment with the subject’s learning outcomes; and an audio feedback file of around five minutes. Audio feedback is the main feedback received by learners for their journal entries.

Educators are directed to include a number of components in each learner’s audio feedback. Following a basic structure, educators first introduce themselves before explaining the audio feedback’s design and how learners can use the feedback for their next assessment task. Educators then discuss the journal entry, using concrete examples drawn from the learner’s work for both positive and critical comments. Educators are also required to provide at least two instances of feedback which links the current assessment with the forthcoming assessment task. The lead educator-in-charge describes this practice as ‘feedforward’:

[T]hey are talking directly about the next assessment piece and how the skill, or what they’ve done in the current assessment piece, is transferred or translated to the next assessment piece.

Educators then conclude their feedback. As educators assess the same cohort of learners across the trimester, they may end their feedback on a personal note – for instance, by referencing a learner’s progress or with well wishes for a future assessment. The learners we spoke to appreciated this personal connection, and educators also find it helpful to follow the same learners across the trimester:

… the more that you can go on the journey with the [learner], you can really see how they are progressing, and you do get that really strong interpersonal relationship that you can build over time by seeing their assessments …

Typically, between 25 and 30 educators are employed each trimester for this subject, so it is essential to have in place processes to manage and support educators. The subject’s educators-in-charge are strongly committed to the philosophy that “[learners] should get exactly the same experience and the same quality of feedback regardless of who’s marking”. This means providing learners with feedback that is consistent in both quality and content; however, ensuring feedback quality is not straightforward across a large team of educators. To ensure that all learners receive high-quality feedback, the subject chairs developed a rigorous moderation process which not only moderates grading and feedback, but supports and develops educators.

The moderation process unfolds as follows. Educators first grade and provide audio feedback on two journal entries from their assigned learner pool. The same two assignments are then blind marked by one of the educators-in-charge, who next compares their blind mark with the educator’s rubric and reviews the audio feedback file, both to determine the educator’s reasoning and to ensure that at least two instances of forward-looking feedback have been included. The educator-in-charge then records an audio feedback file for the educator, following the same structure as the audio feedback provided to learners – offering concrete examples, discussing what might have been missed, and suggesting alternative ways to address concerns. If the educator is determined to be providing feedback at the required standard, they are then able to commence marking. If the educator-in-charge feels the educator is not yet providing feedback in line with subject standards, they will be asked to mark a further two assignments. The moderation process is repeated as required; as the lead educator-in-charge explains, “we would continue that process until we felt like they were marking to the same standard as everybody else”.

The educators-in-charge also maintain a spreadsheet to chart each educator’s progress and grading average. The spreadsheet is updated every two to three days throughout the marking period, and is made available for all educators to view. The educators-in-charge monitor average grades for outliers, and moderate any anomalous educator averages by repeating the moderation process (including audio feedback) for two randomly-selected assignments for that educator. A second round of moderation is also performed when an educator has marked half of their learner cohort, with two assignments selected at random and moderation performed to ensure “they’re not getting tired or missing their feedforward feedback or [have] got some bad habits that we didn’t pick up”.

While moderation is a time- and effort-intensive process, the lead educator-in-charge affirms that the initial investment pays off: “it reduces the [learner] complaints every time … for example, sometimes we might only get four [requests to re-mark] out of 1500 assignments”. In addition, the benefits for educators are significant. As the lead educator-in-charge explains, the subject’s educators-in-charge are motivated by their own experiences as sessional educators to ensure educators receive support and professional development:

I really think that giving feedback – audio feedback in particular – to [educators], where you actually spend some decent amount of time going through their work and talking to them like a person, is so valuable, rather than just saying, ‘Oh, you gave it a HD and I gave it a D’ … I remember that happening to me when I was a sessional [educator], where you get double-marked and you still don’t quite understand why your marking was different.

Providing educators with audio feedback on their own feedback artefacts not only assists in developing their understanding, but models good feedback practices. In addition, moderating educators using the same medium by which they comment on learner work places the educator in their learners’ shoes, as the lead educator-in-charge notes:

We’ve had [educators] say that the moderation process is valuable for them, not only because obviously it’s helping them to develop as [an educator], but because they realise how useful their own feedback is to the [learner].

Turnaround times are also an important aspect of the subject’s feedback design. While the university’s turnaround policy is three weeks, feedback in the subject is scheduled to occur seventeen days after each journal entry’s due date. For the educators-in-charge, the important turnaround timeframe is not the time elapsed since learners submitted their assessment, but the time until the forthcoming assignment is due. This element of the feedback design is closely linked with in-class activities to maximise the impact of forward-looking feedback on learners’ preparation for the next assessment. Learners receive their feedback seven days before their next assignment is due, by which stage time has been spent in class discussing and working on the impending assignment. The lead educator-in-charge explains that this ensures “when [learners] actually get their feedback, and that feedback is talking about that next assessment, they are already in that mindspace … when they listen to it they can clearly make connections”.

This subject’s feedback design has been developed and refined over a number of years. The educator-in-charge was motivated to explore alternatives to written feedback due to several concerns with feedback in the subject. Large learner numbers created a tension between time and budget constraints and the aim of providing “personally meaningful or relevant” feedback to learners. In addition, standards of written feedback were low, with educators in the subject struggling to provide comments of sufficient quality and consistency within allocated marking timeframes. At the same time as these feedback practices became untenable, the university switched its online learning platform to a system with an inbuilt audio feedback function. While exploring the new online platform, the lead educator-in-charge discovered the audio feedback option and decided to trial it as a possible solution to the subject’s feedback difficulties. The subject has now been offering learners audio feedback for eighteen trimesters, with only one educator unwilling to provide audio feedback in this time. For the lead educator-in-charge, audio feedback is an effective means of providing learners with feedback that’s “as meaningful as we can [make it] in the five minutes that we have to give it”. Learners we spoke with told us that they liked the personal feel of audio feedback, and commented that “when you get the verbal feedback, you get a lot of … motivation because they can kind of convey a bit more emotion in what you’ve done well and be a bit more tactful in what you haven’t done so well”. The moderation process and emphasis on forward-looking feedback have both followed the implementation of audio feedback in the subject. As the lead educator-in-charge reflects,

all of the things that have happened – so the moderation process, the feedforward feedback – have all been to address gaps that we’ve found as we’ve built the feedback up, as we’ve found ways or found things that we could see needed to be improved.

Why it worked

The design

In this case, feedback was considered to be successful particularly because of the following key elements:

• Feedback undergoes rigorous moderation: educators’ feedback is moderated at several stages by experienced educators (subject educators-in-charge). This process identifies any inconsistencies in educators’ approaches to feedback provision and grading, and allows corrective interventions to take place before a large volume of assignments have been marked, reducing the demand for later re-marking procedures. The rigorous moderation process allows the subject chairs to ensure feedback quality and consistency across a large number of educators of diverse skill and experience levels.
• Assessment and feedback design are closely linked: the subject’s major assessment piece, three linked reflective journals, is designed to ensure that assessment and feedback work together. Feedback for each journal links assignments and explicitly references the next assessment, providing learners with recommendations and areas for improvement. Turnaround times perform an important role in the interplay between assessment and feedback, and are designed so that learners receive feedback linked with their impending assignment at a time when they have already commenced working on or thinking about their draft.
• Feedback is personalised through educator continuity and audio feedback: educators are assigned a cohort of learners to mark across the whole trimester. This continuity ensures learners journals are marked by the same educator throughout, allowing educators to observe a learner’s progress and a personal connection to develop. Learners told us that this makes them feel like “more than just a number on a page – that your [educator knows] you a bit better”. Feedback is provided in an audio format which is conversational and allows meaning to be conveyed through tone and verbal cues. The personalised feel of audio feedback also works well with the journals, which are reflective and personal in nature. 

Educator-in-charge Jaclyn Broadbent explains what worked in this subject and why.

Enablers

Some of the enabling factors for this feedback design included:

• Long-term subject leadership with a clear vision for future development: the continued leadership of this subject by the lead educator-in-charge for an extended period of time (eighteen trimesters) has enabled the subject to develop in a considered, planned and thoughtful manner. The lead educator-in-charge has a clear vision for and commitment to the subject, and is continually reflecting on, adjusting and improving the subject, with changes planned several trimesters in advance.
• Feedback mechanisms and resources provide extensive support for educators: educators receive audio feedback from an educator-in-charge on their moderated audio feedback artefacts for learners. This moderation and feedback process places educators in their learners’ shoes, models good feedback practice and behaviours, and provides ongoing professional development. Educators also receive a seventy-page marking handbook containing examples, videos explaining rubrics and marking criteria, and scripts to facilitate the audio feedback process. In addition, the educators-in-charge hold regular marking meetings, including opportunities for mentorship from senior educators.
• Technical systems facilitate a simple audio feedback process: the university’s online learning management system, CloudDeakin, allows educators to easily provide non-textual feedback to learners. An audio feedback function is built into CloudDeakin’s marking platform, providing educators with a simple one-step recording process.

Challenges

Some of the challenges for this feedback design included:

• Moderation is a time- and effort-intensive process: the short turnaround times for learner feedback mean shorter timeframes for educators-in-charge to moderate educators’ feedback. This can potentially become a workload issue, with the lead educator-in-charge acknowledging that educator moderation often takes place outside business hours. While this initial investment of time and effort typically results in fewer learner queries and requests for re-marking, educators should be aware that rigorous moderation requires significant commitment.
• Ensuring consistency at scale is an ongoing process: inducting educators into new systems and ensuring consistency across a large number of educators is an ongoing process. Even with extensive support mechanisms, expected standards may not be immediately realised. This underlines the need for a supportive and thorough moderation process.
• Assessment design must be considered from the outset: an initial outlay of effort and planning is required to ensure assessment design supports linked assignments and feedback. Providing learners with forward-looking feedback works best when assignments are aligned clearly and coherently with one another, which also helps to ensure that feedback linking past assignments with upcoming assignments is comprehensible for learners. How assessment and feedback will work together must be considered and incorporated into subject design from the outset.

What the literature says

Learners generally consider audio feedback to be a highly acceptable feedback format. Learners commonly describe audio feedback as ‘personalised’, ‘supportive’ and ‘motivating’ (Gould & Day, 2013; Voelkel & Mello, 2014), and may also feel that audio feedback offers a more informal, conversational style of feedback. Gould and Day (2013) found that audio feedback may help build educator-learner relationships, with learners considering educators more ‘approachable’ after listening to audio feedback. Similarly, Ice et al. (2007) report that learners receiving audio feedback were more likely to feel that their educators care about their learning. However, educators should be aware that the more personalised nature of audio feedback can have negative emotional effects for some learners. Some studies report that learners find listening to audio feedback stressful, and that hearing their educator’s voice may amplify critical comments (Gould & Day, 2013; Voelkel & Mello, 2014).

Moderation procedures in Australian higher education are typically ad hoc and the responsibility of individual institutions (Beutel, Adie & Lloyd, 2017). Moderation research is almost exclusively concerned with grading rather than feedback provision, and moderation is recognised as an under-researched and disputed concept. Subjects with large learner numbers typically require multiple marking staff, and ensuring consistency amongst a team of educators can be challenging for educators-in-charge (Beutel et al., 2017; Saunders & Davis, 1998). Moderation can be resisted by educators, who may regard moderation as a judgement on professional competency (Beutel et al., 2017). The challenges of coordinating moderation processes for a large team of educators can also be exacerbated by tight turnaround times (Saunders & Davis, 1998). Dialogic moderation procedures (such as consensus or social moderation) are recommended to help develop shared understandings of criteria and standards across large marking teams (Beutel et al., 2017; Bloxham, Hughes & Adie, 2016).

Management and moderation procedures for sessional educators have received limited attention in the literature. Smith and Coombe (2006) report that universities typically have few procedures in place to monitor marking quality and diagnose areas of concern. This lack of formal processes is particularly concerning given unsystematic moderation and marking procedures around sessional educators place academic standards at risk (Percy et al., 2008). Sessional educators typically receive limited training and almost no ongoing professional development (Smith & Coombe, 2006). Insufficient performance management and limited professional development opportunities not only fail to adequately support sessional educators, they can jeopardise learner learning and place institutional quality outcomes at risk (Percy et al., 2008). Smith and Coombe (2006) found that supervision practices for sessional educators are largely inadequate and informal, and in some cases even non-existent. Crimmins et al. (2016) suggest that systematic moderation processes provide an opportunity to professionally develop sessional educators, and found that sessional educators recognise and appreciate moderation feedback as a valuable chance to develop their marking practices. 

Moving forwards

Advice for educators

The participants in this case offered several suggestions for educators wishing to trial the feedback design:

• Choose one element and build from it: select part of the feedback design that you think will work best for you, and trial that first. The lead educator-in-charge explains that, while the feedback design of the subject can initially seem overwhelming, the current feedback design “didn’t just happen overnight”. Instead, it was built up slowly, beginning with audio feedback and adding elements as areas for improvement were identified. While embarking on a complete overhaul of a subject may seem tempting, the lead educator-in-charge advises that “doing one small thing at a time, and building on top of it, [is] what makes it work.”
• Use concrete examples and signpost: provide learners with clear, concrete examples drawn from their own work. This helps to build learners’ evaluative judgement and understanding of assessment standards. If using audio feedback, ensure you clearly signpost each element of the assignment before discussing it – for instance, refer to the paragraph’s theme or contention. The lead educator-in-charge advises that it can take time to get used to signposting.
• Align your assessment and feedback designs: mapping out your assessment and feedback designs will help identify ways in which assessments can be linked or reconfigured, and how feedback can be integrated. While this planning may require an initial outlay of time and effort – the lead educator-in-charge told us that “it took me a lot of time with lots of pieces of paper on my desk” – a cohesive assessment and feedback design will help maximise the impact of forward-looking feedback.
• Choose the feedback format that best suits your assessment: as the learners and educators we spoke to noted, one size doesn’t fit all. In this subject, the conversational tone of audio feedback complemented the reflective nature of the journal entries and allowed a personal connection between educator and learner to develop. However, this personalised approach will not suit all assessments or contexts.

Advice for institutions

This case offers several useful insights for leaders within institutions wishing to support similar feedback designs:

• Feedback needs to be valued at an institutional level: the lead educator-in-charge in this case described herself as “intrinsically motivated” to provide learners with quality feedback, but observed that not all educators possess the same levels of motivation. It should be made clear at an institutional level that feedback is valuable, to encourage educators who may not be as motivated to prioritise high-quality feedback. As the lead educator-in-charge stated, “if the university makes it clear that [feedback] is valuable, then it will be a priority to teaching staff”.
• Aim for continuity in teaching leadership roles: subject systems, educator resources and curricular designs need to be established over time. This subject has benefited from the clear vision and long-term commitment of the lead educator-in-charge, who has been in her role for over five years. Departments should recognise the need for staffing continuity – particularly for large, core subjects – in senior leadership roles (subject educator-in-charge level) over a sustained time period. This ensures a long-term vision for the subject can be designed and enacted.
• Invest in educator development: this subject’s feedback design is committed not only to providing learners with high-quality feedback, but to supporting and developing educators. The moderation process, educator handbook and marking meetings all operate on the principle that investing time, energy and care into developing educators’ skills leads to long-term benefits for both educators-in-charge and the educators themselves. As the lead educator-in-charge explains:

Recording audio feedback and giving formative feedback to the [educators] is so important because it’s investing time in their development, … [it] actually helps you in the long run because the better they are the less mistakes they make, which is good in the end … [and] it’s good for [educators] because then they can take those skills to another subject or another job.

Departments should consider this approach of investing initial resources to develop sessional educators, in order to reap long-term gains.

• Technological support: non-textual feedback formats such as audio feedback need to be supported through an institution’s technology and online platforms. In particular, learning management systems must be able to facilitate a digital feedback strategy. Support for educators, such as training and resources, is also recommended.

Resources

The Technology-Mediated Assessment Feedback project offers a range of resources for educators interested in trying audio feedback in their subjects. For practical tips on how to get started with audio feedback, including a video explanation, check out the project’s ‘How’ page. For suggestions on how to structure feedback recordings, including a sample structure, check out the project’s ‘What’ page.

To learn more about the assessment design of this subject, its development, and the challenges of teaching at scale, check out the paper co-authored by the case study’s lead educator-in-charge: Applying formative practices to summative assessment: A case study of a large class in the pursuit of sustainable assessment.

References

• Beutel, D., Adie, L., & Lloyd, M. (2017). Assessment moderation in an Australian context: processes, practices, and challenges. Teaching in Higher Education, 22(1), 1-14. doi:10.1080/13562517.2016.1213232
• Bloxham, S., Hughes, C., & Adie, L. (2016). What’s the point of moderation? A discussion of the purposes achieved through contemporary moderation practices. Assessment & Evaluation in Higher Education, 41(4), 638-653. doi:10.1080/02602938.2015.1039932
• Crimmins, G., Nash, G., Oprescu, F., Alla, K., Brock, G., Hickson-Jamieson, B., & Noakes, C. (2016). Can a systematic assessment moderation process assure the quality and integrity of assessment practice while supporting the professional development of casual academics? Assessment & Evaluation in Higher Education, 41(3), 427-441. doi:10.1080/02602938.2015.1017754
• Gould, J. & Day, P. (2013). Hearing you loud and clear: learner perspectives of audio feedback in higher education. Assessment & Evaluation in Higher Education, 38(5), 554-566. doi:10.1080/02602938.2012.660131
• Ice, P., Curtis, R., Phillips, P. & Wells, J. (2007) Using asynchronous audio feedback to enhance teaching presence and learners’ sense of community. Journal of Asynchronous Learning Networks, 11(2), 3-25.
• Percy, A., Scoufis, M., Parry, S., Goody, A., Hicks, M., Macdonald, I., … Sheridan, L. (2008). The RED Report, Recognition – Enhancement – Development: The contribution of sessional teachers to higher education. Sydney: Australian Learning and Teaching Council. Retrieved from http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1139&context=asdpapers
• Saunders, M. N. K. & Davis, S. M. (1998). The use of assessment criteria to ensure consistency of marking: some implications for good practice. Quality Assurance in Education, 6(3), 162-171. doi:10.1108/09684889810220465
• Smith, E. & Coombe, K. (2006). Quality and qualms in the marking of university assignments by sessional staff: an exploratory study. Higher Education, 51(1), 45-69. doi:10.1007/s10734-004-6376-7
• Voelkel, S. & Mello, L. V. (2014). Audio feedback – better feedback? Bioscience Education, 22(1), 16-30. doi:10.11120/beej.2014.00022