b) An understanding of your target learners

Supporting Statements

As noted in Core Area 1(c), I planned, developed, and managed a joint e-portfolio system for social work students at University of Brighton (UoB) and University of Sussex (US) from 2017-2020. The complexity of the system meant that I was working with five distinct groups of learners: students (UoB and US), staff (UoB and US), external work placement staff. The system was built on the Mahara e-portfolio platform with an internal structure mirroring the two institutions for data protection purposes. The system also needed to accommodate a work placement structure where students from both universities could potentially interact in groups owned by an external practice-based educator¹. The initial structure had an emphasis on groupings of students, this was requested by a work placement staff representative and academic staff. The e-portfolio template for each student was based on the London and Southeast England HEI (Higher Education Institution) Agreed Common Practice Learning Documents for Social Work. The documentation was complex with a lot of discipline-specific terminology. My challenge was to rapidly get up-to-speed with the terminology, harmonise workflows between the two universities, and train students and staff in the use of the system (with learning technologists at US). While delivering the training, I met and spoke with external work placement staff who used the system. These discussions caused me to theorise that the system might be more complex than it needed to be.

I conducted a usability study to test my theory in July 2018 as part of the annual report². This featured the results of questionnaires for each user group (staff, students, external work placement staff). The questionnaires were based on the ‘IBM computer usability satisfaction questionnaires’ (Lewis, 1995) with added questions allowing open-text responses. I also established a categorised list of improvements for technical consideration based on staff, student, and externals’ feedback. The initial usability in 2018 was rated by users at 58%, as compared to the benchmark of 50% for a system to be considered usable. In response to the usability study data, I made the following changes³: 

• Redesigned the student e-portfolio template, reducing it from 7 pages to 1 page with collapsible sections.

• Redesigned the internal structure and simplified the menus seen by users and by system role. My key discovery from the data was that placements are very individualised. So, although some stakeholders liked the idea of groups, they were not needed or used as they did not map to the students’ real-world tasks.

• Changes including clearer group naming and revisions to the electronic sign-off workflow.

I re-ran the same usability study in September 2019, the survey response was low and underepresentative (8.5%; SEE p.16²). Usability was rated by users at 60% (+2%). The qualitative feedback from the study suggests that the improvements may have had a more significant impact on student experience than represented by the figures; simplifying supervision and marking workflows. On a less positive note, I also uncovered usability issues with Mahara’s base functionality which were not feasible to resolve. The e-portfolio system was discontinued in 2020 due to changes in funding, but based on the usability data, I also believe that the solution had reached the end of its functional lifespan. Mahara required high volume manual work to serve the needs of both institutions and although this was anticipated it would have been unsustainable in the long term due to the level of specialised technical and administrative staff expertise required. The system did however see the students though their placements during the pandemic when wet ink signatures were not possible and for that I will always be thankful.

References

Lewis, J.R. (1995) 'IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use', International Journal of Human-Computer Interaction, 7:1. pp. 57-78.

Evidence