Discovering user needs for an Assessment Operations System
So, let’s start at the beginning.
User research at the beginning of a project is exploratory- usually done to understand the goals, motivations, frustrations, and behavior of the intended users and understand their environmental context and constraints. We pay special attention to unmet user needs and pain points, any workarounds that the user employs to get the job done now.
Project Aurora aimed to develop a new Assessment Operations System (AOS) to replace the current system offered by International Baccalaurate Organisation to schools for the management of multiple facets of the assessment of graduating students. The central goals of the project are to improve the user experience of the assessments systems, and to reduce administrative burden for IB staff and the IB Community.
Preparation
To successfully deliver on the objectives of the project, I needed to engage with end-users at different stages to ensure that the product delivered is efficient, easy to use and enjoyable for them. To this end, I planned to organize comprehensive user research at different stages with users in schools. I wanted to ask a lot of questions to know more about our future AOS user and the two most important school- facing modules planned for development first.
Research plan and stakeholder conversations
Our research plan included research questions that sought insights into demographics, technical ability, technology used in schools, physical environment as well as administrative processes and other curricula taught in the school and of course, the usage of IB systems including IBIS.
We leveraged two types of research sessions to carry out the investigations:
face-to-face interviews to develop a deep understanding of users’ goals, motivations, frustrations and behaviour (i.e. reported behaviour)
observations of users engaging with the current IBIS to develop a better understanding of how they currently use and interact with the system (i.e. observed behaviour)
Additionally, we also did some background study from existing studies and surveys done before to inform us before these sessions.
Process- recruiting, interviewing and analysis
Recruiting users through different channels brought us in touch with more internal stakeholders who also had their own insights from interacting and being close to different end users. Our interview sessions were remote, as travelling to users spread out all over the world would make this a time and resource-consuming operation and time was of the utmost essence in this project.
We recruited in all the three regions and we talked to IB educators in schools who had established a well- oiled machinery and the IBIS process and interactions was like clockwork for them, whereas some struggled for various reasons.
Deliverables
The outputs and deliverables of this research phase were:
presentation of reports of the interviews and observations at regular intervals to the AOS project team
an analysis of the findings based on the recordings and reports, reflected in:
Personas – realistic representations of our key audience segments that help the project team empathize with end-users and created a shared understanding of their needs
Customer journey maps – a visual representation of a series of user goals and actions on a timeline that can inform design decisions
Storytelling through Comics encapsulating certain journeys and painpoints of users
Composite user journeys- combining personas and their user journeys
Composite user journeys- combining personas and their user journeys
Composite user journeys- combining personas and their user journeys
Creativity in user research
Exploring Comics as a storytelling tool to present user research insights
As a part of the design process, I always end up creating large consolidated and synthesized journey maps. In order to improve the user experience of stakeholders who receive such user research insight artefacts and reports, I explore different ways this could be made bite- sized to be more impactful and tangible for them to take action.
When I created the persona and journey map above, I wanted to provide a bite- sized insight artefact that would help the team and business stakeholders better understand some painpoints that a large cross-section of our target users go through.
I have hardly met anyone who dislikes comics. They are fun to read, engaging one can breeze through them to grasp the essence of the story. I decided to create a narrative of one of the most important pasts of the user journey here.
Comics look easy but when you start creating them they are many aspects to be considered- flow, story arc, consistency in visuals and so on. I worked on a few iterations where I played around with what I wanted to highlight in the story.
The final iteration talked about the major painpoints but also visualized what some users expected out of the service and the systems provided by the organisation for these set of tasks.
This was an experiment that I conducted to present user insights to an audience who has never seen a format like this for such a purpose. It was well received by the stakeholders, but I couldn't create more as they require a substantial amount of time and as a user research team of one, some times I needed to priortize on deliverables.
Successes and failures and lessons learned
We were delayed in conducting this research because of certain unavoidable circumstances. But it was conducted, and the product team welcomed the insights received. Although all insights could not be immediately operationalized, it brought the product owners closer to the user’s context and they could validate the assumptions made during development of the new features. The deliverables also informed other stakeholders involved in the planning and strategy to support the transition of all users from the old to the new systems.
Since all our interview and observation sessions were remote, and sometime having a coordinator in Spain who was stuck due to travel restrictions in a raging pandemic coordinate with his teacher in coastal China while uploading internal assessment marks was challenging but very important and informative to observe. Our end users have to create their own special workarounds to get the job done.
Some interesting technology related lessons were also learnt about the operations of this research sessions. How certain pop-up windows were not recorded during the observation sessions and we had to consider that when we planned and designed our next session with the right tool and approach.
Next steps
As per the research plan, as we progress through the different phases of the project, our research questions will be different at each stage. We have covered the first stage leading to the development of the product, we plan for usability tests with a representative sample set of end users in the next stage before we launch the product to all end users. Additionally, we have also started collecting data about user satisfaction and success currently delivered by IBIS to provide us with a benchmark for AOS.