I encourage you all to read the blog of NHS ePortfolio developer @zingmatter, and look at the presentation he presented at AMEE 2012 (a recent Medical Education Conference) “Assessing NHS ePortfolio behaviour: variations in the online activity of doctors as they progress through training.”
I was at the presentation at AMEE 2012 and, although the presentation title may nor sound gripping, I was fascinated to hear what could be learnt from the vast amounts of data ready and waiting to be analysed on NHS ePortfolio site use. The development team (including @zingmatter) had done a great job of drilling down into some of the data, using Google Analytics and internal tracking, in order to filter out some meaningful information from the thousands and thousands of logins and episodes over a year.
However, my first thoughts on seeing the conclusions of the presentation were “they’re asking the wrong questions” and “if they wanted to know that they should have just asked the trainees.” Many of the peaks and troughs seen on the graphs were entirely predictable (ie pre-ARCP), and some of the conclusions drawn by the developers on “depth of use” were weak. I could explain away many of the findings, as I know how trainees use the site is a function of what hoops are put in front of them to jump through. I was also sceptical about the conclusion that trainees change their behaviour in relation to the ePortfolio over the course of their training. FYs and ST6s may interact differently with the site, but there are so many confounders that a snapshot comparison is not a valid way to assess this: a longitudinal study would be required.
Despite these reservations, reading @zingmatter’s blog gives me hope for the future, as the developers at NES are committed to engaging with the needs of users. In our often passionate discussions on social media (including this blog and twitter) we must remember that we come from very different perspectives, and have unique sets of knowledge and skills.
As @zingmatter points out:
“There is a balance between college needs and trainee needs in the design of an e-portfolio and possibly this type of data can help inform this debate.”
We also have to make sure we are not misdirecting our frustration at the wrong people, and potentially alienating them:
“while I’m happy to ask simple questions about user flow, user experience and so on, questions about the educational implications of this data have not been well addressed as it’s not really in my sphere of knowledge (or in my job description). I would see the research I presented at this conference as a ‘this is the kind of thing we can do’ exercise that should lead on to better designed questions that will allow us to understand how best to develop an e-portfolio that supports effective learning and development through the effective delivery of a training programme.”
I really hope we can work together to ask the right questions and use all the data we have to inform the process. All we need now is the Royal Colleges on board and we can really maximise the potential of the ePortfolio.
Just imagine a world in which trainees didn’t hate the NHS ePortfolio. It has the potential to be a useful tool to encourage self-directed learning, provide evidence of experience and achievements, act as a showcase for job applications and excellence awards, and strengthen the relationship between trainee and trainer. This world is far away, but perhaps we are starting to see the path forward…