What to do with all that data?

I encourage you all to read the blog of NHS ePortfolio developer @zingmatter, and look at the presentation he presented at AMEE 2012 (a recent Medical Education Conference) “Assessing NHS ePortfolio behaviour: variations in the online activity of doctors as they progress through training.”

With thanks to http://www.acunu.com/ for the badge!

I was at the presentation at AMEE 2012 and, although the presentation title may nor sound gripping, I was fascinated to hear what could be learnt from the vast amounts of data ready and waiting to be analysed on NHS ePortfolio site use. The development team (including @zingmatter) had done a great job of drilling down into some of the data, using Google Analytics and internal tracking,  in order to filter out some meaningful information from the thousands and thousands of logins and episodes over a year.

The Prezi can be seen in the “elastic elephant” blog

However, my first thoughts on seeing the conclusions of the presentation were “they’re asking the wrong questions” and “if they wanted to know that they should have just asked the trainees.” Many of the peaks and troughs seen on the graphs were entirely predictable (ie pre-ARCP), and some of the conclusions drawn by the developers on “depth of use” were weak. I could explain away many of the findings, as I know how trainees use the site is a function of what hoops are put in front of them to jump through. I was also sceptical about the conclusion that trainees change their behaviour in relation to the ePortfolio over the course of their training. FYs and ST6s may interact differently with the site, but there are so many confounders that a snapshot comparison is not a valid way to assess this: a longitudinal study would be required.

Despite these reservations, reading @zingmatter’s blog gives me hope for the future, as the developers at NES are committed to engaging with the needs of users. In our often passionate discussions on social media (including this blog and twitter) we must remember that we come from very different perspectives, and have unique sets of knowledge and skills.

As @zingmatter points out:

“There is a balance between college needs and trainee needs in the design of an e-portfolio and possibly this type of data can help inform this debate.”

We also have to make sure we are not misdirecting our frustration at the wrong people, and potentially alienating them:

“while I’m happy to ask simple questions about user flow, user experience and so on, questions about the educational implications of this data have not been well addressed as it’s not really in my sphere of knowledge (or in my job description). I would see the research I presented at this conference as a ‘this is the kind of thing we can do’ exercise that should lead on to better designed questions that will allow us to understand how best to develop an e-portfolio that supports effective learning and development through the effective delivery of a training programme.”

I really hope we can work together to ask the right questions and use all the data we have to inform the process. All we need now is the Royal Colleges on board and we can really maximise the potential of the ePortfolio.

Just imagine a world in which trainees didn’t hate the NHS ePortfolio. It has the potential to be a useful tool to encourage self-directed learning, provide evidence of experience and achievements, act as a showcase for job applications and excellence awards, and strengthen the relationship between trainee and trainer. This world is far away, but perhaps we are starting to see the path forward…

Advertisements

2 responses to “What to do with all that data?

  1. Very interesting to read your perspectives. It shows why quantitative data can only take us so far.
    By the way, have you seen the discussion about an app for CPD that has started on the LinkedIn group?
    http://lnkd.in/WPycBs
    It’s partly morphed in to a discussion of #openbadges, something that we might hear more about in the future!

  2. As you’ve seen recently on twitter, I’ve been looking at educational aspects of work based assessment use in terms of trainee competence and behaviour (pub: Med Ed, 2012 & 2013). The eportfolio data let us do this (anonymised and aggregated trainee results rather than eportfolio site stats), and in the end we found a whole lot of not-a-lot; WPBA outcomes don’t tell us much about whether trainees are competent or not.

    The flip side of this is that, if online assessments are quick, accessible (eg via smartphone app), formative and relevant, and if the various eportfolio systems are integrated and allow for easy and meaningful data visualisation (for trainees and supervisors to monitor progress, and for program directors to observe trends), then we could get meaningful data out of them. Those findings can then drive improvements in training and assessment for individuals and on macro levels.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s