Monthly Archives: September 2012

NHS Hackday and the ePortfolio Data Liberation Front

I spent this weekend in Liverpool at NHS Hackday. I had no idea what to expect. I had never met anyone there before and only knew a few names from twitter and google groups conversations in the weeks running up to the Hackday. I wasn’t completely sure I knew what a Hackday was.

I was astounded. 

I spend a lot of my life getting frustrated by the slow pace of change and the massive inefficiencies in the way that we work. I want to be freed up to spend time teaching, learning, writing, thinking, talking to patients and providing care. I hate unnecessary paperwork and bureaucracy. I hate meetings that don’t achieve anything.

NHS Hackday was a breath of fresh air. A diverse group of people with totally different backgrounds, most of whom had never met,  got together, discussed problems and solved them. In a weekend!

I will describe here what happened at the Hackday, what our project “The NHS ePortfolio Data Liberation Front” achieved and why it won 2nd place. There is far more info about how it is run, by who, and why on the NHS Hackday site. You can also see an interview with Carl (from OpenHealthcare) on Youtube:

 What follows are my personal impressions.

The HACKDAY CoNCEPT

On Saturday morning, whilst people were registering and getting coffee all those with ideas for projects wrote them on a board. Everyone gathered in the main hall and each idea had 2minutes to pitch. After all the pitches, people gathered around signs indicating each idea, and people formed groups. Then the work began. Groups discussed their vision, their proposed solution, and thrashed out conceptual and technical details. Fuelled by enthusiasm, tea, coffee and wotsits, software developers created things out of thin air (OK, out of data and code, blood, sweat and tears). Health professionals like me, who couldn’t code, were on hand to give context to the projects and point out real-world hurdles, which could then be worked around.

The NHS ePortfolio Data Liberation Front

Our group consisted of me (full of ideas, no understanding of code), Nicolas Tollervy, a developer (a genius with lots of patience and an incredible ability to work round every problem the project presented him with)  and Marcus Baw, (a GP who can code a bit and is a RCGP Health Informatics Group member, who was a great bridge and font of knowledge on NHS informatics issues).

We discussed some of the problems with the current NHS ePortfolio and possible workarounds. Since the code is not open and there is no API this was no simple problem.

We discussed the urgent need for an app to make trainees and trainers lives easier, and make WPBAs educationally valid. Any app would have to be able to get data into the ePortfolio so that a WPBA showed up not just in the personal library section as any random document, but in the WPBA section. With no code and no API this would be a great challenge.

We decided to focus on the fact that my data is locked in a vault in my ePortfolio. Whilst it is in there I can do nothing with it.

I want to liberate it, as I could then do anything I want with it! Ideas include:

  • visualise my achievements and progression
  • present the data in a way that my supervisor can see, understand and give feedback on
  • present the data in a way that makes it clear I have achieved all the competencies required by the JRCPTB for ARCPs and CCT
  • integrate the data into my CV, my online CV, an alternative ePortfolio (mahara, Googlios etc), use it for job applications
  • allow me to take the data with me into another role (progression or change of career path) eg Foundation Trainee –> Emergency medicine ACCS trainee –> GP trainee –> GP (all use different ePortfolio systems)

Not only is there a practical need for this, but the more we talked about it the more I realised that this is bigger than practicalities. It’s a philosophical argument. It’s my data. About me. I want it liberated. I can already download a PDF so clearly no-one disputes the fact that the data is mine and I have a right to it, but a PDF is useless.

@ntoll worked incredibly hard (with breaks for coffee, sandwiches, a trip to the pub and a curry house), came up against many problems and found ways around them all. We modified our plan as we went along, and decided that the best use of our time would be to do a ‘proof of concept’ and focus on a particular data set within the ePortfolio (there’s a lot of data in there, and it’s not organised as logically as you might imagine!). By the time we reached the submission deadline of 12.00 on Sunday we had something to show for our efforts. @ntoll made some finishing touches and we put together a brief presentation.

All 15 projects that had been selected from the pitches presented (a strict 5min and 1min for questions) to a panel of judges including: @MarkPriceDavies (chair), Ian Gilmore, Dr Farath Arshad, Zeinab Abdi, Francis Irving @frabcus, Dan Lynch @MethodDan, and Lilian Wiles. They deliberated and at 17.00 announced the winners.

The Other Projects

You can see more details of the projects on the NHS Hackday site, and get all the code through the wiki and on github, since all projects are open and shared. There were lots of fantastic projects but those that particularly caught my attention were:

  • AskIt (a general purpose question asking android app for any questionnaire you need – Waterlow, MUST score, falls assessment etc. Simple, effective, important!)
  • Making sense of patient comments (data visualisation from sources such as NHS Choices – massive potential applications)
  • CoIncidence Gate: a Conflict of Interest tool (scraped data from conflict of interest statements on Pubmed – something like 480,000 papers analysed!! Again, follow the link for more discussion on the massive potential applications of this project)
  • BleepBleep (making in-hospital communication better. An end to having to call switchboard. An end to the bleep! I trialled this, and am keen to help get it into hospitals now! Stop wasting time on hold)
  • GAAG: Guidelines at a Glance (there are well-studied barriers to doctors using guidelines, meaning patients don’t get best care. GAAG provides quick access to personalised most-used bits of guidelines on an app. Lots of potential for social add-ons, highlighting when guidelines change, seeing what peers use, rating bits of guidelines. See presentation for more info. Can’t wait to use it!)
  • Bloodcount (haematologists sit at very advanced microscopes counting different normal/abnormal cell types using very un-advanced technology = clicker and pen and paper. Bloodcount is a desktop system of a counter with keyboard shortcuts, reference normal and abnormal cells, report generation and learning function. Hard to describe to do it justice. A worthy winner!)
  • wtfdoc (an NHS jargon buster for patients and relatives as an app. Has a database, and if a term is unknown it will crowdsource answer through twitter and other sources. V clever!)

Why I think We Won a Prize

Our project won the First Scraperwiki prize for scraping, and came joint second overall on the day. I think the reasons we won are multiple:

  • @ntoll achieved amazing things writing novel code to scrape data out of a closed system and generate a .json file of hierarchical data that could then be used. In just a day and a half this was some achievement!
  • our pitch was powerful as this is an issue for all doctors of all specialities at all levels, especially with revalidation now a reality. Facilitating learning for healthcare professionals is in all our interests as a society
  • the concept of data liberation goes beyond this project. Who owns the data in public databases? Who owns the data in the NHS? What right does an individual have to their own data? What right does an institution have to keep it from them?

What next

I owe a huge thank you to the organisers, supporters, volunteers and participants at NHS Hackday Liverpool 2012.  And a special thanks to Scraperwiki for providing prizes including my beautiful new Google Nexus 7! This weekend I saw innovation in action, providing real, practical solutions to the day-to-day problems facing those who work in and use the NHS. Some of these solutions are now in use – today! Others will be worked on outside the Hackdays or at the next one. I have had my mind opened to new ways of working and have returned to work today full of enthusiasm and inspiration.

There’s no going back now. I’m a doctor who loves geeks who love the NHS, and I have the T-shirt to prove it. 

Where do we go now?

Last week I saw a film at the BFI called “Where do we go now?” The film is great, but the content is not very relevant to Postgraduate Medical Training. The title, however, got me thinking….

The Royal Colleges and NES, who make the NHS physician ePortfolio, are drawing up ‘roadmaps’ which set out a vision for future directions. This is great, as their aims, objectives and ideas are being carefully constructed. But where is the trainee input to this process?

Where do we go now?

Where do we go now?

So, if the JRCPTB and NES are going to have ‘roadmaps’, I propose we write our own. I have started a Googledoc. You can edit it by clicking this link:

If  you don’t participate when given the chance don’t complain when you have to do 100 WPBAs a week, link every curriculum item to 50 pieces of evidence and there’s still no app in 2020

Go on, contribute. Put your ideas on paper. Make change happen. If you don’t other people will make decisions for you. I doubt you’ll like them.

A Comment Cloud

This is a wordle, made up of all the comments left on this site:

The prominent words are those that have featured more frequently, and include: ePortfolio, think, learning, NHS, work, trainees, system, need, training, good, open source, evidence, use, app and people. 

Interesting.

What to do with all that data?

I encourage you all to read the blog of NHS ePortfolio developer @zingmatter, and look at the presentation he presented at AMEE 2012 (a recent Medical Education Conference) “Assessing NHS ePortfolio behaviour: variations in the online activity of doctors as they progress through training.”

With thanks to http://www.acunu.com/ for the badge!

I was at the presentation at AMEE 2012 and, although the presentation title may nor sound gripping, I was fascinated to hear what could be learnt from the vast amounts of data ready and waiting to be analysed on NHS ePortfolio site use. The development team (including @zingmatter) had done a great job of drilling down into some of the data, using Google Analytics and internal tracking,  in order to filter out some meaningful information from the thousands and thousands of logins and episodes over a year.

The Prezi can be seen in the “elastic elephant” blog

However, my first thoughts on seeing the conclusions of the presentation were “they’re asking the wrong questions” and “if they wanted to know that they should have just asked the trainees.” Many of the peaks and troughs seen on the graphs were entirely predictable (ie pre-ARCP), and some of the conclusions drawn by the developers on “depth of use” were weak. I could explain away many of the findings, as I know how trainees use the site is a function of what hoops are put in front of them to jump through. I was also sceptical about the conclusion that trainees change their behaviour in relation to the ePortfolio over the course of their training. FYs and ST6s may interact differently with the site, but there are so many confounders that a snapshot comparison is not a valid way to assess this: a longitudinal study would be required.

Despite these reservations, reading @zingmatter’s blog gives me hope for the future, as the developers at NES are committed to engaging with the needs of users. In our often passionate discussions on social media (including this blog and twitter) we must remember that we come from very different perspectives, and have unique sets of knowledge and skills.

As @zingmatter points out:

“There is a balance between college needs and trainee needs in the design of an e-portfolio and possibly this type of data can help inform this debate.”

We also have to make sure we are not misdirecting our frustration at the wrong people, and potentially alienating them:

“while I’m happy to ask simple questions about user flow, user experience and so on, questions about the educational implications of this data have not been well addressed as it’s not really in my sphere of knowledge (or in my job description). I would see the research I presented at this conference as a ‘this is the kind of thing we can do’ exercise that should lead on to better designed questions that will allow us to understand how best to develop an e-portfolio that supports effective learning and development through the effective delivery of a training programme.”

I really hope we can work together to ask the right questions and use all the data we have to inform the process. All we need now is the Royal Colleges on board and we can really maximise the potential of the ePortfolio.

Just imagine a world in which trainees didn’t hate the NHS ePortfolio. It has the potential to be a useful tool to encourage self-directed learning, provide evidence of experience and achievements, act as a showcase for job applications and excellence awards, and strengthen the relationship between trainee and trainer. This world is far away, but perhaps we are starting to see the path forward…

Surgical spirit: what the surgeons think of their ePortfolio

An article published recently in the Journal of Surgical Education looks at the experience of surgical trainees and their ePortfolio. As a Medical Registrar I am in danger of being disowned by my colleagues for suggesting that we may be able to learn something from the surgeons! But in relation to ePortfolio use, many parallels can be drawn between the experience of surgical and physician trainees.

The surgical ePortfolio (ISCP) became mandatory for British surgical trainees 5 years ago, with a compulsory £125 annual fee. In 2008 widespread dissatisfaction was reported. This article (by Pereira and Dean) surveyed 359 users across all specialities and geographical areas. Although ratings improved between 2005 and 2008 trainees were underwhelmed overall. Unfortunately the article is not open access, and is behind a paywall, so I have selected some quotes for discussion below.

My love don’t cost a thing (but my training does….):

“An evaluation by ASiT estimated conservatively the upward spiralling costs of surgical training to the trainee to be £130,000 even before the introduction of MMC, with ISCP and its mandatory annual fee amounting to an additional £1000 over 8 years of surgical training.”

No medic would claim to be poorly paid, but there must be honesty and transparency with regard to the significant financial burden placed on trainees. This is likely to become more pressing as graduates leave medical school with escalating debts. Value for money is high on the agenda.

The current cost of the physician ePortfolio is only £18 per trainee per year, but perhaps this needs review, especially in the context of calls for investment to improve functionality. Trainees have a poor understanding of the costs of training and there is a disconnect between payment of JRCPTB fees and any visible outcomes in terms of education and training. Surely a lesson for all Colleges and higher bodies is that greater engagement and consultation with trainees could help prevent widespread and growing resentment.

A teacher affects eternity; he can never tell where his influence stops 

Sir William Osler, a great clinical teacher

“..incentive for trainer and assessor engagement remains lacking. It is important that trainers are properly recognized and rewarded for the time that they spend assessing and supervising trainees if obliged to use increasingly time-consuming methods, and we would welcome any system that encourages them.”

We must spare a thought for the Consultants who are striving to support us in our professional development. Demands on their time come from all directions and, unfortunately, postgraduate education and training is often the thing that loses out and gets pushed to the bottom of the mounting to-do pile. The system needs to reward and encourage senior clinicians so that they make time to give high quality feedback to trainees during WPBA completion. But this is a long term aim that feels intangible and unattainable. In the short term, reducing the time it takes to complete WPBA paperwork will make everyone happer. An app seems the quickest way to achieve this.

A call for EBT: Evidence Based Training

“Recently the JCST has specified a minimum of 40 WPBAs per year to be completed as a ‘quality indicator’ for surgical training and career progression…Regional training programs have set directives for mandatory WBAs per annum, ranging from a minimum JCST dictat of 40 to the 80 required in London. These present a great challenge upon time available to any practicing surgeon.”

“…a recent systematic review that includes our first survey suggests that there is no evidence that they [WPBAs] improve physician performance. It goes on to conclude that multisource feedback may be helpful, but that individual factors, context of feedback, and presence of facilitation (ie mentoring) may improve trainee responses.”

These sentiments will sound familiar to physicians, many of whom also feel frustrated at the widespread adoption of WBPAs, for which there is limited evidence of value for trainees in the real world. Valid concerns have been raised about the difficulties of applying theoretically helpful frameworks and tools to the realities of clinical life, and it is unclear where the numbers set by training boards have come from.

“ISCP has improved its interface, but it and other electronic portfolios deliver an increasingly overwhelming bureaucratic burden of WBAs and domains of evidence to include in a portfolio. These have rapidly become entrenched in postgraduate physician training in the UK, spreading a plague of box-ticking exercises that continue to increase year on year….It is of particular concern that so many trainees (80%) felt that ISCP did not improve their training after a modal average of over three years using it.”

Again these feelings will be familiar to many of those who have commented on this site and engaged with the debate on twitter. Time is precious. Many feel that the current demands on trainees, coupled with inadequate technology, steals  it away from busy trainees and trainers.

Perhaps it is time to ask the question, who is the ePortfolio for? Is it a learning tool for trainees? Is it an evidence vault for Royal Colleges to check off competencies of registered members? It is unclear to me what the aims of the NHS physician ePortfolio was at its inception. Has this been reassessed as it has expanded and evolved? These is great potential to improve the ePortfolio so that it serves the needs of trainees, trainers, assessors and higher bodies better. We have an opportunity to seek clarification and contribute to making the aims and expectations explicit. Let’s not let it pass us by.

The authors of the paper conclude:

“The performance of ISCP has improved in the 4 years since its inception with proportionately less negative feedback. British surgeons remain dissatisfied with several of its tools, in particular its workplace-based assessments. Half a decade on, these assessments remain without appropriate evidence of validity despite increasing demands upon trainees to complete quotas of them. With reduced permitted training hours, the growing online bureaucratic burden continues to demoralize busy surgical trainers and trainees.”

These conclusions should ring alarm bells not only for the Royal Colleges, but for the wider community of healthcare leaders. The NHS faces many challenges, and a demoralized workforce will struggle to face them. Physician and surgical trainees feel overburdened and undervalued. The system needs to change. Who will lead this change? And where will the ePortfolio fit in? Answers on a postcard…..

E.A. Pereira B.J. Dean.British SurgeonsExperiences of a Mandatory Online Workplace Based Assessment Portfolio Resurveyed Three Years On. Journal of Surgical Education. J Surg Educ. (2012) doi: 10.1016/j.jsurg.2012.06.019

A. Miller, J. Archer Impact of workplace based assessment on doctors’ education and performance: A systematic review. BMJ, 341 (2011), p. c5064

E.A. Pereira, B.J. Dean British surgeons’ experiences of mandatory online workplace-based assessment J R Soc Med, 102 (2009), pp. 287–293

S.A. Welchman Educating the surgeons of the future: The successes, pitfalls and principles of the ISCP. Bull R Coll Surg Engl, 94 (2012) online

W.C. Leung. Competency based medical training. Review, 325 (2002), pp. 693–696