Tag Archives: Royal College

NHS Hackday and the oPortfol.io

This weekend I was at NHS Hackday. Doctors, other healthcare workers, students, patients, organisations and software developers came together in their free time to make stuff that could make the NHS better.

Thanks to @londonlime

Thanks to @londonlime

I was astounded by the last Hackday. I didn’t think my expectations could be surpassed. They were.

The projects were diverse, aiming to solve everyday problems at all levels of the NHS. You can see the list on the Google doc and they’ll be on the wiki soon. The highly deserved winner was OpenHeart. The team used the amazing open source electronic health record at Moorfields Hospital, Open Eyes, and adapted it for use in Cardiology. The end result was stunning. It will save hours of doctors’ time, will create patient records that are much more understandable for patients themselves and for GPs, and will improve communication and therefore the quality of care.

Another favourite was Dementia scrapbook, an app to allow family and friends to contribute to a virtual scrapbook of memories and reminders. It has an easy to use touch interface that can be used by carers or people with dementia themselves. Dementia is common and this takes a very patient-centred approach to solving problems many of us may face in the future. I hope to see it available soon on the app store.

Cellcountr, initially hacked at the Liverpool hackday, was built on with additional features such as data visualisation and a customisable keyboard. It will be launched in the next month at a Pathology conference, and will then make a real difference to doctors, and their ability to accurately diagnosis patients with haematological conditions.

So what did we do? We created oPortfolio, an open API which allows trainee doctors to record learning events online, offline and on the go. It includes a webapp, a mobile-friendly site, an iPhone app, and an android app that all synch data. From nothing to all this in 36 hours! The team were incredible: full of talent, patience, and creativity.

Oportfolio

What does it do?

It solves an immediate need to log learning events on the go (see examples below). It lays the foundation for a more complex system to log assessments and meetings. With (quite a bit) more work it could be a simple open portfolio that doctors who are not currently in a training programme (eg LATs, people doing fellow jobs in between F2 and speciality training) could use to track their professional development. The funding model would have to be clarified as development and hosting is not free! It could also be an arena to experiment with and showcase new ways of organising a professional portfolio that could usefully feed into the debate on what and who a portfolio is for. It could highlight how different systems talking to each other and 3rd party apps and plug-ins have the potential to improve a core product. Another fabulous creation was Quicklog, an app to log personal development in performing procedures on the go. They built in data visualisation to encourage reflection and chart progress. It would be fantastic if the data from Quicklog could be integrated into a portfolio system. Anyone who is interested (and understands it!)  should look at the code on github for oPortfolio and Quicklog!

Screen Shot 2013-01-28 at 11.55.34

What does it not do?

It is not a replacement for the current ePortfolio system(s). The NES ePortfolio and others (eg surgical portfolio) are complex structures build up over years, with thousands and thousands of pounds of investment. Many have questioned whether they are value for money and I can’t answer that but good software does cost. Existing systems have  layers of access rights and methods of data extraction since these were priorities for the bodies who paid for them. They have cloud hosting and data security. People have spent years making them do what they do and it would be crazy to think they could be replicated in a weekend. They have their problems and must be improved but they are here to stay until a better alternative exists.

The focus of building a model Oportfolio was the user experience. If it was developed further it could fulfil a need for trainees who are not in a current training programme, who currently use various cobbled together documents on Evernote, phone notes apps, word documents and paper to record their learning and showcase their achievements when applying for jobs. With regular end-user input it could be beautiful, and a joy to use!

I am sure that our exploits this weekend will appear highly challenging and controversial to some. But I am not controversial. I have always highlighted the frustrations felt by trainees (which are well known) but advocated for engagement with all interested parties: individual trainees, trainers/educational supervisors, LETBs, Trusts, Royal Colleges, current ePortfolio provider NES, the GMC and HEE. We need to get our heads together and think about what the future of training will look like, what tools are needed to enhance learning, and how they will be funded.

The NHS can’t keep putting up with unintuitive, inflexible IT that doesn’t match the realities of practice. As demonstrated at NHS Hackday; intelligence, enthusiasm, creativity, a few humous sandwiches and some coffee can create magic. But that magic needs support and investment to make it sustainable. Muir Gray says change in the NHS will come from the bottom up. He is one of a few inspirational people at the top supporting projects in which frontline staff make a difference. We could do with a few more like him….

Advertisements

What to do with all that data?

I encourage you all to read the blog of NHS ePortfolio developer @zingmatter, and look at the presentation he presented at AMEE 2012 (a recent Medical Education Conference) “Assessing NHS ePortfolio behaviour: variations in the online activity of doctors as they progress through training.”

With thanks to http://www.acunu.com/ for the badge!

I was at the presentation at AMEE 2012 and, although the presentation title may nor sound gripping, I was fascinated to hear what could be learnt from the vast amounts of data ready and waiting to be analysed on NHS ePortfolio site use. The development team (including @zingmatter) had done a great job of drilling down into some of the data, using Google Analytics and internal tracking,  in order to filter out some meaningful information from the thousands and thousands of logins and episodes over a year.

The Prezi can be seen in the “elastic elephant” blog

However, my first thoughts on seeing the conclusions of the presentation were “they’re asking the wrong questions” and “if they wanted to know that they should have just asked the trainees.” Many of the peaks and troughs seen on the graphs were entirely predictable (ie pre-ARCP), and some of the conclusions drawn by the developers on “depth of use” were weak. I could explain away many of the findings, as I know how trainees use the site is a function of what hoops are put in front of them to jump through. I was also sceptical about the conclusion that trainees change their behaviour in relation to the ePortfolio over the course of their training. FYs and ST6s may interact differently with the site, but there are so many confounders that a snapshot comparison is not a valid way to assess this: a longitudinal study would be required.

Despite these reservations, reading @zingmatter’s blog gives me hope for the future, as the developers at NES are committed to engaging with the needs of users. In our often passionate discussions on social media (including this blog and twitter) we must remember that we come from very different perspectives, and have unique sets of knowledge and skills.

As @zingmatter points out:

“There is a balance between college needs and trainee needs in the design of an e-portfolio and possibly this type of data can help inform this debate.”

We also have to make sure we are not misdirecting our frustration at the wrong people, and potentially alienating them:

“while I’m happy to ask simple questions about user flow, user experience and so on, questions about the educational implications of this data have not been well addressed as it’s not really in my sphere of knowledge (or in my job description). I would see the research I presented at this conference as a ‘this is the kind of thing we can do’ exercise that should lead on to better designed questions that will allow us to understand how best to develop an e-portfolio that supports effective learning and development through the effective delivery of a training programme.”

I really hope we can work together to ask the right questions and use all the data we have to inform the process. All we need now is the Royal Colleges on board and we can really maximise the potential of the ePortfolio.

Just imagine a world in which trainees didn’t hate the NHS ePortfolio. It has the potential to be a useful tool to encourage self-directed learning, provide evidence of experience and achievements, act as a showcase for job applications and excellence awards, and strengthen the relationship between trainee and trainer. This world is far away, but perhaps we are starting to see the path forward…

Surgical spirit: what the surgeons think of their ePortfolio

An article published recently in the Journal of Surgical Education looks at the experience of surgical trainees and their ePortfolio. As a Medical Registrar I am in danger of being disowned by my colleagues for suggesting that we may be able to learn something from the surgeons! But in relation to ePortfolio use, many parallels can be drawn between the experience of surgical and physician trainees.

The surgical ePortfolio (ISCP) became mandatory for British surgical trainees 5 years ago, with a compulsory £125 annual fee. In 2008 widespread dissatisfaction was reported. This article (by Pereira and Dean) surveyed 359 users across all specialities and geographical areas. Although ratings improved between 2005 and 2008 trainees were underwhelmed overall. Unfortunately the article is not open access, and is behind a paywall, so I have selected some quotes for discussion below.

My love don’t cost a thing (but my training does….):

“An evaluation by ASiT estimated conservatively the upward spiralling costs of surgical training to the trainee to be £130,000 even before the introduction of MMC, with ISCP and its mandatory annual fee amounting to an additional £1000 over 8 years of surgical training.”

No medic would claim to be poorly paid, but there must be honesty and transparency with regard to the significant financial burden placed on trainees. This is likely to become more pressing as graduates leave medical school with escalating debts. Value for money is high on the agenda.

The current cost of the physician ePortfolio is only £18 per trainee per year, but perhaps this needs review, especially in the context of calls for investment to improve functionality. Trainees have a poor understanding of the costs of training and there is a disconnect between payment of JRCPTB fees and any visible outcomes in terms of education and training. Surely a lesson for all Colleges and higher bodies is that greater engagement and consultation with trainees could help prevent widespread and growing resentment.

A teacher affects eternity; he can never tell where his influence stops 

Sir William Osler, a great clinical teacher

“..incentive for trainer and assessor engagement remains lacking. It is important that trainers are properly recognized and rewarded for the time that they spend assessing and supervising trainees if obliged to use increasingly time-consuming methods, and we would welcome any system that encourages them.”

We must spare a thought for the Consultants who are striving to support us in our professional development. Demands on their time come from all directions and, unfortunately, postgraduate education and training is often the thing that loses out and gets pushed to the bottom of the mounting to-do pile. The system needs to reward and encourage senior clinicians so that they make time to give high quality feedback to trainees during WPBA completion. But this is a long term aim that feels intangible and unattainable. In the short term, reducing the time it takes to complete WPBA paperwork will make everyone happer. An app seems the quickest way to achieve this.

A call for EBT: Evidence Based Training

“Recently the JCST has specified a minimum of 40 WPBAs per year to be completed as a ‘quality indicator’ for surgical training and career progression…Regional training programs have set directives for mandatory WBAs per annum, ranging from a minimum JCST dictat of 40 to the 80 required in London. These present a great challenge upon time available to any practicing surgeon.”

“…a recent systematic review that includes our first survey suggests that there is no evidence that they [WPBAs] improve physician performance. It goes on to conclude that multisource feedback may be helpful, but that individual factors, context of feedback, and presence of facilitation (ie mentoring) may improve trainee responses.”

These sentiments will sound familiar to physicians, many of whom also feel frustrated at the widespread adoption of WBPAs, for which there is limited evidence of value for trainees in the real world. Valid concerns have been raised about the difficulties of applying theoretically helpful frameworks and tools to the realities of clinical life, and it is unclear where the numbers set by training boards have come from.

“ISCP has improved its interface, but it and other electronic portfolios deliver an increasingly overwhelming bureaucratic burden of WBAs and domains of evidence to include in a portfolio. These have rapidly become entrenched in postgraduate physician training in the UK, spreading a plague of box-ticking exercises that continue to increase year on year….It is of particular concern that so many trainees (80%) felt that ISCP did not improve their training after a modal average of over three years using it.”

Again these feelings will be familiar to many of those who have commented on this site and engaged with the debate on twitter. Time is precious. Many feel that the current demands on trainees, coupled with inadequate technology, steals  it away from busy trainees and trainers.

Perhaps it is time to ask the question, who is the ePortfolio for? Is it a learning tool for trainees? Is it an evidence vault for Royal Colleges to check off competencies of registered members? It is unclear to me what the aims of the NHS physician ePortfolio was at its inception. Has this been reassessed as it has expanded and evolved? These is great potential to improve the ePortfolio so that it serves the needs of trainees, trainers, assessors and higher bodies better. We have an opportunity to seek clarification and contribute to making the aims and expectations explicit. Let’s not let it pass us by.

The authors of the paper conclude:

“The performance of ISCP has improved in the 4 years since its inception with proportionately less negative feedback. British surgeons remain dissatisfied with several of its tools, in particular its workplace-based assessments. Half a decade on, these assessments remain without appropriate evidence of validity despite increasing demands upon trainees to complete quotas of them. With reduced permitted training hours, the growing online bureaucratic burden continues to demoralize busy surgical trainers and trainees.”

These conclusions should ring alarm bells not only for the Royal Colleges, but for the wider community of healthcare leaders. The NHS faces many challenges, and a demoralized workforce will struggle to face them. Physician and surgical trainees feel overburdened and undervalued. The system needs to change. Who will lead this change? And where will the ePortfolio fit in? Answers on a postcard…..

E.A. Pereira B.J. Dean.British SurgeonsExperiences of a Mandatory Online Workplace Based Assessment Portfolio Resurveyed Three Years On. Journal of Surgical Education. J Surg Educ. (2012) doi: 10.1016/j.jsurg.2012.06.019

A. Miller, J. Archer Impact of workplace based assessment on doctors’ education and performance: A systematic review. BMJ, 341 (2011), p. c5064

E.A. Pereira, B.J. Dean British surgeons’ experiences of mandatory online workplace-based assessment J R Soc Med, 102 (2009), pp. 287–293

S.A. Welchman Educating the surgeons of the future: The successes, pitfalls and principles of the ISCP. Bull R Coll Surg Engl, 94 (2012) online

W.C. Leung. Competency based medical training. Review, 325 (2002), pp. 693–696

The Bigger Picture

The NHS ePortfolio started as a local pilot project. It’s use was then expanded and expanded. There was no budget for this, and no vision for how trainees might want it to look and feel. It fulfills it’s purpose, is functional and allows trainees to record meetings and assessments. But is “functional” really good enough?

Here are some quotes from the ePortfolio site, and the JRCPTB (who fund the Physician version):

The NES (NHS Education for Scotland) ePortfolio has grown rapidly since its inception in August 2005 and now comprises over 20 versions for over 35,000 healthcare trainees within Scotland (Nursing, Midwifery, Dentistry and Pharmacy), across the United Kingdom (Medicine), and The Republic of Ireland (Medicine).

This ePortfolio is designed to support learning by providing a secure record of appraisal discussions, an ongoing personal development plan, workplace assessments plus reflection on clinical and other learning events. The ePortfolio links to the relevant GMC approved curricula appropriate to your stage of training.

The ePortfolio is designed to help gather and organise evidence in a way that is trainee centred and user friendly. The emphasis for this is on you as the trainee with support from your supervisor.

If the emphasis is on the trainee to gather and organise evidence, then the emphasis should be on the trainee to determine how that is done. There has been a lack of investment in the ePortfolio which has resulted in the current slow, unimaginative, frustrating version. This needs to change. As trainees we need to demand more.

Direct feedback to developers is needed to bring about improvements in day to day usability. Feedback to Royal Colleges etc is needed to communicate the need for investment in the overall structure and function of the system. We need an app and we need an overhaul of how the ePortfolio works in order to encourage trainee  and supervisor engagement. There is no obvious way in which this feedback can be given, hence this site….

Practice what you (aspire to) preach

What I don’t think the Colleges etc realise is that the ePortfolio is the only concrete thing we see as a result of handing over large amounts of money in JRCPTB fees.  Therefore in surveys it is said to be “poor value for money.”

Even more importantly the ePortfolio itself is a reflection of how training is currently viewed. It feels to many that we are aiming for competence – how uninspiring! What happened to striving for excellence? What happened to valuing the diversity and individuality of talent and interest amongst trainees? What happened to mentorship? Many trainees feel like just a number on a rota spreadsheet, getting through the day, jumping through hoops for assessments that often become tick-box exercises of no educational value. Of course competency in the required knowledge and skills is essential, but competency is a MINIMUM requirement. It often feels that this is all that is expected. Many trainees do extraordinary things but their supervisors and deaneries would never know this. One trainee’s portfolio looks just like any others, the only difference being whether they have done 6 mini-cex’s or 7.

If we want to inspire excellence, we need to show this in everything we do, and in every system we create. Trainees need a system that allows them to not only demonstrate that they have reached the minimum standard to progress, but that they are so much more than the sum of a few WPBAs. They need to be proud of their portfolio. Currently the system is failing them, with the danger of producing uninspired, undervalued, underachieving, burnt out doctors. Bad for trainees, bad for Trusts, bad for patients.

Time for a fresh look at what we expect of trainees, and what trainees can expect of the system.