Tag Archives: WPBAs

Surgical spirit: what the surgeons think of their ePortfolio

An article published recently in the Journal of Surgical Education looks at the experience of surgical trainees and their ePortfolio. As a Medical Registrar I am in danger of being disowned by my colleagues for suggesting that we may be able to learn something from the surgeons! But in relation to ePortfolio use, many parallels can be drawn between the experience of surgical and physician trainees.

The surgical ePortfolio (ISCP) became mandatory for British surgical trainees 5 years ago, with a compulsory £125 annual fee. In 2008 widespread dissatisfaction was reported. This article (by Pereira and Dean) surveyed 359 users across all specialities and geographical areas. Although ratings improved between 2005 and 2008 trainees were underwhelmed overall. Unfortunately the article is not open access, and is behind a paywall, so I have selected some quotes for discussion below.

My love don’t cost a thing (but my training does….):

“An evaluation by ASiT estimated conservatively the upward spiralling costs of surgical training to the trainee to be £130,000 even before the introduction of MMC, with ISCP and its mandatory annual fee amounting to an additional £1000 over 8 years of surgical training.”

No medic would claim to be poorly paid, but there must be honesty and transparency with regard to the significant financial burden placed on trainees. This is likely to become more pressing as graduates leave medical school with escalating debts. Value for money is high on the agenda.

The current cost of the physician ePortfolio is only £18 per trainee per year, but perhaps this needs review, especially in the context of calls for investment to improve functionality. Trainees have a poor understanding of the costs of training and there is a disconnect between payment of JRCPTB fees and any visible outcomes in terms of education and training. Surely a lesson for all Colleges and higher bodies is that greater engagement and consultation with trainees could help prevent widespread and growing resentment.

A teacher affects eternity; he can never tell where his influence stops 

Sir William Osler, a great clinical teacher

“..incentive for trainer and assessor engagement remains lacking. It is important that trainers are properly recognized and rewarded for the time that they spend assessing and supervising trainees if obliged to use increasingly time-consuming methods, and we would welcome any system that encourages them.”

We must spare a thought for the Consultants who are striving to support us in our professional development. Demands on their time come from all directions and, unfortunately, postgraduate education and training is often the thing that loses out and gets pushed to the bottom of the mounting to-do pile. The system needs to reward and encourage senior clinicians so that they make time to give high quality feedback to trainees during WPBA completion. But this is a long term aim that feels intangible and unattainable. In the short term, reducing the time it takes to complete WPBA paperwork will make everyone happer. An app seems the quickest way to achieve this.

A call for EBT: Evidence Based Training

“Recently the JCST has specified a minimum of 40 WPBAs per year to be completed as a ‘quality indicator’ for surgical training and career progression…Regional training programs have set directives for mandatory WBAs per annum, ranging from a minimum JCST dictat of 40 to the 80 required in London. These present a great challenge upon time available to any practicing surgeon.”

“…a recent systematic review that includes our first survey suggests that there is no evidence that they [WPBAs] improve physician performance. It goes on to conclude that multisource feedback may be helpful, but that individual factors, context of feedback, and presence of facilitation (ie mentoring) may improve trainee responses.”

These sentiments will sound familiar to physicians, many of whom also feel frustrated at the widespread adoption of WBPAs, for which there is limited evidence of value for trainees in the real world. Valid concerns have been raised about the difficulties of applying theoretically helpful frameworks and tools to the realities of clinical life, and it is unclear where the numbers set by training boards have come from.

“ISCP has improved its interface, but it and other electronic portfolios deliver an increasingly overwhelming bureaucratic burden of WBAs and domains of evidence to include in a portfolio. These have rapidly become entrenched in postgraduate physician training in the UK, spreading a plague of box-ticking exercises that continue to increase year on year….It is of particular concern that so many trainees (80%) felt that ISCP did not improve their training after a modal average of over three years using it.”

Again these feelings will be familiar to many of those who have commented on this site and engaged with the debate on twitter. Time is precious. Many feel that the current demands on trainees, coupled with inadequate technology, steals  it away from busy trainees and trainers.

Perhaps it is time to ask the question, who is the ePortfolio for? Is it a learning tool for trainees? Is it an evidence vault for Royal Colleges to check off competencies of registered members? It is unclear to me what the aims of the NHS physician ePortfolio was at its inception. Has this been reassessed as it has expanded and evolved? These is great potential to improve the ePortfolio so that it serves the needs of trainees, trainers, assessors and higher bodies better. We have an opportunity to seek clarification and contribute to making the aims and expectations explicit. Let’s not let it pass us by.

The authors of the paper conclude:

“The performance of ISCP has improved in the 4 years since its inception with proportionately less negative feedback. British surgeons remain dissatisfied with several of its tools, in particular its workplace-based assessments. Half a decade on, these assessments remain without appropriate evidence of validity despite increasing demands upon trainees to complete quotas of them. With reduced permitted training hours, the growing online bureaucratic burden continues to demoralize busy surgical trainers and trainees.”

These conclusions should ring alarm bells not only for the Royal Colleges, but for the wider community of healthcare leaders. The NHS faces many challenges, and a demoralized workforce will struggle to face them. Physician and surgical trainees feel overburdened and undervalued. The system needs to change. Who will lead this change? And where will the ePortfolio fit in? Answers on a postcard…..

E.A. Pereira B.J. Dean.British SurgeonsExperiences of a Mandatory Online Workplace Based Assessment Portfolio Resurveyed Three Years On. Journal of Surgical Education. J Surg Educ. (2012) doi: 10.1016/j.jsurg.2012.06.019

A. Miller, J. Archer Impact of workplace based assessment on doctors’ education and performance: A systematic review. BMJ, 341 (2011), p. c5064

E.A. Pereira, B.J. Dean British surgeons’ experiences of mandatory online workplace-based assessment J R Soc Med, 102 (2009), pp. 287–293

S.A. Welchman Educating the surgeons of the future: The successes, pitfalls and principles of the ISCP. Bull R Coll Surg Engl, 94 (2012) online

W.C. Leung. Competency based medical training. Review, 325 (2002), pp. 693–696

An app-ortunity

Quite reasonably I have been asked how an NHS ePortfolio app would benefit doctors, and what it would have to do to be worth any investment. In my opinion the need for an app is driven by the need to make WPBAs more relevant. An app would put control back in the hands of trainees, and make life significantly easier for trainers/assessors. This would reduce resentment towards WBPAs and would save an unimaginably huge amount of time for a stressed, squeezed, overworked profession.

The current situation:

I am doctor in training (this covers everyone who is not yet a Consultant/GP partner). I am required to complete a certain number of WPBAs to progress. One day I am at work and am on call admitting new patients to hospital. I think I’ve made a pretty thorough assessment of a patient with a condition I’ve not encountered before and ask my Consultant if, after presentation of the case on the post-take ward round they can fill in a mini-CEX. They say yes.

I present my case during the round and the Consultant provides some useful immediate feedback on my assessment, including a recommendation to read a recent review on the subject in an academic medical journal. However, the Consultant has another 7 patients to review after this and can’t stop to find a computer, login, wait for it to load up, access the NHS ePortfolio website, login and complete the assessment. “Send me a ticket” they say, with a genuine intent to complete is as soon as possible. My shift gets busier and after 13hours at work I go straight to bed when I get home. The next day i am very busy and forget to send the ticket via email. I remember when I get home but realise I don’t know the Consultant’s email address. It’s a weekend so I’m not in for another 2 days. I set a reminder with an alarm on my phone and on Monday the alarm prompts me to retrieve the email  from the hospital system and I send the ticket from the NHS ePortfolio site.

A week later the assessment has not been completed and I send a reminder. Three days after this I bump into the Consultant in the lunch queue and gently remind them about the mini-CEX. They make excuses, feel bad, and promise to do it ASAP.

A week after this the Consultant finally has some time for admin and discovers my reminder email in their inbox. They login and struggle to remember anything about the patient or the feedback they gave me. They have an overall impression of whether I’m any good or not and complete the assessment mainly based on this overall view, rather than the specifics of the case we discussed. I get an email to say that the assessment has been completed. At a later date I login and read the comments, which are brief, and get no educational benefit from the recording of the episode. I do however feel less stressed as that’s one less assessment to get ticked off. I can’t remember the author of the review recommended by the Consultant and never quite get round to searching for it.

A possible future situation:

One day I am at work and am on call admitting new patients to hospital. I think I’ve made a pretty thorough assessment of a patient with a condition I’ve not encountered before and ask my Consultant if, after presentation of the case on the post-take ward round they can fill in a mini-CEX. They say yes.

I present my case during the round and the Consultant provides some useful immediate feedback on my assessment. I get out my smartphone and login to the NHS ePortfolio app. I bring up the mini-CEX form and we complete it together adding comments based on the feedback the Consultant has just given, including the recommendation to read a recent review by author X in journal Y. There is a prompt to enter the Consultant’s email address so that they can validate the mini-CEX as an accurate representation of the assessment, and I input this as the Consultant dictates it. I save the form. The Consultant continues with the post-take ward round. I continue to admit new patients.

When I get home my phone picks up my wifi signal, and the ePortfolio app automatically synchs with my account so that the mini-CEX is uploaded. An email is sent to my Consultant and me to inform us of this new entry on my ePortfolio. I don’t have to waste time chasing up multiple assessments like this, so actually get round to looking up the review recommended by the Consultant, and learn something that will benefit my future patients.

It is essential that an NHS ePortfolio app:

  • is cross platform (iPhone, android etc)
  • can perform most functions offline with synching later with the main site. Most NHS hospitals have no wifi and poor phone sinal coverage. If an app required wifi it would be of no use to many, many, users

Another possible function would be to record reflection-in-action – essentially quick notes about things that happen that are particularly challenging, satisfying etc. There would then be scope to comment on this in the portfolio later (reflection-on-action).  Professionals must be reflective to learn and develop, but there is debate around the value of writing down these reflections. An app would at least make the process easier for those that wished to do this.

Oh, and of course ideally it would be free. But I’d pay £0.69 to make my life easier, wouldn’t you?

The NHS ePortfolio fan club

A recent twitter post directed me to the blog of Dr Fiona Pathiraja, which I encourage you to read. She asks:

Am I the only member of the ePortfolio fan club? If the recent vitriol on Twitter is anything to go by, one would be forgiven for thinking that the fan club comprises n=1.

She suggests that much of the negative press is unfair, and that medics don’t know how lucky we are, especially when compared to the private sector.

However, looking more closely her comments are a mix of praise for the concept of an ePortfolio itself, the structured nature of medical training programmes, and the job security that so many others rightly envy. She is honest about the shortcomings of the NHS ePortfolio in its current form and the need for change.

Of course, the ePortfolio is far from perfect. The end user is typically generation Y and expects technology to have the beautiful aesthetics and seamless functionality of their i-products. Improving functionality e.g. linking curriculum items to assessments, and aesthetics is essential for an improved ePortfolio.

This is exactly the point. We expect more. We deserve more. It is time for investment in the NHS ePortfolio to bring its functionality and aesthetics into the era of generation Y.

She also goes beyond the technical aspects of the current ePortfolio and makes requests for: a better and more transparent evidence-base for assessment tools including WPBAs; more training for supervisors on use of assessments; and the need to treat medics as adult learners:

As a cohort of intelligent, motivated young people, we are able to take responsibility for adult learning, but need to be treated like adults in order to do so.  If the ePortfolio gods are listening—engage with us, take constructive feedback, and try to improve the portfolio to inspire the medical generation Y.

I couldn’t have said it better myself. This site is a first step in connecting users, developers, and Royal Colleges.  If we can all work together to effect these changes, I might just join the fan club myself!