Tag Archives: assessment

News from the trenches

If you have not yet already seen the blog by Dr Fitz “NHS ePortfolio WPBAs in CMT: are they educationally useful?” I recommend you make yourself a cup of tea and have a read. He has taken a significant amount of time out of his day as a medical trainee to document his experiences of using WorkPlace Based Assessments (WPBAs) in the “real world.” He has done this not because he likes to rant, but because he is genuinely interested in how he can support his own learning, and ensure that the assessments he undertakes are valid and useful. He is like many of us – he wants to be a better doctor – but wonders whether the system currently helps or hinders this.

teacup

Dr Fitz seems representative of many Core Medical Trainees (CMTs). In 16 months he has undertaken 51 assessments (ACATs*, CbDs*, miniCEXs*, DOPS* and Teaching Assessments) and 2 rounds of MSFs* (360 degree assessments) with 35 responses from colleagues in all. This is a little more than the minimum requirement for his ARCP (annual appraisal) and is a good bank of data on which to reflect. Well, it should be. Dr Fitz looks in detail at the contents of his ePortfolio and wonders what it really tells him. Of particular concern is the documented feedback that is at the heart of these assessments:

“Unfortunately the stats don’t look good. Over the course of 9 ACATs, covering the management of 55 patients over 12 months, I received 127 words of feedback. That is 14.1 words per ACAT and 2.3 words per patient seen. About 6 tweets.”

He is quick to point out that this does not necessarily reflect the amount or quality of verbal feedback he received, but

“…the educational benefit of my on-calls and the instruction I received from my consultants was separate to, not part of, my ACATs.”

Sadly the story is very similar for his CbDs, miniCexs and DOPS. The quality of the feedback documented on his ePortfolio is poor, and is of no use to him when he reviews his ePortfolio to prepare for his ARCP and consider what should be on his Personal Development Plan (PDP).

His conclusion is balanced and reflects the feelings of many trainees who have commented here previously:

“Overall, my experiences of the NHS ePortfolio assessments for CMT is that whilst they may act as a record of learning, they fail to be a useful educational tool in themselves. This is mainly due to the discord between how they are supposed to be completed and how they are completed in practice. Teaching, supervision and education is happening, but it is in spite of WPBAs rather than because of them.”

enthusiasm for portfoliosHe doesn’t end there and has several suggestions for improvements. They mirror comments already on this blog and range from urgent technical improvements (really, do I have to mention an app yet again?), to faculty development. This is just the kind of input Training Programme Directors, and National Programme leads want and need. Trainees have reasonable and real concerns about their training, and are engaged and enthusiastic about improving it.

Harnessing this enthusiasm will be vital.

If you haven’t already contributed to the Shape of Training review please do so. This is a collaboration between several higher bodies including the General Medical Council, Medical Education England, the Academy of Medical Royal Colleges, the Medical Schools Council, NHS Scotland, NHS Wales and the Conference of Postgraduate Deans of the UK. The review is considering what changes are needed to postgraduate medical training to make sure it continues to meet the needs of patients and health services in the future. This includes options to support greater training and workforce flexibility, and how to address the tensions between obtaining training and providing a service. You have until February to make your voice heard.

* ACAT= acute care assessment tool, CbD = case-based discussion, miniCex = mini Clinical encounter, DOPS = directly observed procedure

An app-ortunity

Quite reasonably I have been asked how an NHS ePortfolio app would benefit doctors, and what it would have to do to be worth any investment. In my opinion the need for an app is driven by the need to make WPBAs more relevant. An app would put control back in the hands of trainees, and make life significantly easier for trainers/assessors. This would reduce resentment towards WBPAs and would save an unimaginably huge amount of time for a stressed, squeezed, overworked profession.

The current situation:

I am doctor in training (this covers everyone who is not yet a Consultant/GP partner). I am required to complete a certain number of WPBAs to progress. One day I am at work and am on call admitting new patients to hospital. I think I’ve made a pretty thorough assessment of a patient with a condition I’ve not encountered before and ask my Consultant if, after presentation of the case on the post-take ward round they can fill in a mini-CEX. They say yes.

I present my case during the round and the Consultant provides some useful immediate feedback on my assessment, including a recommendation to read a recent review on the subject in an academic medical journal. However, the Consultant has another 7 patients to review after this and can’t stop to find a computer, login, wait for it to load up, access the NHS ePortfolio website, login and complete the assessment. “Send me a ticket” they say, with a genuine intent to complete is as soon as possible. My shift gets busier and after 13hours at work I go straight to bed when I get home. The next day i am very busy and forget to send the ticket via email. I remember when I get home but realise I don’t know the Consultant’s email address. It’s a weekend so I’m not in for another 2 days. I set a reminder with an alarm on my phone and on Monday the alarm prompts me to retrieve the email  from the hospital system and I send the ticket from the NHS ePortfolio site.

A week later the assessment has not been completed and I send a reminder. Three days after this I bump into the Consultant in the lunch queue and gently remind them about the mini-CEX. They make excuses, feel bad, and promise to do it ASAP.

A week after this the Consultant finally has some time for admin and discovers my reminder email in their inbox. They login and struggle to remember anything about the patient or the feedback they gave me. They have an overall impression of whether I’m any good or not and complete the assessment mainly based on this overall view, rather than the specifics of the case we discussed. I get an email to say that the assessment has been completed. At a later date I login and read the comments, which are brief, and get no educational benefit from the recording of the episode. I do however feel less stressed as that’s one less assessment to get ticked off. I can’t remember the author of the review recommended by the Consultant and never quite get round to searching for it.

A possible future situation:

One day I am at work and am on call admitting new patients to hospital. I think I’ve made a pretty thorough assessment of a patient with a condition I’ve not encountered before and ask my Consultant if, after presentation of the case on the post-take ward round they can fill in a mini-CEX. They say yes.

I present my case during the round and the Consultant provides some useful immediate feedback on my assessment. I get out my smartphone and login to the NHS ePortfolio app. I bring up the mini-CEX form and we complete it together adding comments based on the feedback the Consultant has just given, including the recommendation to read a recent review by author X in journal Y. There is a prompt to enter the Consultant’s email address so that they can validate the mini-CEX as an accurate representation of the assessment, and I input this as the Consultant dictates it. I save the form. The Consultant continues with the post-take ward round. I continue to admit new patients.

When I get home my phone picks up my wifi signal, and the ePortfolio app automatically synchs with my account so that the mini-CEX is uploaded. An email is sent to my Consultant and me to inform us of this new entry on my ePortfolio. I don’t have to waste time chasing up multiple assessments like this, so actually get round to looking up the review recommended by the Consultant, and learn something that will benefit my future patients.

It is essential that an NHS ePortfolio app:

  • is cross platform (iPhone, android etc)
  • can perform most functions offline with synching later with the main site. Most NHS hospitals have no wifi and poor phone sinal coverage. If an app required wifi it would be of no use to many, many, users

Another possible function would be to record reflection-in-action – essentially quick notes about things that happen that are particularly challenging, satisfying etc. There would then be scope to comment on this in the portfolio later (reflection-on-action).  Professionals must be reflective to learn and develop, but there is debate around the value of writing down these reflections. An app would at least make the process easier for those that wished to do this.

Oh, and of course ideally it would be free. But I’d pay £0.69 to make my life easier, wouldn’t you?