Tag Archives: WPBA

News from the trenches

If you have not yet already seen the blog by Dr Fitz “NHS ePortfolio WPBAs in CMT: are they educationally useful?” I recommend you make yourself a cup of tea and have a read. He has taken a significant amount of time out of his day as a medical trainee to document his experiences of using WorkPlace Based Assessments (WPBAs) in the “real world.” He has done this not because he likes to rant, but because he is genuinely interested in how he can support his own learning, and ensure that the assessments he undertakes are valid and useful. He is like many of us – he wants to be a better doctor – but wonders whether the system currently helps or hinders this.

teacup

Dr Fitz seems representative of many Core Medical Trainees (CMTs). In 16 months he has undertaken 51 assessments (ACATs*, CbDs*, miniCEXs*, DOPS* and Teaching Assessments) and 2 rounds of MSFs* (360 degree assessments) with 35 responses from colleagues in all. This is a little more than the minimum requirement for his ARCP (annual appraisal) and is a good bank of data on which to reflect. Well, it should be. Dr Fitz looks in detail at the contents of his ePortfolio and wonders what it really tells him. Of particular concern is the documented feedback that is at the heart of these assessments:

“Unfortunately the stats don’t look good. Over the course of 9 ACATs, covering the management of 55 patients over 12 months, I received 127 words of feedback. That is 14.1 words per ACAT and 2.3 words per patient seen. About 6 tweets.”

He is quick to point out that this does not necessarily reflect the amount or quality of verbal feedback he received, but

“…the educational benefit of my on-calls and the instruction I received from my consultants was separate to, not part of, my ACATs.”

Sadly the story is very similar for his CbDs, miniCexs and DOPS. The quality of the feedback documented on his ePortfolio is poor, and is of no use to him when he reviews his ePortfolio to prepare for his ARCP and consider what should be on his Personal Development Plan (PDP).

His conclusion is balanced and reflects the feelings of many trainees who have commented here previously:

“Overall, my experiences of the NHS ePortfolio assessments for CMT is that whilst they may act as a record of learning, they fail to be a useful educational tool in themselves. This is mainly due to the discord between how they are supposed to be completed and how they are completed in practice. Teaching, supervision and education is happening, but it is in spite of WPBAs rather than because of them.”

enthusiasm for portfoliosHe doesn’t end there and has several suggestions for improvements. They mirror comments already on this blog and range from urgent technical improvements (really, do I have to mention an app yet again?), to faculty development. This is just the kind of input Training Programme Directors, and National Programme leads want and need. Trainees have reasonable and real concerns about their training, and are engaged and enthusiastic about improving it.

Harnessing this enthusiasm will be vital.

If you haven’t already contributed to the Shape of Training review please do so. This is a collaboration between several higher bodies including the General Medical Council, Medical Education England, the Academy of Medical Royal Colleges, the Medical Schools Council, NHS Scotland, NHS Wales and the Conference of Postgraduate Deans of the UK. The review is considering what changes are needed to postgraduate medical training to make sure it continues to meet the needs of patients and health services in the future. This includes options to support greater training and workforce flexibility, and how to address the tensions between obtaining training and providing a service. You have until February to make your voice heard.

* ACAT= acute care assessment tool, CbD = case-based discussion, miniCex = mini Clinical encounter, DOPS = directly observed procedure

Acronym Soup: RCP, JRCPTB, WPBA, SLE, AoP, AoMRC ATDG

The Royal College of Physicians

This week I met with the Royal College of Physicians in London to discuss the NHS ePortfolio. After the many discussions I have had with trainees and trainers on this blog, on Twitter and in person I felt well prepared. I was sure the RCP should be taking a keen interest in the ePortfolio and it’s place in the wider context of Postgraduate Education, and through some brief email conversations I felt we were likely to have common ground. But as I walked through the falling leaves of Regent’s Park, I wondered what they would make of me and my graffiti-decorated blog…

It would have been unrealistic to expect to leave the meetings with all the answers, but I truly believe we have taken the first step towards a better solution. There are lots of details to work out, and I found myself discussing things like commissioning and procurement – yet another new language to learn! What was so positive was that there was an acknowledgement that trainees (and trainers) are dissatisfied with the ePortfolio, and that the concerns being raised are not idle complaining, but are educationally valid.

This is an important issue because it affects thousands of doctors, and because it relates to other areas of Postgraduate Training. The RCP wants to support trainees in their professional development and acknowledge that the ePortfolio is part of this process. There is also a realisation that technology has moved on, that the ePortfolio was not future-proofed, and that it is important to take stock and think about how to move forward. Financial contraints will impact on these plans and we need to be both realistic and imaginative in our decision making. The fact that the new RCP revalidation tools for Consultants have no interface or link with the trainee ePortfolio is another example of a lack of joined up thinking which we must avoid in future.

I was happy to hear that the RCP have almost finalised the details of the new ePortfolio User Reference group, which will have several trainee representatives. Hopefully this will not only impact on decision making, but also improve communication, so that we don’t in future have changes appear with no warning. It was also encouraging to hear more about the research being done on WPBAs/ SLEs. The RCP are putting considerable time and resources into reviewing the use and utility of WPBAs and the results of this research is likely to have a significant impact on the assessment systems for all medical specialties and shape training for the next 10 years. This is a great opportunity to engage with the College and have a real impact on Postgraduate Training.

There are many questions that do not yet have answers, in particular the question of funding and commissioning ePortfolio systems in the future. All doctors have a core set of common needs and I believe it is essential that we define these together and press for collaboration across Colleges, and open-ness in every part of the process.  The Trainee Roadmap is a first attempt at this, and I encourage you to contribute. The RCP seem to truly be putting trainees’ needs at the heart of what they do, and are (slowly) responding to the concerns of the thousands of you who have visited this site. But the structures of these organisations are complex and my attempts to find a way through the committees and funding streams of the RCP, the JRCPTB and others is already bringing on a headache.

We need to make sure this stays at the top of the agenda for the College, and I look forward to follow-up meetings in the future. I hope the other Colleges are watching, and are considering their own strategies. Otherwise they may find their trainees asking questions they can’t answer….

The Academy of Medical Royal Colleges Trainee Doctor Group

Coincidentally, in the same week I also went to give a presentation at the AoMRC ATDG (seriously, the number of acronyms in the world of medical education is mind-boggling).

There were many nods of assent as I described my frustrations with the current ePortfolio, and the demands placed on trainees and trainers to complete activities with little educational value. The representatives at the ATDG come from a wide spectrum of specialities including O&G, General Surgery, Anaesthetics and Intensive Care, Emergency Medicine, Pathology, General Practice, Ophthalmology, Psychiatry, Radiology and Medicine. Various ePortfolio systems are used by these trainees and some, such as the Haematologists, have to use two systems which, of course, do not talk to each other.

There was also great enthusiasm for sharing information, and for collaboration across Colleges. There was agreement that there are common needs for tools that support learning and professional development, capture workplace learning, log assessments, and provide evidence for appraisal. These needs are shared by doctors of all specialities and span the start of FY1 to retirement (in fact, since four UK medical schools use the NHS ePortfolio, these needs span Undergraduate as well as Postgraduate training). Technical aspects were touched on, and some absolutes were identified: such as a single sign on for all systems; a set of core standards for any ePortfolio used by doctors; and the need for flexibility for Colleges/Specialities and individuals. There was enthusiasm and hope for a future in which there is a simple but flexible ePortfolio system, with mobile support, that truly supports learning, and that makes people smile not scream when they login!

This committee is unique in bringing together trainees across specialities. It’s also full of lovely and enthusiastic people. The representatives are going back to their respective College trainee groups to gather information on what systems are used, and what trainees think of them. I hope we can then finalise a Trainee Roadmap and Core Requirements document, that will help us move forward.

Watch this space!

NHS Hackday and the ePortfolio Data Liberation Front

I spent this weekend in Liverpool at NHS Hackday. I had no idea what to expect. I had never met anyone there before and only knew a few names from twitter and google groups conversations in the weeks running up to the Hackday. I wasn’t completely sure I knew what a Hackday was.

I was astounded. 

I spend a lot of my life getting frustrated by the slow pace of change and the massive inefficiencies in the way that we work. I want to be freed up to spend time teaching, learning, writing, thinking, talking to patients and providing care. I hate unnecessary paperwork and bureaucracy. I hate meetings that don’t achieve anything.

NHS Hackday was a breath of fresh air. A diverse group of people with totally different backgrounds, most of whom had never met,  got together, discussed problems and solved them. In a weekend!

I will describe here what happened at the Hackday, what our project “The NHS ePortfolio Data Liberation Front” achieved and why it won 2nd place. There is far more info about how it is run, by who, and why on the NHS Hackday site. You can also see an interview with Carl (from OpenHealthcare) on Youtube:

 What follows are my personal impressions.

The HACKDAY CoNCEPT

On Saturday morning, whilst people were registering and getting coffee all those with ideas for projects wrote them on a board. Everyone gathered in the main hall and each idea had 2minutes to pitch. After all the pitches, people gathered around signs indicating each idea, and people formed groups. Then the work began. Groups discussed their vision, their proposed solution, and thrashed out conceptual and technical details. Fuelled by enthusiasm, tea, coffee and wotsits, software developers created things out of thin air (OK, out of data and code, blood, sweat and tears). Health professionals like me, who couldn’t code, were on hand to give context to the projects and point out real-world hurdles, which could then be worked around.

The NHS ePortfolio Data Liberation Front

Our group consisted of me (full of ideas, no understanding of code), Nicolas Tollervy, a developer (a genius with lots of patience and an incredible ability to work round every problem the project presented him with)  and Marcus Baw, (a GP who can code a bit and is a RCGP Health Informatics Group member, who was a great bridge and font of knowledge on NHS informatics issues).

We discussed some of the problems with the current NHS ePortfolio and possible workarounds. Since the code is not open and there is no API this was no simple problem.

We discussed the urgent need for an app to make trainees and trainers lives easier, and make WPBAs educationally valid. Any app would have to be able to get data into the ePortfolio so that a WPBA showed up not just in the personal library section as any random document, but in the WPBA section. With no code and no API this would be a great challenge.

We decided to focus on the fact that my data is locked in a vault in my ePortfolio. Whilst it is in there I can do nothing with it.

I want to liberate it, as I could then do anything I want with it! Ideas include:

  • visualise my achievements and progression
  • present the data in a way that my supervisor can see, understand and give feedback on
  • present the data in a way that makes it clear I have achieved all the competencies required by the JRCPTB for ARCPs and CCT
  • integrate the data into my CV, my online CV, an alternative ePortfolio (mahara, Googlios etc), use it for job applications
  • allow me to take the data with me into another role (progression or change of career path) eg Foundation Trainee –> Emergency medicine ACCS trainee –> GP trainee –> GP (all use different ePortfolio systems)

Not only is there a practical need for this, but the more we talked about it the more I realised that this is bigger than practicalities. It’s a philosophical argument. It’s my data. About me. I want it liberated. I can already download a PDF so clearly no-one disputes the fact that the data is mine and I have a right to it, but a PDF is useless.

@ntoll worked incredibly hard (with breaks for coffee, sandwiches, a trip to the pub and a curry house), came up against many problems and found ways around them all. We modified our plan as we went along, and decided that the best use of our time would be to do a ‘proof of concept’ and focus on a particular data set within the ePortfolio (there’s a lot of data in there, and it’s not organised as logically as you might imagine!). By the time we reached the submission deadline of 12.00 on Sunday we had something to show for our efforts. @ntoll made some finishing touches and we put together a brief presentation.

All 15 projects that had been selected from the pitches presented (a strict 5min and 1min for questions) to a panel of judges including: @MarkPriceDavies (chair), Ian Gilmore, Dr Farath Arshad, Zeinab Abdi, Francis Irving @frabcus, Dan Lynch @MethodDan, and Lilian Wiles. They deliberated and at 17.00 announced the winners.

The Other Projects

You can see more details of the projects on the NHS Hackday site, and get all the code through the wiki and on github, since all projects are open and shared. There were lots of fantastic projects but those that particularly caught my attention were:

  • AskIt (a general purpose question asking android app for any questionnaire you need – Waterlow, MUST score, falls assessment etc. Simple, effective, important!)
  • Making sense of patient comments (data visualisation from sources such as NHS Choices – massive potential applications)
  • CoIncidence Gate: a Conflict of Interest tool (scraped data from conflict of interest statements on Pubmed – something like 480,000 papers analysed!! Again, follow the link for more discussion on the massive potential applications of this project)
  • BleepBleep (making in-hospital communication better. An end to having to call switchboard. An end to the bleep! I trialled this, and am keen to help get it into hospitals now! Stop wasting time on hold)
  • GAAG: Guidelines at a Glance (there are well-studied barriers to doctors using guidelines, meaning patients don’t get best care. GAAG provides quick access to personalised most-used bits of guidelines on an app. Lots of potential for social add-ons, highlighting when guidelines change, seeing what peers use, rating bits of guidelines. See presentation for more info. Can’t wait to use it!)
  • Bloodcount (haematologists sit at very advanced microscopes counting different normal/abnormal cell types using very un-advanced technology = clicker and pen and paper. Bloodcount is a desktop system of a counter with keyboard shortcuts, reference normal and abnormal cells, report generation and learning function. Hard to describe to do it justice. A worthy winner!)
  • wtfdoc (an NHS jargon buster for patients and relatives as an app. Has a database, and if a term is unknown it will crowdsource answer through twitter and other sources. V clever!)

Why I think We Won a Prize

Our project won the First Scraperwiki prize for scraping, and came joint second overall on the day. I think the reasons we won are multiple:

  • @ntoll achieved amazing things writing novel code to scrape data out of a closed system and generate a .json file of hierarchical data that could then be used. In just a day and a half this was some achievement!
  • our pitch was powerful as this is an issue for all doctors of all specialities at all levels, especially with revalidation now a reality. Facilitating learning for healthcare professionals is in all our interests as a society
  • the concept of data liberation goes beyond this project. Who owns the data in public databases? Who owns the data in the NHS? What right does an individual have to their own data? What right does an institution have to keep it from them?

What next

I owe a huge thank you to the organisers, supporters, volunteers and participants at NHS Hackday Liverpool 2012.  And a special thanks to Scraperwiki for providing prizes including my beautiful new Google Nexus 7! This weekend I saw innovation in action, providing real, practical solutions to the day-to-day problems facing those who work in and use the NHS. Some of these solutions are now in use – today! Others will be worked on outside the Hackdays or at the next one. I have had my mind opened to new ways of working and have returned to work today full of enthusiasm and inspiration.

There’s no going back now. I’m a doctor who loves geeks who love the NHS, and I have the T-shirt to prove it.