The perfect ePortfolio

Thanks to @londonlime

Thanks to @londonlime

This weekend I am at NHS Hackday in Oxford. I have written about hackdays before. I am a huge enthusiast. It is amazing to see doctors, other healthcare workers, patients, organisations and software developers coming together in their free time to make stuff that could make the NHS better. A hackday IS agile software development, speeded up, with people motivated not by profit but by intellectual curiosity and a desire to make the world a little bit better.

There are some amazing people coming to NHS Hackday Oxford. Some of them are interested in rethinking a professional ePortfolio. The one we have currently is competent. It does a job. The creators at NES are great, but they are constrained by their history and location. In my opinion an IT project that supports thousands of healthcare professionals’ development should not be run by project managers in a Scottish NHS health board. Their customers are Royal Colleges, not ePortfolio users. However great NES are at their job are they really the best people to make the perfect ePortfolio?

As trainees we want more. In relation to the software we want seamless functionality, we want flexibility, we want personalisation, we want visualisation of data, we want speed, we want interoperability, we want openness and APIs, we want mobile offline data entry, we want intuitive navigation, we want reliability, we want to be encouraged and inspired, we want beauty.

Too much to ask? I hope not. 

I need to form these vague statements into some specifics between now and tomorrow. Wish me luck!

Where’s your user group?

A major contributor to trainees’ frustration with the ePortfolio has been the lack of user feedback. Trainees and trainers using the system day in day out have not had a clear and accountable way to give feedback to the Colleges which purchase the ePortfolio on their behalf. Those at NES managing the system, and the developers weaving their magic in binary don’t necessarily know how it is used on the front line. It has certainly not been an agile development process.

Screen Shot 2013-01-25 at 10.28.38

But there is no point looking back. We are here and must consider how we make things better for the future. Change is slow but it is possible.

The JRCPTB/Royal College of Physicians have actively responded to the pleas of trainees and have created an ePortfolio reference group.

The objectives of the group are to:

  1. Provide feedback and appropriate consultation to the JRCPTB on issues of ePortfolio management including performance, usability and required features.
  2. Assist the JRCPTB in the prioritisation of developments to ensure they are delivered in the best order for all our stakeholders.
  3. Ensure JRCPTB is informed of changes to training that may influence how or when the ePortfolio is used
  4. Assist JRCPTB where required in creation and delivery of appropriate communication about the ePortfolio

The group comprises ten members:

  • 3 trainee representatives – one to be from CMT
  • 3 local administrators from different areas of the UK and represent both Trust and Deanery administrators
  • 3 clinicians from different specialties
  • 1 JRCPTB administrator

In fact there are 4 trainee representatives. Lucky me, I am one of them! You can get in touch with me here, or on Twitter. You can also email the JRCPTB directly with your ePortfolio-related ideas: ePortfolioideas@jrcptb.org.uk.

This is a great step forward. If you’re a physician trainee you now have a direct line of communication to the JRCPTB to influence their decisions. It’s not perfect, it’s not agile, but it’s a start.

If you are not a physician trainee perhaps you should ask your College where your user group is……

Feedback from NES events

Late last year, NES who run the NHS ePortfolio, held a series of user group meetings. The objectives included “providing an opportunity for users to speak their minds and share their ideas to improve the system.

The results of the feedback events were collated by NES, and they published a summary. So what can we learn from this?

Screen Shot 2013-01-19 at 11.07.28

Firstly we must question whether the results are representative of the users of the ePortfolio. Unfortunately the feedback events were during normal working hours and very few trainees or clinicians were able to attend. It is clear from the feedback that there were a significant number of administrators present. Their views are important but are over-represented in this sample. The sample size of the feedback events is also very small. This is important considering the large number of users and the fact that the use of the system by different Colleges varies.

Despite this there is some useful information contained in the summary, which is grouped into categories:

  • what users like about the NHS ePortfolio (mainly that an ePortfolio is better than paper)
  • dislikes about the technical implementation and website design
  • dislikes related to the educational content or usage requirements of the ePortfolio
  • dislikes relating to the governance of the ePortfolio
  • what technical changes users would like made to the system
  • what new features users would like

The results are not surprising. Many of the comments align with those on this site. Users want an app, want the site to be more intuitive to use and navigate, and want it to have better and more flexible functionality. They want interoperability with other systems and they want clarity on the purpose of the ePortfolio “is it a learning tool or a recording tool?” They also want a lower burden of assessments.

NES must be congratulated for attempting to engage users and gain feedback, and for their openness in releasing the results, but I can’t believe there is anything in the comments that they did not know already. The problem is less lack of information, and more lack of action. But as NES highlights in their post “who makes decisions about the ePortfolio?” in the bigger picture the power really rests with the Colleges. 

Is it time to rethink how we use portfolios and workplace-based assessments in Postgraduate training? How can we adapt the software we have so that it works for us and we maximise the benefits of an ePortfolio for lifelong learning?  

What do you want from an ePortfolio?

Discussion of the NHS ePortfolio has been reignited on Twitter this evening, sparked by the publication of a paper in The Clinical Teacher “Evaluation of an established ePortfolio.”

The conclusions are:

“Continued support is needed for both trainees and supervisors in portfolio-building skills and in using the e-portfolio as an educational tool. Trainee tailored feedback is needed to ensure that portfolio-based assessment promotes lifelong, self-directed and reflective learners.”

My favourite quote is:

“Trainees remain unconvinced about the educational value of the e-portfolio”

An understatement perhaps?

The NHS ePortfolio

The comments this evening included some by people who have not been part of previous Twitter discussions, but they echo many sentiments I have heard before, and many comments on this site. They include frustration about the software itself, as well as complaints relating to the burden and perceived utility of workplace-based assessments.

So, it’s 2013, a New Year. Time to take a fresh look at the world and think about how we can turn all this frustration into positive change.

  • What do you think is the purpose of an ePortfolio?
  • What do you, as a trainee, want from an ePortfolio?
  • What value do you find in workplace-based assessments?

I will be writing more as soon as I get time – on funding and commissioning, on the need for trainees to join together across specialities, and on the future in the eyes of the various stakeholders. In the meantime please leave your comments here.

We need your voice. We need a better system – sometime before I become a Consultant would be nice……

2012 in review

The WordPress.com stats helper monkeys have prepared a 2012 annual report for me. Aren’t they nice?

Here’s an excerpt:

4,329 films were submitted to the 2012 Cannes Film Festival. This blog had 13,000 views in 2012. If each view were a film, this blog would power 3 Film Festivals

Click here to see the complete report.

This blog has had a lot of support this year. Thank you to everyone who contributed, commented, argued, debated, encouraged and supported it’s ideals along the way.

2012 was the year of debate about the purpose and functionality of the NHS ePortfolio. What will 2013 bring?

News from the trenches

If you have not yet already seen the blog by Dr Fitz “NHS ePortfolio WPBAs in CMT: are they educationally useful?” I recommend you make yourself a cup of tea and have a read. He has taken a significant amount of time out of his day as a medical trainee to document his experiences of using WorkPlace Based Assessments (WPBAs) in the “real world.” He has done this not because he likes to rant, but because he is genuinely interested in how he can support his own learning, and ensure that the assessments he undertakes are valid and useful. He is like many of us – he wants to be a better doctor – but wonders whether the system currently helps or hinders this.

teacup

Dr Fitz seems representative of many Core Medical Trainees (CMTs). In 16 months he has undertaken 51 assessments (ACATs*, CbDs*, miniCEXs*, DOPS* and Teaching Assessments) and 2 rounds of MSFs* (360 degree assessments) with 35 responses from colleagues in all. This is a little more than the minimum requirement for his ARCP (annual appraisal) and is a good bank of data on which to reflect. Well, it should be. Dr Fitz looks in detail at the contents of his ePortfolio and wonders what it really tells him. Of particular concern is the documented feedback that is at the heart of these assessments:

“Unfortunately the stats don’t look good. Over the course of 9 ACATs, covering the management of 55 patients over 12 months, I received 127 words of feedback. That is 14.1 words per ACAT and 2.3 words per patient seen. About 6 tweets.”

He is quick to point out that this does not necessarily reflect the amount or quality of verbal feedback he received, but

“…the educational benefit of my on-calls and the instruction I received from my consultants was separate to, not part of, my ACATs.”

Sadly the story is very similar for his CbDs, miniCexs and DOPS. The quality of the feedback documented on his ePortfolio is poor, and is of no use to him when he reviews his ePortfolio to prepare for his ARCP and consider what should be on his Personal Development Plan (PDP).

His conclusion is balanced and reflects the feelings of many trainees who have commented here previously:

“Overall, my experiences of the NHS ePortfolio assessments for CMT is that whilst they may act as a record of learning, they fail to be a useful educational tool in themselves. This is mainly due to the discord between how they are supposed to be completed and how they are completed in practice. Teaching, supervision and education is happening, but it is in spite of WPBAs rather than because of them.”

enthusiasm for portfoliosHe doesn’t end there and has several suggestions for improvements. They mirror comments already on this blog and range from urgent technical improvements (really, do I have to mention an app yet again?), to faculty development. This is just the kind of input Training Programme Directors, and National Programme leads want and need. Trainees have reasonable and real concerns about their training, and are engaged and enthusiastic about improving it.

Harnessing this enthusiasm will be vital.

If you haven’t already contributed to the Shape of Training review please do so. This is a collaboration between several higher bodies including the General Medical Council, Medical Education England, the Academy of Medical Royal Colleges, the Medical Schools Council, NHS Scotland, NHS Wales and the Conference of Postgraduate Deans of the UK. The review is considering what changes are needed to postgraduate medical training to make sure it continues to meet the needs of patients and health services in the future. This includes options to support greater training and workforce flexibility, and how to address the tensions between obtaining training and providing a service. You have until February to make your voice heard.

* ACAT= acute care assessment tool, CbD = case-based discussion, miniCex = mini Clinical encounter, DOPS = directly observed procedure

Open Badges in MedEd?

Conversations started online lead to all sorts of places. Today they led me to Leeds School of Medicine to learn about Open Badges in Healthcare Education.

There’s a lot to say on this subject so please follow the links to learn more. I have tried to summarise some of the key opportunities and challenges and how they relate specifically to doctor’s ePortfolios and Postgraduate Training. This is not because Open Badges are limited to this context, but because this blog is focused on these issues. Overall my personal opinions are:

  • there is a need for a better method to collate and accredit learning, particularly informal learning. CVs are limiting
  • this is not a threat to established models of accreditation, or to institutions, but a complimentary system
  • Open badges have great potential as a mechanism of celebrating excellence and supporting medical trainees in differentiating themselves
  • as with all technology/software we need to be clear about what it can and can’t do, and not expect a panacea
  • as doctors we are currently limited by the fact that existing systems do not interface with other systems. No API = no possibility for progress. The whole point is that open badges cut across multiple systems and contexts, so they must not exist in yet another silo, separate from eg the ePortfolio. We need progress in this urgently
  • the drive for these new ideas and opportunities will come from the community, not institutions. Most institutions are likely to be sceptical initially but catch on later once the principle is established (examples would include Hospital Trusts, Royal Colleges, LETBs, Universities…). I would love to be proved wrong about this!

The Concept

Open badges are a mechanism for accrediting learning that happens everywhere. They are more than a graphic that you slap onto your online profile. The image has inbuilt into its’ code some essential data:

  • criteria for attaining the badge
  • the issuer
  • the earner
  • link to evidence

Doug put my picture into his presentation, so I’m returning the ‘favour’ You can find out more about him here: http://dougbelshaw.com/blog/about/

This is important as this means they can’t be simply copied or faked, and there is a way to probe what the badge means by checking the criteria and evidence. Much more information is on the Mozilla wiki, site, and collated blogs.

Future developments such as the ability to encode verification and endorsement mechanisms will be essential to win over the evidence-focused and reputation-aware landscape of healthcare education.

Doug’s presentation from today is on slideshare.

Opportunities for Postgraduate Training

Learning does not just happen in lecture theatres. We know this and demonstrate it every day. As medical students and doctors we piece together our education and training from a diverse range of informal and formal providers and contexts.

Last week I had many learning opportunities: I sat in on a specialist clinic, attended a Radiology meeting, completed an e-learning module, read several articles through UptoDate and pubmed searches, read a blog, contributed to debates on Twitter and followed links from tweets to journal papers, went to a seminar, and had a case discussion with a senior colleague. Trying to log all of this in existing tools is a challenge. My clinical NHS ePortfolio is certainly not flexible enough to facilitate this, and is unlikely to ever be, since it’s structure is dictated by committees of people who don’t use it. But the core system doesn’t have to fulfil every need if it interfaces with other systems. Open Badges could fit into this model very well.

Things that could be badged in medical education (a small and not at all exhaustive list):

  • attendance at a teaching session (1 badge), attendance at 80% of all ‘mandatory’ teaching sessions in your trust (separate badges build up like cheeses in trivial pursuit to unlock ‘St Elsewhere FY teaching attendance badge’)
  • contribution to #twitjc as a one off (1 badge), on a regular basis (higher badge), as host (super-badge)
  • completion of an online e-learning module (1 badge), completion of 15  modules relevant to your training programme (build up to unlock higher badge – BMJ already follow this principle with Silver/Platinum user. Wouldn’t it be great to draw other providers together along similar principles?)
  • giving a peer-led teaching session (1 badge), running a teaching programme as a result of an identified need (higher badge), having an impact ie safer handover/fewer bacteraemias/better compliance with antibiotic policy (super-badge)
  • completing generic mandatory training modules – many of which have been moved online (eg hand-washing, manual handling, equality and diversity), which could then be transported to a new Trust. A junior doctor could show their badges and not have to repeat generic training when they move jobs at 4 months, and instead spend time doing locally-specific training and then meeting the team (this was discussed at the workshop – great idea that I do not take credit for!)

Challenges

Many of these are discussed in more detail on the Mozilla pages (with additional technical, pedagogical, conceptual and philosophical considerations).

  • Equivalence: is my equality and diversity training module at St Elsewhere NHS Trust equivalent to your E&D module at St Somewhere Else? And will you accept is as such? Who decides?
  • Fake-ability: someone could set themselves up as a fake version of Highly Prestigious University and issue badges in their name. Future developments of verification (eg verified accounts) and endorsement would help with this
  • Over-exposure: If you can badge anything, does this devalue them as a currency? Not necessarily. There will be hierarchies of badge, just like there are hierarchies and power structures in other spheres. A peer-issued badge for ‘general awesomeness‘ would be value-less for a job application and no badge earner would choose to display this badge in that context. In contrast a University issued badge for ‘significant contribution to bedside Undergraduate Teaching’ or a Trust issued badge for ‘leadership in quality improvement’ would be worth significantly more.
  • Validity: What if someone earned a badge for hand-washing, but couldn’t then demonstrate the skill? Theory does not necessarily transfer to practice, but this is not a problem of open badges themselves, but of all teaching and learning
  • Admin support: the creation and issuing of badges needs thought and planning. It does not have to mean lots of additional work but of course would require initial startup resources. The discussion of beenfits vs costs needs to be fleshed out, and systems worked out to make badge issuing something people feel supported in doing, and will seek to do. The guys from myknowledgemap and reallymanagingassessment tools have thought about this already, and done a lot of the work. They have envisaged ways to support less code-literate mortals (like me) in the process. We can learn from their expertise, share the work, share costs, share, share, share….

What’s inspiring about the Mozilla Open Badges project is the level of transparency  and collaboration. The Mozilla Foundation is a global non-profit whose only motive is to “promote openness, innovation and participation in the internet”. This open-ness is incredibly powerful. They share all their code and processes, which allows others to create widgets and plug-ins and build layers of code/systems on top of their core system. This does not challenge or threaten Mozilla. It enhances their product and makes it more attractive and useful. This is a general lesson we can learn and apply to other contexts. The most obvious example would be NHS Hackday. Imagine what we could do with an NHS ePortfolio API (for a start we could already have built an app)! Imagine if Open Badges interfaced with the NHS ePortfolio! That would be one way to gather all the informal (and formal) lifelong-learning that is already taking place, and strongly encouraged by Royal Colleges, the GMC, and other healthcare bodies – from medical school entry to retirement.

I think an important point is that Open Badges are not a gimic. They are not a niche internet project. They are a response to a much bigger challenge to traditional learning. You may not think they are the right response, but they should be seen in this context. There has been a shift in how and where learning occurs which is challenging but positive. Better informed, engaged and motivated clinicians provide better patient care: what we’re all here for in the end. We need systems that facilitate and celebrate learning in new contexts. As learners we yearn for it it. As institutions we have a duty to support it. As patients we should welcome it. Open Badges could be such a system.

Thanks to Leeds School of Medicine for being so open and welcoming; to myknowledgemap for being so positive about collaborating; to Tim from NES for putting up with my ranting; and to Doug from Mozilla for his pragmatism,  enthusiasm and expert workshop facilitation.