Tag Archives: OpenSource

Open Badges in MedEd?

Conversations started online lead to all sorts of places. Today they led me to Leeds School of Medicine to learn about Open Badges in Healthcare Education.

There’s a lot to say on this subject so please follow the links to learn more. I have tried to summarise some of the key opportunities and challenges and how they relate specifically to doctor’s ePortfolios and Postgraduate Training. This is not because Open Badges are limited to this context, but because this blog is focused on these issues. Overall my personal opinions are:

  • there is a need for a better method to collate and accredit learning, particularly informal learning. CVs are limiting
  • this is not a threat to established models of accreditation, or to institutions, but a complimentary system
  • Open badges have great potential as a mechanism of celebrating excellence and supporting medical trainees in differentiating themselves
  • as with all technology/software we need to be clear about what it can and can’t do, and not expect a panacea
  • as doctors we are currently limited by the fact that existing systems do not interface with other systems. No API = no possibility for progress. The whole point is that open badges cut across multiple systems and contexts, so they must not exist in yet another silo, separate from eg the ePortfolio. We need progress in this urgently
  • the drive for these new ideas and opportunities will come from the community, not institutions. Most institutions are likely to be sceptical initially but catch on later once the principle is established (examples would include Hospital Trusts, Royal Colleges, LETBs, Universities…). I would love to be proved wrong about this!

The Concept

Open badges are a mechanism for accrediting learning that happens everywhere. They are more than a graphic that you slap onto your online profile. The image has inbuilt into its’ code some essential data:

  • criteria for attaining the badge
  • the issuer
  • the earner
  • link to evidence

Doug put my picture into his presentation, so I’m returning the ‘favour’ You can find out more about him here: http://dougbelshaw.com/blog/about/

This is important as this means they can’t be simply copied or faked, and there is a way to probe what the badge means by checking the criteria and evidence. Much more information is on the Mozilla wiki, site, and collated blogs.

Future developments such as the ability to encode verification and endorsement mechanisms will be essential to win over the evidence-focused and reputation-aware landscape of healthcare education.

Doug’s presentation from today is on slideshare.

Opportunities for Postgraduate Training

Learning does not just happen in lecture theatres. We know this and demonstrate it every day. As medical students and doctors we piece together our education and training from a diverse range of informal and formal providers and contexts.

Last week I had many learning opportunities: I sat in on a specialist clinic, attended a Radiology meeting, completed an e-learning module, read several articles through UptoDate and pubmed searches, read a blog, contributed to debates on Twitter and followed links from tweets to journal papers, went to a seminar, and had a case discussion with a senior colleague. Trying to log all of this in existing tools is a challenge. My clinical NHS ePortfolio is certainly not flexible enough to facilitate this, and is unlikely to ever be, since it’s structure is dictated by committees of people who don’t use it. But the core system doesn’t have to fulfil every need if it interfaces with other systems. Open Badges could fit into this model very well.

Things that could be badged in medical education (a small and not at all exhaustive list):

  • attendance at a teaching session (1 badge), attendance at 80% of all ‘mandatory’ teaching sessions in your trust (separate badges build up like cheeses in trivial pursuit to unlock ‘St Elsewhere FY teaching attendance badge’)
  • contribution to #twitjc as a one off (1 badge), on a regular basis (higher badge), as host (super-badge)
  • completion of an online e-learning module (1 badge), completion of 15  modules relevant to your training programme (build up to unlock higher badge – BMJ already follow this principle with Silver/Platinum user. Wouldn’t it be great to draw other providers together along similar principles?)
  • giving a peer-led teaching session (1 badge), running a teaching programme as a result of an identified need (higher badge), having an impact ie safer handover/fewer bacteraemias/better compliance with antibiotic policy (super-badge)
  • completing generic mandatory training modules – many of which have been moved online (eg hand-washing, manual handling, equality and diversity), which could then be transported to a new Trust. A junior doctor could show their badges and not have to repeat generic training when they move jobs at 4 months, and instead spend time doing locally-specific training and then meeting the team (this was discussed at the workshop – great idea that I do not take credit for!)

Challenges

Many of these are discussed in more detail on the Mozilla pages (with additional technical, pedagogical, conceptual and philosophical considerations).

  • Equivalence: is my equality and diversity training module at St Elsewhere NHS Trust equivalent to your E&D module at St Somewhere Else? And will you accept is as such? Who decides?
  • Fake-ability: someone could set themselves up as a fake version of Highly Prestigious University and issue badges in their name. Future developments of verification (eg verified accounts) and endorsement would help with this
  • Over-exposure: If you can badge anything, does this devalue them as a currency? Not necessarily. There will be hierarchies of badge, just like there are hierarchies and power structures in other spheres. A peer-issued badge for ‘general awesomeness‘ would be value-less for a job application and no badge earner would choose to display this badge in that context. In contrast a University issued badge for ‘significant contribution to bedside Undergraduate Teaching’ or a Trust issued badge for ‘leadership in quality improvement’ would be worth significantly more.
  • Validity: What if someone earned a badge for hand-washing, but couldn’t then demonstrate the skill? Theory does not necessarily transfer to practice, but this is not a problem of open badges themselves, but of all teaching and learning
  • Admin support: the creation and issuing of badges needs thought and planning. It does not have to mean lots of additional work but of course would require initial startup resources. The discussion of beenfits vs costs needs to be fleshed out, and systems worked out to make badge issuing something people feel supported in doing, and will seek to do. The guys from myknowledgemap and reallymanagingassessment tools have thought about this already, and done a lot of the work. They have envisaged ways to support less code-literate mortals (like me) in the process. We can learn from their expertise, share the work, share costs, share, share, share….

What’s inspiring about the Mozilla Open Badges project is the level of transparency  and collaboration. The Mozilla Foundation is a global non-profit whose only motive is to “promote openness, innovation and participation in the internet”. This open-ness is incredibly powerful. They share all their code and processes, which allows others to create widgets and plug-ins and build layers of code/systems on top of their core system. This does not challenge or threaten Mozilla. It enhances their product and makes it more attractive and useful. This is a general lesson we can learn and apply to other contexts. The most obvious example would be NHS Hackday. Imagine what we could do with an NHS ePortfolio API (for a start we could already have built an app)! Imagine if Open Badges interfaced with the NHS ePortfolio! That would be one way to gather all the informal (and formal) lifelong-learning that is already taking place, and strongly encouraged by Royal Colleges, the GMC, and other healthcare bodies – from medical school entry to retirement.

I think an important point is that Open Badges are not a gimic. They are not a niche internet project. They are a response to a much bigger challenge to traditional learning. You may not think they are the right response, but they should be seen in this context. There has been a shift in how and where learning occurs which is challenging but positive. Better informed, engaged and motivated clinicians provide better patient care: what we’re all here for in the end. We need systems that facilitate and celebrate learning in new contexts. As learners we yearn for it it. As institutions we have a duty to support it. As patients we should welcome it. Open Badges could be such a system.

Thanks to Leeds School of Medicine for being so open and welcoming; to myknowledgemap for being so positive about collaborating; to Tim from NES for putting up with my ranting; and to Doug from Mozilla for his pragmatism,  enthusiasm and expert workshop facilitation. 

Surgical spirit: what the surgeons think of their ePortfolio

An article published recently in the Journal of Surgical Education looks at the experience of surgical trainees and their ePortfolio. As a Medical Registrar I am in danger of being disowned by my colleagues for suggesting that we may be able to learn something from the surgeons! But in relation to ePortfolio use, many parallels can be drawn between the experience of surgical and physician trainees.

The surgical ePortfolio (ISCP) became mandatory for British surgical trainees 5 years ago, with a compulsory £125 annual fee. In 2008 widespread dissatisfaction was reported. This article (by Pereira and Dean) surveyed 359 users across all specialities and geographical areas. Although ratings improved between 2005 and 2008 trainees were underwhelmed overall. Unfortunately the article is not open access, and is behind a paywall, so I have selected some quotes for discussion below.

My love don’t cost a thing (but my training does….):

“An evaluation by ASiT estimated conservatively the upward spiralling costs of surgical training to the trainee to be £130,000 even before the introduction of MMC, with ISCP and its mandatory annual fee amounting to an additional £1000 over 8 years of surgical training.”

No medic would claim to be poorly paid, but there must be honesty and transparency with regard to the significant financial burden placed on trainees. This is likely to become more pressing as graduates leave medical school with escalating debts. Value for money is high on the agenda.

The current cost of the physician ePortfolio is only £18 per trainee per year, but perhaps this needs review, especially in the context of calls for investment to improve functionality. Trainees have a poor understanding of the costs of training and there is a disconnect between payment of JRCPTB fees and any visible outcomes in terms of education and training. Surely a lesson for all Colleges and higher bodies is that greater engagement and consultation with trainees could help prevent widespread and growing resentment.

A teacher affects eternity; he can never tell where his influence stops 

Sir William Osler, a great clinical teacher

“..incentive for trainer and assessor engagement remains lacking. It is important that trainers are properly recognized and rewarded for the time that they spend assessing and supervising trainees if obliged to use increasingly time-consuming methods, and we would welcome any system that encourages them.”

We must spare a thought for the Consultants who are striving to support us in our professional development. Demands on their time come from all directions and, unfortunately, postgraduate education and training is often the thing that loses out and gets pushed to the bottom of the mounting to-do pile. The system needs to reward and encourage senior clinicians so that they make time to give high quality feedback to trainees during WPBA completion. But this is a long term aim that feels intangible and unattainable. In the short term, reducing the time it takes to complete WPBA paperwork will make everyone happer. An app seems the quickest way to achieve this.

A call for EBT: Evidence Based Training

“Recently the JCST has specified a minimum of 40 WPBAs per year to be completed as a ‘quality indicator’ for surgical training and career progression…Regional training programs have set directives for mandatory WBAs per annum, ranging from a minimum JCST dictat of 40 to the 80 required in London. These present a great challenge upon time available to any practicing surgeon.”

“…a recent systematic review that includes our first survey suggests that there is no evidence that they [WPBAs] improve physician performance. It goes on to conclude that multisource feedback may be helpful, but that individual factors, context of feedback, and presence of facilitation (ie mentoring) may improve trainee responses.”

These sentiments will sound familiar to physicians, many of whom also feel frustrated at the widespread adoption of WBPAs, for which there is limited evidence of value for trainees in the real world. Valid concerns have been raised about the difficulties of applying theoretically helpful frameworks and tools to the realities of clinical life, and it is unclear where the numbers set by training boards have come from.

“ISCP has improved its interface, but it and other electronic portfolios deliver an increasingly overwhelming bureaucratic burden of WBAs and domains of evidence to include in a portfolio. These have rapidly become entrenched in postgraduate physician training in the UK, spreading a plague of box-ticking exercises that continue to increase year on year….It is of particular concern that so many trainees (80%) felt that ISCP did not improve their training after a modal average of over three years using it.”

Again these feelings will be familiar to many of those who have commented on this site and engaged with the debate on twitter. Time is precious. Many feel that the current demands on trainees, coupled with inadequate technology, steals  it away from busy trainees and trainers.

Perhaps it is time to ask the question, who is the ePortfolio for? Is it a learning tool for trainees? Is it an evidence vault for Royal Colleges to check off competencies of registered members? It is unclear to me what the aims of the NHS physician ePortfolio was at its inception. Has this been reassessed as it has expanded and evolved? These is great potential to improve the ePortfolio so that it serves the needs of trainees, trainers, assessors and higher bodies better. We have an opportunity to seek clarification and contribute to making the aims and expectations explicit. Let’s not let it pass us by.

The authors of the paper conclude:

“The performance of ISCP has improved in the 4 years since its inception with proportionately less negative feedback. British surgeons remain dissatisfied with several of its tools, in particular its workplace-based assessments. Half a decade on, these assessments remain without appropriate evidence of validity despite increasing demands upon trainees to complete quotas of them. With reduced permitted training hours, the growing online bureaucratic burden continues to demoralize busy surgical trainers and trainees.”

These conclusions should ring alarm bells not only for the Royal Colleges, but for the wider community of healthcare leaders. The NHS faces many challenges, and a demoralized workforce will struggle to face them. Physician and surgical trainees feel overburdened and undervalued. The system needs to change. Who will lead this change? And where will the ePortfolio fit in? Answers on a postcard…..

E.A. Pereira B.J. Dean.British SurgeonsExperiences of a Mandatory Online Workplace Based Assessment Portfolio Resurveyed Three Years On. Journal of Surgical Education. J Surg Educ. (2012) doi: 10.1016/j.jsurg.2012.06.019

A. Miller, J. Archer Impact of workplace based assessment on doctors’ education and performance: A systematic review. BMJ, 341 (2011), p. c5064

E.A. Pereira, B.J. Dean British surgeons’ experiences of mandatory online workplace-based assessment J R Soc Med, 102 (2009), pp. 287–293

S.A. Welchman Educating the surgeons of the future: The successes, pitfalls and principles of the ISCP. Bull R Coll Surg Engl, 94 (2012) online

W.C. Leung. Competency based medical training. Review, 325 (2002), pp. 693–696

Can we make ePortfolio open source? a guest post from Karen Beggs

My first question is WHY?

Here are the main issues I hear about:

  • A lot of trainees aren’t happy with workplace based assessments
  • Internet speed is an issue in some NHS locations
  • Some people don’t like using an ePortfolio
  • Some people want to have more input into ePortfolio design
  • Some trainees want their seniors to be more engaged with their learning
  • There is a common misunderstanding that College membership fees are used solely to pay for the ePortfolio

So what are we already doing about these issues?

  • We are eliciting feedback directly from the wider ‘user’ community through social media to find out what usability improvements we can make…and get them done.

We’ve started this already… following a conversation last month with a trainee who was frustrated by the curriculum linking process, our architect made a simple change that was deployed a few days later (see demo here), reducing the number of clicks needed to make multiple links. We have also introduced a twitter feed, visible on the www.nhseportfolios.org home page).

We are moving to a more elastic hosting environment so that as the system gets busier it can engage more resources to deal with the increased load. We aim to have this fully implemented by autumn 2012.

The NHS ePortfolio team do not make decisions about assessment processes, training requirements or the use of specific workplace based assessments. Expertise in these areas usually lies with the Colleges, Postgraduate Deans and the Academy of Medical Royal Colleges. However, we can:

  • Help connect trainees with ideas (or complaints) with the relevant people, whether College, user group, developers or others. In July 2012 I attended the AoMRC Trainee Doctors Committee, and as a result will recommend to the Chair of the specialty ePortfolio User Group that a representative from the committee sits on that group. Some trainees are not aware of their own College decision making processes, and we will pass on contact details as required.

Would open source address any of these issues?

As far as I can see, NO!

But why don’t we just hand over the code to the large community of willing, enthusiastic OSS developers?

  • It’s not as easy as just handing out ‘the code’. Open source software must comply with a number of criteria (see www.opensource.org), many of which would contradict the current NHS ePortfolio license terms.
  • Who would fund re-writing and re-negotiating software licenses for the existing 25 or so organisations using the NHS ePortfolio? What if one of these organisations objects? It’s an integrated application with many shared features, so to separate out one ‘Customer’ would require a large-scale re-write. That seems to defeat the purpose.
  • The ePortfolio is integrated with a number of external (usually College run) systems and moving to an OSS model would have implications for each of these systems. Would Colleges want to pay to conduct a thorough risk assessment before signing up? And would they then want to pay for any adjustments needed to maintain the integrity of their own systems?
  • There would still have to be stringent controls over the quality of the code submitted. This would require a quality control team – possibly a larger one than we have at the moment. Who would pay for this?

I’m not sure I quite follow the argument… get rid of our current team of developers (some of whom have been with us for over 4 years), keep fingers crossed that some OSS developers can meet our commitments, beef up our QA team so they can check the code of the unknown OSS developers…. Seems that we increase our risks (of not meeting SLAs), decrease predictability (how can we hold anyone to a delivery date if we don’t employ them?) and end up with a QA team but lose our development expertise (the current team wouldn’t hang around for long – why would they?). I can’t see a sustainable business model in here unless we were to maintain a large core team – and if we do that, where are the assumed cost savings of OSS?

I have heard arguments that OSS is cheaper overall, but I don’t really see that cost is the problem (see My first question is WHY? above). It seems to me that the per capita charges for the ePortfolio are pretty reasonable. There is currently no charge made for any supervisor (educational or clinical), programme director, administrator, ARCP panel member or assessor using the ePortfolio. Per capita charges are based only on trainees at present. Would OSS have any impact on this? I can’t see that it would.

Final thoughts

If we were starting from scratch we would look at OSS as one of the options. We would probably look at an off-the-shelf ePortfolio too. We would be foolish not to. But we are not starting from scratch. We have an established, bespoke ePortfolio that is used across the professions (we have versions for Dentists, Nurses & Midwives, Pharmacists, Doctors and Undergraduates), is integrated with a number of external systems and capturing over a million forms submitted by ‘assessors’ every year. Each version has a custom set of features, making it adaptable and cost effective (sharing an underlying code base and database).

Many of the problems we hear about relate to complaints about the educational processes, and changes are already underway to address these (eg move to Supervised Learning Events in Foundation from August 2012). We contribute to these discussions when appropriate.

We have developed good relationships with our broad range of Customers, and continue to work with them to improve our change control and development processes. We work within the constraints of the NHS, which impacts our management of finance, procurement, stakeholder management, technology and decision making, as well as our governance arrangements.

We have an established application and an experienced team, whose expertise and commitment cannot be underestimated. Our development costs are at the lower end of the market, and maintenance charges are extremely good value. We can bring in additional specific expertise as and when we need to.

I can’t help but think the suggestion to move NHS ePortfolio to OSS is a solution to the wrong problem.

To Open Source or not to Open Source

Recent discussions around possible solutions to the need for NHS ePortfolio development have led to the suggestion that harnessing Open Source Software (OSS) may be the answer.

So what is Open Source Software (OSS) I hear you cry?

Open Source is collaboratively developed, freely available software or application. You may now hear people using the term ‘Open Source’ across a wide variety of different sectors in order to describe a more open, networked and user generated way of developing ideas and projects. While the term applied originally only to the source code of software, it is now being applied to many other areas.

But it is not just for new projects. For established projects, the software or application itself does not have to be dismantled.  There can be huge benefits of opening up the code to outsiders who then use the software, fix bugs, submit patches, file  bug reports, and create new content. Often for free. In times of ever-contracting training budgets and with austerity in the national consciousness, free sounds very appealing.

NHS Hackday is the most relevant example of this concept, and does a good job of explaining why OSS fits with the concept of openness in healthcare and academia, and why it is financially beneficial.  Examples of projects so far include an app to aid safe handover (in line with recommendations from the acute care toolkit, and clinical governance principles), and OpenBNF (an open source app for access to the tax-funded British National Formulary of medications, currently only available at a cost of £30 via a private provider). I think the projected cost savings may be over-enthusiastic, but the model of using the knowledge of “coalface” clinicians, and harnessing technological expertise for the public good is clearly powerful. This same argument could be made for the NHS ePortfolio as in the short-term all time saved means more time for patient care or educationally valuable activities. And in the longterm a better ePortfolio could facilitate better education and training, ultimately producing better doctors.

There are Open Source evangelists:

Carl is an evangelist,  Ben Goldacre is an evangelist

And there are those with legitimate concerns about control, and whether Open Source would deliver what is needed. As Karen Beggs (ePortfolio project manager at NES) points out it is no panacea: we must look critically at our needs and apply the right solutions. Here are some responses to questions about security and maintaining control, and an insight into the potential hidden problems of not using OSS.

Encouragingly, OSS in healthcare is not new (examples), and OpenSource in ePortfolios is not new (see Mahara) so there is already work to build on.

A vision of collaboration, openness, and harnessing clinical knowledge to create rapid solutions to real-world problems, working from the ground up instead of the top down is incredibly powerful and one the NHS and education communities should celebrate.

Will OSS be the cure for the NHS ePortfolio’s chronic disease? If you’re a geek who loves the NHS maybe you can help us find out..

All ePortfolios are equal, but some are more equal than others

The NHS ePortfolio is not the only kid on the block. Since starting this site I have been pointed in the direction of some alternatives (thanks for all suggestions). There are specific requirements for NHS trainees which mean that it is more likely we will always all have to use the same software, rather than being able to break out and use different versions. Most obviously this is the need to collate standard Workplace Based Assessments. However, we can window shop, and return with a long shopping list…

1. The GPs didn’t just get mad, they got creative

This site seems to have been created due to a need for a FREE ePortfolio tool that could do 360 degree appraisals, and have the ability to log a number of different types of CPD activity. Clearly very GP focused, and I can’t get in to see how user-friendly and intuitive it is. The testimonials make it sound great! Likely to be limited in scope, flexibility and future directions as created for a specific purpose and audience. I get the impression it’s aimed at GP Partners more than trainees (who have to use the NHS GP Portfolio).

2. Pebblepad

Claims to be able to fulfill multiple functions which sounds good:

  • Assessment (formal and informal)
  • Advancement (promotion or transition)
  • Appraisal (self – peer – 360)
  • Accreditation (professional bodies)
  • Application (course, job, funding)
  • Articulation (informative story telling)
Looks quite corporate, not easy to move from different areas of content. Seems to have been well thought out and to have considered the need to be able to display the contents in a useful way, and export this for multiple purposes. Has flexibility in terms of what content can be stored, including files, pictures and links. But somehow still doesn’t feel like what I’m looking for.  

3. Mahara

Aims to be a learner-centred Personal Learning Environment. Specific features include the ability to choose which aspects of the portfolio are public and private; the ability to create a digital CV; a blogging area (maybe it’s better than calling it reflection?) and the ability to interface with Moodle. Also, it’s Open Source. We love Open Source! Mahara ticks a lot of boxes for me, especially the fact that there has been some thought about how to display features in a very visual way, but it feels a little juvenile. And it is difficult to see content – it requires lots of clicks to do anything. The fewer the clicks the lower my frustration levels!

4. Googlios

Of course, why did I not realise that Google would be ahead of the game on this? The Googlio comes the closest to my idea of how the NHS ePortfolio should look and feel.  It is easy to navigate, with different sections accessed in multiple ways, meaning very few clicks to do anything. It presents an individualised “showcase” of what’s inside with a standard set of bits of information, but with the ability for the individual to highlight their most significant achievements and curate their own “front page”. It works with Googledocs, Word and Excel so the information contained within it is exportable for other purposes, and documents can be dynamic (I’m thinking about the great potential for a log-book here). It is easy to embed links, blogs, and other content from all sorts of different applications, and they are displayed in a helpful way. It’s not perfect as it needs a timeline amongst other things. But they completely sold me by using Leonardo da Vinci as inspiration. “The Googlio philosophy could transform 21st century education, using 21st century tools. This has the potential to stir a return, a revolution, a rebirth, a revival of educational renaissance.” I’m not really sure what this means but I’m excited. This is the kind of model the NHS ePortfolio team should follow. ePortfolio Gods, please look at this!

I have now used Googlios (or Google sites as it’s also called) to create my own teaching portfolio. Having spent a while getting to grips with the interface I remain a big fan. The NHS ePortfolio has a lot to learn about user satisfaction…

I was also directed to the Academy of Medical Royal Colleges project MIPS, which aims “to develop an initial set of ePortfolio standards to encourage modularisation of systems and secure interoperability between ePortfolio systems (NHS, Collegiate and others).” I think this is a word-y way of saying they want to lay down some ground rules for how ePortfolios should work, and how they should work together. Sounds great! And it’s so great to see a project like this working across Colleges and specialities. We all have a core set of very similar needs, so should be pooling resources and ideas. I can’t see any signs of progress or deadlines on the site, however. Hopefully there’s lots going on with this project in the  shadows…

Know of an ePortfolio you think is better? Have experience of using one of those featured above? Comment below.

NB. Since writing the original post I have also been directed to an NHS Hackday presentation on standards for an ePortfolio and  a site defining  Government IT standards, which look like great starting points for agreeing standards that must be achieved in using data, including that in the ePortfolio. Lots to think about.