Can we make ePortfolio open source? a guest post from Karen Beggs

My first question is WHY?

Here are the main issues I hear about:

  • A lot of trainees aren’t happy with workplace based assessments
  • Internet speed is an issue in some NHS locations
  • Some people don’t like using an ePortfolio
  • Some people want to have more input into ePortfolio design
  • Some trainees want their seniors to be more engaged with their learning
  • There is a common misunderstanding that College membership fees are used solely to pay for the ePortfolio

So what are we already doing about these issues?

  • We are eliciting feedback directly from the wider ‘user’ community through social media to find out what usability improvements we can make…and get them done.

We’ve started this already… following a conversation last month with a trainee who was frustrated by the curriculum linking process, our architect made a simple change that was deployed a few days later (see demo here), reducing the number of clicks needed to make multiple links. We have also introduced a twitter feed, visible on the www.nhseportfolios.org home page).

We are moving to a more elastic hosting environment so that as the system gets busier it can engage more resources to deal with the increased load. We aim to have this fully implemented by autumn 2012.

The NHS ePortfolio team do not make decisions about assessment processes, training requirements or the use of specific workplace based assessments. Expertise in these areas usually lies with the Colleges, Postgraduate Deans and the Academy of Medical Royal Colleges. However, we can:

  • Help connect trainees with ideas (or complaints) with the relevant people, whether College, user group, developers or others. In July 2012 I attended the AoMRC Trainee Doctors Committee, and as a result will recommend to the Chair of the specialty ePortfolio User Group that a representative from the committee sits on that group. Some trainees are not aware of their own College decision making processes, and we will pass on contact details as required.

Would open source address any of these issues?

As far as I can see, NO!

But why don’t we just hand over the code to the large community of willing, enthusiastic OSS developers?

  • It’s not as easy as just handing out ‘the code’. Open source software must comply with a number of criteria (see www.opensource.org), many of which would contradict the current NHS ePortfolio license terms.
  • Who would fund re-writing and re-negotiating software licenses for the existing 25 or so organisations using the NHS ePortfolio? What if one of these organisations objects? It’s an integrated application with many shared features, so to separate out one ‘Customer’ would require a large-scale re-write. That seems to defeat the purpose.
  • The ePortfolio is integrated with a number of external (usually College run) systems and moving to an OSS model would have implications for each of these systems. Would Colleges want to pay to conduct a thorough risk assessment before signing up? And would they then want to pay for any adjustments needed to maintain the integrity of their own systems?
  • There would still have to be stringent controls over the quality of the code submitted. This would require a quality control team – possibly a larger one than we have at the moment. Who would pay for this?

I’m not sure I quite follow the argument… get rid of our current team of developers (some of whom have been with us for over 4 years), keep fingers crossed that some OSS developers can meet our commitments, beef up our QA team so they can check the code of the unknown OSS developers…. Seems that we increase our risks (of not meeting SLAs), decrease predictability (how can we hold anyone to a delivery date if we don’t employ them?) and end up with a QA team but lose our development expertise (the current team wouldn’t hang around for long – why would they?). I can’t see a sustainable business model in here unless we were to maintain a large core team – and if we do that, where are the assumed cost savings of OSS?

I have heard arguments that OSS is cheaper overall, but I don’t really see that cost is the problem (see My first question is WHY? above). It seems to me that the per capita charges for the ePortfolio are pretty reasonable. There is currently no charge made for any supervisor (educational or clinical), programme director, administrator, ARCP panel member or assessor using the ePortfolio. Per capita charges are based only on trainees at present. Would OSS have any impact on this? I can’t see that it would.

Final thoughts

If we were starting from scratch we would look at OSS as one of the options. We would probably look at an off-the-shelf ePortfolio too. We would be foolish not to. But we are not starting from scratch. We have an established, bespoke ePortfolio that is used across the professions (we have versions for Dentists, Nurses & Midwives, Pharmacists, Doctors and Undergraduates), is integrated with a number of external systems and capturing over a million forms submitted by ‘assessors’ every year. Each version has a custom set of features, making it adaptable and cost effective (sharing an underlying code base and database).

Many of the problems we hear about relate to complaints about the educational processes, and changes are already underway to address these (eg move to Supervised Learning Events in Foundation from August 2012). We contribute to these discussions when appropriate.

We have developed good relationships with our broad range of Customers, and continue to work with them to improve our change control and development processes. We work within the constraints of the NHS, which impacts our management of finance, procurement, stakeholder management, technology and decision making, as well as our governance arrangements.

We have an established application and an experienced team, whose expertise and commitment cannot be underestimated. Our development costs are at the lower end of the market, and maintenance charges are extremely good value. We can bring in additional specific expertise as and when we need to.

I can’t help but think the suggestion to move NHS ePortfolio to OSS is a solution to the wrong problem.

Advertisements

11 responses to “Can we make ePortfolio open source? a guest post from Karen Beggs

  1. Hi everyone,

    (firstly, a declaration: I’m a bit of an open source evangelist and have posted comments in the past about the Mahara e-porttfolio tool. I’m not an NHS employee – I’m a supplier of systems to the NHS – but I’m not posting here in any commercial capacity).

    I wonder whether the discussion would be better focused on open STANDARDS rather than open source systems? For example, there is an agreed e-portfolio data exchange standard called LEAP2A (easily Google-able) which would allow the NHS e-portfolio to interact with other portfolio and learning systems. There was work recently with AoMRC to look at this (http://www.mips.org.uk/) but I’m not sure where it got to.

    This approach, with some standard APIs, might also encourage a marketplace of third party commercial and non-commercial (open source) people to develop plugins, Apps and other features that could enhance the e-portfolio experience, and create a more joined-up learning process, without messing with or throwing away any of the historical hard work and development?

    Tim

  2. from https://nhseportfoliorevolution.wordpress.com/2012/08/15/can-we-make-eportfolio-open-source-a-guest-post-from-karen-beggs/

    Karen

    Carl

    My first question is WHY?

    Here are the main issues I hear about:

    A lot of trainees aren’t happy with workplace based assessments
    Internet speed is an issue in some NHS locations Is it? Or is it the speed of your ePortfolio service which is internet speed + speed of your product?
    Some people don’t like using an ePortfolio
    Some people want to have more input into ePortfolio design True. Perhaps this is they don’t like the design of your product?
    Some trainees want their seniors to be more engaged with their learning
    There is a common misunderstanding that College membership fees are used solely to pay for the ePortfolio
    I think it’s a big issue that the ePortfolio costs as much to develop as it does

    So what are we already doing about these issues?

    We are eliciting feedback directly from the wider ‘user’ community through social media to find out what usability improvements we can make…and get them done.
    The medical and technical NHS Hack day community has also donated considerable time and expertise to the public good to try and help you some of which is here: http://wiki.nhshackday.com/wiki/NHS%20ePortfolios

    We’ve started this already… following a conversation last month with a trainee who was frustrated by the curriculum linking process, our architect made a simple change that was deployed a few days later (see demo here ), reducing the number of clicks needed to make multiple links. We have also introduced a twitter feed, visible on the http://www.nhseportfolios.org home page).
    We are moving to a more elastic hosting environment so that as the system gets busier it can engage more resources to deal with the increased load. We aim to have this fully implemented by autumn 2012.
    The NHS ePortfolio team do not make decisions about assessment processes, training requirements or the use of specific workplace based assessments. Expertise in these areas usually lies with the Colleges, Postgraduate Deans and the Academy of Medical Royal Colleges. However, we can:
    Help connect trainees with ideas (or complaints) with the relevant people, whether College, user group, developers or others. In July 2012 I attended the AoMRC Trainee Doctors Committee, and as a result will recommend to the Chair of the specialty ePortfolio User Group that a representative from the committee sits on that group. Some trainees are not aware of their own College decision making processes, and we will pass on contact details as required.
    Would open source address any of these issues?

    As far as I can see, NO!

    Care to provide an argument here?

    Here’s mine (Carl):

    I would like the ePortfolio, and digital services in general, to be as good, and as good value, as they can be. This involves innovation. Sometimes you don’t know how broken what you have is until someone makes an alternative.

    So in order to address some of the issues you identify AND facilitate future innovation the ePortfolio should be open source because there are talented people outside your organization who are motivated to improve your product and they can’t at present.

    They could make it faster, improve its design and usability, and help you to not disappoint your users.

    Greater scrutiny of your work would lead to fewer bugs and might, by facilitating comparison with other projects, help you to run the project in a more cost effective way.

    Finally, if what you’re doing is so great than by sharing the source code you make it possible for other parts of government and the public to use your code. This prevents waste, no need to reinvent the wheel, helps your excellent product to diffuse, and gets more people invested in making it even better.

    But why don’t we just hand over the code to the large community of willing, enthusiastic OSS developers?
    It’s not as easy as just handing out ‘the code’. Open source software must comply with a number of criteria (see http://www.opensource.org ), many of which would contradict the current NHS ePortfolio license terms.
    Citation please. What are your license terms? Why/How did you choose them? Do they represent good value for money for the tax payer?

    Who would fund re-writing and re-negotiating software licenses for the existing 25 or so organisations using the NHS ePortfolio? What if one of these organisations objects? It’s an integrated application with many shared features, so to separate out one ‘Customer’ would require a large-scale re-write. That seems to defeat the purpose.

    You haven’t told us how much it would cost or why it would cost anything. Have you asked them if they’d object? Why would they? Re “it’s an integrated application” Are you confusing the source code with the licence here!?

    The ePortfolio is integrated with a number of external (usually College run) systems and moving to an OSS model would have implications for each of these systems. Would Colleges want to pay to conduct a thorough risk assessment before signing up? And would they then want to pay for any adjustments needed to maintain the integrity of their own systems?

    Again without talking about real world examples here this is quite meaningless and seems a bit confused… Hard to know what you mean really. Can you provide an example? Hopefully you’re not confusing the source code with the licence again.

    There would still have to be stringent controls over the quality of the code submitted. This would require a quality control team – possibly a larger one than we have at the moment. Who would pay for this?

    Hopefully you employ some software developers already who are able to assess the quality of source code. They might have more time to do this with the help from external guys making the product better.

    I’m not sure I quite follow the argument… get rid of our current team of developers (some of whom have been with us for over 4 years), keep fingers crossed that some OSS developers can meet our commitments, beef up our QA team so they can check the code of the unknown OSS developers…. Seems that we increase our risks (of not meeting SLAs), decrease predictability (how can we hold anyone to a delivery date if we don’t employ them?) and end up with a QA team but lose our development expertise (the current team wouldn’t hang around for long – why would they?). I can’t see a sustainable business model in here unless we were to maintain a large core team – and if we do that, where are the assumed cost savings of OSS?

    Have you asked your current team? Many software developers like to publish their work and have it peer reviewed. It helps them to become better software developers and shows off what they can do to the world. Not for me to suggest how/whether you should restructure your development team. If you did there are more options available than keeping all of them/getting rid of all of them – that’s a false dichotomy.

    I have heard arguments that OSS is cheaper overall, but I don’t really see that cost is the problem

    “NHS Project manager doesn’t see cost as a problem”!! I would suggest that value for money is, and should, always be a very important consideration when buying something.

    (see My first question is WHY? above). It seems to me that the per capita charges for the ePortfolio are pretty reasonable. There is currently no charge made for any supervisor (educational or clinical), programme director, administrator, ARCP panel member or assessor using the ePortfolio. Per capita charges are based only on trainees at present. Would OSS have any impact on this? I can’t see that it would.

    re supervisors paying etc, I think that’s an irrelevant consideration to this argument

    Final thoughts
    If we were starting from scratch we would look at OSS as one of the options. We would probably look at an off-the-shelf ePortfolio too. We would be foolish not to. But we are not starting from scratch.

    Sometimes it’s a good idea to revisit things as if starting from scratch… but again irrelevant to whether or not you release your code under an open source licence.

    We have an established, bespoke ePortfolio that is used across the professions (we have versions for Dentists, Nurses & Midwives, Pharmacists, Doctors and Undergraduates), is integrated with a number of external systems and capturing over a million forms submitted by ‘assessors’ every year. Each version has a custom set of features, making it adaptable and cost effective (sharing an underlying code base and database).
    Many of the problems we hear about relate to complaints about the educational processes, and changes are already underway to address these (eg move to Supervised Learning Events in Foundation from August 2012). We contribute to these discussions when appropriate.
    We have developed good relationships with our broad range of Customers, and continue to work with them to improve our change control and development processes. We work within the constraints of the NHS, which impacts our management of finance, procurement, stakeholder management, technology and decision making, as well as our governance arrangements.
    We have an established application and an experienced team, whose expertise and commitment cannot be underestimated. Our development costs are at the lower end of the market, and maintenance charges are extremely good value. We can bring in additional specific expertise as and when we need to.

    irrelevant to question to open source or not

    I can’t help but think the suggestion to move NHS ePortfolio to OSS is a solution to the wrong problem.

    I can’t help thinking how hard it must be to manage a technology project without a technical background.

    It’d be great to have you along to NHS Hack Day Liverpool to show you what we’re up to. Maybe you could be part of the team that builds an open-source NHS ePortfolio alternative? 🙂

  3. formatting fail 😦

    here is better http://bit.ly/PYQGDH

    best, Carl

  4. Some strong opinions on both sides. Our common aims are to make the product better for all users, and there is an urgent need for this (eg what is happening with an app?). A face to face meet up at NHS Hackday sounds like a good opportunity for all.

    More comments and views on the open source vs not open source debate would be very welcome.

  5. To quote Jamie Zawinski (a key developer in the open-sourcing of Firefox – http://www.jwz.org/gruntle/nomo.html): “Open source does work, but it is most definitely not a panacea. If there’s a cautionary tale here, it is that you can’t take a dying project, sprinkle it with the magic pixie dust of “open source,” and have everything magically work out”

    In this context, I believe the issues with the NHS e-portfolio system are unlikely to be related to the difficulty of implementing feature X (which open source could lessen the load of); I believe the problem is that what the people that control the purse-strings want from the system is very different to what the users want from the system; I would go further and say that the e-portfolio system is likely to be exactly what the management stakeholders asked for. If the users were able to code up their own features, they would still have to persuade these people to include them into the main branch of the code.

    This is just my 2 cents as a software developer who has watched his wife bash her head against the wall trying to use e-portfolio. Its the standard conflict of interest that cripples must bespoke software – the priorities of the people who pay for it are very different from the priorities of the people who have to use it day-in day-out. It would sadly take a very radical change in the way the NHS commissions software to fix this problem.

    • Thanks for your comments – really helpful. I hope we can find a middle ground, with the ePortfolio serving the needs of the Colleges to track trainee’s progress and provide “proof” of progression (whether current WPBAs robustly do this is another debate….). This should be in parallel with trainees’ needs for a useable, fast, transparent system, which actively encourages reflective learning (not necessarily in the current format), flexibility of content and clear goal setting. If a radical change to commissioning is needed, let’s start asking for a radical change! Trainees and Colleges can all benefit from greater user engagement and input.

  6. Background on me – currently a Doctor, used to be a software developer. Use open source on every computer in the house. Open source enthusiast.

    I don’t see open source software improving the ePortfolio system. I doubt there would be much (if any) interest from the wider OSS community in working on a piece of software that has a very specific user base and is very much a bespoke piece of software.

    Who then would work on it? You would be looking at the union of the groups of people who use actually use ePortfolio, people who have the technical skills to write this sort of software and people who actually care enough to want to do this without being paid for it. I can’t see this being a large group of people. I fall in the first two groups. Definitely not in the third one.

    There is some frustration about the ePortfolio itself (when it is slow, when the user interface is a bit clunky) but I really think much of the frustration is actually do the educational framework set by the Colleges which the ePortfolio has to serve.

    All of that said, I still would support the open sourcing of the software. Not in the expectation of any serious development efforts from any outside parties – but simply because I believe that software within the NHS should be developed in an open and transparent way.

    I really think that there is a misunderstanding of what it would mean to move the ePortfolio to OSS. You wouldn’t sack the current development team and expect a wider community to support the development. That would be stupid. All you would do is make the code publicly available and invite contributions. Just because it’s open source software doesn’t mean you can’t get paid to develop it.

  7. Hi

    My background – junior doctor but former management consultant (although no IT expertise). Nothing to add on the open source argument, I’ll leave that to the techies!

    However I’d be interested in being put in touch with the relevant people in the colleges as you offered (mine is RCP, physicians) as I think there’s a lot of scope for improvement. As an ex-management consultant, I’m passionate about feedback. But the current system doesn’t facilitate this (and the IT is not the culprit, just a handy scapegoat). The most glaring example is being able to choose your own assessors for your appraisal – only a fool chooses people who will give negative, if honest or constructive, feedback. So this exercise becomes meaningless. And having obligatory targets of the number of forms filled in by consultants risks turning these into a tick box exercise – most of us would benefit more from registrar feedback as we interact far more closely with them, and the number of forms should be flexible depending on the jobs you’ve done etc. For example, a trainee rotating through Intensive Care should have many more DOPS forms for procedures, but far fewer mini-CEX forms as there is less opportunity for these with unconscious patients.

    Also I can’t help but comment on the cost. You may think the cost is reasonable, but £165pa versus the ZERO cost of maintaining your own paper portfolio (as it was before) is significant. For a system that isn’t obviously better, and at times more onerous. This is one of a myriad “reasonable” costs that are a significant burden on the junior doctor – GMC registration fees of £390pa, medical indemnity (£75 this year for me, rising each year, but £100s for GP trainees), college membership fees £140pa, compulsory exams (MRCP currently costs £408 for part 1, another £408 for part 2 written, £640 for PACES). And we don’t earn that much – basic pay for doctors does not exceed £40K pa until we reach consultant level, for most of us >9 years after leaving medical school! So whilst we cannot hold NHS ePortfolio to account any more than we can the GMC, please do think carefully about every penny you spend as we will have to pay for it.

    • Thanks for your comments. I think it’s important to look at the evidence for certain WPBAs – I believe that the MSF has the most robust evidence of all the WPBAs, and that this holds even if you choose your own assessors (as long as you have enough). I’m working on a post that looks at this so watch this space. Perhaps greater transparency and communication with trainees and trainers regarding the evidence base would be a good first step, to allow more critical review and productive discussions as to future directions. Your points regarding numbers of assessments in different posts is interesting – you make a good case for more flexibility around numbers of different assessment tools.

      Regarding cost, I agree “please do think carefully about every penny you spend as we will have to pay for it,” but where does £165 pa come from? Only £18 per trainee per year is spent on the ePortfolio. I am unsure as to where the rest of the JRPCTB fees go, but I believe another trainee has put in a FOI request on this. Transparency would benefit the Colleges as the current lack of understanding as to where their money goes leads to resentment towards these institutions and bodies.

      It’s not cheap being a trainee! With future generations having even greater debt on graduation, value for money is likely to be high on their priority list…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s