27. Working with an electronic portfolio

Introduction

CBD results in residency programs collecting a large number of resident assessments. This is largely due to the assessment of entrustable professional activities (EPAs), but programs also incorporate assessment modalities such as in-training evaluation reports, written and oral examination scores, procedure logs, non-EPA narrative assessments, progress on learning plans, etc. This variety of assessment data should provide a more comprehensive assessment of resident progress that results in more data-driven decisions regarding resident learning plans and promotion.

However, this large volume of assessment data needs to be amalgamated and visualized effectively to support valid interpretations. This can be supported through electronic portfolios, tools that allow visualization of collected data for learners, academic advisors, and competence committees. These tools serve a dual-purpose of promoting learning and providing more comprehensive assessment data for promotion decisions.

The ability of the learning management systems available within each university to create an effective competency-based e-portfolio varies. There is no single best approach to developing an e-portfolio, and the approach taken by one program or institution may differ from others based upon their needs and the resources that are available. As learning management systems become more sophisticated we anticipate that they will be better able to meet the assessment needs of programs and learning needs of residents in addition to supporting continuous quality improvement (CQI) through program evaluation and faculty development.

Within this chapter, we will outline how e-portfolios can be effectively used and/or developed in your residency program.

Establish an e-portfolio for your residents and Competence Committee

The value of an e-portfolio

Both learners and competence committees can benefit from visualization of assessment data using e-portfolios. Learners can review their assessments, track their progress, and identify both areas of strength and skills requiring further development. In some programs this interpretation is supported by faculty academic advisors. Competence committees often need to review large amounts of assessment data. They are able to function more efficiently when they can easily navigate through the assessment data. E-portfolios can present data using visualizations that demonstrate trends in performance and sources of variation in assessment (e.g. is a lower rating for one EPA evaluation due to a lower performance or a stricter rater?).

Taking stock of what’s available already

If you happen to be part of the team establishing the first e-portfolio in your training program it is important to understand what is already available.  Does your Postgraduate Office provide an e-portfolio within your learning management system (e.g. Elentra, one45, Medsys, etc)? Does the tool that you are using to collect assessment data allow data to be exported and uploaded to external e-portfolio software? Have other training programs at your institution developed their own e-portfolio that your program could adopt? Finding the answers to these questions in collaboration with your Postgraduate office will help you determine what your options are while ensuring that efforts are not duplicated within institutions.

Key Functionalities

An effective e-portfolio allows viewers to navigate between assessments efficiently in a way that allows assessment of a resident’s overall performance relative to the expectation of the program of assessment. This may include “bench marking” with other residents at a similar level.  EPA data is often presented by stage with entrustability scores graphed over time to provide a ‘growth-curve’ for a resident. Visualizations should ideally incorporate the contextual information required for that EPA (e.g. faculty or resident assessor, direct or indirectly observed, specific clinical presentations, case complexity, etc) and link to corresponding narrative comments. If your program’s e-portfolio does not have the functionality to track contextual variables, it will be important to work with them to find a solution, as this information must be tracked in real time.

Resources for further development

Your program may have unique needs that are not fully met by the e-portfolio tools available. For instance, you may want to incorporate other assessment modalities beyond EPAs into your visualizations. If you decide to develop new visualizations, collaborate closely with your Faculty/Department as you will need to investigate whether technical support is available to meet these needs.

Continuously refine your e-portfolio

As CBD matures and your program evolves, the needs of Program Directors, Competence Committees, and residents are likely to change. It is almost certain that these changes will require your e-portfolio to be modified. We recommend speaking with your institution to determine when and how modifications to the e-portfolio will be made for your program so that you can be prepared to contribute to its evolution. We recommend tracking suggestions for modifications that arise over time so that your ideas are readily available when the next opportunity to improve your e-portfolio arises.

Extend the use of your e-portfolio to support continuous quality improvement

Once all of your residents’ information is amalgamated into an electronic portfolio, consideration can be given to how else this information can be used to improve your program. The use of learner assessment data to support CQI of residency programs is a developing field that requires more sophistication than the establishment of learner-specific portfolios. It is prudent to work with your CC chair to decide what CQI information your program would like to track, work with your PGME to develop functionality to track it within the e-portfolio, and create a faculty/resident strategy to implement improvement initiatives.

Grouping resident assessment data by rotation can provide insight into what is occurring on each of your program’s rotations. For example, your residents may complete an intensive care rotation with a goal of becoming competent in central line placement as measured by an EPA. A review of the EPAs completed on ICU may find that this EPA is not being completed as expected. You could then try to determine why this is the case: Are EPAs not being completed on ICU at all? Are fewer central lines placed on ICU than expected? Are fellows or residents from other services placing them instead? The answers to these questions may lead to interventions that can improve your program.

When resident assessment data is grouped by faculty member it can be used for ongoing faculty development initiatives. For example, there may be a group of faculty who provide minimal narrative feedback on the EPAs they complete. Audit and feedback mechanisms are used within clinical CQI initiatives to highlight practice variation for clinicians and can be used in these cases for education. These results could be explored confidentially with these faculty members with the goal of gaining a better understanding of the results and determining how their assessments can be improved.

Ensure the security of the assessment data

Resident assessment data is sensitive information that must be managed with care. There are institutional policies and protocols surrounding the access and distribution of student data that should be used to guide the safe and appropriate use of this data. These policies address complicated issues that arise as residents are assessed by a complex web of supervisors across rotations, programs, and institutions. Specific recommendations include ensuring that assessment data only be accessible to individuals who need it to conduct essential tasks and that it be stored securely using institutional usernames and passwords.

Avoid over-interpreting the assessment information

The interpretation of resident assessment data is complex and multifaceted. Even when presented effectively, misinterpretation is possible. Assessment data should be reviewed within the broader context of the program of assessment. For example, low numbers or entrustability ratings on EPA observations could be the result of many things including poor resident performance, faculty inexperience/uncertainty with the new assessment system, variations in rotation scheduling, resident personal difficulties, or any number of other explanations.  Be aware that additional context is often needed to interpret assessment data accurately.  Programs should consider how other these factors might impact residents’ ability to meet the assessment guidelines outlined by each specialty’s national assessment program and what processes are in place to mitigate these factors.

Conclusion

In summary, electronic portfolios are essential tools to run a CBD assessment program. They facilitate understanding by combining and displaying assessment data and its variability over time. These features help learners to understand their progress and focus their learning, Competence Committees to effectively assess resident progress and develop learning plans, and program directors to engage in continuous quality improvement at the faculty and program levels. E-portfolio development requires institutional support and ongoing stakeholder engagement to ensure that the provided analytics and visualizations promote learning and development within your program.

Further reading

1. Boscardin, Christy, et al. “Twelve tips to promote successful development of a learner performance dashboard within a medical education program.” Medical Teacher 40.8 (2018): 855-861.
2. Heeneman, Sylvia, and Erik W. Driessen. “The use of a portfolio in postgraduate medical education–reflect, assess and account, one for each or all in one?.” GMS Journal for Medical Education 34.5 (2017).
3. Thoma, Brent, et al. “Developing a dashboard to meet Competence Committee needs: a design-based research project.” Canadian Medical Education Journal 11.1 (2019): e16-e34.
4. Tochel, Claire, et al. “The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12.” Medical Teacher 31.4 (2009): 299-318.
5. Van Tartwijk, Jan, and Erik W. Driessen. “Portfolios for assessment and learning: AMEE Guide no. 45.” Medical Teacher 31.9 (2009): 790-801.