ABSTRACT
Background: Clinical networks have been found to provide benefits such as better and more standardized patient care and greater satisfaction for practitioners. In September 2017 the BC Emergency Medicine Network (EM Network) was launched to help practitioners deliver the best care. The EM Network’s website was developed to provide individual practitioners with access to clinical resources, research and innovation initiatives, continuing professional development, and real-time support. Since surprisingly little is known about how clinical networks are best structured and developed, the plans for the EM Network included an early evaluation process to document and guide growth.
Methods: Overall function of the EM Network after 1 year of operation was evaluated by analyzing membership and online engagement data and by conducting an online quantitative survey and subsequent qualitative interviews to obtain member feedback. Google Maps, Google Analytics, and Twitter Analytics were used, as well as PARTNER (Program to Analyze, Record, and Track Networks to Enhance Relationships), a validated social network analysis tool. The BC Ethics Harmonization Initiative advised that formal ethics approval was not needed because the study fits within a quality improvement framework.
Results: During the study period, the EM Network consisted of 622 of 1400 eligible members (44%) from 79 of 108 emergency care sites in BC (73%). Each month an average of 999 active users visited the website. While survey respondents indicated the EM Network is credible and respected, many were unaware of its purpose and offerings. Averaged scores for the perceptions of survey respondents regarding three network values (power/influence, involvement, resource contribution) ranged from 2.36 to 2.52, with 3.00 being considered good. When survey respondents were asked if they felt supported in their work by the Network, the majority said they felt “supported” or “somewhat supported.”
Conclusions: Our findings highlight the need for early evaluation after a network is launched to identify development needs. While our results must be interpreted cautiously because the EM Network is young, membership and online engagement data and member feedback indicate we need to increase awareness of offerings and encourage more online dialogue. Regular re-evaluation is planned to monitor progress and strengthen this initiative.
A review of membership and online engagement data and an analysis of survey and interview results provide valuable insights for those interested in creating and improving clinical networks that support practitioners.
The benefits of clinical networks are numerous and include better and more standardized patient care and less stress and more satisfaction for practitioners. In some situations, networks can eliminate the need to transfer patients to a higher level of care and thus can reduce costs and ensure continued local availability of ambulance crews for other patients.
In September 2017 the BC Emergency Medicine Network (EM Network) was launched to support the delivery of evidence-informed, patient-centred care in all 108 emergency departments (EDs) and diagnostic and treatment centres in BC.[1] The EM Network operates under the oversight of a Management Team and an Advisory Committee that includes patient partners. Through the EM Network website (www.bcemergencynetwork.ca [20]), individual practitioners can access resources and services in four functional programs: clinical resources, research and innovation, continuing professional development, and real-time support. These publicly accessible resources are designed to facilitate communication, physician engagement, patient engagement, and evaluation to achieve the EM Network’s vision: “Exceptional emergency care. Everywhere.”[2] As well, the website features a secure area where members can engage with other members in a discussion forum and access the member directory. Accessing the secure area requires approval by EM Network management to obtain a user name and password.
Together, the publicly accessible and secure parts of the website provide members with a practical point-of-care tool. For example, if an emergency practitioner is working alone and a patient presents with a condition seen rarely or not seen before by the practitioner, the practitioner can quickly access a brief synopsis of the condition (e.g., symptomatic atrial fibrillation) and its management in the form of a two-page summary or a 5-minute video. In future, if more help is needed, the practitioner will be able to connect synchronously with another practitioner in BC and be guided through the management of the condition virtually, which in the case of symptomatic atrial fibrillation might include electrical cardioversion. This technology-supported component of the EM Network for peer-to-peer clinician support has been piloted in one location and is still in development.
Although networks, including clinical networks, are recognized as important for disseminating information and standardizing evidence-informed care, surprisingly little is known about how they are best structured and developed.[3-6]
Knowing that we need to learn more about what makes networks successful, the challenges networks face, the evolution of a network life cycle, and the best strategies for success,[4,7-13] we included a formative evaluation in the design of the EM Network to assess overall network function. Our intention was to document early successes and obtain baseline data for future evaluations and to guide growth and improve the EM Network.
Overall function of the EM Network was evaluated regarding membership, online engagement, and member perceptions of value and progress. The BC Ethics Harmonization Initiative (https://bcethics.ca [21]) advised that the study fits within a quality improvement framework, precluding the need for a formal ethics application and approval process.
Membership data from 12 September 2017 (launch) to 31 August 2018 were exported from the EM Network website into an Excel spreadsheet for analysis. Membership in the EM Network is currently restricted to BC physicians practising in an emergency care setting with the exception of EM Network management staff, patient partners, and a few out-of-province content contributors. Members were plotted by primary hospital site using Google Maps. Members were classified as rural if their primary place of practice was considered rural by the BC Ministry of Health’s Rural Practice Subsidiary Agreement, which evaluates each community on its level of isolation.[14]
To determine how many users, both EM Network members and nonmembers, were accessing the website and their frequency of use, data from 1 September 2017 to 31 August 2018 were analyzed using Google Analytics. Users were defined as having a unique IP address.
To measure the online engagement of members, we analyzed the number of posts and replies in the members-only discussion forum. Additionally, we analyzed Twitter activity using Twitter Analytics to gain a broader understanding of how users interact with the EM Network.
Member feedback was obtained through an online survey and individual interviews. This component of the EM Network’s evaluation framework was developed based on literature recommendations[15-18] and in collaboration with EM Network members and patient partners.
In February and March 2018, a quantitative survey was conducted using PARTNER (Program to Analyze, Record, and Track Networks to Enhance Relationships), a validated social network analysis tool developed though an evaluation of over 150 community networks.[19] The PARTNER tool maps who is connected to whom, and provides a visual representation of the number and quality of relationships, the trust between partners, the value each partner brings to the relationship, resource contributions, and the roles that each partner plays. The tool requires one respondent per site or organization to answer questions, so one member from each of the emergency departments with members at the time of the study was invited to participate, thus ensuring a balanced geographic distribution and an appropriate rural/urban mix, with a rural site defined as a primary hospital in a community with a population of less than 10 000.[20] In order to minimize bias, EM Network Management Team members and Advisory Committee members were not eligible to participate. The survey was administered first to 77 members and a second time to 57 different members using the same selection strategy in order to increase the sample size. In both cases, survey respondents were invited via email and were sent three reminders. In total, 46 members responded. The two data sets were merged prior to analysis. For the purposes of the PARTNER survey, any redundancies were removed to achieve only one response per site, as the software is only able to recognize one response per site.
Twenty-one respondents to the quantitative survey were then asked to indicate whether they were willing to have a follow-up semi-structured interview. Both purposive and random sampling techniques were used to ensure a balanced geographic distribution and an appropriate rural/urban mix when selecting the 21 subjects. Survey respondents who had not completed the survey or only partially completed the survey were included in the interview invitation process in an effort to minimize selection bias.
The analysis of membership and online engagement data indicated that after 1 year the EM Network had members throughout the province and a multitude of active users of the website. The analysis of survey data indicated members perceive the Network to be credible and have a small to fair amount of overall value, while interview results indicated a general lack of awareness of the Network’s purpose and function.
On 31 August 2018, membership in the EM Network stood at 622 of approximately 1400 eligible physicians (44%), and these members were practising at 79 of 108 emergency care sites (73%) throughout BC, as shown in Figure 1 [22]. Looking at the rural/urban mix, 54% were from urban sites and 46% from rural sites.
From 1 September 2017 to 31 August 2018, 11 154 individuals with a unique IP address accessed the EM Network website. Figure 2 [23] shows that 999 users, on average, visited the site each month and 136 web pages were viewed each day. Users accessed the website from British Columbia (61%), other sites in Canada (12%), and outside of Canada (27%). In the members-only discussion forum, 27 topics were posted, with 9 of these coming from EM Network management, and 94 replies were posted, with 29 coming from EM Network management. Most posts (78%) received at least one reply. The most popular discussion topics concerned operational/administration issues, cardiovascular conditions, and toxicology.
It is important to note that interaction also occurred outside the online discussion forum. At the end of the study, the EM Network had over 550 Twitter followers. During the study, the EM Network received an average of 27 “mentions” on Twitter per month, 581 “impressions” per day (the number of times the EM Network’s Tweets are seen by others), and two “likes” per day.
A questionnaire was emailed to 134 EM Network members selected from across the province, and responses were received from 46 (34%). Respondents were asked about the EM Network’s progress in achieving its goals in each of the four program areas: clinical resources, research and innovation, continuing professional development, and real-time support. When asked if they could usually find what they were searching for on the EM Network website, 58% said they could and the remaining 42% answered “not applicable.” Respondents felt progress had been made in the following areas:
Respondents indicated that progress was less evident in other areas:
PARTNER software[19] was used to diagram the whole EM Network based on respondent perceptions, as shown in Figure 3 [24], which used a baseline set of indicators to depict the EM Network in its early phase of development. Each urban and rural node represents an ED or diagnostic and treatment centre with at least one emergency practitioner. These are shown along with nodes for external organizations that emergency practitioners contact regularly: BC Ambulance Service, BC Patient Transfer Network, Emergency Physician Online Support, BC Drug and Poison Information Centre, Rapid Access to Consultative Expertise, and STARS (Shock Trauma Air Rescue Service). Lines on the diagram indicate connections between sites and organizations. Although the location of the nodes in the diagram does not signify value, the size of each node does. The larger the node the more value respondents perceived that site or organization to have.
Value was measured using a combined score for level of power/influence, level of involvement, and level of resource contribution. Figure 4 [25] shows averaged scores for these three values, which were rated by survey respondents as 1 (no value), 2 (a small amount), 3 (a fair amount), or 4 (a great deal), with scores of 3 and higher being considered good.
When surveyed, subjects were asked if they felt more supported by the EM Network, and of the 22 survey respondents who completed that question, most indicated they felt “supported” or “somewhat supported,” as shown in Figure 5 [26].
Of 21 members then invited to take part in semi-structured interviews, 16 (76%) participated. When all comments were analyzed, five concerns emerged:
Despite these findings, many interview subjects said the EM Network is a credible and respected source and felt collaboration would likely improve over time as awareness and momentum builds.
After the first year of operation, the EM Network reviewed its membership and online engagement data and collected member feedback. The purpose was to document early successes and obtain baseline data to guide EM Network development and to use in future evaluations. We found almost three-quarters of EDs and diagnostic and treatment centres in BC had at least one registered EM Network member. However, despite the wide reach of membership, our results suggest that the EM Network needs to increase its efforts to create the awareness, trust, and collaboration required for a high-functioning, effective network. Further evaluation will help us understand the gaps that members judge to be most important.
We found it encouraging that many respondents indicated they already view the EM Network as credible and respected. Possible reasons for this include but are not limited to:
Our EM Network membership findings indicate that urban members are overrepresented at 54%, based on previous research showing that urban practitioners represent only 45% of all emergency practitioners in BC (unpublished data from 2016 UBC Department of Emergency Medicine survey by Marsden, Archibald, and Christenson), underscoring the need to reach more rural practitioners. Our membership findings also indicate we need to enhance current engagement strategies and consult with our partners about ways to encourage more rural practitioner involvement. Patient partners who are also active contributors to EM Network development may play a pivotal role in advocacy and community awareness.
With regard to online engagement, we found high website usage rates across BC. Over half of survey respondents indicated they were “usually” or “always” able to find what they were searching for. We were encouraged by these results though substantial effort is still needed to expand web content. The “not applicable” responses to survey questions are difficult to interpret as they could indicate that respondents did not have enough experience with the website to answer the question, or that they had not used the website to look for information. Although the reply rate is high for the members-only discussion forum, a small percentage of active users were identified and the EM Network Management Team members have played a significant role in initiating discussions. The use of online discussion forums is low in emergency medicine in general, and it will likely take time to change current culture and foster more member-driven discussion of issues. Results obtained during the EM Network development phase show that relatively few BC emergency practitioners use social media, which suggests that a behavioral shift will need to occur for the EM Network to support a larger number of important and interactive online discussions. We will continue to explore strategies to accelerate physician-to-physician interaction through the website.
With regard to online engagement by way of Twitter, we found the EM Network has a strong following, but that the majority of followers are not EM Network members. Instead, most followers are partners and health organizations, health care workers, researchers, students, and members of the general public. It can be argued that having a strong following from nonmembers legitimizes the EM Network and demonstrates its significance to society. Furthermore, Twitter activity shows that online discussion forum participation is not the only form of engagement, and other forms need to be considered. Twitter activity also indicates that different individuals prefer other (and potentially multiple) ways to interact with the EM Network.
Network success has been described as occurring when “members perceive it to be achieved.”[5] One of the main objectives of the evaluation was to learn whether members believe the EM Network is fulfilling its vision and mission, whether goals for the four programs are being met, and whether members are feeling more supported in their work with the EM Network in place. Results from the qualitative interviews show that members believe that there has been noticeable progress regarding:
It was not surprising that no progress was seen in implementing the real-time support program since only a pilot in one location has occurred so far. It was also not surprising to find that external organizations were perceived as more valuable than individual EDs. Measuring value is important for an effective network to ensure an appreciation of all members within the network. It will be informative to see how the perceptions of members change as the EM Network matures, and how this is reflected in the number of connections between sites and the rating of their relative value.
Overall, it is noteworthy that all but one survey respondent felt more supported or somewhat more supported with the EM Network in place. As the purpose of the EM Network is to support emergency practitioners, we see this as an indicator of success.
While early evaluation provides multiple benefits, there are also known risks. Most notably, studies have shown that the effectiveness of a network “is not likely to be demonstrated in the early years . . . for networks, the added time needed to establish trusting relationships and meaningful activity is a factor that must not be underestimated.”[3] Consequently, members may be less likely to respond to evaluation surveys (as illustrated by our study), which further reinforces the perception that members are not engaged. Exposing missteps through an evaluation can also inhibit membership growth and engagement and diminish enthusiasm in those charged with building the network or providing sponsorship. This may explain why little information on developing networks currently exists. We planned this early evaluation with the belief that these risks were worth taking since such preliminary findings might show where and how we could increase our success. Thus, given the early stage of EM Network development, we expected to find a general lack of awareness and engagement and were not surprised that the scores for the value of individual organizations in the EM Network were relatively low. We were gratified to find the overall structure and approach to support was not challenged, that the website itself is viewed positively, and that authorship and leadership are trusted. Finally, our findings suggest practitioners already feel more supported, despite our early stage of network growth, and have provided valuable and specific recommendations for improvement:
The chief limitation of this study is the low survey response rate. One reason for this may be the email address used for the survey request, which came from PARTNER and would have been unfamiliar to most recipients. Another reason for the low response rate may be poor member understanding and awareness of the EM Network, an initiative many knew too little about to provide feedback on, a notion supported by the responses of interview subjects who stated they were unable to comment on the relatively young network.
As a result of the low response rate, the PARTNER analysis considered a limited number of sites and relationships, and consequently found a relatively low perception of value for the EM Network. In addition, the PARTNER software was able to handle comments from only one member per emergency medicine site. Thus, despite repeating the survey twice and merging the data (something never done before), the resulting sample size of 134 was still modest relative to total EM Network membership.
This evaluation, undertaken in the first year of EM Network operation, highlights initial successes and identifies areas where further efforts are needed. A low level of awareness of the EM Network and engagement of the emergency medicine community still exists, and little to no perceived changes were reported to date in clinical behavior or job satisfaction. Analytics show there is frequent use of the website and membership is growing steadily. While membership stood at 44% of all EM practitioners in BC at the end of the study period in 2018, membership reached 737 (53%) as of 24 March 2019. We believe our findings are encouraging and appropriate for this early stage of network development, and that they provide valuable insights and strategies for others interested in creating and improving clinical networks. The BC Emergency Medicine Network will continue to gather data, evaluate, and make adjustments as necessary. By doing this we can expand network functions, document what success looks like for clinical networks, and fulfill the EM Network’s mission of sharing, supporting, and innovating to improve patient care in BC.
Dr Marsden is paid to serve as the lead for the clinical resources program of the BC Emergency Medicine Network. Ms Drebit, Ms MacKinnon, and Ms Archibald as well as Drs Abu-Laban, Khazei, and Lund all receive a salary from the BC Emergency Medicine Network. Dr Lindstrom receives consulting fees from the BC Emergency Medicine Network. The remaining authors have no competing interests to declare.
This article has been peer reviewed.
1. Christenson J. A network to improve emergency patient care by facilitating practitioners to effectively support practitioners. Healthc Manage Forum 2014;27:132-135.
2. Abu-Laban RB, Drebit SK, Lindstrom RR, et al. The British Columbia Emergency Medicine Network: A paradigm shift in a provincial system of emergency care. Cureus 2018;10:e2022.
3. Popp JK, Milward HB, MacKean G, et al. Inter-organizational Networks: A review of the literature to inform practice. Washington, DC: IBM Center for the Business of Government; 2014. Accessed 6 March 2019. www.businessofgovernment.org/sites/default/files/Inter-Organizational%20Networks.pdf [27].
4. Provan KG, Lemaire RH. Core concepts and key ideas for understanding public sector organizational networks: Using research to inform scholarship and practice. Public Admin Rev 2012;72:638-648.
5. Varda DM, Retrum JH. Collaborative performance as a function of network members’ perceptions of success. Public Perform Manag Rev 2015;38:632-653.
6. McInnes E, Middleton S, Gardner G, et al. A qualitative study of stakeholder views of the conditions for and outcomes of successful clinical networks. BMC Health Serv Res 2012;12:49.
7. Woodland R, Hutton M. Evaluating organizational collaborations: Suggested entry points and strategies. Am J Eval 2012;33:366-383.
8. Willis CD, Riley BL, Best A, Ongolo-Zogo P. Strengthening health systems through networks: The need for measurement and feedback. Health Policy Plan 2012;27:62-66.
9. Willis C, Kernoghan A, Riley B, et al. Outcomes of interorganizational networks in Canada for chronic disease prevention: Insights from a concept mapping study, 2015. Prev Chronic Dis 2015;12:E199.
10. Provan KG, Milward HB. Do networks really work? A framework for evaluating public-sector organizational networks. Public Admin Rev 2001;61:414-423.
11. Popp JK, Dolinski CM, Adair CE, et al. How do you evaluate a network? A Canadian child and youth health network experience. Can J Prog Eval 2005;20:123-150.
12. Provan KG, Fish A, Sydow J. Interorganizational networks at the network level: A review of the empirical literature on whole networks. J Management 2007;33:479.
13. Robeson P. Networking in public health: Exploring the value of networks to the National Collaborating Centres for Public Health. Hamilton, ON: National Collaborating Centre for Methods and Tools; 2009.
14. 2012 Rural Practice Subsidiary Agreement. Accessed 6 March 2019. https://www2.gov.bc.ca/assets/gov/health/practitioner-pro/appendix_c_rural_subsidiary_agreement.pdf [28].
15. Cunningham FC, Ranmuthugala G, Westbrook JI, Braithwaite J. Net benefits: Assessing the effectiveness of clinical networks in Australia through qualitative methods. Implement Sci 2012;7:108.
16. Haines M, Brown B, Craig J, et al. Determinants of successful clinical networks: The conceptual framework and study protocol. Implement Sci 2012;7:16.
17. Hill C. Network literature review: Conceptualizing and evaluating networks. Calgary: Southern Alberta Child and Youth Health Network; 2002. Accessed 6 March 2019. www.virtualhospice.ca/Assets/LiteratureReview_20090119154347.pdf [29].
18. McInnes E, Haines M, Dominello A, et al. What are the reasons for clinical network success? A qualitative study. BMC Health Serv Res 2015;15:497.
19. Visible Network Labs. PARTNER [Program to analyze, record, and track networks to enhance relationships]. Accessed 6 March 2019. www.partnertool.net [30].
20. du Plessis V, Beshiri R, Bollman R, Clemenson H. Definitions of “rural.” Ottawa: Statistics Canada; 2002. Accessed 6 March 2019. https://ageconsearch.umn.edu/bitstream/28031/1/wp020061.pdf [31].
All authors are affiliated with the BC Emergency Medicine Network and serve as either Management Team or Advisory Committee members. Dr Marsden is the lead for the EM Network’s clinical resources program and works in the emergency departments of Mount Saint Joseph Hospital, St. Paul’s Hospital, and Vancouver General Hospital. Ms Drebit is the manager of the EM Network. Dr Lindstrom is the EM Network’s evaluation lead. He is also an adjunct professor in the School of Population and Public Health at the University of British Columbia and in the School of Leadership Studies at Royal Roads University. Ms MacKinnon provides support for the EM Network’s research and innovation program and the EM Network’s communications strategy. Ms Archibald provides administrative and communications support for the EM Network. Dr Abu-Laban is the lead for the EM Network’s research and innovation program and an emergency physician at Vancouver General Hospital. He is also an associate professor and research director of the UBC Department of Emergency Medicine. Ms Eggers is a patient partner on the EM Network’s Advisory Committee. She is also a member of the Northern Health Critical Care Network Consensus Group. Dr Ho is the lead for the EM Network’s real-time support program and an emergency physician at Vancouver General Hospital. He is also a professor and the lead for digital emergency medicine in the UBC Department of Emergency Medicine. Dr Khazei is the lead for the EM Network’s continuing professional development program, and practises emergency and hyperbaric medicine at Vancouver General Hospital. Dr Lund is the EM Network’s communications advisor and an emergency physician at Royal Columbian Hospital and Eagle Ridge Hospital. He also works with BC Emergency Health Services as an online support physician. Mr Martin is a patient partner on the EM Network’s Advisory Committee. Dr Christenson is the EM Network’s executive lead and a professor and head of the UBC Department of Emergency Medicine.
Links
[1] https://bcmj.org/node/7230
[2] https://bcmj.org/author/julian-marsden-md
[3] https://bcmj.org/author/sharla-drebit-msc-mba
[4] https://bcmj.org/author/ronald-r-lindstrom-msc-phd
[5] https://bcmj.org/author/carolyn-mackinnon-ba
[6] https://bcmj.org/author/chantel-archibald
[7] https://bcmj.org/author/riyad-b-abu-laban-md-mhsc
[8] https://bcmj.org/author/kim-eggers
[9] https://bcmj.org/author/kendall-ho-md-frcpc
[10] https://bcmj.org/author/afshin-khazei-md
[11] https://bcmj.org/author/adam-lund-md-med
[12] https://bcmj.org/author/ed-martin-ba
[13] https://bcmj.org/author/jim-christenson-md
[14] https://bcmj.org/sites/default/files/BCMJ_Vol61_No4-emergency-med-network.pdf
[15] https://bcmj.org/print/articles/bc-emergency-medicine-network-evaluation-approach-and-early-findings
[16] https://bcmj.org/printmail/articles/bc-emergency-medicine-network-evaluation-approach-and-early-findings
[17] http://www.facebook.com/share.php?u=https://bcmj.org/print/articles/bc-emergency-medicine-network-evaluation-approach-and-early-findings
[18] https://twitter.com/intent/tweet?text=The BC Emergency Medicine Network: Evaluation approach and early findings&url=https://bcmj.org/print/articles/bc-emergency-medicine-network-evaluation-approach-and-early-findings&via=BCMedicalJrnl&tw_p=tweetbutton
[19] https://bcmj.org/javascript%3A%3B
[20] https://www.bcemergencynetwork.ca/
[21] https://bcethics.ca
[22] https://bcmj.org/sites/default/files/public/BCMJ_Vol61_No4-emergency-med-network-web-fig1.jpg
[23] https://bcmj.org/sites/default/files/public/BCMJ_Vol61_No4-emergency-med-network-web-fig2.jpg
[24] https://bcmj.org/sites/default/files/public/BCMJ_Vol61_No4-emergency-med-network-web-fig3.jpg
[25] https://bcmj.org/sites/default/files/public/BCMJ_Vol61_No4-emergency-med-network-web-fig4.jpg
[26] https://bcmj.org/sites/default/files/public/BCMJ_Vol61_No4-emergency-med-network-web-fig5.jpg
[27] http://www.businessofgovernment.org/sites/default/files/Inter-Organizational%20Networks.pdf
[28] https://www2.gov.bc.ca/assets/gov/health/practitioner-pro/appendix_c_rural_subsidiary_agreement.pdf
[29] http://www.virtualhospice.ca/Assets/LiteratureReview_20090119154347.pdf
[30] https://partnertool.net/
[31] https://ageconsearch.umn.edu/bitstream/28031/1/wp020061.pdf
[32] https://bcmj.org/modal_forms/nojs/webform/176?arturl=/articles/bc-emergency-medicine-network-evaluation-approach-and-early-findings&nodeid=67
[33] https://bcmj.org/%3Finline%3Dtrue%23citationpop