Background

The benefits of clinical networks are numerous and include better and more standardized patient care and less stress and more satisfaction for practitioners. In some situations, networks can eliminate the need to transfer patients to a higher level of care and thus can reduce costs and ensure continued local availability of ambulance crews for other patients.

In September 2017 the BC Emergency Medicine Network (EM Network) was launched to support the delivery of evidence-informed, patient-centred care in all 108 emergency departments (EDs) and diagnostic and treatment centres in BC.[1] The EM Network operates under the oversight of a Management Team and an Advisory Committee that includes patient partners. Through the EM Network website (www.bcemergencynetwork.ca), individual practitioners can access resources and services in four functional programs: clinical resources, research and innovation, continuing professional development, and real-time support. These publicly accessible resources are designed to facilitate communication, physician engagement, patient engagement, and evaluation to achieve the EM Network’s vision: “Exceptional emergency care. Everywhere.”[2] As well, the website features a secure area where members can engage with other members in a discussion forum and access the member directory. Accessing the secure area requires approval by EM Network management to obtain a user name and password.

Together, the publicly accessible and secure parts of the website provide members with a practical point-of-care tool. For example, if an emergency practitioner is working alone and a patient presents with a condition seen rarely or not seen before by the practitioner, the practitioner can quickly access a brief synopsis of the condition (e.g., symptomatic atrial fibrillation) and its management in the form of a two-page summary or a 5-minute video. In future, if more help is needed, the practitioner will be able to connect synchronously with another practitioner in BC and be guided through the management of the condition virtually, which in the case of symptomatic atrial fibrillation might include electrical cardioversion. This technology-supported component of the EM Network for peer-to-peer clinician support has been piloted in one location and is still in development.

Although networks, including clinical networks, are recognized as important for disseminating information and standardizing evidence-informed care, surprisingly little is known about how they are best structured and developed.[3-6]

Knowing that we need to learn more about what makes networks successful, the challenges networks face, the evolution of a network life cycle, and the best strategies for success,[4,7-13] we included a formative evaluation in the design of the EM Network to assess overall network function. Our intention was to document early successes and obtain baseline data for future evaluations and to guide growth and improve the EM Network.

Methods

Overall function of the EM Network was evaluated regarding membership, online engagement, and member perceptions of value and progress. The BC Ethics Harmonization Initiative (https://bcethics.ca) advised that the study fits within a quality improvement framework, precluding the need for a formal ethics application and approval process.

Membership

Membership data from 12 September 2017 (launch) to 31 August 2018 were exported from the EM Network website into an Excel spreadsheet for analysis. Membership in the EM Network is currently restricted to BC physicians practising in an emergency care setting with the exception of EM Network management staff, patient partners, and a few out-of-province content contributors. Members were plotted by primary hospital site using Google Maps. Members were classified as rural if their primary place of practice was considered rural by the BC Ministry of Health’s Rural Practice Subsidiary Agreement, which evaluates each community on its level of isolation.[14]

Online engagement

To determine how many users, both EM Network members and nonmembers, were accessing the website and their frequency of use, data from 1 September 2017 to 31 August 2018 were analyzed using Google Analytics. Users were defined as having a unique IP address.

To measure the online engagement of members, we analyzed the number of posts and replies in the members-only discussion forum. Additionally, we analyzed Twitter activity using Twitter Analytics to gain a broader understanding of how users interact with the EM Network.

Member feedback

Member feedback was obtained through an online survey and individual interviews. This component of the EM Network’s evaluation framework was developed based on literature recommendations[15-18] and in collaboration with EM Network members and patient partners.

In February and March 2018, a quantitative survey was conducted using PARTNER (Program to Analyze, Record, and Track Networks to Enhance Relationships), a validated social network analysis tool developed though an evaluation of over 150 community networks.[19] The PARTNER tool maps who is connected to whom, and provides a visual representation of the number and quality of relationships, the trust between partners, the value each partner brings to the relationship, resource contributions, and the roles that each partner plays. The tool requires one respondent per site or organization to answer questions, so one member from each of the emergency departments with members at the time of the study was invited to participate, thus ensuring a balanced geographic distribution and an appropriate rural/urban mix, with a rural site defined as a primary hospital in a community with a population of less than 10 000.[20] In order to minimize bias, EM Network Management Team members and Advisory Committee members were not eligible to participate. The survey was administered first to 77 members and a second time to 57 different members using the same selection strategy in order to increase the sample size. In both cases, survey respondents were invited via email and were sent three reminders. In total, 46 members responded. The two data sets were merged prior to analysis. For the purposes of the PARTNER survey, any redundancies were removed to achieve only one response per site, as the software is only able to recognize one response per site.

Twenty-one respondents to the quantitative survey were then asked to indicate whether they were willing to have a follow-up semi-structured interview. Both purposive and random sampling techniques were used to ensure a balanced geographic distribution and an appropriate rural/urban mix when selecting the 21 subjects. Survey respondents who had not completed the survey or only partially completed the survey were included in the interview invitation process in an effort to minimize selection bias.

Results

The analysis of membership and online engagement data indicated that after 1 year the EM Network had members throughout the province and a multitude of active users of the website. The analysis of survey data indicated members perceive the Network to be credible and have a small to fair amount of overall value, while interview results indicated a general lack of awareness of the Network’s purpose and function.

Membership

On 31 August 2018, membership in the EM Network stood at 622 of approximately 1400 eligible physicians (44%), and these members were practising at 79 of 108 emergency care sites (73%) throughout BC, as shown in Figure 1. Looking at the rural/urban mix, 54% were from urban sites and 46% from rural sites.

Online engagement

From 1 September 2017 to 31 August 2018, 11 154 individuals with a unique IP address accessed the EM Network website. Figure 2 shows that 999 users, on average, visited the site each month and 136 web pages were viewed each day. Users accessed the website from British Columbia (61%), other sites in Canada (12%), and outside of Canada (27%). In the members-only discussion forum, 27 topics were posted, with 9 of these coming from EM Network management, and 94 replies were posted, with 29 coming from EM Network management. Most posts (78%) received at least one reply. The most popular discussion topics concerned operational/administration issues, cardiovascular conditions, and toxicology.

It is important to note that interaction also occurred outside the online discussion forum. At the end of the study, the EM Network had over 550 Twitter followers. During the study, the EM Network received an average of 27 “mentions” on Twitter per month, 581 “impressions” per day (the number of times the EM Network’s Tweets are seen by others), and two “likes” per day.

Member feedback

A questionnaire was emailed to 134 EM Network members selected from across the province, and responses were received from 46 (34%). Respondents were asked about the EM Network’s progress in achieving its goals in each of the four program areas: clinical resources, research and innovation, continuing professional development, and real-time support. When asked if they could usually find what they were searching for on the EM Network website, 58% said they could and the remaining 42% answered “not applicable.” Respondents felt progress had been made in the following areas:

• Development of a structure to share clinical and operational solutions and tools (33%).
• Fulfillment of the vision and mission of the EM Network (31%).
• Dissemination of important knowledge for critical decision making through continuing professional development (11%).
• Clinical innovation to improve care across BC (8%).

Respondents indicated that progress was less evident in other areas:

• Integration of continuing professional development opportunities to acquire and maintain necessary EM skills such as increased simulation program capacity (5%).
• Implementation of real-time support (5%).
• System innovation (5%).
• Clinical resources knowledge translation (2%).

PARTNER software[19] was used to diagram the whole EM Network based on respondent perceptions, as shown in Figure 3, which used a baseline set of indicators to depict the EM Network in its early phase of development. Each urban and rural node represents an ED or diagnostic and treatment centre with at least one emergency practitioner. These are shown along with nodes for external organizations that emergency practitioners contact regularly: BC Ambulance Service, BC Patient Transfer Network, Emergency Physician Online Support, BC Drug and Poison Information Centre, Rapid Access to Consultative Expertise, and STARS (Shock Trauma Air Rescue Service). Lines on the diagram indicate connections between sites and organizations. Although the location of the nodes in the diagram does not signify value, the size of each node does. The larger the node the more value respondents perceived that site or organization to have.

Value was measured using a combined score for level of power/influence, level of involvement, and level of resource contribution. Figure 4 shows averaged scores for these three values, which were rated by survey respondents as 1 (no value), 2 (a small amount), 3 (a fair amount), or 4 (a great deal), with scores of 3 and higher being considered good.

When surveyed, subjects were asked if they felt more supported by the EM Network, and of the 22 survey respondents who completed that question, most indicated they felt “supported” or “somewhat supported,” as shown in Figure 5.

Of 21 members then invited to take part in semi-structured interviews, 16 (76%) participated. When all comments were analyzed, five concerns emerged:

1. General lack of awareness of the EM Network’s purpose and structure.
2. Lack of engagement regarding EM Network activities.
3. Little to no change perceived in collaborative behavior due to the EM Network.
4. Limited improvement perceived in job satisfaction due to the EM Network.
5. Lack of knowledge of EM Network offerings.

Despite these findings, many interview subjects said the EM Network is a credible and respected source and felt collaboration would likely improve over time as awareness and momentum builds.

Conclusions

After the first year of operation, the EM Network reviewed its membership and online engagement data and collected member feedback. The purpose was to document early successes and obtain baseline data to guide EM Network development and to use in future evaluations. We found almost three-quarters of EDs and diagnostic and treatment centres in BC had at least one registered EM Network member. However, despite the wide reach of membership, our results suggest that the EM Network needs to increase its efforts to create the awareness, trust, and collaboration required for a high-functioning, effective network. Further evaluation will help us understand the gaps that members judge to be most important.

We found it encouraging that many respondents indicated they already view the EM Network as credible and respected. Possible reasons for this include but are not limited to:

• The Network’s affiliation with the UBC Department of Emergency Medicine.
• Reputable EM Network members with name recognition.
• Resonance with the EM Network vision.
• Clinically relevant website content.
• Plans to implement real-time support.

Our EM Network membership findings indicate that urban members are overrepresented at 54%, based on previous research showing that urban practitioners represent only 45% of all emergency practitioners in BC (unpublished data from 2016 UBC Department of Emergency Medicine survey by Marsden, Archibald, and Christenson), underscoring the need to reach more rural practitioners. Our membership findings also indicate we need to enhance current engagement strategies and consult with our partners about ways to encourage more rural practitioner involvement. Patient partners who are also active contributors to EM Network development may play a pivotal role in advocacy and community awareness.

With regard to online engagement, we found high website usage rates across BC. Over half of survey respondents indicated they were “usually” or “always” able to find what they were searching for. We were encouraged by these results though substantial effort is still needed to expand web content. The “not applicable” responses to survey questions are difficult to interpret as they could indicate that respondents did not have enough experience with the website to answer the question, or that they had not used the website to look for information. Although the reply rate is high for the members-only discussion forum, a small percentage of active users were identified and the EM Network Management Team members have played a significant role in initiating discussions. The use of online discussion forums is low in emergency medicine in general, and it will likely take time to change current culture and foster more member-driven discussion of issues. Results obtained during the EM Network development phase show that relatively few BC emergency practitioners use social media, which suggests that a behavioral shift will need to occur for the EM Network to support a larger number of important and interactive online discussions. We will continue to explore strategies to accelerate physician-to-physician interaction through the website.

With regard to online engagement by way of Twitter, we found the EM Network has a strong following, but that the majority of followers are not EM Network members. Instead, most followers are partners and health organizations, health care workers, researchers, students, and members of the general public. It can be argued that having a strong following from nonmembers legitimizes the EM Network and demonstrates its significance to society. Furthermore, Twitter activity shows that online discussion forum participation is not the only form of engagement, and other forms need to be considered. Twitter activity also indicates that different individuals prefer other (and potentially multiple) ways to interact with the EM Network.

Network success has been described as occurring when “members perceive it to be achieved.”[5] One of the main objectives of the evaluation was to learn whether members believe the EM Network is fulfilling its vision and mission, whether goals for the four programs are being met, and whether members are feeling more supported in their work with the EM Network in place. Results from the qualitative interviews show that members believe that there has been noticeable progress regarding:

• Fulfilling the vision and mission of the EM Network.
• Establishing a clinical resources structure and repository.
• Facilitating continuing professional development.
• Supporting research and innovation.

It was not surprising that no progress was seen in implementing the real-time support program since only a pilot in one location has occurred so far. It was also not surprising to find that external organizations were perceived as more valuable than individual EDs. Measuring value is important for an effective network to ensure an appreciation of all members within the network. It will be informative to see how the perceptions of members change as the EM Network matures, and how this is reflected in the number of connections between sites and the rating of their relative value.

Overall, it is noteworthy that all but one survey respondent felt more supported or somewhat more supported with the EM Network in place. As the purpose of the EM Network is to support emergency practitioners, we see this as an indicator of success.

Risks of early evaluation

While early evaluation provides multiple benefits, there are also known risks. Most notably, studies have shown that the effectiveness of a network “is not likely to be demonstrated in the early years . . . for networks, the added time needed to establish trusting relationships and meaningful activity is a factor that must not be underestimated.”[3] Consequently, members may be less likely to respond to evaluation surveys (as illustrated by our study), which further reinforces the perception that members are not engaged. Exposing missteps through an evaluation can also inhibit membership growth and engagement and diminish enthusiasm in those charged with building the network or providing sponsorship. This may explain why little information on developing networks currently exists. We planned this early evaluation with the belief that these risks were worth taking since such preliminary findings might show where and how we could increase our success. Thus, given the early stage of EM Network development, we expected to find a general lack of awareness and engagement and were not surprised that the scores for the value of individual organizations in the EM Network were relatively low. We were gratified to find the overall structure and approach to support was not challenged, that the website itself is viewed positively, and that authorship and leadership are trusted. Finally, our findings suggest practitioners already feel more supported, despite our early stage of network growth, and have provided valuable and specific recommendations for improvement:

• Increasing face-to-face visits of targeted communities.
• Maintaining or accelerating communication strategies to increase engagement.
• Providing new techniques that encourage member contributions and comments in order to grow and improve content.

Study limitations

The chief limitation of this study is the low survey response rate. One reason for this may be the email address used for the survey request, which came from PARTNER and would have been unfamiliar to most recipients. Another reason for the low response rate may be poor member understanding and awareness of the EM Network, an initiative many knew too little about to provide feedback on, a notion supported by the responses of interview subjects who stated they were unable to comment on the relatively young network.

As a result of the low response rate, the PARTNER analysis considered a limited number of sites and relationships, and consequently found a relatively low perception of value for the EM Network. In addition, the PARTNER software was able to handle comments from only one member per emergency medicine site. Thus, despite repeating the survey twice and merging the data (something never done before), the resulting sample size of 134 was still modest relative to total EM Network membership.

Summary

This evaluation, undertaken in the first year of EM Network operation, highlights initial successes and identifies areas where further efforts are needed. A low level of awareness of the EM Network and engagement of the emergency medicine community still exists, and little to no perceived changes were reported to date in clinical behavior or job satisfaction. Analytics show there is frequent use of the website and membership is growing steadily. While membership stood at 44% of all EM practitioners in BC at the end of the study period in 2018, membership reached 737 (53%) as of 24 March 2019. We believe our findings are encouraging and appropriate for this early stage of network development, and that they provide valuable insights and strategies for others interested in creating and improving clinical networks. The BC Emergency Medicine Network will continue to gather data, evaluate, and make adjustments as necessary. By doing this we can expand network functions, document what success looks like for clinical networks, and fulfill the EM Network’s mission of sharing, supporting, and innovating to improve patient care in BC.

Competing interests

Dr Marsden is paid to serve as the lead for the clinical resources program of the BC Emergency Medicine Network. Ms Drebit, Ms MacKinnon, and Ms Archibald as well as Drs Abu-Laban, Khazei, and Lund all receive a salary from the BC Emergency Medicine Network. Dr Lindstrom receives consulting fees from the BC Emergency Medicine Network. The remaining authors have no competing interests to declare.

This article has been peer reviewed.

All authors are affiliated with the BC Emergency Medicine Network and serve as either Management Team or Advisory Committee members. Dr Marsden is the lead for the EM Network’s clinical resources program and works in the emergency departments of Mount Saint Joseph Hospital, St. Paul’s Hospital, and Vancouver General Hospital. Ms Drebit is the manager of the EM Network. Dr Lindstrom is the EM Network’s evaluation lead. He is also an adjunct professor in the School of Population and Public Health at the University of British Columbia and in the School of Leadership Studies at Royal Roads University. Ms MacKinnon provides support for the EM Network’s research and innovation program and the EM Network’s communications strategy. Ms Archibald provides administrative and communications support for the EM Network. Dr Abu-Laban is the lead for the EM Network’s research and innovation program and an emergency physician at Vancouver General Hospital. He is also an associate professor and research director of the UBC Department of Emergency Medicine. Ms Eggers is a patient partner on the EM Network’s Advisory Committee. She is also a member of the Northern Health Critical Care Network Consensus Group. Dr Ho is the lead for the EM Network’s real-time support program and an emergency physician at Vancouver General Hospital. He is also a professor and the lead for digital emergency medicine in the UBC Department of Emergency Medicine. Dr Khazei is the lead for the EM Network’s continuing professional development program, and practises emergency and hyperbaric medicine at Vancouver General Hospital. Dr Lund is the EM Network’s communications advisor and an emergency physician at Royal Columbian Hospital and Eagle Ridge Hospital. He also works with BC Emergency Health Services as an online support physician. Mr Martin is a patient partner on the EM Network’s Advisory Committee. Dr Christenson is the EM Network’s executive lead and a professor and head of the UBC Department of Emergency Medicine.