Developmental Evaluation of a Collective Impact Initiative: Insights for Foundations

Citation metadata

Date: June 2018
From: The Foundation Review(Vol. 10, Issue 2)
Publisher: Grand Valley State University
Document Type: Report
Length: 6,347 words
Lexile Measure: 1450L

Document controls

Main content

Abstract: 

Keywords: Collective impact, developmental evaluation, realist framework

Full Text: 

Introduction

Seven years ago, the publication of Kania and Kramer's (2011) influential paper on collective impact caught the attention of organizations across sectors, including nonprofit organizations and philanthropies (Cabaj & Weaver, 2016; Cooper, 2017; Easterling, 2013; Lynn, Breckinridge, Denault, & Marvin, 2015). The Colorado Health Foundation was one of the organizations that saw the potential of collective impact to help tackle complex, systems-level health issues in Colorado.

In 2013, the Foundation embarked on a collective impact initiative focused on health care delivery system and payment reform (DSPR), an area in which the Foundation had made large investments for many years and where significant partnerships were already established. It was conceptualized as a statewide effort intended to align actors and realize greater impact from the Foundation's investments. By the end of 2016, the collective impact initiative had been dissolved by mutual agreement of the initiative's steering committee and the Foundation.

This article describes the collective impact initiative and the role that developmental evaluation--and a realist framework--played in aiding both the initiative's steering committee and the Foundation in making decisions about its accomplishments and future. It highlights the developmental evaluation approach, how that informed decisions, and how it helped surface broader insights about doing highly collaborative work.

Context

The term "collective impact" was first named and described in a 2011 article in the Stanford Social Innovation Review by John Kania and Mark Kramer (Kania & Kramer, 2011). The authors suggested that the nonprofit sector traditionally supported isolated impact--directing resources to individual organizations thought to be the best change-makers in specific areas. They also suggested that this strategy had not resulted in the innovation needed to address large, complex social problems, and that what was needed was cross-sector coalitions engaged with those outside the nonprofit sector--a strategy they dubbed collective impact. Such problems require a systemic approach to social impact that focuses on the relationships between organizations and the progress toward shared objectives. Kania and Kramer further proposed that five conditions--now known as the five pillars of collective impact--were needed for successful collective impact initiatives:

1. a common agenda,

2. shared measurement systems,

3. mutually reinforcing activities,

4. continuous communication, and

5. backbone support organizations.

In the years since Kania and Kramer's article, philanthropic organizations, nonprofit organizations, government, private industry, consultants, and others have embraced the concept of collective impact. While entities have engaged in varying forms of collaboration for years, (1) what Kania and Kramer tried to do in their 2011 article was to synthesize and bring structure to a collaborative approach that is more rigorous than typical collaboration.

The Colorado Health Foundation embarked on a DSPR collective impact initiative in 2013, with an original collaborative group that included executive representation from 11 Colorado organizations. The initiative's genesis was twofold. First, there was a 2011 request by the Foundation's board to find a way to create greater value for the considerable investments that it had been making for many years in the DSPR space. The board's direction catalyzed Foundation staff to consider new ways of supporting coordination among grantee organizations so as to reduce duplication of efforts, create better alignment among organizations working on the topic, and realize greater impact. In early 2013, as the idea of collective impact gained momentum, a grantee stepped forward to suggest that it might be a suitable backbone organization to organize and drive the work among actors in the DSPR space. Though the Foundation did not fund that initial request, it did assess how it might use a collective impact approach to support its DSPR grantees, how interested organizations might pursue such an effort together, and what the Foundation's own role would be if the initiative was pursued.

The Foundation knew it would need external expertise to help execute a collective impact approach, since the concept was new to both leadership and staff. To this end, the Foundation in early 2013 engaged a consulting firm that specialized in managing the processes associated with setting up and executing collective impact initiatives. The consultants assessed stakeholder readiness, conducted landscape scans, and began to assist the newly formed steering committee to put in place the building blocks necessary to create the five pillars of collective impact. The readiness assessment helped the Foundation explore:

* the environmental context around whether there was a belief that coordinated action could lead to greater impact on this issue, and a propensity towards trust;

* the potential to align around a common vision and strategies for achieving it;

* stakeholder interest in engagement; and

* strength of key stakeholders related to roles and responsibilities.

The consultant's assessment showed that most of these criteria were met. The Foundation was particularly excited to learn that stakeholders believed there was considerable potential for a collaborative process to create greater impact. Stakeholders clearly said that the Foundation could play a unique role as partner, leader, and funder in launching the work. The results left the Foundation with the impression that there was end, it decided to tie its entire funding in this area to what emerged from the collective impact effort. In practical terms, this meant that the Foundation would not fund DSPR grants outside of collective impact, which positioned the steering committee to prioritize and design bodies of work that the Foundation would then fund.

The Foundation also sought to create funding alignment with the emerging collective impact initiative by lining up existing grant timing and expectations. This created a single point in time when existing grants for organizations represented on the steering committee ended, and the Foundation could make new grants that included the requirement that grantees participate in, and align portions of their work with, the goals determined by the collective impact initiative. In addition, the Foundation provided funding for backbone support and evaluation through contracts that were directly held by the foundation. The backbone function initially consisted of support from a facilitation consultant, with the expectation that this function would be formalized later in a backbone organization.

The Foundation demonstrated its organizational commitment to rapid-cycle learning from the beginning and partnered with external evaluators to support that learning. The steering committee chose to partner with evaluators at the Georgia Health Policy Center at Georgia State University through a competitive bidding process in late 2014, and the evaluators began work in May 2015--a time at which the Foundation was also in the middle of a national search for a new CEO. By September 2015, the new CEO was in place and was interested in exploring how effectively the collective impact approach was achieving what had been intended.

Methods

Developmental evaluation formed the basis of the collective impact evaluation; it is defined as an approach to understanding the activities of a program operating in a dynamic or novel environment characterized by complex interactions (Norman, 2011). The method focuses on strategic learning rather than simply assessing outcomes. It examines activities in context and provides feedback to the overall process (Patton, 2010). Collective impact initiatives address challenges that are complex and adaptive (Kania & Kramer, 2013; Mann, 2014; Weaver, 2014a). Since delivery system and payment reform was seen by the Foundation as complex, adaptive, and requiring innovation, and due to the organizational desire for strategic learning to inform their own practice (in addition to that of the broader field), developmental evaluation was seen by the Foundation and evaluation team as a good fit for this initiative. According to Patton (2006), the tools and techniques of developmental evaluation should be utilization focused, with measures and tracking mechanisms developed as outcomes emerge. The approach uses rapid, real time feedback, and the aim is to nurture learning.

The Foundation's commitment to evaluation as a learning tool was rooted in an interest in real-time improvement, and understanding what actions influenced success or failure more broadly in deeply collaborative work. The process of working together to create collaborative change is very complex and is constantly impacted by many uncontrollable factors (Minyard, Phillips, & Baker, 2016). As such, the evaluators added a realist evaluation lens to help unravel the web of conditions and actions that influenced success or failure. The method explores the relationships among context (organizational setting and external constraints), mechanisms (reasoning and resources), and outcomes (intended and unintended results) (Pawson & Tilley, 1997). The realist framework assumes that innovations, programs, and interventions work only in particular contexts and that the purpose of evaluation is to find those conditions: Which mechanisms work, in which contexts, to produce which outcomes (Greenhalgh et al., 2009)? By understanding the interactions of these factors and how they enabled or inhibited outcomes, it was hoped that the findings would be more useful not only to the Foundation, but also to decision-makers beyond this particular initiative.

Consistent with Patton's (2006) call for the use of tools and techniques that match the developmental needs of evaluation users, the evaluation team used a variety of approaches to collect and analyze data around the collective impact initiative:

* Document review. The evaluators began in spring 2015 by reviewing more than 200 documents and emails that had been generated over the previous two years. Notes were abstracted from the documents by two researchers, and thematic analysis was conducted.

* Key informant interviews. The evaluators conducted 20 semistructured key informant interviews with individuals who were currently, or had been, connected to the initiative. The 12 interview questions covered thoughts on collective impact, the Foundation's role in the work, the initiative's funding structure, steering committee membership and dynamics, and ideas about short-term and long-term success. A thematic analysis was conducted of these data.

* Polling. In June 2015, the evaluation team administered a poll to gauge the steering committee's opinions about whether the initiative in which it was engaged was developmental, the degree to which the group had adaptive capacity, and the degree to which it was ready for developmental evaluation (Cabaj, 2014a). In August 2015, the team created a short survey to assess the five pillars of collective impact (Preskill, Parkhurst, & Juster, 2014) and explore what the group believed it had accomplished over the past two years of work.

* Participant observation. From May 2015 through July 2016, the evaluation team observed 17 steering committee meetings. At least two, and often three, evaluation team members documented observations with structured meeting notes that aligned with the context, mechanism, and outcomes of the realist framework. Notes were compared and synthesized by theme.

* Feedback loops. The evaluators established a number of feedback loops in order to collect, process, and reflect information back to stakeholders. The feedback loops included monthly calls with a member of the Foundation evaluation team, the steering group facilitators, and a subgroup of steering committee members who served as a four-member evaluation advisory group; and periodic check-ins with internal and external stakeholders. The establishment of an evaluation advisory group is a standard of Georgia Health Policy Center's evaluation practice in alignment with the Centers for Disease Control and Prevention's evaluation guidelines to engage stakeholders (Milstein, Wetterhall, & Group, 2000). Conversations with the steering committee facilitators enabled the evaluators to be kept apprised of local dynamics and served as a check on what evaluators were observing during steering committee meetings. Regular dialogue with the evaluation advisory group enabled the evaluators to pressure test information with a subset of the steering committee prior to sharing with the larger group. Monthly meetings and calls with the Foundation enabled staff to reflect on how the work was unfolding, explore how they were showing up as actors in it, and consider how that was impacting the group's progress.

* Sense-making. Although the use of a realist framework is method neutral, an important aspect of the approach is pattern recognition (Pawson & Tilley, 1997). The evaluators referred to this process of pattern recognition as sense-making. The evaluation team leveraged the opportunity of reviewing data and information with a number of stakeholders, including the steering committee, the Foundation's staff and philanthropy committee, and the facilitators so that everyone was engaged in identifying patterns that were emerging from the work. Soon after joining the initiative, the evaluation team worked with the facilitators to establish a portion of the monthly steering committee meeting that would be spent reviewing evaluation data and sharing feedback to help identify patterns in what was unfolding. The evaluation team also met internally on a quarterly basis to make sense of the information that was emerging.

Results

The initial interviews conducted by the evaluation team in May 2015 provided the first systematic information the Foundation received about participants' perceptions of the collective impact work to date. Overall, data showed that steering committee members were supportive of collective impact. They acknowledged it was a long process with many moving parts, especially since Colorado was concurrently involved in many federal health reform efforts; one steering committee member commented that Colorado was "eating from the all-you-can-eat health reform buffet." However, they asserted that they wanted to take some kind of action soon. Said one: "We need progress, not process." They acknowledged there had been past issues around trust among organizations and individuals on the steering committee, but felt that, over time, trust had improved. Several expressed concern about how the initiative would continue to be managed given that the backbone function was spread across the participating organizations--a situation they referred to as a distributed backbone model. Regarding the composition of the steering committee, members were pleased by the addition to the group in late 2014 of providers and insurers that were not Foundation grantees.

Steering committee members remarked that they saw the Foundation as a champion, and acknowledged its potential to build partnerships. "This would not have happened without the Foundation bringing us together," one member observed. But the data also illuminated challenges. While members reported that nothing would be possible without the Foundation's funding, they said it "increases the sense of competitiveness among [steering committee] members, contributing to uncertainty about future funding and lack of trust among members." One steering committee member commented, "It's hard for the Foundation not to own it"--an observation that reflected the Foundation's own struggle with what its role should be. As one staff member put it, "We're not sure how to balance grant monitoring and how to be a partner at the table."

A poll of steering committee members administered in June 2015 revealed that it wasn't clear whether members were thinking about the work as a complex adaptive problem. (see Figure 1.) About 50 percent of members disagreed with the statement, "The challenge we want to address is difficult to define"; and more than 60 percent disagreed with the statement, "The factors that contribute to progress in meeting the challenge are unknown or unclear." Almost 90 percent disagreed that "We have a history of innovation and tackling complex challenges"; and almost half the members disagreed that "We have the patience to experiment with new approaches and generate results."

In September 2015, the evaluation team polled the steering committee members to gauge how much progress they felt they had made along three of the five pillars of collective impact. The poll did not assess shared measurement or backbone support, as the committee had not yet addressed them. On a five-point scale, the group gave itself a "two" for pursuing a common agenda, continuous communication, and mutually reinforcing activities. The members rated themselves highest in attendance and in participation in subcommittee meetings, and lowest in communicating with stakeholders, developing a collective plan of action, and the degree to which they held each other accountable. Prior to polling the steering committee, the evaluation team had separately created its own ratings using the same scale, based on all its data and observations to date; the team triangulated its assessments with those of the steering committee and discovered that they had very similar conclusions.

Between October and December 2015, the evaluation team reported back to the Foundation and the collective impact steering committee about their conclusions to date, based on all the data collected and sense-making with the various stakeholders. A number of important concerns surfaced about the usefulness of the collective impact framework and the steering committee members' commitment to it. This feedback helped the Foundation recognize that its approach had created unintentional challenges and barriers for steering committee members and their organizations that were hindering progress, and it shared the concern that collective impact might not be the most appropriate way for this group of organizations to collaborate.

The October 2015 steering committee meeting provided an important opportunity for the committee and the Foundation to consider what had been learned from the evaluation so far, and how this should inform next steps. Some steering committee members shared that the work around collective impact had always been too focused on what the Foundation wanted, and they were not comfortable having the collective impact work tied to organizational funding. Committee members also reflected on the group's challenge in figuring out its goal and direction. "In our heart of hearts," remarked one member, "do we want this to continue? Are we bringing value?" Another said, "I don't give us a good prognosis. There are too many groups working on this. We don't add value, but we are a think tank. Maybe we should focus on a narrower goal." In contrast, other steering committee members still felt the group could come together to play a key role in the state's work around delivery system and payment reform, and that it could help stimulate systems change. "What was exciting about this initially," said one member, "was that it was an opportunity to fill the gaps that the Foundation misses." Another reflected, "I would like to see what the dynamic is around the table without the Foundation present." As a result of the evaluation findings and discussion with the steering committee, the Foundation concluded that its presence was more detrimental than helpful to the group. In December 2015, they announced a decision to step off the steering committee and remove the requirement that grantees participate in collective impact (the grants otherwise remained the same). The Foundation continued to provide funding for facilitation and evaluation support.

The steering committee sustained its monthly meetings, but quickly decided (in January 2016) to switch from collective impact to a learning network model of working together. Though it was no longer on the committee, the Foundation was interested in helping it set its own goals and expectations of success. The evaluation team polled the steering committee to determine what members would consider evidence of progress by April 2016: 26 percent said establishing concrete goals and objectives; 22 percent said evidence of two or more partners working together; and 13 percent said evidence of alignment around a common goal. When asked if the committee was moving in a positive direction, the members gave their group a rating of 5.2 on a 10-point scale.

In April 2016, the point at which they wished to see signs of progress, steering committee members offered meeting feedback such as, "I find myself becoming more disengaged in this work the more we revisit old conversations and focus on our past," and "I think we will continue to lose momentum and participant morale until we become much more specific." Others expressed more optimism: "I think there is potential for the conversation to go either way.... I think it would be a shame to call it quits."

At the July 2016 meeting, the steering committee announced that it wanted to decline the Foundation's offer to continue financial support for facilitation and evaluation; at the time, members said they were willing to chip in small amounts of funding to support additional facilitation if that was needed. But evaluation data continued to suggest that the group was struggling with its purpose. "I don't know what my organization gets out of participating in this group," said one member. "I don't know how much longer we can continue to spend staff time and energy on just talking about things, with no actual outcomes." Another observed, "I think there is progress in that the group realizes something is wrong. However, the group should stop trying to force-fit a reason for meeting and be brave enough to stop doing so, if there truly is no need." That was the last time they met as a group. Their vision to continue meeting on a regular basis as a learning network did not come to fruition.

Reflections

In the evaluators' analysis, observations did not fit neatly into the realist framework categories of context, mechanism, and outcomes. For example, a mechanism or outcome in one instance appeared to be context in another. Still, through participant observation, qualitative analysis, and internal sense-making over 14 months, the evaluation team identified several dominant patterns that emerged in the initiative's dynamics:

* Group progress and morale. The evaluators observed that the steering committee's action or progress toward goals and the resultant increase or decrease in morale appeared to be mutually reinforcing--the more progress members made, the greater their morale; the greater their morale, the more progress they made or at least they perceived they had made. Likewise, stagnation or backsliding appeared to reduce morale, and low morale appeared to reduce inclination toward action. Additionally, evaluators observed that variations in members' buy-in for collective action--the degree to which participants felt committed to the group and its goals and their resulting level of focused engagement--was sometimes an impediment to action and, in turn, a damper to morale. When there was strong buy-in, there appeared to be greater action and progress. When buy-in was low, action and morale appeared to be reduced.

* New information and continuity of direction. The evaluators observed that steering committee members were consistently enthusiastic to learn. When new data and information were presented, members often left meetings in high spirits, with positive morale and energy. However, sometimes the new information, instead of reinforcing the direction the group had previously set, shifted its focus to a new objective. In such instances, new data negatively influenced the group's continuity of direction, which in turn hampered progress.

* Leadership. It appeared from the evaluation team's observations and conversations with steering committee members that lack of agreement on a formalized leadership structure left the group somewhat adrift. The group did not invest authority in the external facilitator to help keep itself focused and hold members accountable for their own decisions, nor did the group agree to a leader from within until after members had decided to abandon the collective impact model.

* Foundation influence. Conversations with Foundation staff revealed that they recognized the potentially adverse impact of real or perceived power differentials, and took specific steps to mitigate the perception (e.g., by not having more than one Foundation staff member present in steering committee meetings and specifically not participating on the evaluation advisory group with other committee members). Despite those efforts, the perception of power dynamics around the Foundation's presence negatively impacted progress. Even after the Foundation removed grant contingencies that had required participation in collective impact, committee members continued to express concern about aligning their work with future Foundation priorities to ensure continued organizational funding.

Insights for Foundations

The Colorado Health Foundation embarked on a collective impact journey with its partners in order to align its funded work within DSPR, to realize greater value from its DSPR investments, to reduce duplication, and, ultimately, to improve the health of Coloradans. Because the field of collective impact is still emerging, few case studies exist to guide new work--particularly in the areas of health and health care. The Foundation and the evaluation team have reflected deeply on this experience and explored insights about collective impact itself and about the Foundation's thinking and approach more broadly. The insights shared here represent exploratory thinking based on the experience of this specific collective impact effort, as well as a consideration of the broader literature on collective impact.

Innovators of Change

Those participating in collective impact acknowledge that its implementation is complex and even unnatural. But what may be missing from early reflections on these efforts is acknowledgement that the collective impact process is still in many ways experimental. While the concept of collaboration is, of course, not new, collective impact is innovative, particularly in the realm of health systems, which have seen fewer applications of the framework than the field of social services. Collective impact participants must embrace their role as innovators and be accepting that the road to greater impact may be unpredictable.

Applying collective impact to DSPR at a state level was a novel experiment, without documented precedent, and it was an approach the Foundation had not attempted before. Both steering committee participants and the Foundation acknowledged that the work was difficult, but there may not have been recognition enough that the work was groundbreaking and required a willingness from everyone involved to change their behavior as they entered unchartered territory. Says Cabaj:

The only way to move the needle on community issues is to embrace an
adaptive approach to wrestling with complexity. This means replacing
the paradigm of predetermined solutions and "plan the work and work the
plan" stewardship with a new style of leadership that encourages bold
thinking, tough conversations and experimentation, planning that is
iterative and dynamic, and management organized around a process of
learning-by-doing (2014b, p. 111).

In hindsight, the Foundation recognized that although at the time it believed it was approaching the work in an open way, it actually adopted a more "formulaic" mindset, believing that if the Foundation provided the process supports, the group could simply put in place the five pillars of collective impact and move forward in an aligned way. Instead, the Foundation discovered that this type of collaborative work relied on much more than good process, and it developed more nuanced understandings of the roles that trust, power, and organizational dynamics have in the success of a collaboration. The Foundation also came to understand that the focus on using the model of collective impact hampered its ability to recognize when that process wasn't actually leading to effective collaboration. This helped the Foundation recognize the importance of not getting attached to a particular approach, but rather entering the work with a learning mindset that allows for experimentation about what it will take to effectively collaborate in a given context, and having the flexibility to modify the approach depending on what is discovered during the process of working together.

Who Initiates Collective Impact Matters

The Colorado Health Foundation is the third-largest health philanthropy in the United States. When the Foundation proposed collective impact as a way to realign its work, it was flexing its convening power to bring stakeholders to the table around an important issue. This was widely seen by stakeholders as positive, and in fact, early data from stakeholders who indicated that the Foundation was uniquely positioned to drive this work forward was a key driver in the Foundation's decision to move forward with a collective impact approach.

This was also the opposite of how most collective impact movements have started. In many cases, stakeholders approach a funder to support a collective impact movement that has already been emerging, whereas in this case a funder proposed collective impact as a model and asked stakeholders to come to the table. While its intentions were laudable and the Foundation attempted to mitigate its perceived influence, the imbalance of power created from the beginning by the funder being the one to propose collective impact may have been insurmountable.

The Importance of Leadership Leadership is important for accomplishing collaborative goals, and the structure, process, and individuals involved are all important (Huxham & Vangen, 2000). Leadership is often provided by an individual who possesses a commitment to stewardship of the collaborative (Emerson, Nabatchi, & Balogh, 2012). Leadership is also a theme that has been highlighted in the collective impact literature, and it is one that emerged throughout the course of this initiative. Two years after their seminal paper on collective impact, Kania and Kramer (2013) reflected on how collective impact influenced complexity and the role of leadership: "Our own experience, and that of several leading practitioners, has shown that the principles of adaptive leadership are extremely useful in guiding the collective impact process" (p. 7). Adaptive leadership is needed in unpredictable, complex situations involving multiple partners. This leadership can come from within the group or it can be ceded to a high-performing backbone organization. A funder taking on the leadership role introduces another layer of complexity to the dynamic:

When funders proactively create networks in support of an identified
cause, the vested interest in achieving desired results may lead to the
problem of funders trying to direct activities rather than acting as
facilitators to draw out the collective wisdom of the participants.
This temptation to direct the group may undermine the very
collaboration required to create change. (Mann, 2014, p. 59).

In the case of this initiative, the group operated for several years without a leader, deciding instead to practice shared decision-making among the group as it distributed the role of a backbone organization among its partners. As was previously highlighted, this was not always effective, potentially impacting the group's direction, progress, and accountability. On several occasions, the Foundation's program officer attempted to provide leadership by focusing the group on metrics, goals, and strategic plans. Later, the program officer encouraged the group to name a chair, and this did result in them agreeing to an internal leader. However, not long after this the group decided to adopt a looser structure as a learning network.

Money Complicates Things One of the primary motivations of the Foundation choosing collective impact was to realign its investments in DSPR. Those investments had day-to-day implications for the participating, funded partners. So, it was perhaps no surprise that when a partnership was formed, it was a self-selected group of Foundation-funded organizations that stepped forward. They were, appropriately, working in their organizations' own best interests in wanting to have a say in how the effort proceeded. In retrospect, the Foundation learned that tying grantee funding to expectations around collective impact, as well as tying its own strategic funding in the DSPR space to the activities of the steering committee, was counterproductive.

A potentially more effective path would have been to fund the initiative in a low-cost and low-risk way (e.g., funding backbone functions and small actions by the group), which would have supported a healthier dynamic around relationships and funding. This was an important insight that shifted the Foundation's mindset about how to approach experimentation; it recognized that although the Foundation had the risk tolerance and interest in engaging in novel experiments, it is critical to right-size investments in experimentation so that appropriate supports are available, and no stakeholder feels so over-invested that it cannot recognize or discontinue an experiment that is not proving effective.

The Role of the Backbone Organization

The literature indicates that the most successful collective impact efforts can be tied back to a strong backbone support function (Bartczak, 2014; Kubisch, Auspos, Brown, Buck, & Dewar, 2011; Pearson, 2014; Weaver, 2014a). In the case of this work, every time the group discussed backbone support it returned to the idea of distributing the functions among the members. While this may have made sense to them at the time, none of the stakeholders had time to be able to truly commit to completing backbone functions, given that each stakeholder had their own organization to manage.

An independent backbone organization may have been able to continue the work between meetings to move the group's agenda forward. While stakeholders did commit time and resources to various subcommittees, the results were piecemeal and may not have been as coordinated had they been the responsibility of one dedicated organization. In hindsight, this is an area where the Foundation may have been justified in being more directive.

The Role of Evaluation

Within the context of collective impact, evaluation is a role often taken on by a backbone organization. The Foundation wanted to include evaluation as a core part of collective impact and, in collaboration with the steering committee, chose to hire an outside evaluation partner to fill this role. Complex change initiatives like this group's attempt to impact delivery system and payment reform call for a kind of evaluation that is neither formative nor summative. Approaching this effort as a developmental evaluation positioned the evaluation as a process of co-learning between the evaluators and those implementing change.

As evaluators, the Georgia Health Policy Center established data collection strategies, regular feedback loops, and sense-making tied into decision points. The findings from the evaluation helped both the Foundation and steering committee understand how the group was functioning as a collective impact initiative and the role the Foundation was playing as a funder. The information raised red flags that led the Foundation to reassess its own assumptions, approach, and role in the work. Although the Foundation already had a strong commitment to learning and evaluation, the experience with collective impact reinforced its view that learning is a critical component that has to be embedded early and engaged in intentionally and often.

Conclusion

Collective impact continues to evolve (Cabaj & Weaver, 2016), and some have even questioned its validity as means of effective community engagement (Wolff, 2016). Seven years into its practice, though, it still draws great interest. Not all collective impact initiatives will succeed, and the field can learn from initiatives that are both successful and not so successful. The experience of the Colorado Health Foundation helped it see the importance of approaching any collaborative process with a learning mindset and dovetailing that with an openness to doing things in a fundamentally different way than it had before, including how it thought about the Foundation's role, its interactions with stakeholders during collaboration, and the way it engaged in experimentation. Developmental evaluation, with its frequent cycles of data collection and sense-making--not just among the evaluation team but with those doing the work--was a critical support for learning and adaptation throughout the collective impact process.

Acknowledgments

Support for this open access article is provided by Georgia Health Policy Center, Andrew Young School of Policy Studies, Georgia State University.

References

BARTCZAK, L. (2014). The role of grantmakers in collective impact. Stanford Social Innovation Review, 12(4), 8-10.

CABAJ, M. (2014a). Developmental evaluation: Diagnostic checklist. Edmonton, Alberta, Canada: Here to There Consulting. Retrieved from http://www.tamarackcommunity.ca/library/developmental-evaluation-diagnostic-checklist

CABAJ, M. (2014b). Evaluating collective impact: Five simple rules. The Philanthropist, 26(1), 109-124.

CABAJ, M., & WEAVER, L. (2016). Collective impact 3.0 An evolving framework for community change. Community Change Series 2016, 1-14. Retrieved from https://www.collectiveimpactforum.org/sites/default/files/Collective%20Impact%203.0.pdf

COOPER, K. R. (2017). Nonprofit participation in collective impact: A comparative case. Community Development, 48(4), 499-514. DOI: 10.108 0/15575330.2017.1332654

EASTERLING, D. (2013). Getting to collective impact: How funders can contribute over the life course of the work. The Foundation Review,

5(2), 67-83. https://doi.org/10.9707/1944-5660.1157

EMERSON, K., NABATCHI, T., & BALOGH, S. (2012). An integrative framework for collaborative governance. Journal of Public Administration Research and Theory, 22(1), 1-29.

GREENHALGH, T., HUMPHREY, C., HUGHES, J., MAC-FARLANE, F., BUTLER, C., & PAWSON, R. A. Y. (2009). How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Quarterly, 87(2), 391-416. DOI: 10.1111/j.1468-0009.2009.00562.x

HUXHAM, C., & VANGEN, S. (2000). Leadership in the shaping and implementation of collaboration agendas: How things happen in a (not quite) joined-up world. Academy of Management Journal, 43(6), 1159-1175.

KANIA, J., & KRAMER, M. (2011). Collective impact. Stanford Social Innovation Review, 9(1), 36-41.

KANIA, J., & KRAMER, M. (2013). Embracing emergence: How collective impact addresses complexity. Stanford Social Innovation Review. Retrieved from https://ssir.org/articles/entry/social_progress_through_collective_impact

KUBISCH, A., AUSPOS, P., BROWN, P., BUCK, E., & DEWAR, T. (2011). Voices from the field III: Lessons and challenges for foundations based on two decades of community-change efforts. The Foundation Review, 3(1), 138-149. https://doi.org/10.4087/FOUNDATIONREVIEW-D-11-00010

LYNN, J., BRECKINRIDGE, K., DENAULT, A., & MARVIN, C. (2015). When backbone organizations become the funder: The use of fiscal intermediaries in the context of collective impact. The Foundation Review, 7(4), 81-96. https://doi.org/10.9707/1944-5660.1268

MANN, C. (2014). The role of philanthropy in collective impact. The Philanthropist, 26(1), 55.

MILSTEIN, B., WETTERHALL, S., & GROUP, C. E. W. (2000). A framework featuring steps and standards for program evaluation. Health Promotion Practice, 1(3), 221-228.

MINYARD, K., PHILLIPS, M. A., & BAKER, S. (2016). The philanthropic collaborative for a healthy Georgia: Building a public-private partnership with pooled funding. The Foundation Review, 8(1), 74-87. https://doi.org/10.9707/1944-5660.1285

NORMAN, C. (2011). What is developmental evaluation? Retrieved from https://censemaking.com/2011/11/19/what-is-developmental-evaluation

PATTON, M. Q. (2006). Evaluation for the way we work. Nonprofit Quarterly, 13(1), 28-33.

PATTON, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guilford Press.

PAWSON, R., & TILLEY, N. (1997). Realistic evaluation. Thousand Oaks, CA: Sage.

PEARSON, H. (2014). Collective Impact: Venturing on an Unfamiliar Road. The Philanthropist, 26(1), 49-53.

PRESKILL, H., PARKHURST, M., & JUSTER, J. (2014). Guide to evaluating collective impact. Washington, DC: FSG. Available online at https://www.fsg.org/publications/guide-evaluating-collective-impact

WEAVER, L. (2014a). The promise and peril of collective impact. The Philanthropist, 26(1), 11-19.

WEAVER, L. (2014b). Q & A with John Kania and Fay Hanleybrown. The Philanthropist, 26(1), 125-132.

WOLFF, T. (2016). Ten places where collective impact gets it wrong. Global Journal of Community Psychology Practice, 7(1), 1-13.

Glenn Landers, Sc.D., is director of health systems at the Georgia Health Policy Center and a research assistant professor in the Andrew Young School of Policy Studies at Georgia State University. Correspondence concerning this article should be addressed to Glenn Landers, Georgia Health Policy Center, Georgia State University, 55 Park Place, 8th Floor, Atlanta, GA, 30303 (email: glanders@gsu.edu)

Kelci Price, Ph.D., is senior director of learning and evaluation at the Colorado Health Foundation.

Karen Minyard, Ph.D., is CEO of the Georgia Health Policy Center and a research professor with the Department of Public Management and Policy in the Andrew Young School of Policy Studies at Georgia State University.

Glenn Landers, Sc.D., Georgia State University; Kelci Price, Ph.D., Colorado Health Foundation; and Karen Minyard, Ph.D., Georgia State University

Key Points

* The 2011 publication of John Kania and Mark Kramer's influential paper, "Collective Impact," caught the attention of organizations across sectors, including nonprofit organizations and philanthropies. The Colorado Health Foundation was one of the organizations that saw the potential of collective impact to help tackle the state's complex, systems-level health issues.

* This article describes a collective impact initiative and the role that developmental evaluation--and a realist framework--played in aiding both the initiative's steering committee and the Colorado Health Foundation in making decisions about the initiative's accomplishments and future.

* The article highlights the developmental evaluation approach, how that informed decisions, and how it helped surface broader insights about the many challenges of doing highly collaborative work.

(1) In a 2014 interview, Faye Hanleybrown and John Kania were careful to say that Kania and Kramer had no interest in copyrighting the term "collective impact" (Weaver, 2014b).

doi: 10.9707/1944-5660.1417

Source Citation

Source Citation   

Gale Document Number: GALE|A550998355