This popular saying about the two sides of a pancake provides salient counsel to evaluators, researchers, and program leaders: there are multiple ways of understanding complex social issues. Through experiences gained from working with clients, peer evaluators, and researchers, we have learned this important lesson: conveying insights and findings on evidence-based approaches to policymakers and others working to improve social issues requires applying an interdisciplinary and culturally responsive lens. Here’s why. 

When we focus on one side of the proverbial pancake, we often create barriers to well-informed assessments.  This one-sided approach has the unintentional consequence of forcing our work into an artificial silo in such a deleterious way that we may miss important cultural context and nuance. 

For example, when evaluating an educational success program, if we neglect to disaggregate gender self-identification or racial and ethnic demographic data (when such data are available), we miss an important opportunity to examine systemic issues affecting individuals in different and profound ways.  As well, if we conduct an assessment of community health, yet exclude economic and political analysis, built environment factors, and the presence of indigenous anchor institutions and leaders as central areas of inquiry, we are sacrificing important context.    

Let’s drill down further and examine the economic inequities facing so-called opportunity youth (youth ages 16-24 who are out of school or the workforce).  According to the Aspen Institute’s Forum for Community Solutions, it is estimated that there are now 5.5 million opportunity youth – also commonly referred to as disconnected youth – in the United States. Unless we are attentive to factors such as reliable transportation access, trust in employer institutions, social networks, job preparedness, prior interactions with the legal system, and family supports, among other factors, we will lose sight of critical factors in the lives of youth.  Piecemeal fixes resulting from under-informed assessments may leave communities, funders, and policymakers without viable and sustainable pathways to youth and family success.   

To further illustrate these points, I was struck by a piece published in the Washington Post, following the death of Freddie Gray.  The reporter offers a comprehensive look at the systemic failures at critical junctures during Freddie Gray’s life and tragic death. 

At one point in his life, Mr. Gray was an opportunity youth.  He lived in a community marked by tough systemic issues, such as staggering unemployment, poor access to sustainable living wage positions, the absence of fresh foods, and the lack of affordable healthier alternatives to fast food restaurants.  The reporter also discussed how environmental factors, such as early exposure to lead toxins, may have affected Mr. Gray’s brain development.  Additionally, by the time Mr. Gray reached his 18th birthday, he had received several out-of-school suspensions, which contributed to missed critical in-classroom learning time.   

The story of Freddie Gray clearly illustrates why it is so essential to apply a mindset that considers the mosaic of lived experiences and deeply rooted systemic issues in a community. When evaluators miss these factors, and neglect opportunities to identify more holistic evaluation designs, we perpetuate system failures. 

Let’s be clear, multi-dimensional approaches to evaluation may add time, further complexity, and costs.  And approaching our work in such a comprehensive manner may require us to sharpen our abilities to work across disciplines, understand cultural awareness, integrate lived experiences into our thinking, and strengthen respect for cultural assets.  But the upsides far outweigh the costs, and the outcomes are stronger.
​
At the heart of CREA’s mission is an unapologetic commitment to understand the character and influences of diverse cultural norms and practices when designing and conducting evaluations. That’s why the need for evaluators to work across disciplines ought to be shouted from the roof tops.  Years of experience confirm that we simply cannot sweep in and out of communities without considering our cultural biases (intentional and unintentional), riding solo without including partners working in other disciplines.  Further, we must stretch ourselves to tap deeply and respectfully into new wells of expertise and knowledge, on an ongoing basis, to understand and consider both sides of the proverbial pancake.   

 By: Melvin Hall and Robin Miller
​
​Over the prior decade or more, the evaluation profession has made great strides around issues of diversity. From programs such as the Graduate Education Diversity Initiative sponsored by the American Evaluation Association to its cultural competence statement to the founding of CREA to each of the many other domestic and international achievements on issues of diversity and inclusion, our professional evaluation community continues to step in the right direction. Our progress, however, invites worry, as we too often hear satisfaction simply with netting more evaluators from under-represented groups within our junior ranks, a worthwhile achievement but one which falls short of the larger goal of attacking longstanding patterns of segregation in theory development, scholarship, leadership, and practice.

Our fondest hope as we continue to address critical issues on diversity is that we remain vigilant we are not advocating diversity for diversity’s sake. We pursue diversity  because we strive to create a socially just profession in which evaluators from traditionally under-represented and disenfranchised backgrounds enjoy the supports, status, recognition, opportunities, and depth of professional connections other members of the profession have long had the privilege to take for granted. With this expanded participation in evaluation, we also seek a broadened base of thought about how evaluation work is accomplished, and the nature of its impact on all communities. Diversity and inclusion are not the same things. We can be diverse without creating inclusivity and without meaningful and enduring change in how our profession and its primary institutions function. That is not the future professional community to which we aspire.

Not all contexts are created equal within professional associations or outside them when it comes to re-creating evaluation as a profession characterized by heterogeneous professional networks and inclusive leadership at every career stage and in every corner. An emergent line of research in community psychology points to a troubling paradox: diversity often comes at the expense of sense of community and belonging. Broader access to the profession alone does not provide for the meaningful integration of different values, perspectives, and priorities that bridge the spaces and structures of practice to facilitate deep connections among all practitioners and theorists.  When we fail to nurture inclusivity, justice, and the honoring of values and multiple perspectives explicitly as part of diversity efforts, the potential is there to undermine the truly inclusive community we seek. Advancing diversity as a matter of justice requires we focus on meaningful inclusion across the full spectrum of professional practice.

​

By: Waapalaneexkweew / Nicole Bowman (Mohican/Lunaape), PhD

Koolamalsi njoos (Greetings, colleagues and friends):
 
I greet you as a representative of the least known people on Turtle Island and Mother Earth: Indigenous people.  My Indigenous ancestors are Lenaape-Munsee and Mohican Indians, known to Europeans as “Delaware Indians.” I am proud to be part of an evaluation community at CREA that is leading the way to encompass the history, values, perspectives, strengths, and contributions that our collective and strong diversity represents.
 
As we embark on another “Thanksgiving” and “Native American history month” I’d like to offer a different, historical Indigenous perspective, “re-writing and re-righting” (Smith, 1999) the western narrative that often excludes, is ignorant of, or provides a romanticized version of Indigenous people’s perspectives, experiences, and contributions. 
 
Current controversy provides a significant example for reflection and consideration.  Within both Indigenous and academic communities, I’ve been asked about the conflict surrounding the Dakota Access Pipeline (DAPL) and the #NoDAPL counter-movement launched by human, water, and earth rights protectors from Indigenous and non-Indigenous communities.  #NoDAPL seeks to block a $3.8B, 1,100-mile fracked-oil pipeline currently under construction from the Bakken shale fields of North Dakota to Peoria, Illinois. DAPL crosses Lakota Treaty Territory at the Standing Rock Sioux Reservation where it would be laid underneath the Missouri River, endangering a source of fresh water for the Standing Rock Sioux and 8 million people living downstream. DAPL also impacts sites that are sacred to the Standing Rock Sioux and other Indigenous nations. 
 
As a culturally responsive community of evaluators, we all must understand the cultural, contextual, historical, and legal implications of DAPL and similar projects with Indigenous communities and Tribal nations.  Like many projects – research, evaluation, and otherwise – conducted with Indigenous people, DAPL has deep, destructive, and illegal historical roots.  The project violates treaty rights afforded the Standing Rock Sioux as a sovereign nation, reflecting the lack of recognition of Tribes as sovereign nations that stretches back to the earliest treaties enacted in 1774 between colonists and Indigenous Tribal nations.  How many research and evaluation projects have been enacted with Indigenous communities to “extract” knowledge and data without due consideration for Indigenous ownership of that knowledge and data?   
 
Reflecting on the DAPL project reminds us that our broader evaluation community is also about power, networks, and resources.  We only need to look at who gets funded, published, promoted, elected, or represented on key commissions, editorial review boards, or “invite only” events to see that while we talk about inclusion, equity, and diversity, there is little evidence of it in our “community” of practice.  What business interests, amount of dedicated resources, and professional practices are most prevalent in our evaluation community?  It depends on the conference and context.  But the interests, practices and viewpoints of Indigenous and other marginalized groups are consistently and significantly absent!
 
DAPL and the #NoDAPL movement also remind us that our broader evaluation community privileges the written word and values certain evaluation voices – largely white and male – more than others.  I welcome emerging voices that tell different stories in different ways – voices like those of Carl Sack, a graduate student at the University of Wisconsin-Madison who uses critical cartography as a culturally-responsive and scientifically-sound method to provide a counter-narrative to the lack of media coverage of the historic, Indigenous and #NoDAPL perspective.

By: Jennifer C. Greene

Evaluation is the social practice of valuing.
Evaluation is the social practice of gathering and using empirical information to make judgments about the merit and worth of a given social or educational intervention somewhere in the world. Both our empirical data and our judgments are grounded in particular values, which are too often unnamed and thereby under-used to advance our culturally responsive agenda of cultural respect and socio-political equity.

Where do values show up in evaluation practice?
First, our data are significantly shaped by the evaluation approach and the key evaluation questions that frame and steer a particular evaluation study, and by the different methodologies that support these different evaluation approaches and questions. A policy-oriented evaluation often asks questions about statistically significant outcome attainment, on the average and sometimes also as disaggregated by targeted subgroups, and uses an experimental methodology to address these questions. A participatory evaluation often asks questions about the quality of program services and their match to priority community needs, and uses a case study or participatory methodology to address these questions. And second, our judgments of program success are anchored in a set of quality criteria, too rarely explicitly named or justified. From a policy evaluation perspective, a “good” program has statistically significant average outcome effects, resting on utilitarian and instrumental understandings of the common good. From a participatory perspective, a “good” program addresses important needs of the targeted community, resting on democratic, pluralistic, and community-oriented understandings of the common good.

In culturally responsive evaluation, how can underlying values importantly and consequentially inform evaluation practices?
Privileged in culturally responsive evaluation (CRE) are the needs and interests of cultural minorities in our society who have yet to attain fair and equitable access to the resources and opportunities needed for the development of each person’s full human potential. The boundary lines defining these minorities are constructed along racial, ethnic, gender, economic, geographic, and identity lines. CRE thus, in theory, advances values of inclusion and pluralism, of contextuality and history, of fairness and equity, and of socio-political justice. In practice, CRE work can more consequentially advance these values by clearly naming and justifying them as integral to this evaluation approach, its key questions and audiences, and its accompanying designs and methods. Even more powerfully, the commitments of CRE can be advanced with a clear articulation and justification of – in tandem with a conversation about – the values that constitute the criteria by which a program is judged as worthy or not.

References
Boyce, A.S., DeStefano, L., & Greene, J.C. (September 2015). Values-Engaged, Educative Evaluation
Approach. MSPnet.
 
Greene, J.C., Boyce, A., & Ahn, J. (2011). A values-engaged educative approach for evaluating education programs: A guidebook for practice. University of Illinois at Urbana-Champaign, IL. (http://comm.eval.org/EVAL/Resources/ViewDocument/?DocumentKey=f3c734c0-8166-4ba4-9808-a07e05294583)
 
Greene, J.C. (2006). Toward a methodology of mixed methods social inquiry. Research in the Schools. Special Issue: New Directions in Mixed Methods Research, 13(1), 93-99.
 
Hall, J.N., Greene, J.C., & Ahn, J. (2012). Values-engagement in evaluation: Ideas, implications, and illlustrations. American Journal of Evaluation, 33(2), 195-207.

Julnes, G. (Ed.) (2013). Promoting valuation in the public interest: Informing policies for judging value in evaluation. New Directions for Evaluation no. 133.
 

By: Katherine Tibbetts & Herb Lee
​
After the first CREA Conference in 2014, the Consuelo Foundation agreed to host an ongoing discussion with approximately 20 participants representing community-based organizations, non-profit organizations working with both community and educational institutions including the University of Hawaiʻi, the Kamehameha Schools, Liliʻuokalani Trust, the Office of Hawaiian Affairs, the Hawaiʻi-Pacific Evaluation Association, and the Native Hawaiian Educational Council. This hui (gathering) has been meeting to share our work and approaches, highlight key issues we face, and develop and affirm our dedication to culturally responsive approaches to evaluation and assessment.

CREA-Hawaiʻi is dedicated to idea that “Aʻohe pau ka ʻike i ka halau hoʻokahi” (all knowledge is not learned in one school).

Our mission is to use a Native Hawaiian lens to offer empowering, values-based evaluation approaches that support insights and conclusions which, in turn, promote equity and justice in the diversity that is Hawai'i. To do this we will weave together and share cultural knowledge, tools, and practices that are grounded in and affirm a Native Hawaiian worldview.

We look forward to continuing our dialog and sharing new ways of thinking, observing, exploring, with CREA and all its partners as we seek to identify the practices that best to serve our respective communities in the 21st century. There is much research to share and stories of success that have brought us to this point in time. We hope to have many work products in the months and years ahead that can add to the knowledge and opportunity that CREA has provided. For this we are forever grateful and humbled.
​
Our first work product will be a Native Hawaiian evaluation framework that can be used by evaluators, program managers, communities, and funders to guide evaluation practice from their various positions. The framework will lay the foundation for the next steps of training and advocacy. This work is being partially funded by the Faster Forward Fund established by Michael Scriven. We are also looking forward to establishing a series of unconferences where we can explore questions and promising approaches in greater depth and with a wider pool of participants.