On March 10th, Washington Evaluators and The Evaluators Institute convened a panel of evaluation experts to discuss "Evaluation Policies and Approaches Across Administrations." On a Friday evening, more than 50 evaluators joined to reflect on how evaluation policy has been shaped over the years by different Presidents and to what degree evidence was used to inform policy decisions across administrations. The panelists each offered interesting perspectives from their various experiences. Here are several points from my own remarks that are especially relevant to our evaluation community in DC.
Our Shared Goal
While the transition of power in the Executive Branch of government affects how policies are implemented, and which are prioritized, it's important to remember that evaluators and policymakers share a common goal – to improve the lives of people. Despite differences in political views and philosophies, which will undoubtedly vary, we must strive to acknowledge our shared goals, because this is precisely what evaluation seeks to achieve – better, more efficient programs and policies that improve people lives.
We use evaluation to improve programs that help keep homeless veterans off the streets, ensure children have enough food to eat, protect our health by limiting pollution in our air and water, help the unemployed find work, and provide opportunity to those who need it most. It is because our common goals are the same that determining how to best achieve our policy aims is a moral imperative.
In thinking about the relationship between evaluators and policymakers, we must also acknowledge that evaluation is inherently political. We work in a profession that is designed to pass judgment on policies and programs, to announce winners and losers. Pericles said that "just because you do not take an interest in politics, does not mean politics will not take an interest in you." And this is exactly where evaluators often find themselves; it's where we thrive. But even then, we must recognize that evaluation needs to be conducted in an apolitical manner, maximizing the credibility of findings with an appropriate level of independence.
New Priorities in New Administrations, Yet Common Challenges Persist
New administrations will always introduce new priorities, but that does not lessen the value of evaluation for its two primary purposes: accountability and learning. We often have questions in new administrations about what priorities will be the real focus – and whether evaluation will be part of that discussion. In practice, we know that in recent years, whether in times of increasing or declining budgets, agencies have articulated a priority for building new evidence.
In 2015, Kathy Newcomer and I wrote a working paper comparing the evidence initiatives in the George W. Bush Administration and the Barack Obama Administration. What you see looking across them is that there are a striking number of similarities, most of which have nothing to do with political philosophies. Instead, they have everything to do with the shared goal of making government better, and improving the lives of Americans.
In the Bush Administration, efforts to implement the Program Assessment Rating Tool (PART) included specific questions about evaluation that we know today encouraged some agencies to pursue evaluation, bolstering evaluation capacity. During the Bush Administration, Federal agencies started developing clearinghouses for evaluation results and increased funding for targeted evaluation activities across government. Then, the Obama Administration pursued the "thousand flowers bloom" strategy building on the Bush efforts by requesting even more funding for evaluation, pushing out initiatives to encourage pay for success, and even creating new, formal evaluation positions in some parts of government.
Looking across these initiatives, Kathy Newcomer and I distilled three key challenges that persist. First, we need to identify how to better balance the purposes of evidence-building activities, recognizing there is a need for methods in evaluation to address both the learning and accountability purposes. Addressing both purposes also means we need to better synergize the existing infrastructure for evidence-building in government, starting with a recognition that the canyon between performance measurement and evaluation must be bridged.
Second, attention must be given to acquiring and sustaining audiences for evidence. And this is the case regardless of which political party controls the White House. For years we have simply assumed that the establishment of demand for evaluation, met with sufficient supply, presents one strategy for developing and maintaining a use case for performance information. But neither administration made serious strides in helping their political appointees embrace evaluation or other evidence initiatives.
Third, initiatives must be implemented effectively. This point seems so obvious it's almost not worth saying – except that we must be explicit about the role effective implementation has in initiatives' success. Even the best designed, theory-based initiative must face the realities of implementation in the complex, institutional structures of government. In practice, the perverse incentives introduced with mandating performance and evaluation activities may lead implementers to focus merely on measurable outcomes, rather than the desirable. This is one of the reasons why more cross-agency collaboration is needed, to break down silos of topical emphases to look at how programs interact. For far too long, cross-agency collaboration has been an oxymoron in government and it's time to address this limitation.
Moving Forward: What You Can Do
Given the many challenges facing the evaluation community regardless of who is in the seat of power, what can be done to move forward? Here are several brief suggestions:
1. Stay Calm. Transitions are a routine part of democratic society. It takes time for agency leaders to settle in and to learn about the priorities before them. Evaluation may not be the first thing that a new appointee turns to, but inevitably it will be part of the discussion – everyone wants to know what they achieved at the end of an appointment. This is one strategy for beginning to build a better audience for evaluation.
2. Articulate the value of evaluation as a community. We can do this by bolstering our professional networks of evaluators, both inside and outside of government to facilitate appropriate connections and ensure the multiple purposes of evaluation are represented. In the DC area we have Washington Evaluators to facilitate this networking, and there are several other communities of practice available to individuals within the government. With a broader community we can help present evaluation as a tool for more than informing budget decisions – though we hope that it is among those tools – but one that helps promote learning and continuous improvement in policies and programs.
3. Inform decision-makers about the usefulness of evaluation. In order for evaluation to be useful, decision-makers will need to understand how existing evaluations can be used to help shape and frame new policies, or even review old ones. These available evaluations allow us to find opportunities to continue to educate the public and their leaders about the role many important government programs play in achieving our shared goal for the country, especially in helping to implement initiatives effectively. Evaluators can help facilitate informed use of evaluation, noting the relevance of existing evaluations and identifying important caveats as new policy decisions are formulated.
4. Help agencies find opportunities to bolster their analyses. Similar to the suggestion above about highlighting the usefulness of evaluation, we can actually strive to make evaluation more useful too! One strategy might be to incorporate in our evaluations analysis of disparities in policies that exist across subgroups of our population. I recently wrote at greater length about the latest Dialogue on Race and Class, a January event co-sponsored by Washington Evaluators, the American Evaluation Association, and George Washington University. Our community must give greater attention to the evaluation of subgroups because the challenges facing our society are complex and disparities can be inadvertently embedded in policies. We owe it to our fellow citizens to help identify and address such injustices.
At the end of the day, we do not live in a perfect system of government. In fact, change in the policies of our representative democracy was designed to be difficult. But it's what we've got. So let's make the best of it by continuing to learn and improve together, to accomplish our shared goal.
NICK HART, PH.D. is the President of Washington Evaluators in 2017. The views presented here are those of the author and do not represent the official position of the U.S. Government, including the Office of Management Budget and the Commission on Evidence-Based Policymaking.
